A medical image segmentation method, device, electronic equipment, medium and product

Abstract

The present disclosure relates to a medical image segmentation method and device, electronic equipment, medium and product. The method comprises: performing multi-scale feature extraction and coarse segmentation processing on the to-be-segmented medical image through a coarse segmentation network in the image segmentation model to obtain a coarse segmentation feature map; cropping the to-be-segmented medical image according to the coarse segmentation feature map to obtain an object region image; performing uncertainty analysis on the coarse segmentation feature map through a segmentation calibration network in the image segmentation model to generate a calibration prompt mask to calibrate the object region image to obtain a guided feature map; and performing fine segmentation by combining the semantic feature map and the guided feature map through a fine segmentation network in the image segmentation model to output a target segmentation result of the to-be-segmented object. The present embodiment solves the problem of small target segmentation features not being obvious in medical images and improves the segmentation effect of medical images.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, and in particular to a method, apparatus, electronic device, medium and product for segmenting medical images. Background Technology

[0002] With the rapid development of deep learning technology, it has been widely used in medical image analysis, enabling automated segmentation of various tissue structures, such as specific organs, abnormal tissues and related blood vessels. This provides a new technical path for intelligent analysis and medical auxiliary applications of medical images, effectively making up for the shortcomings of manual segmentation and indirect functional monitoring technologies in terms of efficiency and accuracy, and significantly improving the standardization level of medical image processing.

[0003] Limited by the hardware conditions of medical imaging equipment and the inherent complexity of physiological structures, the edges between tissue structures are often blurred, and medical images are subject to significant noise interference. Especially when the tissue structure is small, its edge information is easily lost during feature extraction, which brings many challenges to the application of deep learning technology.

[0004] Taking the recurrent laryngeal nerve (RLN) as an example, the recurrent laryngeal nerve exhibits typical small target characteristics in laparoscopic images: it accounts for a very low percentage of pixels in the laparoscopic field of view, its branch morphology varies significantly among different individuals, and the muscle fibers and blood vessels distributed around it are visually similar to nerve structures, further obscuring the edge information of the recurrent laryngeal nerve.

[0005] Currently, deep learning technology has achieved relatively mature results in large target segmentation tasks, but its transfer effect in small target segmentation tasks is not ideal. In particular, it is not good at capturing the edge details of small targets, and it is easy to have problems such as missed target detection and edge misjudgment. It is difficult to output accurate and complete segmentation results stably, and it cannot meet the high-precision and high-reliability segmentation requirements of medical scenarios. Summary of the Invention

[0006] This disclosure provides a method, apparatus, electronic device, medium, and product for segmenting medical images to address the problem of unclear segmentation features for small targets in medical images and improve the segmentation effect of medical images.

[0007] One aspect of this disclosure provides a method for segmenting medical images, comprising: Acquire a medical image to be segmented, and a trained image segmentation model that matches the object to be segmented. The image segmentation model consists of a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation network is used to extract multi-scale features from the medical image to be segmented to obtain at least two semantic feature maps, and coarse segmentation is performed based on each semantic feature map to obtain a coarse segmentation feature map. The medical image to be segmented is cropped based on the coarse segmentation feature map to obtain the object region image; Uncertainty analysis is performed on the coarse segmentation feature map through the segmentation calibration network to obtain a calibration cue mask, and the object region image is calibrated according to the calibration cue mask to determine the guiding feature map; The fine segmentation network performs fine segmentation based on the semantic feature maps and the guiding feature maps to obtain the target segmentation result corresponding to the object to be segmented.

[0008] Another aspect of this disclosure provides a medical image segmentation apparatus, comprising: The image segmentation model acquisition module is used to acquire the medical image to be segmented and the trained image segmentation model that matches the object to be segmented. The image segmentation model consists of a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation feature map determination module is used to extract at least two semantic feature maps from the medical image to be segmented through the coarse segmentation network, and to perform coarse segmentation processing based on each of the semantic feature maps to obtain a coarse segmentation feature map. The object region image determination module is used to crop the medical image to be segmented based on the coarse segmentation feature map to obtain the object region image; The guided feature map determination module is used to perform uncertainty analysis on the coarse segmentation feature map through the segmentation calibration network to obtain a calibration prompt mask, and to calibrate the object region image according to the calibration prompt mask to determine the guided feature map; The target segmentation result determination module is used to perform fine segmentation based on the semantic feature maps and the guiding feature maps through the fine segmentation network to obtain the target segmentation result corresponding to the object to be segmented.

[0009] Another aspect of this disclosure provides an electronic device, comprising: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the medical image segmentation method according to any embodiment of this disclosure.

[0010] Another aspect of this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the medical image segmentation method according to any embodiment of this disclosure.

[0011] Another aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the medical image segmentation method described in any embodiment of this disclosure.

[0012] The technical solution of this disclosure deeply integrates a coarse-fine segmentation architecture with an uncertainty guidance mechanism to construct a progressive segmentation paradigm driven by a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation network performs multi-scale feature extraction and coarse segmentation to achieve rapid localization of the object to be segmented. Based on the coarse segmentation feature map, the medical image to be segmented is cropped to obtain the object region image. The segmentation calibration network performs uncertainty modeling to obtain a calibration cue mask and calibrates the object region image, achieving a precise transition from coarse-grained semantic localization to pixel-level anatomical reconstruction. Finally, the fine segmentation network achieves deep fusion of multi-scale semantic features based on the guidance feature map, outputting the target segmentation result of the object to be segmented. The structured prior provided by the segmentation calibration network enhances the anti-interference ability of the image segmentation model and its feature expression ability for edge structures, solving the problem of indistinct segmentation features for small targets in medical images, improving the accuracy of small target object recognition, and enhancing the segmentation accuracy and robustness of medical images.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating a medical image segmentation method provided in one embodiment of this disclosure; Figure 2 This is a schematic diagram of the architecture of a segmentation calibration network provided in one embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating the architecture of a specific example of an image segmentation model provided in one embodiment of this disclosure; Figure 4 A flowchart illustrating another medical image segmentation method provided in one embodiment of this disclosure; Figure 5 This is a schematic diagram of the architecture of a specific example of a coarse segmentation network provided in one embodiment of this disclosure; Figure 6 This is a schematic diagram of the architecture of a specific example of a fine segmentation network provided in one embodiment of this disclosure; Figure 7 A comparative schematic diagram of a target segmentation result provided in one embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of a medical image segmentation device provided in one embodiment of the present disclosure; Figure 9 This is a schematic diagram illustrating the structure of a specific example of a medical image segmentation device provided in one embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0017] It should be noted that the terms "to be divided," "target," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Figure 1This is a flowchart illustrating a medical image segmentation method according to an embodiment of the present disclosure. This embodiment is applicable to the segmentation of medical images, particularly for segmenting small tissue structures within medical images. The method can be executed by a medical image segmentation device, which can be implemented in hardware and / or software and can be configured in a terminal device. Figure 1 As shown, the method includes: S110. Obtain the medical image to be segmented, and the image segmentation model that has been trained and matches the object to be segmented.

[0019] Specifically, the medical image to be segmented refers to medical image data that covers the object to be segmented. The pixel region containing the object to be segmented may be a small percentage of the entire image region. For example, in a lung nodule segmentation task, the medical image to be segmented may be image data of the entire lung or half of the lung, while the nodule as the object to be segmented may be less than 1% of the entire lung; in a liver segmentation task, the medical image to be segmented may be image data of the entire abdomen, while the liver as the object to be segmented only accounts for about 5% to 10% of the abdomen.

[0020] It should be noted that this embodiment is only used as an example to illustrate the pixel proportion of the object to be segmented in the medical image to be segmented, and is intended to highlight the adaptability of this embodiment to small target segmentation scenarios, rather than to limit it.

[0021] For example, the medical imaging equipment corresponding to the medical image to be segmented includes, but is not limited to, direct digital radiography (DR), computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), or ultrasound equipment.

[0022] Regarding the timing of acquiring the medical image to be segmented, specifically, the medical image to be segmented can be a single-frame image acquired before the treatment operation on the target tissue, an image frame captured from a real-time video stream during the treatment operation on the target tissue, or a single-frame image acquired after the treatment operation on the target tissue, wherein the target tissue is the anatomical structure distributed within the object to be segmented.

[0023] Regarding the acquisition method of the medical image to be segmented, for example, the medical image to be segmented can be obtained by preprocessing the original medical image. The preprocessing operations include, but are not limited to, grayscale normalization, noise suppression, contrast enhancement and image registration. It can also be retrieved directly from a cloud server or local storage, or it can be extracted from the real-time data stream transmitted by medical imaging equipment, but it is not limited to the given example.

[0024] This embodiment only provides an exemplary description of the timing of acquiring the medical image to be segmented, the suitable medical imaging equipment, and the acquisition method, and does not limit them. In practical applications, it can be flexibly adjusted according to actual application needs and specific scenarios. Any medical image that meets the image quality requirements and surgical procedure specifications can be included in the segmentation process of this embodiment.

[0025] In one alternative embodiment, the imaging site of the medical imaging device is the thyroid gland, and the object to be segmented is the recurrent laryngeal nerve.

[0026] The recurrent laryngeal nerve is a delicate nerve structure with important physiological functions located in the anatomical region of the thyroid gland. Its overall diameter is about 1.2–2.0 mm, and its burial depth is relatively large. During the treatment process, it is often gradually exposed in the interstitial space. At the same time, due to its own tortuous and variable course, it is difficult to achieve clear identification and precise positioning under conventional imaging methods, thus increasing the difficulty of performing thyroid treatment procedures.

[0027] During thyroid procedures, the recurrent laryngeal nerve is susceptible to structural damage from various external factors, such as traction, heat, and mechanical cutting. This damage can easily lead to serious consequences, including abnormal movement of surrounding tissues, vocal dysfunction, and even loss of voice. Therefore, precise segmentation and real-time tracking of the recurrent laryngeal nerve are crucial for ensuring the safety and reliability of thyroid procedures.

[0028] In this embodiment, the image segmentation model consists of a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. Specifically, the image segmentation model is a hybrid architecture composed of deep learning networks and image processing algorithms. It leverages the functional advantages of different technical paths to achieve hierarchical segmentation processing, sequentially completing the segmentation tasks of initial localization and division, object region calibration, and precise detail recognition.

[0029] S120. Through the coarse segmentation network, multi-scale feature extraction is performed on the medical image to be segmented to obtain at least two semantic feature maps, and coarse segmentation processing is performed according to each of the semantic feature maps to obtain a coarse segmentation feature map.

[0030] Among them, multi-scale feature extraction is used to capture multi-level representations of the medical image to be segmented, from local details to global semantics, covering feature information such as pixel-level texture, organ-level contour and anatomical context. Specifically, the semantic feature map represents the abstract semantic distribution of the medical image to be segmented in a high-dimensional feature space.

[0031] Specifically, the coarse segmentation network includes a multi-scale encoder and a coarse segmentation head. The multi-scale encoder is used to extract features by downsampling the medical image to be segmented step by step, and to use the feature maps of the encoding layers that are greater than a preset layer threshold as semantic feature maps. The coarse segmentation head is used to perform feature fusion and upsampling reconstruction on each of the semantic feature maps to obtain a coarse segmentation feature map.

[0032] The preset layer threshold can be set with reference to the network structure depth of the multi-scale encoder, usually 2 / 3 or more of the total encoding layers. The semantic feature map retains the most discriminative deep semantic information of the medical image to be segmented. The preset layer threshold can also be dynamically adjusted according to the accuracy requirements of the initial localization to ensure that the key anatomical semantics are preserved while taking into account the computational efficiency.

[0033] In a specific example, the multi-scale encoder can use feature extraction architectures such as Swing Transformer, ResNet50, VisionTransformer, or EfficientNet; the coarse segmentation head can use a network architecture such as a U-Net decoder, a lightweight decoding module, or a multi-scale fusion decoding network. This section only provides illustrative examples of the network architectures for the multi-scale encoder and coarse segmentation head; in practical applications, a suitable network architecture can be flexibly selected based on task requirements.

[0034] Specifically, the coarse segmentation feature map represents the initial localization region of the object to be segmented in the medical image to be segmented, reflecting the rough spatial range and morphological outline of the object in the medical image to be segmented. For example, the coarse segmentation feature map can be a probability distribution map, a heat map, or a binary mask map, etc.

[0035] S130. The medical image to be segmented is cropped according to the coarse segmentation feature map to obtain the object region image.

[0036] Specifically, the object region image represents the pixel information of the local region where the object to be segmented is located.

[0037] In one optional embodiment, the coarse segmentation feature map is a binary mask map, and the step of cropping the medical image to be segmented based on the coarse segmentation feature map to obtain an object region image includes: locating the contour edge of the object to be segmented based on the coarse segmentation feature map, and cropping the medical image to be segmented based on the contour edge to obtain an object region image containing the object to be segmented.

[0038] In another optional embodiment, the coarse segmentation feature map is a probability distribution map, and the step of cropping the medical image to be segmented according to the coarse segmentation feature map to obtain an object region image includes: determining the minimum bounding rectangle of the recurrent laryngeal nerve using an adaptive thresholding method based on the probability distribution of the coarse segmentation feature map; and cropping the medical image to be segmented according to the minimum bounding rectangle to obtain an object region image containing the object to be segmented.

[0039] The adaptive thresholding method can perform statistical analysis based on the probability distribution of the coarse segmentation feature map, and dynamically set the threshold according to the statistical results. For example, the statistical results can be obtained by traversing the coarse segmentation feature map using a sliding window and calculating the probability mean and probability variance within the sliding window. For instance, the object region image is obtained by cropping based on the minimum bounding rectangle, using the highest probability in the coarse segmentation feature map as the cropping center.

[0040] Based on the above embodiments, optionally, before executing S140, the method further includes: performing image enhancement on the object region image to obtain an enhanced object region image.

[0041] For example, image enhancement processing includes Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm, gamma correction, Gaussian blur denoising, and normalization preprocessing. Among them, the CLAHE algorithm can enhance the gray-level gradient difference between the object to be segmented and its surrounding tissues, thereby improving the texture recognition of the object to be segmented. Gamma correction can adjust the overall brightness and contrast of the object region image. Gaussian blur denoising can effectively suppress high-frequency noise interference in the object region image. Normalization preprocessing can map pixel values ​​to a uniform numerical range.

[0042] The advantage of applying image enhancement after image cropping is that it highlights the texture and contour features in the object region image that are related to the object to be segmented, providing more discriminative feature inputs for the subsequent segmentation calibration network, thereby further improving the segmentation effect of medical images.

[0043] S140. Through the segmentation calibration network, perform uncertainty analysis on the coarse segmentation feature map to obtain a calibration prompt mask, and calibrate the object region image according to the calibration prompt mask to determine the guiding feature map.

[0044] Uncertainty analysis is a quantitative or qualitative method used to assess the scope and confidence of the impact of uncertainties in input variables, model assumptions, or data structures on model predictions or decision outputs. In this embodiment, uncertainty analysis refers to quantifying the confidence of each pixel feature value in the coarse segmentation feature map using pixel-level confidence estimation, and using a calibration cue mask to characterize regions with high uncertainty values ​​in the coarse segmentation feature map.

[0045] For example, uncertainty analysis methods may include pixel-level entropy calculation, error propagation, or Bayesian approximation inference, but are not limited to the given examples.

[0046] In an optional embodiment, the segmentation calibration network incorporates a Dropout layer, and the uncertainty analysis method is a Monte Carlo simulation method. The uncertainty analysis of the coarse segmentation feature map to obtain a calibration cue mask includes: during the forward propagation of the segmentation calibration network, randomly activating the Dropout layer and repeatedly performing forward propagation N times, with each forward propagation yielding a predicted segmentation image; using the probability variance of the N predicted segmentation images at each pixel location as an uncertainty metric, and marking pixel locations where the uncertainty metric exceeds a preset metric threshold to obtain the calibration cue mask.

[0047] In another optional embodiment, for a real-time video stream scenario, the step of performing uncertainty analysis on the coarse segmentation feature map to obtain a calibration cue mask includes: performing pixel-by-pixel feature extraction on the coarse segmentation feature map to obtain uncertainty measurement data; filtering the coarse segmentation feature map according to preset filtering conditions and the uncertainty measurement data to obtain an initial cue mask; and performing morphological dilation processing on the initial cue mask to obtain a calibration cue mask.

[0048] Specifically, the uncertainty measure data includes the uncertainty measure value corresponding to each pixel position in the coarse segmentation feature map. For example, the uncertainty measure value can be calculated based on statistics such as pixel-level entropy, complementary maximum probability confidence, pixel-level variance, pixel-level probability, or standard deviation of Bayesian posterior distribution.

[0049] Taking pixel-level entropy as an example, uncertain measurement data Satisfy the following formula: in, Entropy value represents the probability that a pixel location in the coarse segmentation feature map belongs to the object to be segmented. The larger the value, the higher the probability. The lower the confidence level, the more likely the pixel location is to be in an area prone to being missed, such as an anatomical edge, a long and thin end, or a blurred texture. Conversely, when the entropy value... When it approaches 0, it indicates that the probability value is... The closer the value is to 0 or 1, the more certain the classification decision of the coarse segmentation network is for that pixel location.

[0050] The preset filtering conditions include a preset metric threshold and / or a preset mask ratio. For example, the preset metric threshold is set to 0.6, and the initial prompt mask is constructed based on the pixel positions with uncertainty metric values ​​greater than 0.6; the preset mask ratio is set to 15%, and the initial prompt mask is constructed by selecting the top 15% of pixel positions with the highest uncertainty metric values ​​from the uncertainty metric data.

[0051] Morphological dilation is a neighborhood operation based on set theory. It uses a structuring element with a specific shape and size to perform a sliding window scan on the initial cue mask. If any pixel position within the area covered by the structuring element is true (or non-zero), the center pixel position of the covered area is set to true (or its local maximum value is taken). For example, the structuring element can be a 3×3 square kernel or a 5×5 circular kernel, and the dilation radius can be adaptively adjusted according to the anatomical scale of the object to be segmented.

[0052] This embodiment reduces the constraint of coarse segmentation feature maps on segmentation edges by employing a morphological dilation mechanism, enabling the calibration cue mask to more fully cover potential misclassified regions. In particular, it can provide stronger spatial inclusiveness for the slender shape of neural structures, thereby providing more complete localization guidance information for downstream networks.

[0053] Specifically, the segmentation calibration network adopts an uncertainty-guided CSC (Coarse-to-SAM Calibration) architecture, which includes a cue-driven network with an adapter. The cue-driven network includes an image encoder, a cue encoder, a mask decoder, an image adapter disposed between the image encoder and the mask decoder, and a cue adapter disposed between the cue encoder and the mask decoder.

[0054] In an optional embodiment, calibrating the object region image according to the calibration cue mask to determine the guide feature map includes: inputting the calibration cue mask and the object region image into the cue driving network to obtain the output guide feature map.

[0055] Figure 2 This is a schematic diagram of the architecture of a segmentation calibration network provided in one embodiment of the present disclosure. Specifically, F0 represents the coarse segmentation feature map, I' represents the object region image, P0 represents the calibration cue mask, and F G The guided feature map is represented by a segmentation calibration network, which includes an uncertainty analysis module and a cue-driven network with an adapter.

[0056] Based on the above embodiments, for example, the image encoder can be selected from ViT-Base architecture, Tiny-ViT architecture, or EfficientNet architecture, the cue encoder can be selected from convolutional neural network or Transformer architecture, the image adapter and cue adapter can be selected from multilayer perceptron, one-dimensional convolutional network or feature mapping network, and the mask decoder can be selected from two bidirectional attention layers and mask prediction head, lightweight U-Net variant or nonlinear fusion layer, etc. Here, only the architecture of each module in the cue-driven network is illustrated by example, and its specific architecture and parameters can be flexibly adjusted according to task requirements.

[0057] Specifically, the cue-driven network achieves collaborative modeling of topological constraints and visual representations. Through cue branches and image branches, it extracts spatial feature information of the calibration cue mask and constructs the visual feature space of the object region image, respectively. Then, through a lightweight decoder, it uses a multi-source feature interaction mechanism to realize the dynamic calibration process from coarse-grained semantic localization to pixel-level anatomical reconstruction. It effectively integrates the advantages of cue guidance with the anatomical structural characteristics of medical images, corrects problems such as edge offset and region omission in the coarse segmentation stage, calibrates the feature representation of uncertain regions in the object region image, and finally outputs a feature map with accurate anatomical edges and complete detailed textures.

[0058] In another optional embodiment, calibrating the object region image according to the calibration cue mask to determine the guide feature map includes: inputting the calibration cue mask and the object region image into the cue driving network to obtain the output calibration feature map; upsampling the calibration feature map to obtain an upsampled feature map, and performing channel alignment and pixel-wise summation based on the upsampled feature map and the coarse segmentation feature map to obtain the guide feature map.

[0059] Among them, the upsampled feature map represents the feature distribution with the same resolution as the object region image, which can clearly present the key details such as the edge and contour of the object to be segmented. The guiding feature map not only retains the global localization information of the coarse segmentation stage, but also makes up for the segmentation error of the coarse segmentation stage, strengthens the geometric consistency and texture coherence of the local anatomical structure, and further improves the segmentation accuracy and reliability of medical images.

[0060] S150. Through the fine segmentation network, fine segmentation is performed based on each semantic feature map and the guiding feature map to obtain the target segmentation result corresponding to the object to be segmented.

[0061] Among them, the guiding feature map is used to provide accurate spatial constraints and semantic guidance for fine segmentation, prompting the fine segmentation process to focus on the key regions, morphological edges and local details of the object to be segmented, effectively weakening the interference of irrelevant background information, and improving the segmentation accuracy and completeness of small targets, weak edges and complex contour regions.

[0062] Specifically, the fine segmentation network includes a feature fusion network and a fine segmentation head. The feature fusion network is used to perform feature fusion on each semantic feature map according to the guiding feature map to obtain a fused semantic feature map. The fine segmentation head is used to perform pixel-level classification on the semantic feature map and output the target segmentation result of the object to be segmented.

[0063] In an optional embodiment, the feature fusion network includes a multi-scale fusion module and a guided fusion module. The step of performing refined segmentation based on the semantic feature maps and the guided feature maps through the fine segmentation network to obtain the target segmentation result corresponding to the object to be segmented includes: performing multi-scale concatenation and channel-weighted fusion on the semantic feature maps using the multi-scale fusion module to obtain an initial fused feature map; performing channel-wise weighted fusion of the initial fused feature map and the guided feature map using the guided fusion module to obtain a fused semantic feature map; and performing pixel-level classification on the fused semantic feature map using the fine segmentation head to output the target segmentation result.

[0064] Specifically, the multi-scale fusion module is used to enhance the contextual information of multi-scale semantics in the object region image. It captures hierarchical semantics from local texture to global structure through multi-scale stitching. The channel attention mechanism dynamically calibrates the response intensity of each channel, strengthens the feature channels that are discriminative for key anatomical structures, and suppresses redundant or noisy channels. The guided fusion module is used to achieve semantic alignment and spatial synergy between the guided feature map and the initial fused feature map. Through channel-wise weighted fusion, the spatial constraint capability of the guided feature map is accurately injected into the semantic feature space, ensuring the interpretability of the guided feature map at the semantic level and the accuracy of its spatial positioning.

[0065] In one alternative embodiment, the fine segmentation head consists of a convolutional layer and a binary activation function.

[0066] Figure 3 This is a schematic diagram illustrating the architecture of a specific example of an image segmentation model provided in one embodiment of this disclosure. Specifically, I represents the medical image to be segmented, F0 represents the coarse segmentation feature map, I' represents the object region image, F1 represents the guiding feature map, and F... out The target segmentation results are represented by the red dashed box, which represents the coarse segmentation network; the blue dashed box, which represents the segmentation calibration network; and the green dashed box, which represents the fine segmentation network. f1-f5 together constitute the multi-scale encoder. The hierarchical feature maps corresponding to f3-f5 are used as semantic feature maps and input to the coarse segmentation head and the fine segmentation network, respectively.

[0067] The technical solution of this embodiment deeply integrates a coarse-fine segmentation architecture with an uncertainty guidance mechanism to construct a progressive segmentation paradigm driven by a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation network performs multi-scale feature extraction and coarse segmentation to achieve rapid localization of the object to be segmented. Based on the coarse segmentation feature map, the medical image to be segmented is cropped to obtain the object region image. The segmentation calibration network performs uncertainty modeling to obtain a calibration cue mask and calibrates the object region image, achieving a precise transition from coarse-grained semantic localization to pixel-level anatomical reconstruction. Finally, the fine segmentation network achieves deep fusion of multi-scale semantic features based on the guidance feature map, outputting the target segmentation result of the object to be segmented. The structured prior provided by the segmentation calibration network enhances the anti-interference ability of the image segmentation model and its feature expression ability for edge structures, solving the problem of indistinct segmentation features for small targets in medical images, improving the accuracy of small target object recognition, and enhancing the segmentation accuracy and robustness of medical images.

[0068] Figure 4 This is a flowchart illustrating another medical image segmentation method provided in one embodiment of the present disclosure. This embodiment further refines the coarse segmentation network and fine segmentation network described in the above embodiments. Figure 4 As shown, the method includes: S210. Obtain the medical image to be segmented, and the image segmentation model that has been trained and matches the object to be segmented.

[0069] S210 in this embodiment is the same as that in the above embodiment. Figure 1 The S110 shown is the same or similar, and will not be described again here.

[0070] S220. Extract multi-scale feature data of the medical image to be segmented through each feature extraction layer in the coarse segmentation network.

[0071] In this embodiment, the multi-scale encoder in the coarse segmentation network includes a backbone network and at least two multi-scale enhancement layers, wherein the backbone network includes at least three cascaded feature extraction layers.

[0072] For example, the backbone network can be a pre-trained ResNet series architecture, ViT architecture, EfficientNet architecture, or other backbone networks with multi-scale feature extraction capabilities. The cascaded feature extraction layers have long-distance dependency modeling capabilities and can adapt to segmentation tasks of slender anatomical structures such as neural structures and blood vessels.

[0073] In this embodiment, the multi-scale feature data includes hierarchical feature maps that correspond one-to-one with the feature extraction layer. The deeper the feature extraction layer, the lower the spatial resolution and the more abstract the semantic information of its corresponding hierarchical feature map.

[0074] S230. Through the multi-scale enhancement layer in the coarse segmentation network, the hierarchical feature map output by the preset feature extraction layer connected upstream is enhanced to obtain a semantic feature map.

[0075] In this embodiment, the preset feature extraction layer is a feature extraction layer in the backbone network whose number of cascaded layers exceeds a preset layer threshold, and the preset layer threshold is not less than 1.

[0076] The preset layer threshold can be dynamically set according to the total number of cascaded layers of the backbone network. For example, when the backbone network adopts the Tiny U-Net encoder architecture, the total number of cascaded layers is 5, and the preset layer threshold can be set to 3 layers; when the backbone network adopts the ViT-Base architecture, the total number of cascaded layers is 12, and the preset layer threshold can be 8 layers.

[0077] In one optional embodiment, the network architecture of the multi-scale enhancement layer is a multi-scale adaptive selective fusion module (MSAF), which includes a preprocessing module, a feature extraction module corresponding to multiple feature scales, a scale fusion module, and an enhancement output module.

[0078] In this embodiment, the step of enhancing the hierarchical feature map output by the upstream connected preset feature extraction layer through the multi-scale enhancement layer to obtain a semantic feature map includes: preprocessing the hierarchical feature map through the preprocessing module to obtain a preprocessed feature map; extracting features from the hierarchical feature map through each of the parallel feature extraction modules to obtain a multi-scale feature set; performing scale-dimension weighted fusion on the multi-scale feature set through the scale fusion module to obtain a fused feature map; and performing channel alignment and pixel-wise summation based on the preprocessed feature map and the fused feature map through the enhancement output module to obtain a semantic feature map.

[0079] In one optional embodiment, the preprocessing module includes cascaded dilated convolutional layers and convolutional layers. The dilated convolutional layers are used to expand the receptive field and capture broader contextual information without reducing spatial resolution; the convolutional layers are responsible for calibrating the feature responses in the channel dimension to improve semantic discriminative power.

[0080] Specifically, each of the feature extraction modules is used to extract feature information at multiple feature scales. In an optional embodiment, each feature extraction module corresponds to a different convolution kernel size, or each feature extraction module corresponds to a different pooling factor.

[0081] In an optional embodiment, the feature extraction module includes a pooling layer and a convolutional layer. The pooling layer is used to downsample the hierarchical feature map at a corresponding ratio, and the convolutional layer performs a convolution operation on the output of the pooling layer to extract local spatial features.

[0082] For example, each feature extraction module corresponds to four downsampling ratios of 4×, 2×, 1× and 0.5×, respectively, to capture multi-granular context from global to local, but is not limited to the given example.

[0083] In one optional embodiment, the scale fusion module includes a stitching unit, a pooling layer, a weight modeling unit, a convolution unit, and a weighted fusion unit; wherein, the stitching unit is used to stitch the multi-scale feature set along the channel dimension to obtain a stitched feature map; the pooling layer is used to perform global pooling processing on the stitched feature map to obtain a global feature map; the weight modeling unit is used to perform spatial dimension weight modeling processing on the global feature map to obtain a spatial weight map; the convolution unit is used to perform multi-scale convolution mapping on the spatial weight map to obtain a scale weight set; and the weighted fusion unit is used to perform a scale-wise weighted summation of the multi-scale feature set according to the scale weight set to obtain a fused feature map.

[0084] For example, the pooling layer can be a max pooling layer, an average pooling layer, or an average max pooling layer. The weight modeling unit is used to provide dynamically learnable scale weights (SW) for the feature map at each feature scale in the multi-scale feature set. For example, the weight modeling unit includes, but is not limited to, a gating mechanism unit, an adaptive convolution unit, or an attention unit.

[0085] Figure 5 This is a schematic diagram of the architecture of a specific example of a coarse segmentation network provided in an embodiment of the present disclosure. Specifically, m1-m3 represent the feature extraction layers in the backbone network, S2 represents the hierarchical feature map output by the second feature extraction layer in the backbone network, f3-f5 represent the cascaded structure of the feature extraction layer and the corresponding multi-scale enhancement layer, and S3 represents the semantic feature map output by the multi-scale enhancement layer cascaded with the third feature extraction layer in the backbone network.

[0086] For example, in the first In a multi-scale enhancement layer, the feature maps are fused. Satisfy the following formula: in, Indicates the number of feature extraction modules. Represents the scale weight set, Indicates the first The feature maps output by each feature extraction module.

[0087] This embodiment extracts features from the hierarchical feature map at multiple feature scales and performs adaptive weighted fusion on the resulting multi-scale feature sets. While preserving global semantic information, it enhances the ability to model objects with slender shapes and small structures that are waiting to be segmented. It avoids the defects of fixed-scale feature extraction, such as loss of key features and incoherent context, and further improves the segmentation accuracy and stability of medical images.

[0088] S240. Using the coarse segmentation head in the coarse segmentation network, coarse segmentation is performed based on each semantic feature map to obtain a coarse segmentation feature map.

[0089] In this embodiment, S240 corresponds to or is similar to the coarse segmentation head in the above embodiments, and will not be described again here.

[0090] S250. The medical image to be segmented is cropped according to the coarse segmentation feature map to obtain the object region image.

[0091] S260. Through the segmentation calibration network, perform uncertainty analysis on the coarse segmentation feature map to obtain a calibration prompt mask, and calibrate the object region image according to the calibration prompt mask to determine the guiding feature map.

[0092] S250-S260 in this embodiment are the same as those in the above embodiment. Figure 1 The S130-S140 shown are the same or similar, and will not be described again here.

[0093] S270. Through the fine segmentation network, fine segmentation is performed based on each semantic feature map and the guiding feature map to obtain the target segmentation result corresponding to the object to be segmented.

[0094] In one alternative embodiment, S270 is the same as in the above embodiment. Figure 1 The S150 shown corresponds to the same or similar.

[0095] In another alternative embodiment, the fine segmentation network includes a fine segmentation head and the semantic features. Figure 1 A corresponding edge optimization network; the step of performing fine segmentation based on the semantic feature maps and the guiding feature maps through the fine segmentation network to obtain the target segmentation result corresponding to the object to be segmented includes: performing localization optimization based on the feature map to be processed and the corresponding semantic feature map through a cascaded edge optimization network to obtain a target feature map; and performing feature mapping and pixel-level classification on the target feature map output by the last edge optimization network through the fine segmentation head to output the target segmentation result corresponding to the object to be segmented.

[0096] In this embodiment, when the edge optimization network is the edge optimization network corresponding to the last semantic feature map, the feature map to be processed is the guiding feature map. When the edge optimization network is the edge optimization network corresponding to other semantic feature maps, the feature map to be processed is the target feature map output by the cascaded previous edge optimization network. The other semantic feature maps are semantic feature maps other than the last semantic feature map.

[0097] Among them, the Edge Modification (EM) network is used to fuse and model the detailed cue information provided by the coarse segmentation network and the localization label information provided by the segmentation calibration network or the previous edge modification network, and output a more accurate edge feature representation. Multiple cascaded edge modification networks progressively optimize edge details on multi-scale semantic feature maps, thereby ensuring the continuity and structural integrity of the edge contour of the object to be segmented.

[0098] For example, edge optimization networks may select convolutional residual architectures, channel attention mechanisms, spatial pyramid pooling, or deconvolutional upsampling, but are not limited to the given examples.

[0099] In one optional embodiment, the edge optimization network includes a channel weight layer, a feature aggregation layer, a spatial weight layer, and an output layer. The step of obtaining a target feature map by performing localization optimization based on the feature map to be processed and the corresponding semantic feature map through the cascaded edge optimization network includes: mapping the semantic feature map to channel dimensions using the channel weight layer to obtain a channel weight matrix; enhancing the semantic feature map using the channel weight matrix using the feature aggregation layer to obtain a channel-enhanced feature map; and summing the channel-enhanced feature map with the feature map to be processed to obtain an aggregated feature map; mapping the aggregated feature map to spatial dimensions using the spatial weight layer to obtain a spatial weight matrix; and enhancing the aggregated feature map using the spatial weight matrix using the output layer to obtain a spatially enhanced feature map; and summing the spatially enhanced feature map with the aggregated feature map to obtain the target feature map.

[0100] The channel weight matrix represents the contribution of each channel to edge localization, and is used to strengthen key semantic channels related to edge localization and suppress interference from redundant noise channels; the spatial weight matrix represents the edge response intensity of each pixel position, and is used to highlight edge feature regions and weaken internal smooth regions, thereby accurately depicting the spatial distribution characteristics of edge structures.

[0101] For example, the channel weight layer may employ average pooling and fully connected layers, SE architecture (Squeeze and Excitation), or channel self-attention mechanism, etc., and the spatial weight matrix may employ average pooling, spatial attention mechanism, or dilated convolution, etc., but is not limited to the examples given above.

[0102] The channel-enhanced feature map represents the semantic feature information after being weighted by the channel weight matrix. Its channel dimension response intensity is highly coupled with the semantics of the edge structure. The aggregated feature map integrates the channel-enhanced feature map and the feature map to be processed. It retains the original spatial details provided by the feature map to be processed and injects semantically guided edge localization priors, so that the feature expression achieves collaborative alignment in both spatial and semantic dimensions.

[0103] In an optional embodiment, the channel weight layer includes a cascaded feature extraction module, a multilayer perceptron, and an activation module; the spatial weight layer includes a cascaded feature extraction module and an activation module. The feature extraction module includes a max pooling module, an average pooling module, and a spatial fusion module. The max pooling module performs max pooling on the input data; the average pooling module performs average pooling on the input data; and the spatial fusion module sums the outputs of the max pooling module and the average pooling module to obtain feature statistics in the spatial dimension. The input data is the semantic feature map or the aggregated feature map.

[0104] Figure 6 This is a schematic diagram of the architecture of a specific example of a fine segmentation network provided in one embodiment of this disclosure. Specifically, EM1-EM3 represent cascaded edge optimization networks, F1 represents the guiding feature map, F2-F3 represent the target feature maps, MLP is a multilayer perceptron, "+" represents the element-wise addition operation of two feature maps, "S" represents the activation function, and "×" represents the element-wise multiplication operation of the feature map and the weight matrix. channel w represents the channel weight matrix. spatial This represents the spatial weight matrix.

[0105] This embodiment uses a cascaded edge optimization network to iteratively optimize edge localization from coarse to fine, and adapts semantic feature maps of different scales in a layered manner. This allows edge details to be fully modeled and enhanced at multiple scales, effectively avoiding the segmentation break and morphological distortion of edge structures, and improving the edge continuity and topological consistency of the target segmentation results.

[0106] Optionally, based on this embodiment, the method further includes: inputting training images from the training image set into the image segmentation model to be trained to obtain predicted segmentation data corresponding to each training image; determining a segmentation loss term corresponding to the training image based on the predicted segmentation data and the true segmentation label of the training image; determining a total loss value based on each segmentation loss term; and updating the model parameters of the image segmentation model through backpropagation based on the total loss value until the training termination condition is met.

[0107] The training image set supports modal images in multiple surgical scenarios. Taking the thyroid gland as an example, the training image set includes thyroid gland images from various approaches such as transthoracic breast, transaxillary, and transoral vestibule, thereby overcoming the problems of scarcity and uniformity in the training image set.

[0108] Specifically, video footage from multiple surgical scenarios is acquired, effective segments are selected from the video footage, and video segments highly relevant to the segmentation task are filtered out. Frame-by-frame truncation and sieving operations are performed on the selected video segments to select keyframes that meet quality requirements as training images. The objects to be segmented in the training images are labeled. After labeling, an expert physician collaborative verification mechanism is used to check and correct labeling errors. The verified labeled data is divided into a dataset, which is divided into a training set, a validation set, and a test set according to a preset ratio to form a training image set, and the model training process is carried out.

[0109] For example, the image segmentation model to be trained can be an untrained image segmentation model or an image segmentation model pretrained on a publicly available medical dataset. Specifically, a transfer learning method is used to fine-tune the training on the training image set provided in this embodiment, thereby improving the convergence speed of the model training. In addition, the image segmentation model to be trained can also be an image segmentation model that has undergone one or more rounds of training, with the sample size and / or complexity of the training image set used in each round of training increasing sequentially, but it is not limited to the example given above.

[0110] In this embodiment, the predicted segmentation data includes coarse segmentation labels and fine segmentation labels corresponding to each of the target feature maps. The loss function corresponding to the segmentation loss term includes, but is not limited to, intersection-over-union (IoU) loss, binary cross-entropy loss, Dice loss, and Tversky loss. Among them, IoU loss is used to measure the degree of overlap between the segmentation labels predicted by the image segmentation model and the true segmentation labels, binary cross-entropy loss focuses on calibrating pixel-level classification confidence, Dice loss enhances the sensitivity to small target regions, and Tversky loss further improves the robustness of the model in class imbalance scenarios by adjusting the weights of false positives and false negatives.

[0111] For example, the total loss value Satisfy the following formula: in, Indicates the first The segmentation loss term corresponding to each training image Indicates the first Cross-Union Loss for each training image Indicates the first The binary cross-entropy loss corresponding to each training image.

[0112] Based on the above training strategies, by replacing the training sample set and fine-tuning the model, generalization and adaptation to data collected by different surgical procedures, different anatomical structures, and even different imaging equipment can be achieved, thereby significantly improving the robustness and transferability of the model in real-world application scenarios.

[0113] The technical solution of this embodiment solves the problem of feature dilution of small targets during full-scale feature extraction by selecting feature extraction layers in the coarse segmentation network with more than a preset threshold number of cascaded layers. It avoids the interference of shallow features with low discriminative power on segmentation accuracy and enhances the output hierarchical feature maps to obtain semantic feature maps. This improves the semantic representation ability of the coarse segmentation network for small target regions and enhances the localization accuracy and segmentation consistency of the image segmentation model for minute anatomical structures. Especially in scenarios where processing operations are underway, the early exposure of the object to be segmented is lower. This embodiment achieves the segmentability of minute anatomical structures, significantly reducing the false negative and false positive rates for minute anatomical structures.

[0114] Based on the above embodiments, the method may optionally further include: after the image segmentation model is trained, compressing the image segmentation model and optimizing the model inference by relying on an inference acceleration architecture.

[0115] For example, the compression method includes, but is not limited to, various forms such as model pruning and weight quantization. The number of model parameters in the compressed image segmentation model is less than 60% of that before compression, so as to reduce the excessive dependence of the image segmentation model on computing resources. The inference acceleration architecture includes, but is not limited to, TensorRT inference graph optimization, ONNX Runtime or OpenVINO, to adapt to different model deployment platforms.

[0116] In one optional embodiment, the model deployment platform is an edge hardware platform. The edge hardware platform is equipped with a multi-threaded operation architecture that separates acquisition, inference, and rendering. It can schedule the processor and image processor to work together in parallel, fully release the hardware computing power, and ensure that the image segmentation process meets the real-time operation indicators of low latency and high frame rate, while taking into account both segmentation accuracy and real-time operation.

[0117] Based on the above embodiments, optionally, the medical image to be segmented is obtained in the following manner: by sequentially reading image frames from the real-time video stream through a video interface connected to the medical imaging equipment, and using the read current image frame as the medical image to be segmented; wherein, the real-time video stream is the image data stream during the device intervention process under the image-guided paradigm.

[0118] Among them, the image-guided paradigm refers to the technical architecture that relies on medical imaging equipment, synchronous transmission technology and visualization system to build an image-guided system, providing operators with real-time and accurate information on tissue structure and instrument location, and is suitable for various instrument intervention application scenarios with strict requirements for real-time performance and positioning accuracy.

[0119] For example, the video interface can adopt standard interface protocols such as HDMI (High Definition Multimedia Interface), SDI (Serial Digital Interface), or MIPI (Mobile Industry Processor Interface). Without changing the original hardware architecture, it can achieve plug-and-play docking with medical imaging equipment, with strong equipment compatibility and low deployment cost.

[0120] In this embodiment, the method further includes: annotating and rendering the medical image to be segmented according to the target segmentation result to obtain an enhanced image frame, and visually displaying the enhanced image on a display terminal.

[0121] For example, the annotation rendering method may be to overlay a highlight mask, outline, text label or spatial coordinate mark on the medical image to be segmented to provide location clues for the object to be segmented, but it is not limited to the example given above.

[0122] In a specific application example, the method further includes: acquiring spatial pose data between the interventional instrument and the object to be segmented; and outputting information to indicate risky operations based on the spatial pose data. For example, the form of the indication information includes, but is not limited to, visual highlighting warnings, voice alarms, flashing indicator lights, or tactile feedback.

[0123] Figure 7 This is a comparative schematic diagram of target segmentation results provided in one embodiment of the present disclosure. Specifically, Figure 7 Four sets of comparison results are shown. Each set, from left to right, consists of a thyroid image, the true segmentation label of the recurrent laryngeal nerve, and the target segmentation label. In the segmentation label, the black area represents the background in the thyroid image, and the white area delineates the anatomical boundary of the recurrent laryngeal nerve.

[0124] The image segmentation model provided in this embodiment has a DICE coefficient of no less than 0.75 in the recurrent laryngeal nerve segmentation task, which significantly reduces the risk of misjudgment and missed detection of the recurrent laryngeal nerve.

[0125] The following are embodiments of the medical image segmentation apparatus provided in this disclosure. This apparatus and the medical image segmentation method described above belong to the same inventive concept. For details not described in detail in the embodiments of the medical image segmentation apparatus, please refer to the content of the medical image segmentation method in the above embodiments.

[0126] Figure 8 This is a schematic diagram of the structure of a medical image segmentation device provided in one embodiment of this disclosure. Figure 8 As shown, the device includes: an image segmentation model acquisition module 310, a coarse segmentation feature map determination module 320, an object region image determination module 330, a guiding feature map determination module 340, and a target segmentation result determination module 350.

[0127] The image segmentation model acquisition module 310 is used to acquire the medical image to be segmented and the trained image segmentation model that matches the object to be segmented. The image segmentation model consists of a coarse segmentation network, a segmentation calibration network and a fine segmentation network. The coarse segmentation feature map determination module 320 is used to extract at least two semantic feature maps from the medical image to be segmented through the coarse segmentation network, and to perform coarse segmentation processing based on each of the semantic feature maps to obtain a coarse segmentation feature map. The object region image determination module 330 is used to crop the medical image to be segmented based on the coarse segmentation feature map to obtain the object region image; The guided feature map determination module 340 is used to perform uncertainty analysis on the coarse segmentation feature map through the segmentation calibration network to obtain a calibration prompt mask, and to calibrate the object region image according to the calibration prompt mask to determine the guided feature map; The target segmentation result determination module 350 is used to perform fine segmentation based on the semantic feature maps and the guiding feature maps through the fine segmentation network to obtain the target segmentation result corresponding to the object to be segmented.

[0128] The technical solution of this embodiment deeply integrates a coarse-fine segmentation architecture with an uncertainty guidance mechanism to construct a progressive segmentation paradigm driven by a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation network performs multi-scale feature extraction and coarse segmentation to achieve rapid localization of the object to be segmented. Based on the coarse segmentation feature map, the medical image to be segmented is cropped to obtain the object region image. The segmentation calibration network performs uncertainty modeling to obtain a calibration cue mask and calibrates the object region image, achieving a precise transition from coarse-grained semantic localization to pixel-level anatomical reconstruction. Finally, the fine segmentation network achieves deep fusion of multi-scale semantic features based on the guidance feature map, outputting the target segmentation result of the object to be segmented. The structured prior provided by the segmentation calibration network enhances the anti-interference ability of the image segmentation model and its feature expression ability for edge structures, solving the problem of indistinct segmentation features for small targets in medical images, improving the accuracy of small target object recognition, and enhancing the segmentation accuracy and robustness of medical images.

[0129] In an optional embodiment, the guided feature map determination module 340 includes: The calibration prompt mask determination unit is used to perform pixel-by-pixel feature extraction on the coarse segmentation feature map to obtain uncertainty measurement data; Based on preset filtering conditions and the uncertainty measurement data, the coarse segmentation feature map is filtered to obtain an initial prompt mask; The initial cue mask is subjected to morphological dilation to obtain a calibration cue mask.

[0130] In an optional embodiment, the segmentation calibration network includes a cue-driven network with an adapter, and a guided feature map determination module 340, comprising: The guidance feature map determination unit is used to input the calibration prompt mask and the object region image into the prompt driving network to obtain the output calibration feature map; The calibration feature map is upsampled to obtain an upsampled feature map, and channel alignment and pixel-by-pixel summation are performed based on the upsampled feature map and the coarse segmentation feature map to obtain a guide feature map.

[0131] In one alternative embodiment, the coarse segmentation network includes a backbone network and at least two multi-scale enhancement layers, wherein the backbone network includes at least three cascaded feature extraction layers; The coarse segmentation feature map determination module 320 includes: A multi-scale feature data extraction unit is used to extract multi-scale feature data of the medical image to be segmented through each of the feature extraction layers. The multi-scale feature data includes hierarchical feature maps that correspond one-to-one with the feature extraction layers. The semantic feature map determination unit is used to perform feature enhancement on the hierarchical feature map output by the upstream connected preset feature extraction layer through the multi-scale enhancement layer to obtain the semantic feature map; The preset feature extraction layer is a feature extraction layer in the backbone network whose number of cascaded layers exceeds a preset layer threshold, and the preset layer threshold is not less than 1.

[0132] In one optional embodiment, the multi-scale enhancement layer includes a preprocessing module, a feature extraction module corresponding to multiple feature scales, a scale fusion module, and an enhancement output module; The semantic feature map determination unit is specifically used for: The preprocessing module preprocesses the hierarchical feature map to obtain a preprocessed feature map, and the parallel feature extraction modules extract features from the hierarchical feature map to obtain a multi-scale feature set. The scale fusion module performs scale-dimension weighted fusion on the multi-scale feature set to obtain a fused feature map. The enhanced output module performs pixel-by-pixel summation based on the preprocessed feature map and the fused feature map to obtain a semantic feature map.

[0133] In one optional embodiment, the scale fusion module includes a stitching unit, a pooling layer, a weight modeling unit, a convolution unit, and a weighted fusion unit; The concatenation unit is used to concatenate the multi-scale feature set along the channel dimension to obtain a concatenated feature map; the pooling layer is used to perform global pooling on the concatenated feature map to obtain a global feature map; the weight modeling unit is used to perform spatial dimension weight modeling on the global feature map to obtain a spatial weight map; the convolution unit is used to perform multi-scale convolution mapping on the spatial weight map to obtain a scale weight set; and the weighted fusion unit is used to perform a scale-wise weighted summation of the multi-scale feature set according to the scale weight set to obtain a fused feature map.

[0134] In an optional embodiment, the fine segmentation network includes a fine segmentation head and the semantic features. Figure 1 One corresponding edge optimization network; The target segmentation result determination module 350 includes: The target feature map determination unit is used to perform localization optimization based on the feature map to be processed and the corresponding semantic feature map through a cascaded edge optimization network to obtain the target feature map. The target segmentation result determination unit is used to perform feature mapping and pixel-level classification on the target feature map output by the last edge optimization network through the fine segmentation head, and output the target segmentation result corresponding to the object to be segmented; Wherein, when the edge optimization network is the edge optimization network corresponding to the last semantic feature map, the feature map to be processed is the guiding feature map; when the edge optimization network is the edge optimization network corresponding to other semantic feature maps, the feature map to be processed is the target feature map output by the cascaded previous edge optimization network; and the other semantic feature maps are semantic feature maps other than the last semantic feature map.

[0135] In one optional embodiment, the edge optimization network includes a channel weight layer, a feature aggregation layer, a spatial weight layer, and an output layer; The target feature map determination unit is specifically used for: The channel weight layer performs channel-dimensional weight mapping on the semantic feature map to obtain the channel weight matrix. The semantic feature map is enhanced by the feature aggregation layer according to the channel weight matrix to obtain a channel-enhanced feature map, and the channel-enhanced feature map and the feature map to be processed are summed to obtain an aggregated feature map. The spatial weight layer is used to perform spatial dimension weight mapping on the aggregated feature map to obtain a spatial weight matrix. The output layer performs feature enhancement on the aggregated feature map according to the spatial weight matrix to obtain a spatially enhanced feature map, and the spatially enhanced feature map and the aggregated feature map are summed to obtain the target feature map.

[0136] In one optional embodiment, the channel weighting layer includes a cascaded feature extraction module, a multilayer perceptron, and an activation module, and the spatial weighting layer includes a cascaded feature extraction module and an activation module. The feature extraction module includes a max pooling module, an average pooling module, and a spatial fusion module. The max pooling module performs max pooling on the input data; the average pooling module performs average pooling on the input data; and the spatial fusion module sums the outputs of the max pooling module and the average pooling module to obtain feature statistics in the spatial dimension. The input data is the semantic feature map or the aggregated feature map.

[0137] In an optional embodiment, the device further includes: The medical image acquisition module is used to sequentially read image frames from the acquired real-time video stream through a video interface connected to the medical imaging equipment, and to use the read current image frame as the medical image to be segmented; wherein, the real-time video stream is the image data stream during the instrument intervention process under the image-guided paradigm; The annotation and display module is used to annotate and render the medical image to be segmented according to the target segmentation result to obtain an enhanced image frame, and to visualize the enhanced image on a display terminal.

[0138] In one alternative embodiment, the imaging site of the medical imaging device is the thyroid gland, and the object to be segmented is the recurrent laryngeal nerve.

[0139] The medical image segmentation apparatus provided in this disclosure can execute the medical image segmentation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0140] Figure 9 This is a schematic diagram illustrating the structure of a specific example of a medical image segmentation device provided in one embodiment of this disclosure. Figure 9 Taking the endoscopic-guided thyroid procedure as an example, the system includes an optical probe for acquiring video streams, a light source host for providing light to the optical probe, a video processing host for preprocessing the video streams, a capture card for reading real-time video streams from the video processing host via an HDMI / SDI interface, an edge hardware platform with an image segmentation model deployed, and a rendering module for rendering and overlaying the target segmentation results in real time and outputting them to the display terminal.

[0141] like Figure 9 As shown, the medical image below the video processing host is the initial image frame captured in real time by the endoscope, while the medical image below the display terminal is the enhanced image frame after segmentation processing by the image segmentation model and superimposed with the recurrent laryngeal nerve segmentation mask.

[0142] Figure 10 This is a schematic diagram of the structure of an electronic device provided according to one embodiment of the present disclosure. The electronic device 10 is intended to represent various forms of digital computers and various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0143] like Figure 10 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor 11. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14.

[0144] The input / output (I / O) interface 15 is also connected to the bus 14. Multiple components in the electronic device 10 are connected to the I / O interface 15, including: input unit 16, output unit 17, storage unit 18, and communication unit 19. The communication unit 19 allows the electronic device 10 to exchange information or data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. The processor 11 executes the various methods and processes described above, such as the medical image segmentation method provided in the above embodiments.

[0146] In some embodiments, the medical image segmentation method provided in the above embodiments can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the medical image segmentation method described above can be performed.

[0147] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.

[0148] The computer program may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowchart and / or block diagram are performed. The computer program may execute entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The systems and technologies described herein can be implemented in any combination of back-end components, middleware components, or front-end components in a computing system that interconnects its components via digital data communication of any form or medium. The computing system may include clients and servers, which typically interact via a communication network, with the client-server relationship established by computer programs running on the respective computers and having a client-server relationship with each other.

[0151] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for segmenting medical images, characterized in that, include: Acquire a medical image to be segmented, and a trained image segmentation model that matches the object to be segmented. The image segmentation model consists of a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation network is used to extract multi-scale features from the medical image to be segmented to obtain at least two semantic feature maps, and coarse segmentation is performed based on each semantic feature map to obtain a coarse segmentation feature map. The medical image to be segmented is cropped based on the coarse segmentation feature map to obtain the object region image; Uncertainty analysis is performed on the coarse segmentation feature map using the segmentation calibration network to obtain a calibration cue mask. The object region image is then calibrated based on the calibration cue mask to determine the guiding feature map. The calibration cue mask represents the region with a high uncertainty metric value in the coarse segmentation feature map. The fine segmentation network performs fine segmentation based on the semantic feature maps and the guiding feature maps to obtain the target segmentation result corresponding to the object to be segmented. The coarse segmentation network includes a backbone network and at least two multi-scale enhancement layers, wherein the backbone network includes at least three cascaded feature extraction layers; The process of extracting at least two semantic feature maps from the medical image to be segmented using multi-scale features includes: Through each of the feature extraction layers, multi-scale feature data of the medical image to be segmented is extracted, and the multi-scale feature data includes hierarchical feature maps that correspond one-to-one with the feature extraction layers. The multi-scale enhancement layer enhances the hierarchical feature map output by the upstream connected preset feature extraction layer to obtain a semantic feature map. The preset feature extraction layer is a feature extraction layer in the backbone network whose number of cascaded layers exceeds a preset layer threshold, and the preset layer threshold is not less than 1. The fine segmentation network includes a fine segmentation head and an edge optimization network that corresponds one-to-one with the semantic feature map; The step of performing refined segmentation based on the semantic feature maps and the guiding feature maps through the refined segmentation network to obtain the target segmentation result corresponding to the object to be segmented includes: By using a cascaded edge optimization network, the localization optimization is performed based on the feature map to be processed and the corresponding semantic feature map to obtain the target feature map; The fine segmentation head performs feature mapping and pixel-level classification on the target feature map output by the last edge optimization network, and outputs the target segmentation result corresponding to the object to be segmented. Wherein, when the edge optimization network is the edge optimization network corresponding to the last semantic feature map, the feature map to be processed is the guiding feature map; when the edge optimization network is the edge optimization network corresponding to other semantic feature maps, the feature map to be processed is the target feature map output by the cascaded previous edge optimization network; and the other semantic feature maps are semantic feature maps other than the last semantic feature map.

2. The method according to claim 1, characterized in that, The uncertainty analysis performed on the coarse segmentation feature map to obtain the calibration hint mask includes: Pixel-by-pixel feature extraction is performed on the coarse segmentation feature map to obtain uncertainty measurement data; Based on preset filtering conditions and the uncertainty measurement data, the coarse segmentation feature map is filtered to obtain an initial prompt mask; The initial cue mask is subjected to morphological dilation to obtain a calibration cue mask.

3. The method according to claim 1, characterized in that, The segmentation calibration network includes a cue-driven network equipped with an adapter. The step of calibrating the object region image based on the calibration cue mask to determine the guiding feature map includes: The calibration cue mask and the object region image are input into the cue driving network to obtain the output calibration feature map; The calibration feature map is upsampled to obtain an upsampled feature map, and channel alignment and pixel-by-pixel summation are performed based on the upsampled feature map and the coarse segmentation feature map to obtain a guide feature map.

4. The method according to claim 1, characterized in that, The multi-scale enhancement layer includes a preprocessing module, a feature extraction module corresponding to multiple feature scales, a scale fusion module, and an enhancement output module; The step of enhancing the hierarchical feature map output by the upstream connected preset feature extraction layer through the multi-scale enhancement layer to obtain a semantic feature map includes: The preprocessing module preprocesses the hierarchical feature map to obtain a preprocessed feature map, and the parallel feature extraction modules extract features from the hierarchical feature map to obtain a multi-scale feature set. The scale fusion module performs scale-dimension weighted fusion on the multi-scale feature set to obtain a fused feature map. The enhanced output module performs pixel-by-pixel summation based on the preprocessed feature map and the fused feature map to obtain a semantic feature map.

5. The method according to claim 4, characterized in that, The scale fusion module includes a stitching unit, a pooling layer, a weight modeling unit, a convolution unit, and a weighted fusion unit; The concatenation unit is used to concatenate the multi-scale feature set along the channel dimension to obtain a concatenated feature map; the pooling layer is used to perform global pooling on the concatenated feature map to obtain a global feature map; the weight modeling unit is used to perform spatial dimension weight modeling on the global feature map to obtain a spatial weight map; the convolution unit is used to perform multi-scale convolution mapping on the spatial weight map to obtain a scale weight set; and the weighted fusion unit is used to perform a scale-wise weighted summation of the multi-scale feature set according to the scale weight set to obtain a fused feature map.

6. The method according to claim 1, characterized in that, The edge optimization network includes a channel weight layer, a feature aggregation layer, a spatial weight layer, and an output layer; The cascaded edge optimization network performs localization optimization based on the feature map to be processed and the corresponding semantic feature map to obtain the target feature map, including: The channel weight layer performs channel-dimensional weight mapping on the semantic feature map to obtain the channel weight matrix. The semantic feature map is enhanced by the feature aggregation layer according to the channel weight matrix to obtain a channel-enhanced feature map, and the channel-enhanced feature map and the feature map to be processed are summed to obtain an aggregated feature map. The spatial weight layer is used to perform spatial dimension weight mapping on the aggregated feature map to obtain a spatial weight matrix. The output layer performs feature enhancement on the aggregated feature map according to the spatial weight matrix to obtain a spatially enhanced feature map, and the spatially enhanced feature map and the aggregated feature map are summed to obtain the target feature map.

7. The method according to claim 6, characterized in that, The channel weighting layer includes a cascaded feature extraction module, a multilayer perceptron, and an activation module; the spatial weighting layer includes a cascaded feature extraction module and an activation module. The feature extraction module includes a max pooling module, an average pooling module, and a spatial fusion module. The max pooling module performs max pooling on the input data; the average pooling module performs average pooling on the input data; and the spatial fusion module sums the outputs of the max pooling module and the average pooling module to obtain feature statistics in the spatial dimension. The input data is the semantic feature map or the aggregated feature map.

8. The method according to any one of claims 1-7, characterized in that, The medical image to be segmented is obtained in the following manner: The system reads image frames from the real-time video stream sequentially through a video interface connected to medical imaging equipment, and uses the read image frame as the medical image to be segmented; wherein, the real-time video stream is the image data stream during the instrument intervention process under the image-guided paradigm. The method further includes: Based on the target segmentation result, the medical image to be segmented is annotated and rendered to obtain an enhanced image frame, and the enhanced image frame is visualized on a display terminal.

9. The method according to claim 8, characterized in that, The imaging site of the medical imaging device is the thyroid gland, and the object to be segmented is the recurrent laryngeal nerve.

10. A medical image segmentation device, characterized in that, include: The image segmentation model acquisition module is used to acquire the medical image to be segmented and the trained image segmentation model that matches the object to be segmented. The image segmentation model consists of a coarse segmentation network, a segmentation calibration network, and a fine segmentation network. The coarse segmentation feature map determination module is used to extract at least two semantic feature maps from the medical image to be segmented through the coarse segmentation network, and to perform coarse segmentation processing based on each of the semantic feature maps to obtain a coarse segmentation feature map. The object region image determination module is used to crop the medical image to be segmented based on the coarse segmentation feature map to obtain the object region image; The guided feature map determination module is used to perform uncertainty analysis on the coarse segmentation feature map through the segmentation calibration network to obtain a calibration prompt mask, and to calibrate the object region image according to the calibration prompt mask to determine the guided feature map; wherein, the calibration prompt mask represents the region with a high uncertainty metric value in the coarse segmentation feature map; The target segmentation result determination module is used to perform fine segmentation based on the semantic feature maps and the guiding feature maps through the fine segmentation network to obtain the target segmentation result corresponding to the object to be segmented; The coarse segmentation network includes a backbone network and at least two multi-scale enhancement layers, wherein the backbone network includes at least three cascaded feature extraction layers. The coarse segmentation feature map determination module includes: A multi-scale feature data extraction unit is used to extract multi-scale feature data of the medical image to be segmented through each of the feature extraction layers. The multi-scale feature data includes hierarchical feature maps that correspond one-to-one with the feature extraction layers. The semantic feature map determination unit is used to perform feature enhancement on the hierarchical feature map output by the upstream connected preset feature extraction layer through the multi-scale enhancement layer to obtain the semantic feature map; The preset feature extraction layer is a feature extraction layer in the backbone network whose number of cascaded layers exceeds a preset layer threshold, and the preset layer threshold is not less than 1. The fine segmentation network includes a fine segmentation head and an edge optimization network that corresponds one-to-one with the semantic feature map; The target segmentation result determination module includes: The target feature map determination unit is used to perform localization optimization based on the feature map to be processed and the corresponding semantic feature map through a cascaded edge optimization network to obtain the target feature map. The target segmentation result determination unit is used to perform feature mapping and pixel-level classification on the target feature map output by the last edge optimization network through the fine segmentation head, and output the target segmentation result corresponding to the object to be segmented; Wherein, when the edge optimization network is the edge optimization network corresponding to the last semantic feature map, the feature map to be processed is the guiding feature map; when the edge optimization network is the edge optimization network corresponding to other semantic feature maps, the feature map to be processed is the target feature map output by the cascaded previous edge optimization network; and the other semantic feature maps are semantic feature maps other than the last semantic feature map.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the medical image segmentation method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the medical image segmentation method according to any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the medical image segmentation method according to any one of claims 1-9.

Images