Image segmentation method and apparatus for vascular imagery

Abstract

The disclosure provides an image segmentation method and device for blood vessel images, which comprises: performing hierarchical feature extraction on a to-be-segmented blood vessel image based on N hierarchical encoding modules in an encoding part of a blood vessel image segmentation model to obtain a bottleneck layer feature image; performing global morphological feature extraction on the bottleneck layer feature image by using a self-attention mechanism based on a global encoding module of the encoding part to obtain a global feature image for representing a target blood vessel in the to-be-segmented blood vessel image; performing hierarchical feature fusion and hierarchical decoding on the global feature image and the encoding feature images output by the N hierarchical encoding modules respectively based on N hierarchical decoding modules in a decoding part of the blood vessel image segmentation model to obtain a target decoding feature image; and generating a segmentation result for the target blood vessel according to the target decoding feature image.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and image segmentation technology, and more specifically, to an image segmentation method and apparatus for vascular images. Background Technology

[0002] Digital subtraction angiography (DSA), as an advanced vascular imaging technique, is an important basis for non-invasive diagnosis and interventional surgical navigation of vascular diseases. In clinical practice, based on specific image segmentation models, real-time and accurate segmentation of blood vessels in DSA images can be achieved, helping doctors to better understand vascular structures, improve the precision and safety of surgery, and reduce surgical risks.

[0003] However, the relevant image segmentation models for vascular imaging face the following four major challenges: terminal vessels are thin and difficult to distinguish, lateral vessels overlap significantly, image quality varies greatly, and individual vessel morphology varies greatly. Summary of the Invention

[0004] In view of this, the present disclosure provides an image segmentation method, apparatus, device, medium, and program product for vascular images.

[0005] One aspect of this disclosure provides an image segmentation method for vascular images, comprising: N hierarchical coding modules in the coding part of a vascular image segmentation model, performing progressive feature extraction on the vascular image to be segmented to obtain a bottleneck layer feature image; wherein the image size of the coded feature image output by each of the N hierarchical coding modules decreases progressively, and the Nth coded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image; each hierarchical coding module includes at least one feature extraction submodule based on dynamic serpentine convolution; N is a positive integer and N≥2; and a global coding module based on the coding part uses a self-attention mechanism to perform global morphological feature extraction on the bottleneck layer feature image to obtain a feature image representing the vascular image to be segmented. The process involves: obtaining a global feature image of the target blood vessel in a segmented blood vessel image; performing step-by-step feature fusion and decoding on the global feature image and the encoded feature images output by each of the N level encoding modules within the decoding part of the blood vessel image segmentation model, to obtain a target decoded feature image; wherein the image size of the decoded feature images output by each of the N level decoding modules increases progressively, and the nth level decoding module in the N level decoding modules corresponds to the nth level encoding module in the N level encoding modules, n=1,…,N, and the first decoded feature image output by the first level decoding module is used as the target decoded feature image; and generating a segmentation result for the target blood vessel based on the target decoded feature image.

[0006] Another aspect of this disclosure provides an image segmentation apparatus for vascular images, comprising: a first processing module for performing progressive feature extraction on the vascular image to be segmented based on N hierarchical coding modules in the coding part of a vascular image segmentation model, to obtain a bottleneck layer feature image; wherein the image size of the coded feature image output by each of the N hierarchical coding modules decreases progressively, and the Nth coded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image; each hierarchical coding module includes at least one feature extraction submodule based on dynamic serpentine convolution; N is a positive integer and N≥2; and a second processing module for performing global morphological feature extraction on the bottleneck layer feature image based on the global coding module of the coding part, using a self-attention mechanism to obtain global features for characterizing the target vascular in the vascular image to be segmented. The system comprises: an image; a global feature image representing the global morphological features of the target blood vessel in the image to be segmented; a third processing module, which, based on the N-level decoding modules in the blood vessel image segmentation model, performs step-by-step feature fusion and decoding on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain the target decoded feature image; wherein the image size of the decoded feature images output by each of the N-level decoding modules increases progressively, the nth-level decoding module in the N-level decoding modules corresponds to the nth-level encoding module in the N-level encoding modules, n=1,…,N, and the Nth-level decoded feature image output by the 1st-level decoding module is used as the target decoded feature image; and a generation module, which generates the segmentation result for the target blood vessel based on the target decoded feature image.

[0007] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the methods described above.

[0008] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0009] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0010] According to embodiments of this disclosure, in image segmentation tasks targeting target blood vessels, the difficulty and complexity of image segmentation are significantly increased due to the combination of delicate and fragile local structures and complex and variable global morphology of blood vessels (which are elongated tubular structures). To address these issues, embodiments of this disclosure propose an image segmentation method for blood vessel images (hereinafter referred to as "this method"), aiming to improve segmentation accuracy and overcome the impact of data quality and morphological differences by capturing fine boundaries and learning the overall morphology. This method can extract features from the blood vessel image to be segmented by progressively using N-level encoding modules in the encoding part of the blood vessel image segmentation model, obtaining a bottleneck layer feature image. Then, based on the global encoding module in the encoding part using a self-attention mechanism, global morphological features are extracted from the bottleneck layer feature image to obtain a global feature image. Furthermore, based on the N-level decoding modules in the decoding part of the blood vessel image segmentation model, progressive feature fusion and progressive decoding are performed on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain a target decoded feature image. Finally, based on the target decoded feature image, a segmentation result for the target blood vessel is generated.

[0011] The vascular image segmentation model of this disclosure adopts a U-shaped network structure of "encoding-layered connection-decoding" and combines a global encoding module based on a self-attention mechanism (such as a global encoding module using a Transformer architecture) to extract global morphological features from the bottleneck layer feature image. By introducing a global encoding module based on a self-attention mechanism into the above U-shaped network structure, the long-distance dependencies of vascular morphological features can be better captured during the encoding stage, helping the model to better maintain the continuity of vascular morphological features, thereby effectively enhancing the model's ability to perceive the global morphology of the target vascular. In addition, each level encoding module includes at least one feature extraction submodule based on dynamic serpentine convolution. Dynamic serpentine convolution can automatically adjust the receptive field of the convolution kernel according to the characteristics of tubular structures to adapt to the morphology of tubular tissue, thereby effectively improving the model's feature extraction ability for slender and curved tubular structures, and thus helping to improve the segmentation accuracy and precision of the target vascular. Understandably, in the encoding part of the vascular image segmentation model, N hierarchical encoding modules use dynamic serpentine convolution. These N hierarchical encoding modules can extract local tubular structure information in the vascular image to be segmented step by step, thereby helping the model learn the terminal parts of intracranial arteries with slender structures and a lot of overlap, while improving the model's ability to segment blurred boundaries. The global encoding module can encode the bottleneck layer features, thereby guiding the model to learn the global morphological information of the blood vessels, so as to alleviate the problem of global information loss caused by local convolution operations and the impact of blood vessel morphology differences on model accuracy. Attached Figure Description

[0012] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 The illustration shows an application scenario of an image segmentation method, apparatus, device, medium, and program product for vascular imaging according to embodiments of the present disclosure.

[0014] Figure 2 A flowchart of an image segmentation method for vascular images according to an embodiment of the present disclosure is shown.

[0015] Figure 3 A comparison diagram of the kernel morphology of standard convolution, deformable convolution, and dynamic serpentine convolution according to embodiments of the present disclosure is shown.

[0016] Figure 4 A schematic diagram of a vascular image segmentation model according to an embodiment of the present disclosure is shown.

[0017] Figure 5 A schematic diagram of a feature extraction submodule according to an embodiment of the present disclosure is shown.

[0018] Figure 6 A schematic diagram of a global encoding module according to an embodiment of the present disclosure is shown.

[0019] Figure 7 A block diagram of an image segmentation apparatus for vascular imaging according to an embodiment of the present disclosure is shown.

[0020] Figure 8 A block diagram of an electronic device suitable for implementing an image segmentation method for vascular imaging, according to an embodiment of the present disclosure, is shown. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0026] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0027] Digital subtraction angiography (DSA) is an imaging technique used in interventional radiology to clearly visualize blood vessels within bone or dense soft tissue environments. For example, during DSA, a contrast agent is first injected into the patient's blood vessels, allowing them to be visualized under X-rays to obtain a two-dimensional X-ray image. Then, through a process of subtraction, enhancement, and re-imaging, bone tissue is removed from the original X-ray image, ultimately resulting in a DSA image that retains only the angiographic vessels.

[0028] According to one embodiment of this disclosure, DSA images may include, for example, 2D-DSA images (two-dimensional DSA images). Taking intracranial and extracranial artery 2D-DSA images as an example, these 2D-DSA images are not only the standard for diagnosing intracranial artery lesions, but also play a crucial role in intracranial artery interventional surgery. In intracranial artery interventional surgery, 2D-DSA images not only reflect the real-time vascular morphology of cerebral blood vessels during the operation, but also provide intraoperative pose information of vascular tissue and the relative pose information of vascular tissue and interventional instruments. Therefore, achieving real-time and accurate segmentation of cerebral blood vessels in 2D-DSA images is a fundamental step in realizing the construction of a three-dimensional navigation scene for interventional surgery. However, the related image segmentation models for intracranial and extracranial artery 2D-DSA images face the following problems:

[0029] (1) The vascular structures at the terminal end of intracranial arteries are delicate and difficult to segment. The internal carotid artery C7 segment, the anterior cerebral artery A1 segment, and the middle cerebral artery M1 segment, located at the terminal end of intracranial arteries, have small vascular radii and low pixel ratios, making it difficult for image segmentation models to learn the vascular features at the terminal end of intracranial arteries during training.

[0030] (2) The terminal vessels of intracranial arteries overlap significantly in lateral images, which can easily lead to missegmentation by the model. The A1 segment of the anterior cerebral artery and the M1 segment of the middle cerebral artery are connected to a rich network of collateral vessels that supply blood to the brain tissue. These collateral vessels overlap significantly with the terminal vessels of intracranial arteries in lateral images, which can mislead the image segmentation model to segment the collateral vessels of the terminal vessels of intracranial arteries in lateral images, thus reducing the segmentation accuracy of the model.

[0031] (3) High-quality 2D-DSA data is scarce. For poor-quality 2D-DSA data, the contrast between the blood vessel region and the background region is low, and the blood vessel boundary is relatively blurred, which will lead to a decrease in the prediction accuracy of the image segmentation model for boundary pixels.

[0032] (4) Significant differences in the overall morphology of blood vessels. Due to genetic variations and different lesion locations, there are certain differences in the morphology of cerebral blood vessels among different individuals. Secondly, during examination and surgery, doctors will observe cerebral blood vessels from multiple angles, which results in significant differences in the 2D-DSA images obtained from different angiography angles. The differences in blood vessel morphology will also lead to certain differences in the data distribution of training data, which will bring certain difficulties to the training of image segmentation models.

[0033] To at least partially address the technical problems existing in related technologies, this disclosure provides an image segmentation method, apparatus, device, medium, and program product for vascular images. The image segmentation method for vascular images includes: N hierarchical coding modules in the coding part of a vascular image segmentation model, which perform stepwise feature extraction on the vascular image to be segmented to obtain a bottleneck layer feature image; wherein the image size of the coded feature image output by each of the N hierarchical coding modules decreases progressively, and the Nth coded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image; each hierarchical coding module includes at least one feature extraction submodule based on dynamic serpentine convolution; N is a positive integer and N≥2; a global coding module based on the coding part uses a self-attention mechanism to perform global morphological feature extraction on the bottleneck layer feature image to obtain a feature image representing the target features in the vascular image to be segmented. The system generates a global feature image of the target blood vessel. Based on the decoding part of the blood vessel image segmentation model, N-level decoding modules perform step-by-step feature fusion and decoding on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain the target decoded feature image. The image size of the decoded feature images output by each of the N-level decoding modules increases progressively. The nth-level decoding module in the N-level decoding modules corresponds to the nth-level encoding module in the N-level encoding modules, n=1,…,N. The first decoded feature image output by the first-level decoding module is used as the target decoded feature image. Finally, based on the target decoded feature image, a segmentation result for the target blood vessel is generated.

[0034] Based on the vascular image segmentation model of this disclosure, for problems (1), (2), and (3), the vascular image segmentation model proposes N hierarchical coding modules based on dynamic serpentine convolution to capture the tubular features of vascular tissue, thereby improving the segmentation accuracy of vascular tissue in local boundary regions. For problem (4), the vascular image segmentation model proposes a global coding module based on a self-attention mechanism to model the long-distance dependence of the image, thereby enabling the model to learn the overall morphological information of blood vessels and obtain more discriminative features, thus mitigating the impact of vascular morphological differences on the model's segmentation accuracy.

[0035] Figure 1 The illustration schematically depicts an application scenario of an image segmentation method, apparatus, device, medium, and program product for vascular imaging according to embodiments of the present disclosure.

[0036] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the image segmentation method for vascular images provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the image segmentation device for vascular images provided in this disclosure embodiment can generally be located in server 105. The image segmentation method for vascular images provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image segmentation device for vascular images provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the image segmentation method for vascular images provided in this disclosure embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the image segmentation device for vascular imaging provided in this embodiment of the present disclosure may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart illustrating an image segmentation method for vascular imaging according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2 As shown, the method includes operations S210~S240.

[0044] In operation S210, the N-level coding modules in the coding part of the vascular image segmentation model extract features from the vascular image to be segmented step by step to obtain the bottleneck layer feature image.

[0045] In this system, the image size of the encoded feature image output by each of the N hierarchical encoding modules decreases progressively, and the Nth encoded feature image output by the Nth hierarchical encoding module is used as the bottleneck layer feature image; each hierarchical encoding module includes at least one feature extraction submodule based on dynamic snake convolution; N is a positive integer and N≥2.

[0046] In operation S220, the global encoding module based on the encoding part uses a self-attention mechanism to extract global morphological features from the bottleneck layer feature image, thereby obtaining a global feature image used to characterize the target blood vessel in the blood vessel image to be segmented.

[0047] In operation S230, the N-level decoding modules in the decoding part of the blood vessel image segmentation model perform step-by-step feature fusion and step-by-step decoding on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain the target decoded feature image.

[0048] Among them, the image size of the decoded feature image output by each of the N level decoding modules increases progressively. The nth level decoding module in the N level decoding modules corresponds to the nth level encoding module in the N level encoding modules, n=1,...,N. The first decoded feature image output by the first level decoding module is used as the target decoded feature image.

[0049] In operation S240, a segmentation result for the target blood vessel is generated based on the target decoded feature image.

[0050] According to embodiments of this disclosure, the vascular image to be segmented may include, for example, a 2D-DSA image, such as an intracranial or extracranial artery 2D-DSA image. The target vessel can be understood as the vascular structure / morphology that needs to be accurately segmented and extracted from the vascular image to be segmented. The target vessel may vary depending on the specific image segmentation task and is not limited herein. It is understood that in clinical medical practice, the accurate segmentation of the target vessel directly affects the accuracy and efficiency of subsequent vascular disease diagnosis and interventional surgical navigation. However, because the vascular morphology (a slender tubular structure) combines small and fragile local structures with complex and variable global morphology, the difficulty and complexity of the image segmentation task are significantly increased.

[0051] According to one embodiment of this disclosure, the vascular image segmentation model may employ a U-shaped network structure of "encoding-layered connection-decoding". Exemplarily, in the encoding stage, the image resolution can be progressively reduced through downsampling operations (such as max pooling) while extracting high-level features to capture contextual features and global information. In the decoding stage, the image resolution and details can be progressively restored through upsampling operations (such as deconvolution), while information loss due to downsampling and upsampling operations is reduced by performing layered connections between the feature maps from the encoding and decoding stages, resulting in a high-resolution segmentation result.

[0052] In one embodiment, the vascular image segmentation model may include an encoding part and a decoding part. The encoding part may include a global encoding module and N cascaded hierarchical encoding modules, and the decoding part may include N cascaded hierarchical decoding modules, where N is a positive integer and N≥2. The nth-level decoding module in the N hierarchical decoding modules corresponds to the nth-level encoding module in the N hierarchical encoding modules, where n=1,…,N.

[0053] The bottleneck layer feature image can be obtained by progressively extracting features from the blood vessel image to be segmented based on N cascaded hierarchical coding modules. The image size of the encoded feature image output by each of the N hierarchical coding modules decreases progressively, and the Nth encoded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image.

[0054] As an example, N can be 3, and the encoding part can include three cascaded hierarchical encoding modules, referred to as the first-level encoding module, the second-level encoding module, and the third-level encoding module, respectively. For instance, the image of the blood vessel to be segmented can be extracted step-by-step based on the aforementioned three hierarchical encoding modules. The first-level encoding module, the second-level encoding module, and the third-level encoding module output the first-encoded feature image, the second-encoded feature image, and the third-encoded feature image, respectively. The image size decreases progressively from the first-encoded feature image to the third-encoded feature image, and the third-encoded feature image can be used as the bottleneck layer feature image.

[0055] A global morphological feature extraction method can be used to extract global morphological features from the bottleneck layer feature image based on a global encoding module and a self-attention mechanism, resulting in a global feature image representing the target blood vessel in the blood vessel image to be segmented. In one embodiment, the global encoding module can, for example, adopt a Transformer architecture. The Transformer architecture is a deep learning model based on a self-attention mechanism. It employs an encoder-decoder architecture, directly models the association between any two positions in a sequence through self-attention, and utilizes fully connected layers and residual structures to achieve efficient parallel computation. This enables efficient processing of sequence data and captures long-distance dependencies.

[0056] It is understandable that the target blood vessels in the vascular images to be segmented are usually long and continuous. The global encoding module with the Transformer architecture can capture the long-distance dependencies of vascular morphological features better and help the model maintain the continuity of vascular morphological features better. Thus, it can better represent the global morphological features of the target blood vessel based on the global feature image and help avoid the breakage and fragmentation of the target blood vessel in the segmentation result.

[0057] Based on N-level decoding modules, a step-by-step feature fusion and decoding process can be performed on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain the target decoded feature image. The image size of the decoded feature images output by each of the N-level decoding modules increases progressively, and the first decoded feature image output by the first-level decoding module is used as the target decoded feature image. The first-level decoding module corresponds to the first-level encoding module.

[0058] As an example, N can be 3, and the decoding part can include three cascaded hierarchical decoding modules, denoted as the third-level decoding module, the second-level decoding module, and the first-level decoding module. The third-level decoding module corresponds to the third-level encoding module, the second-level decoding module corresponds to the second-level encoding module, and the first-level decoding module corresponds to the first-level encoding module. For example, the global feature image and the third-level encoded feature image can be fused and decoded based on the third-level decoding module to output the third decoded feature image. The third decoded feature image and the second encoded feature image can be fused and decoded based on the second-level decoding module to obtain the second decoded feature image. The second decoded feature image and the first encoded feature image can be fused and decoded based on the first-level decoding module to obtain the first decoded feature image. The image size increases progressively from the third decoded feature image to the first decoded feature image, and the first decoded feature image can be used as the target decoded feature image.

[0059] It can generate segmentation results for the target blood vessel based on the target decoded feature image.

[0060] According to embodiments of this disclosure, the encoding portion includes N hierarchical encoding modules, each including at least one feature extraction submodule based on dynamic serpentine convolution. Dynamic serpentine convolution is a convolution method that can automatically adjust the receptive field of the convolution kernel to adapt to the tubular tissue morphology based on the characteristics of the tubular structure. Dynamic serpentine convolution adds a constraint to deformable convolution, whereby the convolution kernel maintains proximity in a certain direction and can only swing within a certain range, thereby enabling the positional distribution of points in the convolution kernel to adapt to the local tubular morphology of the blood vessel.

[0061] In image segmentation tasks targeting blood vessels, the morphology of these vessels is characterized by their elongated and complex shape. The elongated structure occupies only a small portion of the entire image, with a limited number of pixels. Furthermore, these structures are easily affected by complex backgrounds, making it difficult for models to accurately distinguish subtle changes in the target, resulting in fragmented and disjointed segmentation.

[0062] Figure 3 The diagram schematically illustrates a comparison of the kernel morphologies of standard convolution, deformable convolution, and dynamic serpentine convolution according to embodiments of the present disclosure.

[0063] like Figure 3 As shown, for tubular targets, standard convolution cannot adjust the region of interest according to the target features. Although deformable convolution can adaptively learn the region of interest according to the target features, it cannot limit the connectivity of the region of interest for tubular targets. Dynamic snake convolution, on the other hand, limits the connectivity of the region of interest, making it more suitable for segmentation scenarios of tubular targets.

[0064] It is understandable that dynamic snake convolution can optimize tubular target segmentation tasks by utilizing topological geometric constraints, overcoming the limitations of standard convolution and deformable convolution. It is especially suitable for long and complex tubular scenes. It maintains the continuity of the region of interest through an iterative strategy, thereby improving the accuracy and efficiency of segmentation.

[0065] According to embodiments of this disclosure, in image segmentation tasks targeting target blood vessels, the difficulty and complexity of image segmentation are significantly increased due to the combination of delicate and fragile local structures and complex and variable global morphology of blood vessels (which are elongated tubular structures). To address these issues, embodiments of this disclosure propose an image segmentation method for blood vessel images (hereinafter referred to as "this method"), aiming to improve segmentation accuracy and overcome the impact of data quality and morphological differences by capturing fine boundaries and learning the overall morphology. This method can extract features from the blood vessel image to be segmented by progressively using N-level encoding modules in the encoding part of the blood vessel image segmentation model, obtaining a bottleneck layer feature image. Then, based on the global encoding module in the encoding part using a self-attention mechanism, global morphological features are extracted from the bottleneck layer feature image to obtain a global feature image. Furthermore, based on the N-level decoding modules in the decoding part of the blood vessel image segmentation model, progressive feature fusion and progressive decoding are performed on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain a target decoded feature image. Finally, based on the target decoded feature image, a segmentation result for the target blood vessel is generated.

[0066] The vascular image segmentation model of this disclosure adopts a U-shaped network structure of "encoding-layered connection-decoding" and combines a global encoding module based on a self-attention mechanism (such as a global encoding module using a Transformer architecture) to extract global morphological features from the bottleneck layer feature image. By introducing a global encoding module based on a self-attention mechanism into the above U-shaped network structure, the long-distance dependencies of vascular morphological features can be better captured during the encoding stage, helping the model to better maintain the continuity of vascular morphological features, thereby effectively enhancing the model's ability to perceive the global morphology of the target vascular. In addition, each level encoding module includes at least one feature extraction submodule based on dynamic serpentine convolution. Dynamic serpentine convolution can automatically adjust the receptive field of the convolution kernel according to the characteristics of tubular structures to adapt to the morphology of tubular tissue, thereby effectively improving the model's feature extraction ability for slender and curved tubular structures, and thus helping to improve the segmentation accuracy and precision of the target vascular. Understandably, in the encoding part of the vascular image segmentation model, N hierarchical encoding modules use dynamic serpentine convolution. These N hierarchical encoding modules can extract local tubular structure information in the vascular image to be segmented step by step, thereby helping the model learn the terminal parts of intracranial arteries with slender structures and a lot of overlap, while improving the model's ability to segment blurred boundaries. The global encoding module can encode the bottleneck layer features, thereby guiding the model to learn the global morphological information of the blood vessels, so as to alleviate the problem of global information loss caused by local convolution operations and the impact of blood vessel morphology differences on model accuracy.

[0067] According to one embodiment of this disclosure, for the 2D-DSA image segmentation task of cerebral blood vessels, an image segmentation model based on tubular feature extraction and global attention encoding can be used to segment the blood vessel image to be segmented, obtaining the segmentation result for the target blood vessel. This blood vessel image segmentation model fully considers the relationship between the local morphological features and global morphological differences of blood vessels, taking into account both the encoding of local tubular structure information and global morphological information. On the one hand, a feature extraction submodule based on dynamic serpentine convolution operation is used to extract local tubular structure information, thereby alleviating the problem of low segmentation accuracy caused by the thin and heavily overlapping terminal structures of intracranial arteries. On the other hand, a global encoding module based on the Tranformer architecture is used at the bottleneck layer of the model to learn the global semantic dependencies of the blood vessel image, thereby alleviating the problem of large data distribution differences caused by significant differences in blood vessel morphology.

[0068] According to embodiments of this disclosure, the encoding part further includes an initial encoding module, which includes at least one feature extraction submodule; each hierarchical encoding module further includes a downsampling submodule; the N hierarchical encoding modules in the encoding part based on the vascular image segmentation model perform stepwise feature extraction on the vascular image to be segmented to obtain a bottleneck layer feature image, including: based on at least one feature extraction submodule of the initial encoding module, performing feature extraction on the vascular image to be segmented input to at least one feature extraction submodule to obtain and output an initial encoded feature image; based on the first downsampling submodule of the first hierarchical encoding module, performing a downsampling operation on the initial encoded feature image to obtain a first downsampled image. The first downsampled feature image is obtained by performing feature extraction on the first downsampled feature image input to the first feature extraction submodule based on the first level coding module, and outputting the first coded feature image; the (i-1)th coded feature image is downsampled based on the i-th level coding module, and the i-th downsampled feature image is obtained; the i-th downsampled feature image is downsampled based on the i-th level coding module, and outputting the i-th coded feature image; i=2,…,N; and the Nth coded feature image is determined as the bottleneck layer feature image.

[0069] According to one embodiment of this disclosure, the encoding portion of the vascular image segmentation model includes an initial encoding module and N cascaded hierarchical encoding modules. The initial encoding module includes at least one feature extraction submodule, and each hierarchical encoding module includes a downsampling submodule and at least one feature extraction submodule. Exemplarily, N can be 3, and the three hierarchical encoding modules are respectively the first-level encoding module, the second-level encoding module, and the third-level encoding module. Both the initial encoding module and each hierarchical encoding module can include two feature extraction submodules, and the downsampling submodule can be, for example, a max-pooling layer (2×2).

[0070] In one embodiment, performing stepwise feature extraction on the blood vessel image to be segmented based on N hierarchical coding modules to obtain the bottleneck layer feature image may include the following operations:

[0071] For example, features can be extracted from the blood vessel image to be segmented based on the two feature extraction sub-modules of the initial encoding module, and the initial encoded feature image can be obtained and output.

[0072] For example, the initial encoded feature image can be downsampled based on the first downsampling submodule of the first-level encoding module to obtain the first downsampled feature image. Features can then be extracted from the first downsampled feature image based on the two feature extraction submodules of the first-level encoding module to obtain and output the first encoded feature image.

[0073] For example, the first encoded feature image can be downsampled based on the second downsampling submodule of the second-level encoding module to obtain the second downsampled feature image. Features can be extracted from the second downsampled feature image based on the two feature extraction submodules of the second-level encoding module to obtain and output the second encoded feature image.

[0074] For example, the second encoded feature image can be downsampled based on the third downsampling submodule of the third-level encoding module to obtain the third downsampled feature image. Features can be extracted from the third downsampled feature image based on the two feature extraction submodules of the third-level encoding module to obtain and output the third encoded feature image.

[0075] For example, the third encoded feature image can be identified as the bottleneck layer feature image.

[0076] According to embodiments of this disclosure, each hierarchical decoding module includes at least one convolutional unit and an upsampling submodule; based on the N hierarchical decoding modules in the decoding part of the blood vessel image segmentation model, the global feature image and the encoded feature images output by each of the N hierarchical encoding modules are subjected to hierarchical feature fusion and hierarchical decoding to obtain the target decoded feature image, including: when n=N, based on at least one convolutional unit of the Nth hierarchical decoding module, the Nth fused feature image obtained by fusing the global feature image and the Nth encoded feature image is decoded to obtain the Nth fused feature image; based on the Nth hierarchical... The Nth upsampling submodule of the decoding module performs an upsampling operation on the Nth fused feature image to obtain the Nth decoded feature image; when n=N-1,...,1, based on at least one convolutional unit of the nth level decoding module, the nth fused feature image obtained by fusing the (n+1)th decoded feature image and the nth encoded feature image is decoded to obtain the nth fused feature image; based on the nth upsampling submodule of the nth level decoding module, the nth fused feature image is upsampling to obtain the nth decoded feature image; and the first decoded feature image is determined as the target decoded feature image.

[0077] According to one embodiment of this disclosure, the decoding part of the blood vessel image segmentation model includes N cascaded hierarchical decoding modules (N can be, for example, 3), each hierarchical decoding module including at least one convolutional unit and an upsampling submodule. Exemplarily, N can be, for example, 3, and the three hierarchical decoding modules are respectively the third-level decoding module, the second-level decoding module, and the first-level decoding module. Each hierarchical decoding module may include two convolutional units, and the upsampling submodule may be, for example, a deconvolutional layer (2×2).

[0078] In one embodiment, based on N-level decoding modules, performing step-by-step feature fusion and step-by-step decoding on the global feature image and the encoded feature images output by each of the N-level encoding modules to obtain the target decoded feature image may include the following operations:

[0079] For example, the third fused feature image, obtained by fusing the global feature image and the third encoded feature image (i.e., the bottleneck layer feature image), can be decoded using two convolutional units of the third-level decoding module. The third fused feature image can then be upsampled using the third upsampling submodule of the third-level decoding module to obtain the third decoded feature image.

[0080] For example, the second fused feature image, obtained by fusing the third decoded feature image and the second encoded feature image, can be decoded using two convolutional units of the second-level decoding module to obtain the second fused feature image. The second fused feature image can be upsampled using the second upsampling submodule of the second-level decoding module to obtain the second decoded feature image.

[0081] For example, the first fused feature image can be obtained by decoding the first fused feature image obtained by fusing the second decoded feature image and the first encoded feature image based on the two convolutional units of the first-level decoding module. The first fused feature image can be obtained by upsampling the first fused feature image based on the first upsampling submodule of the first-level decoding module.

[0082] For example, the first decoded feature image can be identified as the target decoded feature image.

[0083] According to embodiments of this disclosure, the decoding part further includes a target decoding module, which includes at least one convolutional unit; generating a segmentation result for the target blood vessel based on the target decoded feature image includes: decoding the target feature image obtained by fusing the target decoded feature image and the initial encoded feature image based on at least one convolutional unit of the target decoding module to generate a segmentation result for the target blood vessel.

[0084] According to one embodiment of this disclosure, the decoding part of the blood vessel image segmentation model further includes a target decoding module, which includes at least one convolutional unit. Exemplarily, the target decoding module may include two convolutional units.

[0085] For example, based on the two convolutional units of the target decoding module, the target feature image obtained by fusing the target decoding feature image and the initial encoded feature image can be decoded to generate a segmentation result for the target blood vessel.

[0086] Figure 4A schematic diagram of a vascular image segmentation model according to an embodiment of the present disclosure is shown.

[0087] like Figure 4 As shown, the encoding part 410 of the blood vessel image segmentation model 400M includes an initial encoding module 401, three hierarchical encoding modules, and a global encoding module 405. The three hierarchical encoding modules are the first-level encoding module 402, the second-level encoding module 403, and the third-level encoding module 404. The initial encoding module 401 includes two feature extraction sub-modules, and each hierarchical encoding module includes a max-pooling layer and two feature extraction sub-modules.

[0088] As an example, obtaining the segmentation result P2 of the target vessel from the vascular image P1 to be segmented based on the 400M vascular image segmentation model can include the following operations:

[0089] First, the blood vessel image to be segmented ( R represents the real number field, C is the number of channels (e.g., 3), H and W are the height and width of the image, respectively. P1 is input into the blood vessel image segmentation model 400M. Based on the initial encoding module 401 of the encoding part 410 and three hierarchical encoding modules, features are extracted step by step from the blood vessel image P1 to be segmented, resulting in hierarchical features. .in, This represents the initial encoded feature image output by the initial encoding module 401. This represents the first encoded feature image output by the first-level encoding module 402. This represents the second encoded feature image output by the second-level encoding module 403. This represents the third encoded feature image (i.e., the bottleneck layer feature image) output by the third-level encoding module 404. Then, the bottleneck layer feature image... The input is fed into the global encoding module 405, which encodes the global information of the bottleneck layer feature image based on the global encoding module 405, extracts the overall morphological information of the target blood vessel, and obtains the global feature image. Finally, the bottleneck layer feature image is... and global feature image The input is fed into the decoding section 420 for decoding, and the extracted data is then processed. The corresponding information is progressively added to the decoding section, ultimately yielding the segmentation result P2 for the target blood vessel. Specifically, based on the third-level decoding module 406, global feature images are fused... and the third encoded feature image The third fused feature image (i.e., the bottleneck layer feature image) is decoded and upsampled to obtain the third decoded feature image. Based on the second-level decoding module 407, the third decoded feature image and the second encoded feature image are fused together. The obtained second fused feature image is decoded and upsampled to obtain the second decoded feature image. Based on the first-level decoding module 408, the second decoded feature image and the first encoded feature image are fused. The obtained first fused feature image is decoded and upsampled to obtain the first decoded feature image, which is then determined as the target decoded feature image. Based on the target decoding module 409, the target decoded feature image and the initial encoded feature image are fused. The obtained target feature image is decoded to generate a segmentation result P2 for the target blood vessel.

[0090] According to embodiments of this disclosure, the feature extraction submodule includes a standard convolutional unit, a horizontal dynamic serpentine convolutional unit, a vertical dynamic serpentine convolutional unit, and a residual unit. For any feature extraction submodule, the feature extraction submodule performs the following operations: acquiring an input feature image input to the feature extraction submodule; extracting features from the input feature image based on the standard convolutional unit, the horizontal dynamic serpentine convolutional unit, and the vertical dynamic serpentine convolutional unit of the feature extraction submodule, respectively, to obtain a first intermediate feature image, a second intermediate feature image, and a third intermediate feature image; concatenating the first intermediate feature image, the second intermediate feature image, and the third intermediate feature image to obtain a concatenated feature image; processing the concatenated feature image based on the residual unit of the feature extraction submodule to obtain a residual feature image; fusing the concatenated feature image and the residual feature image to obtain an output feature image, and outputting the output feature image.

[0091] According to embodiments of this disclosure, in dynamic serpentine convolution, the standard convolution kernel is linearized in both the x-axis and y-axis directions. A horizontal dynamic serpentine convolution unit can be understood as a dynamic serpentine convolution in the x-axis direction, and a vertical dynamic serpentine convolution unit can be understood as a dynamic serpentine convolution in the y-axis direction.

[0092] As an example, for a 3×3 convolution kernel Taking dynamic serpentine convolution in the x-axis direction as an example, the convolution kernel... The specific location of each grid cell is represented as follows: In i±c, c=0,1,2,3,4 represents the distance from the center grid. Horizontal distance. Convolution kernel Each grid position The selection process is cumulative. Starting from the central position... Initially, the position away from the center grid depends on the position of the previous grid: Compared to Increase offset ∈[-1,1]. Therefore, the offsets need to be accumulated to ensure that the convolution kernel conforms to a linear morphological structure. Similarly, dynamic serpentine convolution in the y-axis direction... The specific location of each grid cell is represented as follows: Where, in j±c, c=0,1,2,3,4 represents the distance from the center grid ( The vertical distance of the convolution kernel. Each grid position The selection process is cumulative. Starting from the central position... Initially, the position away from the center grid depends on the position of the previous grid: Compared to Increase offset ∈[-1,1].

[0093] For example, the formula for calculating dynamic serpentine convolution in the x-axis direction is shown in Equation (1), and the formula for calculating dynamic serpentine convolution in the y-axis direction is shown in Equation (2).

[0094] (1)

[0095] (2)

[0096] Where c = 0, 1, 2, 3, 4 represents the distance between the grid and the center grid. This represents the cumulative y-axis offset of the convolutional kernel from layer i to layer i+c relative to the previous layer. The summation result is the total y-axis offset of the convolutional kernel at layer i+c. This represents the cumulative offset of each layer of the convolution kernel from layer i to layer j+c in the x-axis direction relative to the previous layer. The summation result is the total offset of the convolution kernel in the x-axis direction of layer j+c.

[0097] Figure 5 A schematic diagram of a feature extraction submodule according to an embodiment of the present disclosure is shown.

[0098] like Figure 5 As shown, for any feature extraction submodule, the feature extraction submodule may include a standard convolutional unit, a horizontal dynamic serpentine convolutional unit, a vertical dynamic serpentine convolutional unit, and a residual unit. For example, the residual unit may include two point convolutional units.

[0099] In addition to using standard convolutional units (3×3 kernel size), horizontal dynamic serpentine convolutional units (7×7 kernel size), and vertical dynamic serpentine convolutional units (7×7 kernel size) to extract features respectively, the feature extraction submodule also introduces a residual unit to perform deep fusion of the features extracted by the aforementioned three convolutional units.

[0100] As an example, for any feature extraction submodule, the specific calculation steps of the feature extraction submodule include:

[0101] The input feature image can be obtained and fed into the feature extraction submodule. For the input feature image The local features can be extracted separately using standard convolutional units, horizontal dynamic serpentine convolutional units, and vertical dynamic serpentine convolutional units, resulting in a first intermediate feature image, a second intermediate feature image, and a third intermediate feature image. These three intermediate feature images can then be concatenated to obtain a concatenated feature image. It can stitch feature images together. The residual feature image is obtained by processing through two point convolutional layers (convolution operation with a kernel size of 1×1 and a stride of 1×1). The features extracted by the three convolutional kernels can be fused together by combining the concatenated feature image and the residual feature image to obtain the final output feature image. .

[0102] According to embodiments of this disclosure, standard convolutional units can extract general local features of blood vessels, while horizontal dynamic serpentine convolutional units and vertical dynamic serpentine convolutional units can focus on extracting tubular features of blood vessels in the x-axis and y-axis directions, respectively. This enables multi-directional, complementary extraction of blood vessel features, enhancing the richness and discriminative power of feature representation. Residual units, by introducing residual connections, allow the stitched feature images in the network to bypass the two point convolutional layers and propagate directly, preventing network degradation and ensuring that gradients can be effectively and directly backpropagated, thereby improving the training efficiency and performance of the network. In the residual unit, the two point convolutional layers can effectively adjust the number of channels in the feature map, achieving feature transformation and fusion. In this way, residual units can better capture image features, improving the accuracy of segmentation tasks.

[0103] According to embodiments of this disclosure, the global encoding module includes a projection unit, a converter unit, and a deformation unit. The global encoding module, based on the encoding portion, uses a self-attention mechanism to extract global morphological features from the bottleneck layer feature image to obtain a global feature image representing the target blood vessel in the blood vessel image to be segmented. This includes: embedding the bottleneck layer feature image into a projection sequence based on the projection unit; encoding the projection feature sequence using a multi-head self-attention mechanism based on the converter unit to obtain an attention feature sequence; and deforming and mapping the attention feature sequence based on the deformation unit to obtain the global feature image.

[0104] Figure 6 A schematic diagram of a global encoding module according to an embodiment of the present disclosure is shown.

[0105] like Figure 6 As shown, the global encoding module includes a projection unit 601, a converter unit 602, and a deformation unit 603. Exemplarily, the projection unit 601 includes an embedded projection layer, the converter unit 602 includes N Transformer layers, where N is an integer and N≥2, and the deformation unit 603 includes a deformation layer and a convolutional layer.

[0106] As an example, consider the bottleneck layer feature image. Abbreviation For example, based on the global encoding module, the global morphological features of the bottleneck layer feature image are extracted using a self-attention mechanism. The resulting global feature image can include the following operations:

[0107] First, the input feature map (i.e., the bottleneck layer feature image) is processed. The feature map is divided into blocks, resulting in N×N feature map sub-blocks. ,For example This represents the i-th feature map sub-block. Based on the projection unit 601, the obtained N×N feature map sub-blocks are embedded and projected respectively to obtain N×N embedding vectors (i.e., projected feature sequences). ,For example Indicates to The i-th embedding vector obtained by projection. Then, the N×N embedding vectors obtained by projection. The input is encoded in a transformer unit 602 consisting of N Transformer layers. The global dependencies between the above N×N embedding vectors are learned to obtain N×N encoded embedding vectors (i.e., attention feature sequences). ,For example Indicates to The i-th attention feature is obtained through encoding. Finally, the deformable layer based on deformable unit 603 will... The size of the feature map sub-blocks is transformed to obtain the globally encoded feature map sub-blocks. , This represents the i-th feature map sub-block after global encoding. The convolutional layer based on deformable unit 603 will... By mapping back to the original feature dimension λ through a 3×3 convolution operation, N×N encoded feature map sub-blocks are obtained. , This represents the i-th encoded feature map sub-block. The final output feature (i.e., the global feature image) is obtained by concatenating the N×N encoded feature map sub-blocks. .

[0108] For any Transformer layer, the input embedding vector First, the input enters a layer normalization module. Then, the normalized result is fed into a multi-head self-attention module. Finally, the output of the multi-head attention module is compared with the embedding vector. Add them together to get the embedding vector. Embedded vector The input is then fed into the second-layer normalization module, and the normalized result is then fed into the multilayer perceptron module. Finally, the output of the multilayer perceptron module is compared with the embedding vector. Add them together to get the final output of the Transformer layer. Where i represents the number of Transformer layers, i=1,…,N.

[0109] The specific calculation formulas for the i-th Transformer layer are shown in equations (3)-(4), where, Let be the embedding vector of the i-th Transformer layer, LayerNorm be the layer normalization, MSA be the multi-head self-attention module, and MLP be the multilayer perceptron module.

[0110] (3)

[0111] (4)

[0112] The multi-head self-attention mechanism is the core of the Transformer layer's computation, and its calculation formulas are shown in equations (5)-(9). For an input embedding vector... The multi-head self-attention mechanism first divides it into M groups of vectors. ,in Then, through three fully connected layers... , , Projecting the embedded vector 𝐄, we get , , Three sets of vectors. Then, by calculating the scaled dot product between each pair of vectors in the same set (X and Y) and normalizing each column, an attention map is obtained. The output is obtained by performing matrix multiplication between 𝐕 and the attention map 𝐀. and transform it into .

[0113] (5)

[0114] (6)

[0115] (7)

[0116] (8)

[0117] (9)

[0118] It is understandable that encoding the global dependencies of vascular images (such as 2D-DSA images) to be segmented has two main advantages: First, it allows for the learning of the overall morphological information of blood vessels, thereby reducing the impact of large differences in vascular morphology on the model's segmentation accuracy. Second, since 2D-DSA images are generally large (commonly 512×512), and most segmentation models can only perform three or four layers of downsampling, the feature map size of their bottleneck layer still reaches 64×64 or 32×32. Long-distance dependencies in the image are not sufficiently modeled and encoded, which leads the model to tend to segment other vascular tissues with the same local structure, resulting in a large number of outliers in the segmentation results. Specifically, in the vascular image segmentation model of this disclosure, although the N-level encoding modules based on dynamic serpentine convolution can effectively extract the tubular structure information of blood vessels, the model is still unable to effectively capture and encode global information due to the local characteristics of convolution operations. To address this issue, a global encoding module is introduced into the bottleneck layer of the model to fully encode the long-distance dependencies of the image, thereby mitigating the loss of global information caused by local convolution operations.

[0119] Figure 7 A block diagram of an image segmentation apparatus for vascular imaging according to an embodiment of the present disclosure is shown schematically.

[0120] like Figure 7 As shown, the image segmentation device 700 for vascular images includes a first processing module 710, a second processing module 720, a third processing module 730, and a generation module 740.

[0121] The first processing module 710 is used to extract features from the blood vessel image to be segmented by N hierarchical coding modules in the coding part of the blood vessel image segmentation model, thereby obtaining the bottleneck layer feature image. The image size of the encoded feature image output by each of the N hierarchical coding modules decreases step by step, and the Nth encoded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image. Each hierarchical coding module includes at least one feature extraction submodule based on dynamic serpentine convolution. N is a positive integer and N≥2.

[0122] The second processing module 720 is used for the global encoding module based on the encoding part. It uses a self-attention mechanism to extract global morphological features from the bottleneck layer feature image to obtain a global feature image that represents the target blood vessel in the blood vessel image to be segmented. The global feature image is used to represent the global morphological features of the target blood vessel in the blood vessel image to be segmented.

[0123] The third processing module 730 is used in the decoding part of the blood vessel image segmentation model to perform step-by-step feature fusion and step-by-step decoding on the global feature image and the encoded feature images output by each of the N level decoding modules to obtain the target decoded feature image. The image size of the decoded feature images output by each of the N level decoding modules increases step by step. The nth level decoding module in the N level decoding modules corresponds to the nth level encoding module in the N level encoding modules, n=1,...,N. The Nth decoded feature image output by the 1st level decoding module is used as the target decoded feature image.

[0124] The generation module 740 is used to generate segmentation results for the target blood vessel based on the target decoded feature image.

[0125] According to embodiments of this disclosure, the encoding portion further includes an initial encoding module, which includes at least one feature extraction submodule; each level encoding module further includes a downsampling submodule; the first processing module 710 may include a first processing submodule, a second processing submodule, a third processing submodule, and a first determination submodule.

[0126] The first processing submodule is used to extract features from the blood vessel image to be segmented, which is input to the initial encoding module, based on at least one feature extraction submodule of the initial encoding module, to obtain and output the initial encoded feature image.

[0127] The second processing submodule is used to perform a downsampling operation on the initial encoded feature image based on the first downsampling submodule of the first-level encoding module to obtain the first downsampled feature image; and to perform feature extraction on the first downsampled feature image input to the first downsampled feature image based on at least one feature extraction submodule of the first-level encoding module to obtain and output the first encoded feature image.

[0128] The third processing submodule is used to perform downsampling operation on the (i-1)th encoded feature image based on the i-th level encoding module's i-th downsampling submodule to obtain the i-th downsampled feature image; and to perform feature extraction on the i-th downsampled feature image input to the at least one feature extraction submodule based on the i-th level encoding module to obtain and output the i-th encoded feature image; i=2,…,N.

[0129] The first determining submodule is used to determine the Nth encoded feature image as the bottleneck layer feature image.

[0130] According to embodiments of this disclosure, the feature extraction submodule includes a standard convolutional unit, a horizontal dynamic serpentine convolutional unit, a vertical dynamic serpentine convolutional unit, and a residual unit. For any feature extraction submodule, the feature extraction submodule performs the following operations: acquiring an input feature image input to the feature extraction submodule; extracting features from the input feature image based on the standard convolutional unit, the horizontal dynamic serpentine convolutional unit, and the vertical dynamic serpentine convolutional unit of the feature extraction submodule, respectively, to obtain a first intermediate feature image, a second intermediate feature image, and a third intermediate feature image; concatenating the first intermediate feature image, the second intermediate feature image, and the third intermediate feature image to obtain a concatenated feature image; processing the concatenated feature image based on the residual unit of the feature extraction submodule to obtain a residual feature image; fusing the concatenated feature image and the residual feature image to obtain an output feature image, and outputting the output feature image.

[0131] According to embodiments of this disclosure, the global encoding module includes a projection unit, a converter unit, and a deformation unit; the second processing module 720 includes:

[0132] The fourth processing submodule is used to embed and project the bottleneck layer feature image based on the projection unit to obtain the projected feature sequence.

[0133] The fifth processing submodule is used to encode the projected feature sequence using a multi-head self-attention mechanism based on the converter unit to obtain the attention feature sequence.

[0134] The sixth processing submodule is used to deform and map the attention feature sequence based on the deformation unit to obtain the global feature image.

[0135] According to embodiments of this disclosure, each level decoding module includes at least one convolutional unit and an upsampling submodule; the third processing module 730 may include a seventh processing submodule, an eighth processing submodule, and a second determination submodule.

[0136] The seventh processing submodule is used to decode the Nth fused feature image obtained by fusing the global feature image and the Nth coded feature image based on at least one convolutional unit of the Nth level decoding module when n=N, to obtain the Nth fused feature image; and to perform an upsampling operation on the Nth fused feature image based on the Nth level decoding module upsampling submodule to obtain the Nth decoded feature image.

[0137] The eighth processing submodule is used to decode the nth fused feature image obtained by fusing the (n+1)th decoded feature image and the nth encoded feature image based on at least one convolutional unit of the nth level decoding module when n=N-1, ...,1, to obtain the nth fused feature image; and to perform an upsampling operation on the nth fused feature image based on the nth level decoding module upsampling submodule to obtain the nth decoded feature image.

[0138] The second determination submodule is used to determine the first decoded feature image as the target decoded feature image.

[0139] According to embodiments of this disclosure, the decoding section further includes a target decoding module, which includes at least one convolutional unit; the generation module 740 includes a generation submodule.

[0140] A generation submodule is used to decode the target feature image obtained by fusing the target decoded feature image and the initial encoded feature image based on at least one convolutional unit of the target decoding module, and generate a segmentation result for the target blood vessel.

[0141] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0142] For example, any plurality of the first processing module 710, the second processing module 720, the third processing module 730, and the generation module 740 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first processing module 710, the second processing module 720, the third processing module 730, and the generation module 740 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first processing module 710, the second processing module 720, the third processing module 730, and the generation module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0143] It should be noted that the image segmentation device part for vascular images in the embodiments of this disclosure corresponds to the image segmentation method part for vascular images in the embodiments of this disclosure. For a detailed description of the image segmentation device part for vascular images, please refer to the image segmentation method part for vascular images, which will not be repeated here.

[0144] Figure 8 A block diagram of an electronic device suitable for implementing an image segmentation method for vascular imaging, according to an embodiment of the present disclosure, is shown schematically. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0145] like Figure 8As shown, an electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a ROM 802 (Read-Only Memory) or a program loaded from a storage portion 808 into a RAM 803 (Random Access Memory). The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0147] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0148] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0149] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0150] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0152] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code enables the electronic device to implement the image segmentation method for vascular imaging provided in the embodiments of this disclosure.

[0153] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0155] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0157] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image segmentation method for vascular images, characterized in that, The method includes: The encoding part of the blood vessel image segmentation model comprises N hierarchical encoding modules that progressively extract features from the blood vessel image to be segmented, resulting in a bottleneck layer feature image. The image size of the encoded feature image output by each of the N hierarchical encoding modules decreases progressively, with the Nth encoded feature image output by the Nth hierarchical encoding module serving as the bottleneck layer feature image. Each hierarchical encoding module includes at least one feature extraction submodule based on dynamic serpentine convolution. N is a positive integer and N≥2. The encoding part includes an initial encoding module, which includes at least one of the aforementioned feature extraction submodules. Each hierarchical encoding module also includes a downsampling submodule. The feature extraction submodule includes standard convolutional units, horizontal dynamic serpentine convolutional units, vertical dynamic serpentine convolutional units, and residual units. The bottleneck layer feature image obtained includes: Based on at least one feature extraction submodule of the initial encoding module, feature extraction is performed on the blood vessel image to be segmented input to the at least one feature extraction submodule to obtain and output the initial encoded feature image; The first downsampling submodule based on the first level coding module performs a downsampling operation on the initial coded feature image to obtain the first downsampled feature image; the first feature extraction submodule based on the first level coding module performs feature extraction on the first downsampled feature image input to the first feature extraction submodule to obtain and output the first coded feature image. Based on the i-th level encoding module, the i-th downsampling submodule performs a downsampling operation on the (i-1)-th encoded feature image to obtain the i-th downsampled feature image; based on at least one feature extraction submodule of the i-th level encoding module, features are extracted from the i-th downsampled feature image input to the at least one feature extraction submodule to obtain and output the i-th encoded feature image; i=2,…,N; and The Nth encoded feature image is determined as the bottleneck layer feature image; For any of the aforementioned feature extraction submodules, the feature extraction submodule is used to perform the following operations: Obtain the input feature image that is fed into the feature extraction submodule; Based on the standard convolutional unit, the horizontal dynamic serpentine convolutional unit, and the vertical dynamic serpentine convolutional unit of the feature extraction submodule, features are extracted from the input feature image to obtain the first intermediate feature image, the second intermediate feature image, and the third intermediate feature image. The first intermediate feature image, the second intermediate feature image, and the third intermediate feature image are stitched together to obtain a stitched feature image. Based on the residual unit of the feature extraction submodule, the stitched feature image is processed to obtain the residual feature image; The stitched feature image and the residual feature image are fused to obtain an output feature image, and the output feature image is then output. Based on the global encoding module of the encoding part, a self-attention mechanism is used to extract global morphological features from the bottleneck layer feature image to obtain a global feature image for characterizing the target blood vessel in the blood vessel image to be segmented. Based on the N-level decoding modules in the decoding part of the blood vessel image segmentation model, the global feature image and the encoded feature images output by each of the N-level encoding modules are subjected to step-by-step feature fusion and step-by-step decoding to obtain the target decoded feature image; wherein, the image size of the decoded feature images output by each of the N-level decoding modules increases progressively, the nth-level decoding module in the N-level decoding modules corresponds to the nth-level encoding module in the N-level encoding modules, n=1,...,N, and the first decoded feature image output by the first-level decoding module is used as the target decoded feature image; each of the level decoding modules includes at least one convolutional unit and an upsampling submodule; The target decoded feature image obtained includes: When n=N, the Nth fused feature image obtained by fusing the global feature image and the Nth encoded feature image is decoded based on at least one convolutional unit of the Nth level decoding module to obtain the Nth fused feature image; the Nth upsampling submodule of the Nth level decoding module performs an upsampling operation on the Nth fused feature image to obtain the Nth decoded feature image. When n = N-1, ..., 1, based on at least one convolutional unit of the nth-level decoding module, the nth fused feature image obtained by fusing the (n+1)th decoded feature image and the nth encoded feature image is decoded to obtain the nth fused feature image; based on the nth-level upsampling submodule of the nth-level decoding module, the nth fused feature image is upsampled to obtain the nth decoded feature image; and The first decoded feature image is determined as the target decoded feature image; and Based on the target decoded feature image, a segmentation result for the target blood vessel is generated.

2. The method according to claim 1, characterized in that, The global encoding module includes a projection unit, a converter unit, and a deformation unit; the global encoding module based on the encoding part uses a self-attention mechanism to extract global morphological features from the bottleneck layer feature image to obtain a global feature image for characterizing the target blood vessel in the blood vessel image to be segmented, including: Based on the projection unit, the bottleneck layer feature image is embedded and projected to obtain a projection feature sequence; Based on the converter unit, the projection feature sequence is encoded using a multi-head self-attention mechanism to obtain an attention feature sequence; Based on the deformation unit, the attention feature sequence is deformed and mapped to obtain the global feature image.

3. The method according to claim 1, characterized in that, The decoding part further includes a target decoding module, which includes at least one convolutional unit; generating a segmentation result for the target blood vessel based on the target decoded feature image includes: Based on at least one convolutional unit of the target decoding module, the target feature image obtained by fusing the target decoded feature image and the initial encoded feature image is decoded to generate a segmentation result for the target blood vessel.

4. An image segmentation device for vascular images, characterized in that, include: The first processing module is used to extract features from the blood vessel image to be segmented by N hierarchical coding modules in the coding part of the blood vessel image segmentation model, thereby obtaining a bottleneck layer feature image. The image size of the encoded feature image output by each of the N hierarchical coding modules decreases progressively, and the Nth encoded feature image output by the Nth hierarchical coding module is used as the bottleneck layer feature image. Each hierarchical coding module includes at least one feature extraction submodule based on dynamic serpentine convolution; N is a positive integer and N≥2. The coding part includes an initial coding module, which includes at least one of the feature extraction submodules. Each hierarchical coding module also includes a downsampling submodule. The feature extraction submodule includes a standard convolution unit, a horizontal dynamic serpentine convolution unit, a vertical dynamic serpentine convolution unit, and a residual unit. The first processing module includes a first processing submodule, a second processing submodule, a third processing submodule, and a first determination submodule. The first processing submodule is used to extract features from the blood vessel image to be segmented input to the at least one feature extraction submodule of the initial encoding module, based on at least one feature extraction submodule of the initial encoding module, to obtain and output the initial encoded feature image; The second processing submodule is used to perform a downsampling operation on the initial encoded feature image based on the first downsampling submodule of the first-level encoding module to obtain a first downsampled feature image; and to perform feature extraction on the first downsampled feature image input to the first feature extraction submodule based on the first-level encoding module to obtain and output the first encoded feature image. The third processing submodule is used to perform a downsampling operation on the (i-1)th encoded feature image based on the i-th level encoding module's i-th downsampling submodule to obtain the i-th downsampled feature image; and to perform feature extraction on the i-th downsampled feature image input to the at least one feature extraction submodule based on the i-th level encoding module to obtain and output the i-th encoded feature image; i=2,…,N; The first determining submodule is used to determine the Nth encoded feature image as the bottleneck layer feature image; The feature extraction submodule is used to perform the following operations: acquiring the input feature image input to the feature extraction submodule; extracting features from the input feature image based on the standard convolution unit, the horizontal dynamic serpentine convolution unit, and the vertical dynamic serpentine convolution unit of the feature extraction submodule, respectively, to obtain a first intermediate feature image, a second intermediate feature image, and a third intermediate feature image; concatenating the first intermediate feature image, the second intermediate feature image, and the third intermediate feature image to obtain a concatenated feature image; processing the concatenated feature image based on the residual unit of the feature extraction submodule to obtain a residual feature image; fusing the concatenated feature image and the residual feature image to obtain an output feature image, and outputting the output feature image. The second processing module is used to extract global morphological features from the bottleneck layer feature image based on the global encoding module of the encoding part using a self-attention mechanism, so as to obtain a global feature image that represents the target blood vessel in the blood vessel image to be segmented; wherein, the global feature image is used to represent the global morphological features of the target blood vessel in the blood vessel image to be segmented. The third processing module is used to perform step-by-step feature fusion and step-by-step decoding on the global feature image and the encoded feature images output by each of the N hierarchical decoding modules in the decoding part of the blood vessel image segmentation model, to obtain the target decoded feature image. The image size of the decoded feature images output by each of the N hierarchical decoding modules increases progressively. The nth hierarchical decoding module corresponds to the nth hierarchical encoding module in the N hierarchical encoding module, n=1,…,N. The Nth decoded feature image output by the 1st hierarchical decoding module is used as the target decoded feature image. Each hierarchical decoding module includes at least one convolutional unit and an upsampling submodule. The third processing module includes a seventh processing submodule, an eighth processing submodule, and a second determination submodule. The seventh processing submodule is used to decode the Nth fused feature image obtained by fusing the global feature image and the Nth encoded feature image based on at least one convolutional unit of the Nth level decoding module when n=N, to obtain the Nth fused feature image; and to perform an upsampling operation on the Nth fused feature image based on the Nth level decoding module upsampling submodule, to obtain the Nth decoded feature image. The eighth processing submodule is used to decode the nth fused feature image obtained by fusing the (n+1)th decoded feature image and the nth encoded feature image based on at least one convolutional unit of the nth level decoding module when n=N-1, ...,1, to obtain the nth fused feature image; and to perform an upsampling operation on the nth fused feature image based on the nth level decoding module upsampling submodule to obtain the nth decoded feature image. The second determining submodule is used to determine the first decoded feature image as the target decoded feature image; and The generation module is used to generate a segmentation result for the target blood vessel based on the target decoded feature image.

5. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 3.

6. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 3.

7. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images