Differential learning model-based blood vessel image segmentation method, device and equipment

Abstract

The invention provides a blood vessel image segmentation method, device and equipment based on a differential learning model, and relates to the field of neural network and image processing. According to the method, 3D blood vessel original image sets of different modalities are obtained and input into a blood vessel image segmentation model, and in the segmentation model, a reconstruction image embedded with modal features is obtained by using a modal difference marking method; inputting the reconstructed image into a 3D UNet encoder to extract low-level features; according to the low-level features, high-level features are generated through a position encoder and a self-attention mechanism; expanding the advanced features into one-dimensional vectors through a linear layer, and optimizing the one-dimensional vectors by adopting a Transform encoder to generate optimized features; secondly, calculating difference features between modals by means of element-by-element subtraction of the features after Transform optimization, so as to obtain difference fusion features; the difference fusion features are remodeled through a Transform encoder, and a final blood vessel segmentation result is obtained. According to the invention, morphological characteristics of small blood vessels can be accurately captured, and the blood vessel image segmentation precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network learning and image processing, and in particular to a method, device and equipment for segmenting blood vessel images based on a difference learning model. Background Art

[0002] Deep neural networks have been widely used in the field of medical image processing, especially in the segmentation of blood vessels in 3D medical images, such as CT and MR angiography image analysis, which helps doctors diagnose diseases and formulate treatment plans more accurately. In recent years, UNet has made major breakthroughs in medical image segmentation. It achieves pixel-by-pixel prediction through end-to-end training. UNet introduces jump connections between encoders and decoders, integrates low-resolution features into high-resolution features, and greatly improves segmentation capabilities.

[0003] However, existing vascular image segmentation methods have significant deficiencies in multimodal data fusion and difference processing. Traditional methods mainly rely on a single imaging modality (such as CTA or MRA), and fail to fully utilize the complementarity of multimodal data, resulting in limited segmentation accuracy and robustness. Especially in the segmentation of complex and small blood vessels, a single modality cannot fully capture vascular characteristics, affecting the accuracy of the segmentation results. In addition, existing methods fail to effectively handle the differences between different modalities (such as visual features, resolution, and noise differences), resulting in inconsistent segmentation results for the same vascular structure under different modalities, affecting the overall performance and application effect of the segmentation model.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a vascular image segmentation method, device and equipment based on a difference learning model to effectively integrate inter-modality differences and improve the accuracy and robustness of vascular segmentation, especially in the segmentation tasks of complex and small blood vessels.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: A blood vessel image segmentation method based on a difference learning model, comprising: S1, obtaining a set of 3D vascular original images of different modalities, and inputting them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module; S2, in the modality difference labeling module, using the modality difference labeling method, the local patches obtained by segmenting each image are fused and reconstructed with the small blocks labeled with the corresponding modality features to obtain a reconstructed image embedded with the modality features; S3, inputting the reconstructed image into a 3D UNet encoder to extract low-level features of each image modality; S4, adds position information to the low-level features of each image modality through the position encoder, and combines the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality; S5, the high-level features are expanded into corresponding one-dimensional vectors through a linear layer, and the one-dimensional vectors of each image modality are optimized using the normalization layer of the Transformer encoder, the multi-head self-attention mechanism and the multi-layer perceptron to capture the global context information of the image and generate optimized features; S6, in the difference learning module, the features optimized by the Transformer encoder are subtracted element by element to calculate the difference features between the modalities, and after convolution and normalization, they are fused with the difference features to obtain the difference fusion features; S7, the difference fusion feature is processed by the Transformer encoder and reshaped into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

[0007] The 3D vascular original image set includes CTA images, MRA images and DSA images; each of the local patches contains local area information of the vascular image; each local patch corresponds to a small block of the same size marked with modal features, namely, a modal feature marked block; the feature labels of the modal feature marked blocks are obtained by continuous training and optimization of the pre-trained model to adapt to the characteristics of a specific image modality and capture the specific information of the image modality.

[0008] Preferably, the local patch of each image is fused and reconstructed with the corresponding modality feature marked block, and the specific steps are as follows: Each original image of the 3D vascular original image set is divided into several local patches, each local patch introduces a feature tag related to the original image modality, and each local patch is fused with the corresponding modality feature tag block through feature fusion. The formula is: ; in, Indicates a local patch The fused patch; [ ] indicates the feature concatenation operation; Encoder(s) indicates the conversion of the modal feature marker block s into the corresponding local patch Feature vectors of the same dimension; Then, through the decoder, all the fused patches are combined and reconstructed to obtain the reconstructed image embedded with the modal features. , the formula is: ; Among them, Decoder is a decoder, represents the Pth local patch, where P is the number of local patches.

[0009] Preferably, the S4 is specifically: Each low-level feature inputted is added with position information through the position encoder; wherein the position encoder is generated by sine and cosine functions; The features with position encoding are input into the multi-head self-attention mechanism, and after linear transformation, the self-attention results of each head are output; Concatenate and linearly transform the self-attention results of all heads to obtain the final self-attention result; The final self-attention result is normalized by layers and then input into a multi-layer perceptron to perform deep nonlinear processing on the normalized features. It is then added to the input low-level features through residual connections to obtain high-level features of the image modality.

[0010] Preferably, the Transformer encoder includes a normalization layer, a multi-head self-attention mechanism and a multi-layer perceptron; the Transformer encoder is used to optimize the one-dimensional vector of each image modality, specifically: The normalized features are input into the multi-head self-attention mechanism, and each self-attention head learns different relationships in the input features in parallel; Then the output results of all heads are concatenated together and mapped to the output space through a linear transformation to obtain the multi-head self-attention result; The multi-head self-attention results are normalized by layers and then input into a multi-layer perceptron to perform deep nonlinear processing on the normalized features. They are then added to the input features through residual connections to generate Transformer optimized features.

[0011] Preferably, the S6 is specifically: Reshape the Transformer optimized features into the same shape as the high-level features, and then calculate the difference features between the image modalities by element-by-element subtraction ; Sending the difference features to the convolution layer and the normalization layer in sequence to obtain normalized difference features; The normalized difference feature is compared with the difference feature Perform residual connection to obtain the fused difference features ; The fused difference features Reshape back to one-dimensional features , which is the difference fusion feature.

[0012] Preferably, it also includes using a loss function to continuously optimize the accuracy of multimodal vascular image segmentation, wherein the loss function Combining multi-class weighted cross entropy loss and , whose expression is: ; ; ; in, represents the true value of the c-th and n-th blood vessel image pixel, , n N, C is the total number of labeled categories; N is the total number of pixels in the vascular image; is the predicted value.

[0013] Preferably, it also includes using a loss function to continuously optimize the vascular image segmentation model to improve the accuracy of the model in multimodal vascular image segmentation, and the loss function used is Combining multi-class weighted cross entropy loss and , the expression is: ; in, and Respectively represent the modal feature labeling blocks corresponding to the i-th local patch through the encoder and Feature representation after encoding; Represents the inner product operation, that is, the dot product of two vectors; represents the L2 norm; represents the total number of local patches; By minimizing the loss function , so that the modal feature markers are cut and The differences between them are reduced to improve the effectiveness of marking.

[0014] The present invention also provides a blood vessel image segmentation device based on a difference learning model, comprising: An input unit, used to obtain a set of 3D vascular original images of different modalities and input them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module; A modality difference labeling unit, used in the modality difference labeling module, to fuse and reconstruct the local patches obtained by segmenting each image with the small blocks labeled with the corresponding modality features by using the modality difference labeling method, so as to obtain a reconstructed image embedded with the modality features; A 3D UNet encoding unit, used for inputting the reconstructed image into a 3D UNet encoder to extract low-level features of each image modality; The position encoding unit is used to add position information to the low-level features of each image modality through the position encoder, and combine the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality; A Transformer optimization unit is used to expand the high-level features into corresponding one-dimensional vectors through a linear layer, and optimize the one-dimensional vectors of each image modality using the normalization layer, multi-head self-attention mechanism and multi-layer perceptron of the Transformer encoder to capture the global context information of the image and generate optimized features; The difference learning unit is used to calculate the difference features between the modalities by element-by-element subtraction of the features optimized by the Transformer encoder in the difference learning module, and then fuse them with the difference features after convolution and normalization to obtain the difference fusion features; The output unit is used to process the difference fusion feature through the Transformer encoder and reshape it into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

[0015] The present invention also provides a vascular image segmentation device based on a difference learning model, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a vascular image segmentation method based on a difference learning model as described above.

[0016] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a vascular image segmentation method based on a difference learning model as described above is implemented.

[0017] In summary, compared with the prior art, the present invention has the following beneficial effects: The present invention obtains multimodal vascular image segmentation by constructing a vascular image segmentation model based on a difference learning model. The model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module. In the modality difference labeling module, the acquired original image is reconstructed using the modality difference labeling method to capture the specific information of the image modality, and a reconstructed image embedded with modality features is obtained to compensate for the differences between different imaging modalities and enhance the understanding and processing capabilities of the multimodal vascular image segmentation model for multimodal images.

[0018] The present invention introduces a multimodal fusion method based on difference learning. By calculating the difference features between different modalities and fusing them, the complementary information between different modal images can be fully mined, thereby overcoming the limitations of traditional single-modal methods in small blood vessel segmentation, improving the accuracy and robustness of blood vessel segmentation, and performing outstanding performance in the task of segmenting small blood vessels. Compared with existing methods, the method of the present invention can more accurately capture the morphological characteristics of small blood vessels, reduce the influence of noise interference and differences between modalities, and obtain accurate and detailed segmentation results.

[0019] In addition, the present invention adopts different loss functions to respectively optimize the modality difference labeling module and the multimodal vascular image segmentation model. By minimizing the loss function, the modality difference labeling is optimized, the differences between different modalities are reduced, and the accuracy of multimodal vascular image segmentation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a blood vessel image segmentation method based on a difference learning model provided in Example 1.

[0022] Figure 2 A structural schematic diagram of a blood vessel image segmentation method based on a difference learning model provided in Example 1.

[0023] Figure 3 This is a structural diagram of the modality difference marking module provided in Example 1.

[0024] Figure 4 This is a visualization diagram of the segmentation result provided in Example 1.

[0025] Figure 5A schematic diagram of a blood vessel image segmentation device based on a difference learning model provided in Example 2.

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Embodiment 1 Embodiment 1 of the present invention provides a vascular image segmentation method based on a difference learning model, which can be implemented by a vascular image segmentation device based on a difference learning model (hereinafter referred to as a segmentation device), and in particular, executed by one or more processors in the segmentation device.

[0029] In this embodiment, the segmentation device may be an electronic device equipped with a processor, the processor having a computer program of the vascular image segmentation method based on the difference learning model and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0030] Existing vascular segmentation methods have significant deficiencies in multimodal data fusion and difference processing. Traditional methods mainly rely on a single imaging modality (such as CTA or MRA), and fail to fully utilize the complementarity of multimodal data, resulting in limited segmentation accuracy and robustness. Especially in the segmentation of complex and small blood vessels, a single modality cannot fully capture vascular characteristics, affecting the accuracy of the segmentation results. In addition, existing methods fail to effectively handle the differences between different modalities (such as visual features, resolution, and noise differences), resulting in inconsistencies in cross-modal segmentation results, further reducing the overall performance of the model. Therefore, this embodiment aims to propose a multimodal fusion method based on difference learning, which effectively integrates the differences between modalities and improves the accuracy and robustness of vascular segmentation, especially in the segmentation tasks of complex and small blood vessels.

[0031] like Figure 1-Figure 2As shown, a blood vessel image segmentation method based on a difference learning model includes steps S1 to S7.

[0032] S1, obtaining a set of 3D vascular original images of different modalities and inputting them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module.

[0033] Specifically, the original 3D vascular image sets of different modalities include but are not limited to: CTA images, MRA images, DSA images, etc. Among them, CTA (Computed Tomography Angiography) images are obtained by intravenous injection of contrast agents to increase the concentration of contrast agents in blood vessels, thereby clearly displaying the vascular structure on CT images. MRA (Magnetic Resonance Angiography) images can display vascular structures without the use of contrast agents. They use the signal differences generated by blood flow to distinguish blood vessels from other tissues. DSA (Digital Subtraction Angiography) is an imaging technology that uses computer processing to remove bone and soft tissue images after injecting contrast agents, leaving only vascular images.

[0034] like Figure 2-3 As shown, the multimodal vascular image segmentation model based on difference learning includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module.

[0035] S2, in the modality difference labeling module, the modality difference labeling method is used to fuse and reconstruct the local patches obtained by segmenting each image with the small blocks marked with the corresponding modality features to obtain a reconstructed image embedded with the modality features.

[0036] like Figure 3 As shown, taking CTA images and MRA images as examples, we reconstruct the original images using the modality difference labeling method to obtain a reconstructed image embedded with modality features to make up for the differences between different imaging modalities (CTA and MRA) and enhance the model's understanding and processing capabilities for multimodal images. In this embodiment, each local patch contains local area information of the vascular image. The modality feature labeling corresponding to each local patch and the feature labeling corresponding to the cut block are continuously trained and optimized by the pre-training model to adapt to the characteristics of a specific image modality and capture the specific information of the image modality.

[0037] In this embodiment, by introducing the fusion strategy of modality-specific markers and local patches, the unique features of different modalities are fully exploited to improve the expressiveness and generalization ability of the model. The local patches of each image are fused and reconstructed with the corresponding feature markers. The specific steps are as follows: Each original image of the 3D blood vessel original image set is divided into several local patches Each local patch introduces a feature marker related to the original image mode, that is, each small block (local patch) corresponds to a small block of the same size used to mark their modal features, that is, the corresponding modal feature marker block For each local patch and its corresponding modal feature marked block s, adopt a feature fusion strategy (such as splicing in this embodiment, or other methods such as weighted fusion according to the situation) to enhance the expression ability of the patch, and fuse each local patch with the corresponding modal feature marked block through feature fusion. The formula is: ; in, Indicates a local patch The fused patch; [ ] indicates the feature concatenation operation; Encoder(s) indicates that the modal feature marker block s is converted into the corresponding local patch through the encoder Feature vectors of the same dimension; i=1,2,…,P; P is the number of local patches.

[0038] The merged patch It not only retains the spatial information of the local image, but also incorporates modality-specific features, thereby better expressing cross-modal features.

[0039] After all local patches have completed feature fusion, all fused patches are combined and reconstructed through the decoder to obtain a reconstructed image embedded with modal features. , the formula is: ; Among them, Decoder is a decoder, represents the Pth local patch.

[0040] S3, input the reconstructed image into the 3D UNet encoder to extract the low-level features of each image modality.

[0041] In this embodiment, the 3D UNet encoder is a variant of the UNet architecture, which is specifically used to process three-dimensional medical image data and is mainly composed of multiple convolutional layers and pooling layers. These layers are stacked to gradually extract and compress the features of the input three-dimensional medical image.

[0042] Convolutional layer: Responsible for extracting local features in the image. In 3D UNet, the convolutional layer uses a three-dimensional convolution kernel to extract features in volume data. Each convolutional layer is usually followed by a nonlinear activation function (such as ReLU) to enhance the nonlinear ability of the network.

[0043] Pooling layer: used to gradually reduce the size of the feature map while increasing the receptive field size. This helps the network capture more global feature information. In 3D UNet, the pooling layer usually uses three-dimensional maximum pooling or average pooling.

[0044] S4, adds position information to the low-level features of each image modality through the position encoder, and combines the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality.

[0045] Furthermore, in the position encoder part of the vascular image segmentation model, each low-level feature of the input is added with position information through the position encoder; wherein the position encoder is generated by sine and cosine functions, aiming to add position information to each input feature, thereby helping the model understand the relative position of each element in the sequence. Through position encoding, the input feature not only contains the original image information, but also introduces the sequential relationship of the data, so that the model can capture the structural information inside the data.

[0046] The features with position encoding are input into the multi-head self-attention mechanism of the Transformer encoder. The multi-head self-attention mechanism generates a query matrix (Q), a key matrix (K), and a value matrix (V) by linearly transforming the input features. Then, the dot product similarity between the query matrix and the key matrix is ​​calculated and normalized to obtain the attention weight matrix. The attention weight matrix is ​​weighted and summed with the corresponding value matrix, the result of each self-attention head is output, and the output results of all heads are concatenated. In this way, the self-attention mechanism can dynamically adjust the weight of each element according to the relationship between the input features, thereby effectively capturing global information.

[0047] After processing the self-attention, the final self-attention result is normalized by layer and input into the multi-layer perceptron to perform deep nonlinear processing on the normalized features. It is then added to the low-level features of the input through residual connection to obtain high-level features of the image modality, ensuring that information can flow effectively.

[0048] Normalization can ensure that the mean and variance of the features are stable, which helps to accelerate training and prevent gradients from disappearing or exploding. The multilayer perceptron of this embodiment consists of multiple fully connected layers, and each layer is usually transformed by a nonlinear activation function (such as a rectified linear unit). The role of the multilayer perceptron is to perform deep nonlinear processing on the normalized features, thereby further improving the expressive power of the model.

[0049] Through the iterative processing of the above steps, the output of the position encoder is a set of high-dimensional feature representations that incorporate global context information. and .

[0050] In this step, high-level features of the input low-level features are extracted through position encoders and self-attention mechanisms. These high-level features can not only effectively capture the complex relationships in the input data, but also provide accurate feature representations for downstream tasks such as blood vessel segmentation.

[0051] S5, the high-level features are expanded into corresponding one-dimensional vectors through linear layers, and the normalization layer, multi-head self-attention mechanism and multi-layer perceptron of Transformer encoder are used to optimize the one-dimensional vectors of each image modality to capture the global context information of the image and generate optimized features.

[0052] In this step, the role of the linear layer is to perform a linear transformation on the high-level features of the input, converting their dimensions to the input dimensions required by the Transformer encoder, that is, expanding them into a one-dimensional vector and The one-dimensional vector of each image modality is then input into the Transformer encoder.

[0053] like Figure 2 As shown, the Transformer encoder structure in this embodiment includes, in order: a normalization layer, a multi-head self-attention mechanism, a normalization layer and a multi-layer perceptron MLP. The normalized input features are input into the multi-head self-attention mechanism and divided into multiple subspaces. Each subspace is processed by an independent self-attention head, and each self-attention head learns different relationships in the input features in parallel. Finally, the output results of all heads are spliced ​​together and mapped to the final output space through a linear transformation to obtain the final self-attention result. In this way, the model can capture rich information in the input data from multiple angles at the same time, enhancing the ability to model complex data.

[0054] The Multi-Head Self-Attention layer is the core part of the Transformer encoder, which allows the model to pay attention to all other elements when processing each input element, thereby learning the global dependencies in the input sequence. The Multi-Head Self-Attention mechanism calculates the results of multiple attention heads in parallel and concatenates them to enhance the representation ability of the model.

[0055] Each Transformer encoder layer also contains residual connections and layer normalization operations. Residual connections help alleviate the gradient vanishing problem in deep networks, while layer normalization helps speed up the training process and improve the stability of the model.

[0056] The multi-head self-attention mechanism captures the global dependencies between different positions (or different modalities) to generate feature representations containing global context information. Through these operations, the Transformer encoder is able to gradually optimize the representation of the input vector so that it contains richer global context information.

[0057] The output of the multi-head self-attention is normalized by layers and then input into a multi-layer perceptron to perform deep nonlinear processing on the normalized features. It is then added to the input features through residual connections to generate optimized features.

[0058] Transformer encoder outputs optimized feature representation and These feature representations not only contain the high-level features of the original image, but also the global dependencies between different locations (or different modalities).

[0059] In this embodiment, the Transformer encoder is an important component of the Transformer model, which is responsible for converting the input sequence information into a set of representation vectors for subsequent feedforward neural network or decoder training and prediction.

[0060] S6, in the difference learning module, the features optimized by Transformer are subtracted element by element to calculate the difference features between the modalities, and after convolution and normalization, they are fused with the difference features to obtain the difference fusion features.

[0061] The difference learning module is used to solve the problem of modal differences in multimodal data processing. Through difference modeling and feature fusion, the complementary information from different modalities (CTA and MRA) is accurately extracted to optimize the segmentation results. First, the two modal features processed by the Transformer encoder are and are reshaped into the same shape as the original high-level features, respectively. and : ; Among them, reshape is the reshaping function.

[0062] This operation ensures the consistency of the features of different modes in shape, providing a unified basis for subsequent difference calculations. Next, the model calculates the difference features between modes by element-by-element subtraction. , the expression is: ; The difference characteristics obtained It describes the difference information between the two image modalities in terms of visual features, resolution and noise, providing a basis for subsequent feature fusion and optimization.

[0063] Then, the difference feature It is sent to the convolution layer and normalized to obtain the normalized difference features, which helps to smooth the difference features, reduce the impact of noise, and enhance the model's ability to capture key features. Differences from the original Perform residual connection to obtain the fused difference features , the expression is: ; The residual connection can retain the useful information of the original difference features, and at the same time, it can deepen the feature expression ability of the model through enhanced learning, thereby improving the recognition ability of complex and small vascular structures. Reshaped back to one-dimensional features , which is the difference fusion feature: ; Through this difference learning module, the model corresponding to the method of the present invention (called DL_Net model) can fully integrate the difference information between different modalities, thereby improving the accuracy and robustness of blood vessel segmentation.

[0064] S7, the difference fusion feature is processed by the Transformer encoder and reshaped into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

[0065] In this embodiment, the difference fusion feature The final blood vessel segmentation result is obtained by reshaping the Transformer encoder part.

[0066] In another preferred embodiment, the present invention further includes using a loss function to continuously optimize the accuracy of the vascular image segmentation model based on the difference learning model. The loss function used Combining multi-class weighted cross entropy loss and , whose expression is: ; ; ; in, represents the true value of the c-th and n-th blood vessel image pixel, , n N, C are the total number of categories; N is the total number of pixels in the vascular image; is the predicted value.

[0067] In addition, the feature labels corresponding to each local patch involved in the modality difference labeling module are continuously trained and optimized through the pre-training model. In order to optimize the modality difference labeling and reduce the differences between different modalities, a loss function based on cosine similarity is adopted. Optimize the representation of the feature markers corresponding to each local patch to reduce the differences between different image modalities. The expression is: ; in, and Respectively represent the modal feature labeling blocks corresponding to the i-th local patch through the encoder and Feature representation after encoding; Represents the inner product operation, that is, the dot product of two vectors; represents the L2 norm; represents the total number of local patches; By minimizing the loss function , so that the modal feature markers are cut and The differences between them are reduced, the representation of the same feature in different modalities is more consistent, while different features maintain their differences, thereby improving the effectiveness of cross-modal feature fusion.

[0068] like Figure 4 As shown, the segmentation results of the method model of the present invention (vascular image segmentation model based on differential learning, namely Ours in the figure) and six common medical image segmentation models (3DUnet, SegResNet, SegResNetVAE, UNETR, SwinUNETR and CSNet3D) are visually compared and displayed. The red frame area of ​​the Ground Truth (the real annotation or reference segmentation result) in the figure shows the small and easily overlooked blood vessels between different blood vessels. In the CT and MR image modalities, our proposed model (namely Ours in the figure) successfully focused on these small blood vessels and performed correct segmentation. Other models either ignored these blood vessels or mistakenly segmented them into other types. It can be seen that the model proposed by ours can pay more attention to the details in the image through differential learning, thereby achieving a more accurate segmentation effect.

[0069] illustrate: Figure 4 In the figure, CT-case A represents a case of CTA images. MR-case A represents a case of MRA images. Ground Truth: refers to the real annotation or reference segmentation result, which is used as a benchmark for evaluating segmentation performance. Ours: The segmentation result obtained by the model of the method of the present invention.

[0070] The following are several commonly used medical image segmentation models: 3DUnet: A 3D image segmentation model based on convolutional neural networks (CNNs), which is widely used in medical image segmentation tasks. The basic principle is: a 3D image is input into an encoder and its dimension is reduced through a series of convolutional layers. The output of the encoder is then upsampled through a series of deconvolutional layers, and the final output is a segmentation result with the same size as the original input.

[0071] SegResNet: A segmentation network based on the ResNet architecture that uses deep residual learning to improve image segmentation performance. Its basic principle is to train a deeper network through residual learning, that is, to learn the residual between the network input and output, rather than directly learning the mapping relationship between the original input and output.

[0072] SegResNetVAE: The SegResNet model combined with a variational autoencoder (VAE) improves the robustness of segmentation by enhancing generation capabilities. VAE is a generative model used to learn data distribution and generate new data samples. It does this by mapping the input data to an encoded vector in a low-dimensional latent space and then mapping the encoded vector back to the original data space through a decoder.

[0073] UNETR: A 3D image segmentation model that combines the Transformer architecture with U-Net, suitable for 3D medical image segmentation.

[0074] SwinUNETR: A 3D image segmentation model that combines Swin Transformer and Convolutional Neural Network (CNN). Swin Transformer is used to capture long-range dependencies while CNN is used to extract local features.

[0075] CSNet3D: A 3D convolutional neural network (CNN) model specifically designed for medical image segmentation.

[0076] In summary, compared with the prior art, the present invention has the following beneficial effects: The multimodal vascular image segmentation model based on difference learning of the present invention first reconstructs the original image into a reconstructed image embedded with modal features in the modal difference labeling module; then uses the 3D UNet encoder to extract the low-level features of each modality of the reconstructed image respectively, and combines the position encoding to generate the high-level features of the modality. Next, the high-level features of each modality are processed by the Transformer encoder to capture global context information. Difference learning is used to model and integrate the differences between different modalities (such as visual features, resolution and noise differences), generate difference features, and fuse them with the original difference features after processing. Finally, the fused features are further encoded to generate vascular segmentation results.

[0077] The present invention introduces a differential learning mechanism to effectively fuse 3D image data of different modalities to improve the accuracy of blood vessel segmentation, especially in the segmentation tasks of small and complex blood vessels.

[0078] Embodiment 2 like Figure 5 As shown, the second embodiment of the present invention further provides a blood vessel image segmentation device based on a difference learning model, comprising: An input unit, used to obtain a set of 3D vascular original images of different modalities and input them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module; A modality difference labeling unit, used in the modality difference labeling module, to fuse and reconstruct the local patches obtained by segmenting each image with the small blocks labeled with the corresponding modality features by using the modality difference labeling method, so as to obtain a reconstructed image embedded with the modality features; A 3D UNet encoding unit, used for inputting the reconstructed image into a 3D UNet encoder to extract low-level features of each image modality; The position encoding unit is used to add position information to the low-level features of each image modality through the position encoder, and combine the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality; A Transformer optimization unit is used to expand the high-level features into corresponding one-dimensional vectors through a linear layer, and optimize the one-dimensional vectors of each image modality using the normalization layer, multi-head self-attention mechanism and multi-layer perceptron of the Transformer encoder to capture the global context information of the image and generate optimized features; The difference learning unit is used to calculate the difference features between the modalities by element-by-element subtraction of the features optimized by the Transformer encoder in the difference learning module, and then fuse them with the difference features after convolution and normalization to obtain the difference fusion features; The output unit is used to process the difference fusion feature through the Transformer encoder and reshape it into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

[0079] Embodiment 3 The third embodiment of the present invention also provides a vascular image segmentation device based on a difference learning model, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the vascular image segmentation method based on the difference learning model as described above.

[0080] Embodiment 4 The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the vascular image segmentation method based on the difference learning model as described above is implemented.

[0081] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0083] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0084] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0085] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0086] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0087] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A blood vessel image segmentation method based on a difference learning model, characterized in that: include: S1, obtaining a set of 3D vascular original images of different modalities, and inputting them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module; S2, in the modality difference labeling module, using the modality difference labeling method, the local patches obtained by segmenting each image are fused and reconstructed with the small blocks labeled with the corresponding modality features to obtain a reconstructed image embedded with the modality features; S3, inputting the reconstructed image into a 3D UNet encoder to extract low-level features of each image modality; S4, adds position information to the low-level features of each image modality through the position encoder, and combines the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality; S5, the high-level features are expanded into corresponding one-dimensional vectors through a linear layer, and the one-dimensional vectors of each image modality are optimized using the normalization layer of the Transformer encoder, the multi-head self-attention mechanism and the multi-layer perceptron to capture the global context information of the image and generate optimized features; S6, in the difference learning module, the features optimized by the Transformer encoder are subtracted element by element to calculate the difference features between the modalities, and after convolution and normalization, they are fused with the difference features to obtain the difference fusion features; S7, the difference fusion feature is processed by the Transformer encoder and reshaped into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

2. The vascular image segmentation method based on the difference learning model according to claim 1 is characterized in that The 3D vascular original image set includes CTA images, MRA images and DSA images; each of the local patches contains the local area information of the vascular image; each local patch corresponds to a small block of the same size marked with modal features, namely, the modal feature marked block; the feature labels of the modal feature marked block are obtained by continuous training and optimization of the pre-trained model to adapt to the characteristics of a specific image modality and capture the specific information of the image modality.

3. The vascular image segmentation method based on the difference learning model according to claim 2 is characterized in that ,The local patch of each image is fused and reconstructed with the corresponding modal feature marked block. The specific steps are as follows: Each original image of the 3D vascular original image set is divided into several local patches, each local patch introduces a feature tag related to the original image modality, and each local patch is fused with the corresponding modality feature tag block through feature fusion. The formula is: ； in, Indicates a local patch The fused patch; [ ] indicates the feature concatenation operation; Encoder(s) indicates the conversion of the modal feature marker block s into the corresponding local patch Feature vectors of the same dimension; Then, through the decoder, all the fused patches are combined and reconstructed to obtain the reconstructed image embedded with the modal features. , the formula is: ； Among them, Decoder is a decoder, represents the Pth local patch, where P is the number of local patches.

4. The vascular image segmentation method based on the difference learning model according to claim 1 is characterized in that , the S4 is specifically: Each low-level feature inputted is added with position information through the position encoder; wherein the position encoder is generated by sine and cosine functions; The features with position encoding are input into the multi-head self-attention mechanism, and after linear transformation, the self-attention results of each head are output; Concatenate and linearly transform the self-attention results of all heads to obtain the final self-attention result; The final self-attention result is normalized by layers and then input into a multi-layer perceptron to perform deep nonlinear processing on the normalized features. It is then added to the input low-level features through residual connections to obtain high-level features of the image modality.

5. The vascular image segmentation method based on the difference learning model according to claim 1 is characterized in that , the Transformer encoder includes a normalization layer, a multi-head self-attention mechanism and a multi-layer perceptron; the Transformer encoder is used to optimize the one-dimensional vector of each image modality, specifically: The normalized features are input into the multi-head self-attention mechanism, and each self-attention head learns different relationships in the input features in parallel; Then the output results of all heads are concatenated together and mapped to the output space through a linear transformation to obtain the multi-head self-attention result; The multi-head self-attention results are normalized by layers and then input into a multi-layer perceptron to perform deep nonlinear processing on the normalized features. They are then added to the input features through residual connections to generate Transformer optimized features.

6. The vascular image segmentation method based on the difference learning model according to claim 1, characterized in that , the S6 is specifically: Reshape the Transformer optimized features into the same shape as the high-level features, and then calculate the difference features between the image modalities by element-by-element subtraction ; Sending the difference features to the convolution layer and the normalization layer in sequence to obtain normalized difference features; The normalized difference feature is compared with the difference feature Perform residual connection to obtain the fused difference features ; The fused difference features Reshape back to one-dimensional features , which is the difference fusion feature.

7. The vascular image segmentation method based on the difference learning model according to claim 1 is characterized in that , also includes, using the loss function to continuously optimize the vascular image segmentation model to improve the accuracy of the model in multimodal vascular image segmentation. The loss function used Combining multi-class weighted cross entropy loss and , whose expression is: ； ； ； in, represents the true value of the c-th and n-th blood vessel image pixel, , n N, C is the total number of labeled categories; N is the total number of pixels in the vascular image; is the predicted value.

8. The vascular image segmentation method based on the difference learning model according to claim 3 is characterized in that , also includes: when the feature labels of the modal feature label blocks are continuously trained and optimized through the pre-training model, a loss function based on cosine similarity is used The representation of the modality feature marker slice corresponding to each local patch is optimized to reduce the difference between different image modalities. The expression is: ； in, and Respectively represent the modal feature labeling blocks corresponding to the i-th local patch through the encoder and Feature representation after encoding; Represents the inner product operation, that is, the dot product of two vectors; represents the L2 norm; represents the total number of local patches; By minimizing the loss function , so that the modal feature markers are cut and The differences between them are reduced to improve the effectiveness of marking.

9. A blood vessel image segmentation device based on a difference learning model, characterized in that: include: An input unit, used to obtain a set of 3D vascular original images of different modalities and input them into a vascular image segmentation model based on difference learning; wherein the vascular image segmentation model includes a modality difference labeling module, a 3D UNet encoder, a position encoder, a difference learning module and a Transformer encoder module; A modality difference labeling unit, used in the modality difference labeling module, to fuse and reconstruct the local patches obtained by segmenting each image with the small blocks labeled with the corresponding modality features by using the modality difference labeling method, so as to obtain a reconstructed image embedded with the modality features; A 3D UNet encoding unit, used for inputting the reconstructed image into a 3D UNet encoder to extract low-level features of each image modality; The position encoding unit is used to add position information to the low-level features of each image modality through the position encoder, and combine the self-attention mechanism to capture the global information of the input features to generate high-level features of the corresponding image modality; A Transformer optimization unit is used to expand the high-level features into corresponding one-dimensional vectors through a linear layer, and optimize the one-dimensional vectors of each image modality using the normalization layer, multi-head self-attention mechanism and multi-layer perceptron of the Transformer encoder to capture the global context information of the image and generate optimized features; The difference learning unit is used to calculate the difference features between the modalities by element-by-element subtraction of the features optimized by the Transformer encoder in the difference learning module, and then fuse them with the difference features after convolution and normalization to obtain the difference fusion features; The output unit is used to process the difference fusion feature through the Transformer encoder and reshape it into the same shape as the high-level feature to obtain the final blood vessel segmentation result.

10. A blood vessel image segmentation device based on a difference learning model, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a blood vessel image segmentation method based on a difference learning model as described in any one of claims 1 to 8.

Images