Neural network-based medical image segmentation method and system

Abstract

Embodiments of the present application provide a neural network-based medical image segmentation system, which comprises a feature extraction module for extracting features from a medical image and a classifier for segmenting the medical image according to the features extracted by the feature extraction module, wherein the feature extraction module comprises an encoder and a decoder constructed based on a convolutional neural network and a transformer connected between the encoder and the decoder. This scheme automatically learns and focuses on important regions at any position in the image via the transformer and suppresses irrelevant interference information, and can effectively exchange information between feature representations of different regions, different sub-pixels in the image, accurately depict the dependence and relationship between pixels, and thus accurately and explicitly capture and model long-distance global context information, thereby improving the segmentation efficiency and segmentation accuracy of medical images.

Description

Technical Field

[0001] This invention relates to the field of medical informatics, and in particular to a system and method for medical image segmentation using neural networks. Background Technology

[0002] The purpose of medical image segmentation is to make anatomical or pathological structural changes in medical images clearer. It often plays a key role in computer-aided diagnosis and smart healthcare. Common medical image segmentation tasks include liver and liver tumor segmentation, brain and brain tumor segmentation, optic disc segmentation, etc. With the development and popularization of medical imaging equipment, segmentation algorithms based on X-ray, computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound images have become important auxiliary tools to help clinicians diagnose diseases, assess prognosis, and plan surgical procedures. Thanks to the rapid development of technologies such as machine learning and deep learning, methods that use large amounts of sample data to train machine learning models such as neural networks as classifiers for automatic segmentation of medical images have been widely applied in the field of medical image segmentation. However, these medical image segmentation methods usually rely on manually designed features, but medical images are relatively complex, blurry, have low contrast, and are noisy, making it difficult to manually design effective features from medical images. In recent years, using neural network algorithms to achieve self-extraction of feature representations from medical images and applying these extracted features to medical image segmentation has become a popular research topic in the field of medical image processing.

[0003] The basic module of commonly used neural networks is the convolutional layer. The receptive field of a single convolutional layer is often limited and localized, only able to capture and process information from local areas of an image, and unable to accurately capture global contextual information reflecting the relationships and positions between pixels in the image. To capture information representations with a global receptive field, multiple convolutional layers and downsampling layers are typically stacked non-linearly in the neural network to expand the receptive field. However, this approach has several drawbacks: First, excessively deep neural networks consume enormous computational resources, making them difficult to apply in real-world scenarios; second, excessive parameters lead to optimization problems, and when applied to small datasets, too many parameters can easily cause overfitting; third, in capturing long-distance global contextual information through multi-layer stacking, object information features need to implicitly jump multiple times between layers to be transmitted to the target region. During this transmission process, useful features are easily lost, and additional harmful interference information is introduced, which may ultimately lead to limited and inaccurate segmentation results in complex segmentation tasks. Summary of the Invention

[0004] Therefore, the objective of this invention is to provide a new and effective medical image segmentation method and system for complex medical images. This objective is achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, a neural network-based medical image segmentation system is provided. This system includes a feature extraction module for extracting features from a medical image and a classifier for segmenting the medical image based on the features extracted by the feature extraction module. The feature extraction module includes an encoder and a decoder constructed based on a convolutional neural network, and a transformer connected between the encoder and decoder. The encoder is used to extract features of different dimensions from the input medical image. The transformer is used to generate features reflecting global contextual information of different regions of the medical image and the relationships between different pixels based on at least a portion of the features extracted by the encoder. The decoder is used to convert the features from the transformer into a feature format suitable for the classifier.

[0006] In some embodiments of the present invention, the transformer may include a serialization module, one or more self-attention modules, one or more cross-attention modules, and a deserialization module; wherein the serialization module is used to convert features from the encoder into a series of feature sequences with the same feature dimension; the self-attention module is used to generate a feature sequence reflecting the relationship between the feature sequences; the cross-attention module is used to correct the feature sequence output from the self-attention module based on a vector formed by the relationship between medical image categories; and the deserialization module is used to convert the feature sequence from the cross-attention module into a feature format suitable for convolutional neural networks.

[0007] In some embodiments of the present invention, the self-attention module may include a multi-head attention layer and a feedforward layer, wherein the sequence from the serialization module is mapped by different parameter matrices into input pairs consisting of a query vector, a key vector, and a value vector, wherein the multi-head attention layer consists of multiple independent attention layers; each attention layer calculates a score between pairs of features in the feature sequence by performing a dot product operation between the query vector and the key vector, the score representing the correlation between the two features; the score is normalized to between 0 and 1 and weighted with the value vector; the outputs of each attention layer are concatenated and aggregated by a linear layer before being output.

[0008] In some embodiments of the present invention, the cross-attention module may include two multi-head attention layers and a feedforward layer. The input of one multi-head attention layer is an input pair consisting of a query vector, a key vector, and a value vector, which is mapped based on the feature sequence reflecting each category in the medical image category. Its output is used as the query vector input of another multi-head attention layer. The key vector and value vector of the other multi-head attention layer come from the output of the self-attention module. The output of the other multi-head attention layer is added, normalized, and fed forward before being output to the deserialization module.

[0009] According to a second aspect of the present invention, a method for medical image segmentation using the system of the first aspect of the present invention is also provided, comprising: step S1, extracting features of different dimensions from an input medical image via a convolutional neural network acting as an encoder; step S2) capturing long-range global context information based on at least a portion of the extracted features of different dimensions via a transformer; step S3) converting the output from the transformer into features suitable for image segmentation by a classifier via a convolutional neural network acting as a decoder; and step S4) segmenting the medical image based on the features from the decoder via a classifier and outputting a corresponding segmentation result.

[0010] In some embodiments of the present invention, step S2) may include: converting features from the encoder into a series of feature sequences with the same feature dimension via a serialization module; generating a feature sequence reflecting the relationship between the feature sequences via the self-attention module; correcting the feature sequence from the self-attention module via a vector constructed based on the relationship between medical image categories via a cross-attention module; and converting the feature sequence from the cross-attention module into a feature format suitable for convolutional neural networks via a deserialization module.

[0011] In some embodiments of the present invention, generating a feature sequence reflecting the relationship between the feature sequences via the self-attention module may include: mapping the feature sequence from the serialization module into input pairs consisting of a query vector, a key vector, and a value vector through different parameter matrices; calculating the score between each pair of features in the feature sequence by performing a dot product operation on the query vector and the key vector; normalizing the score to between 0 and 1; weighting it with the value vector; and adding it to the input feature sequence; and aggregating and outputting the calculation results obtained based on each feature sequence.

[0012] According to a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the program, when executed, implements the method described in the second aspect of the above embodiments.

[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0014] This scheme provides an effective segmentation method for complex medical images. It can effectively exchange information between the feature representations of different regions and sub-pixels in the image, accurately characterize the dependencies and relationships between pixels, and thus accurately and explicitly capture and model long-range global context information. Therefore, it can significantly improve the segmentation efficiency and accuracy of complex medical images, thereby assisting doctors in providing better diagnosis and treatment services in a range of downstream applications.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0017] Figure 1 A schematic diagram of a neural network-based medical image segmentation system according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram of the structure of a self-attention module and a cross-attention module according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic flowchart of a neural network-based medical image segmentation method according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0024] In medical image segmentation, target anatomical or pathological structures often possess complex and highly varied shapes, appearances, and textures, posing a significant challenge to accurately segmenting target regions. Global context information, by modeling the relationships and dependencies between pixels in an image, provides rich location and category information, thus facilitating accurate target region segmentation. However, as mentioned above, existing methods that implicitly capture long-range global context information by non-linearly stacking multiple convolutional and downsampling layers in neural networks are not only inefficient but also inaccurate, resulting in reduced performance and inaccurate segmentation results in the final medical image segmentation.

[0025] This invention provides a novel neural network-based medical image segmentation system, comprising a feature extraction module for extracting features from medical images and a classifier for segmenting the medical images based on the features extracted by the feature extraction module. The feature extraction module combines a convolutional neural network model with a transformer model, enhancing the capture of global features of the medical image and dynamically focusing on important regional information while suppressing irrelevant interference. This alleviates the problem of neural networks failing to capture long-range global contextual information, extracting and combining feature representations rich in global and local information from the medical image, effectively improving the robustness and accuracy of medical image segmentation.

[0026] Figure 1 A schematic diagram of a neural network-based medical image segmentation system according to an embodiment of the present invention is shown. Figure 1As shown, the medical image segmentation system includes a feature extraction module and a classifier. The feature extraction module includes an encoder and decoder built based on a convolutional neural network, as well as a transformer connected between the encoder and decoder. This module extracts features for medical image segmentation from the input medical image and provides the extracted features to the classifier. The classifier takes the features from the feature extraction module as input and outputs the segmented medical image result. It should be noted that the classifier can employ any type of machine learning model suitable for medical image segmentation, including but not limited to neural network models, and this document does not impose any restrictions. The encoder and decoder in the feature extraction module are built using convolutional neural networks, such as LeNet, AlexNet, VGG, DenseNet, ResNet, and GoogLeNet. Preferably, VGG is used to construct the encoder and decoder. For ease of description, the following example illustrates that both the encoder and decoder are composed of multi-layer VGG networks. In this embodiment, the classifier, encoder, transformer, and decoder are first trained using a training set comprising a large number of medical image data samples (this can be called the offline training phase). Then, the medical image to be segmented is provided as input to the trained medical image segmentation system for processing, and its output is the segmented medical image (this can be called the online segmentation phase). The medical image data samples used for training include medical images and their corresponding versions marked with segmented regions. Typically, the collected data samples can be divided into a training set and a validation set (e.g., a 4:1 ratio). The training set is used to train each module of the medical image segmentation system, while the validation set is used to test the segmentation performance.

[0027] like Figure 1 As shown, during the training process, the medical images used as samples are first input into an encoder built on a convolutional neural network. The encoder extracts features from the input medical images. Different layers of the convolutional neural network extract features of different dimensions (collectively referred to as multi-scale features). The specific number of layers in the convolutional neural network can be determined according to actual needs. Figure 1 The encoder, constructed through multiple stages of VGG, extracts multi-scale features {X_0, X_1, X_2, X_3, X_4} from the input medical image. In some embodiments, all features extracted by the encoder can be used as input to the transformer. In other embodiments, a subset of features extracted by the encoder can be selected as input to the transformer, while the remaining features are directly used as input to the decoder. Figure 1As shown, features X_2, X_3, and X_4 (i.e., features extracted by the last three layers of the encoder's convolutional neural network) are selected from the features extracted by the encoder as input to the transformer; while features X_0 and X_1 (i.e., features extracted by the first two layers of the encoder's convolutional neural network) are used as input to the decoder. The transformer is used to further capture and model long-range global contextual information based on the multi-scale features obtained by the encoder. The output of the transformer is provided as features to the decoder. The decoder can convert the output from the transformer and the output from the encoder (if any) into features suitable for the classifier and input them into the classifier to obtain the corresponding segmentation result. The corresponding loss is calculated by comparing the segmentation result obtained by the classifier with the labeled segmentation result (i.e., the desired segmentation result) corresponding to the input medical image in the training set, and then the corresponding parameters in the above modules are adjusted and optimized in reverse. The above operations are iterated until the preset segmentation accuracy threshold or a predetermined number of iterations is met.

[0028] Continue to refer to Figure 1 The transformer can include a serialization module, a self-attention module, a cross-attention module, and a deserialization module. The transformer first uses the serialization module to serialize the multi-scale features X_2, X_3, X_4 from the encoder, transforming them into a series of feature sequences with the same feature dimension. For example, in the serialization module, each pixel on each level of the feature map is unfolded along the feature dimension and aligned using a 1*1 convolutional layer. Finally, multiple feature sequences T0 with the same feature dimension are aggregated and input into the self-attention module. The feature dimension of the feature sequences can be set according to actual needs and is not limited here.

[0029] Figure 2 A schematic diagram of the structure of a self-attention module and a cross-attention module according to an embodiment of the present invention is shown. Figure 2 As shown, the self-attention module can contain multi-head attention layers and feedforward layers, where normalization is applied before each layer, and residual connections are applied after each layer. The sequence T0 from the serialization module is first passed through three matrices with different parameter matrices (W). Q W K W V The linear transformation layer is mapped to obtain the input pair for the multi-head attention layer. This input pair consists of a query vector, a key vector, and a value vector, as shown in the following formula:

[0030] query=T0W Q key=T0W k , value = T0W V

[0031] The multi-head attention layer consists of multiple independent attention layers. Each attention layer employs an attention-based structure, the core of which involves examining other sequences within the input sequence while encoding the corresponding sequence. This allows for pairwise concatenation of sequences, capturing global contextual information through modeling of class and positional relationships. The computation process for each attention layer is as follows:

[0032]

[0033] Where d k Here, d is the scaling factor, T represents the transpose, and Softmax() represents the softmax activation function. As shown in the formula above, the score between each pair of features in the sequence is calculated by performing a dot product operation between the query vector and the key vector; this score represents the correlation between the two features. To maintain gradient stability, a scaling factor d is used. k The scores are normalized, then the Softmax() function is used to standardize the values ​​to between 0 and 1. Finally, the scores are weighted with the value vector to enhance relevant features and suppress irrelevant features. Based on this, the multi-head attention layer contains multiple independent attention layers, each focusing on a specific portion of contextual information. The outputs of these attention layers are concatenated and further aggregated through a linear layer. The calculation formula for the multi-head attention layer is as follows:

[0034] MSA(T0)=Concat(SA1(T0),...,SA h (T0))W o

[0035] Where h represents the total number of independent attention layers, W o This is the parameter matrix used to aggregate the linear feature layers. For example... Figure 2 As shown, the output of this multi-head attention layer is further processed through addition and normalization operations, as well as a feedforward layer (FFN), to arrive at the final output. The feedforward layer (FFN) can consist of two linear transformation layers and a nonlinear activation function (ReLU). The overall calculation process of the self-attention module can be represented by the following formula:

[0036] T′0=MSA(T0)+FFN(MSA(T0))

[0037] like Figure 2As shown, the cross-attention module has a similar hierarchical structure to the self-attention module, the difference being that the input also includes a surrogate vector E0, and an additional multi-head attention layer is added in the middle. The cross-attention module aims to use class distribution guidance to correct the contextual information encoded by the self-attention module. In the self-attention module, the contextual information is entirely based on a self-driven mechanism, so erroneous connections between features in the sequence may be established, thus polluting the global contextual information and causing unsatisfactory segmentation results. In the embodiments of this invention, a surrogate vector E is introduced to guide and correct the connections between features. Specifically, the surrogate vector E represents the feature prototype of each category in the target dataset, and one of its sub-vectors represents the feature prototype of a certain category. In one example, one sub-vector of the surrogate vector E0 can represent the feature representation of normal tissue, while another sub-vector can represent the feature information of diseased tissue.

[0038] like Figure 2 As shown, the agent vector E0 is first input into a multi-head attention layer to learn and model the relationships between classes. The learned class relationships serve as the input query vector for the next multi-head attention layer, while the context information T, encoded by the self-attention module, serves as the key and value vector input for that next multi-head attention layer. In this next multi-head attention layer, the learned class relationships and context information are connected, interact, and influence each other. The output of this next multi-head attention layer (denoted as E0)... The sequence is input to the subsequent deserialization module. In some embodiments, the output of the next multi-attention layer can be further processed by addition and normalization operations and a feedforward layer (FFN) to obtain the category distribution E′0 associated with the input context information. Therefore, there can be two outputs across the attention module. With E′0. For example Figure 1 As shown, the class distribution E′0 output by the cross-attention module can be further input into an additional classification supervision module to optimize the modeling of the class distribution. For example, this additional classification supervision module can input the class distribution E′0 of each sample into multiple linear layers, using image-level labels for additional classification supervision training, further optimizing the modeling of the class distribution. In subsequent iterations, the cross-attention module can use the surrogate vector E reflecting the class distribution optimized by the additional classification supervision module to further improve performance. Although... Figure 1 The additional classification supervision module is shown, but it should be understood that this module is optional rather than necessary, and the integrity of the overall system is not affected even without this additional classification supervision module.

[0039] The number of self-attention modules and cross-attention modules in the transformer can be determined according to actual needs and is not limited here. In some embodiments, the transformer may include several consecutive self-attention modules and cross-attention modules. For example, it may include 4 self-attention modules and 2 cross-attention modules. It can be seen that the self-attention modules and cross-attention modules in the transformer can automatically learn and focus important regions at any location in the image and suppress irrelevant interference information. Moreover, it can effectively exchange information between feature representations of different regions and different sub-pixels in the image, accurately characterize the dependencies and relationships between pixels, and thus accurately and explicitly capture and model long-range global context information, thereby improving the segmentation efficiency and accuracy of medical images.

[0040] like Figure 1 As shown, the class distribution E′0 obtained from the self-attention module and the cross-attention module after passing through the transformer is input to the additional classification supervision module, while the feature sequence... The data is then further input into the deserialization module for deserialization, i.e., folding back the 2D convolutional features. Next, the folded-back 2D convolutional features, along with the original multi-scale features X_0 and X_1 from the encoder, are input into the decoder constructed from the convolutional neural network for decoding. For example, the decoder's convolutional neural network can employ a bottom-up decoding structure, where the upper-layer convolutional features are downsampled to the same spatial resolution as the next layer, then added together, and fused through a 1x1 convolutional layer. Max pooling can be used for the downsampling operation. This process is iterated multiple times, ultimately outputting a feature map suitable for a classifier for medical image segmentation, which is then input into the classifier to obtain the corresponding segmentation result.

[0041] To better understand the present invention, the performance of the medical image segmentation system according to an embodiment of the present invention will be illustrated using training on a cell segmentation dataset as an example. The steps are as follows:

[0042] *The cell segmentation dataset was augmented to obtain a large number of data samples, which were then divided into training and validation sets in a 4:1 ratio.

[0043] *During the training phase, training data is input into... Figure 1In the medical image segmentation system shown, the encoder generates multi-scale features based on the input medical image data. These multi-scale features are further input into the transformer for encoding contextual information. The encoded global contextual information is then transmitted to the decoder for decoding and input into the classifier for segmentation prediction. The prediction result obtained by the classifier is compared with the segmentation results labeled in the training data, and a loss function is calculated to adjust and optimize the corresponding parameters. The system is trained a total of 60,000 times on the training set, and its performance is validated on the validation set every 500 times. The model parameters corresponding to the best performance on the validation set are selected to construct the various modules of the medical image segmentation system. The inventors then compared the performance of the trained medical image segmentation system of this invention with commonly used medical segmentation methods on the validation set.

[0044] Table 1 compares the segmentation performance of this invention with other commonly used methods on a cell segmentation dataset, using the intersection-over-union (IoU) metric to measure the segmentation results. In Table 1, parameters represent the network storage size, computational cost is the total number of addition and multiplication operations performed per iteration (in millions), classes one through five represent the segmentation accuracy for each segmentation category in the dataset, and the final average is the average segmentation performance across the five object classes. Table 1 shows that this invention outperforms other methods without increasing computational cost.

[0045] Table 1

[0046] method parameter computational load Category 1 Category 2 Category 3 Category 4 Category 5 average UNet 24.56 38.25 82.85 65.48 62.29 40.11 75.57 65.26 UNet++ 25.09 84.29 82.03 67.58 62.79 40.79 77.21 66.08 CENet 34.36 41.39 82.73 68.25 63.15 41.12 77.27 66.50 AttentionUnet 25.09 40.06 82.74 65.42 62.09 38.60 76.02 64.97 This invention 23.78 39.71 84.22 68.21 65.04 48.30 78.70 68.90

[0047] Figure 3 A flowchart illustrating a method for medical image segmentation using the neural network-based medical image segmentation system according to an embodiment of the present invention is shown. The method includes: step S1) extracting features of different dimensions from an input medical image via a convolutional neural network acting as an encoder; step S2) capturing long-range global contextual information by a transformer based on selecting at least a portion of the extracted features of different dimensions; step S3) converting the output from the transformer into features suitable for image segmentation by a classifier via a convolutional neural network acting as a decoder; and step S4) segmenting the medical image by the classifier based on the features from the decoder and outputting the corresponding segmentation result.

[0048] More specifically, in step S1, features of different dimensions are extracted from the input medical image via a convolutional neural network acting as an encoder. As mentioned above, different layers of the convolutional neural network extract features of different dimensions (collectively referred to as multi-scale features), and the specific number of layers in the convolutional neural network can be determined according to actual needs.

[0049] In step S2), a transformer captures long-range global contextual information based on selecting at least a portion of the extracted features from different dimensions. As mentioned above, in some embodiments, all features extracted by the encoder can be used as input to the transformer. In other embodiments, a portion of the features extracted by the encoder can be selected as input to the transformer, while the remaining features are directly used as input to the subsequent decoder. The transformer first transforms the features from the encoder into a series of feature sequences with the same feature dimension through a serialization module and inputs them to a self-attention module. The self-attention module captures global contextual information reflecting the relationships between feature sequences by concatenating the input sequences pairwise. Then, a cross-attention module further refines the contextual information encoded by the self-attention module using the relationships between categories in the target dataset. The feature sequences refined by the cross-attention module are transformed into a feature format suitable for convolutional neural networks via a deserialization module and passed to the convolutional neural network as the decoder.

[0050] In step S3), the output from the transformer and the output from the encoder (if any) are transformed into features suitable for the classifier via a convolutional neural network acting as a decoder and then input into the classifier.

[0051] In step S4), the classifier takes the features from the decoder as input and outputs the corresponding segmentation result.

[0052] In another embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program or executable instructions are stored. When the computer program or executable instructions are executed by a processor or other computing unit, the technical solution described in the foregoing embodiments is implemented. The implementation principle is similar and will not be repeated here. In the embodiments of the present invention, the computer-readable storage medium can be any tangible medium capable of storing data and readable by a computing device. Examples of computer-readable storage media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical or non-optical data storage devices. The computer-readable storage medium may also include computer-readable media distributed across a network-coupled computer system so that computer programs or instructions can be stored and executed in a distributed manner.

[0053] References to "various embodiments," "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a specific feature, structure, or property described in connection with the said embodiment, included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that the combination is not illogical or inoperable.

[0054] The terms "comprising," "having," and similar expressions used in this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. "A" or "an" does not exclude multiple instances. Furthermore, the elements in the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0055] Although the present invention has been described through the above embodiments, the present invention is not limited to the embodiments described herein, and includes various changes and variations without departing from the scope of the present invention.

Claims

1. A neural network-based medical image segmentation system, comprising a feature extraction module for extracting features from medical images and a classifier for segmenting the medical images based on the features extracted by the feature extraction module, wherein the feature extraction module includes an encoder and a decoder constructed based on a convolutional neural network and a transformer connected between the encoder and the decoder, wherein: The encoder is used to extract features of different dimensions from the input medical image; The transformer is used to generate features based on at least a portion of the features extracted by the encoder, which can reflect global contextual information that reflects different regions of the medical image and the relationships between different pixels. The decoder is used to convert features from the transformer into a feature format suitable for the classifier; The transformer includes a serialization module, one or more self-attention modules, one or more cross-attention modules, and a deserialization module; wherein: The serialization module is used to convert features from the encoder into a series of feature sequences with the same feature dimension; The self-attention module is used to generate feature sequences that reflect the relationships between the feature sequences; The cross-attention module is used to correct the feature sequence output from the self-attention module based on a vector formed by the relationship between medical image categories; The deserialization module is used to convert feature sequences from the cross-attention module into a feature format suitable for convolutional neural networks.

2. The system according to claim 1, wherein the self-attention module comprises a multi-head attention layer and a feedforward layer, wherein the sequence from the serialization module is mapped by different parameter matrices into input pairs consisting of a query vector, a key vector, and a value vector, wherein the multi-head attention layer consists of multiple independent attention layers; each attention layer calculates a score between pairs of features in the feature sequence by performing a dot product operation between the query vector and the key vector, the score representing the correlation between the two features; the score is normalized to between 0 and 1 and weighted with the value vector; the outputs of each attention layer are concatenated and aggregated by a linear layer before being output.

3. The system according to claim 1, wherein the cross-attention module comprises two multi-head attention layers and a feedforward layer, wherein the input of one multi-head attention layer is an input pair consisting of a query vector, a key vector, and a value vector, which is mapped based on the feature sequence reflecting each category in the medical image category, and its output serves as the query vector input of the other multi-head attention layer, wherein the key vector and value vector of the other multi-head attention layer are derived from the output of the self-attention module, and the output of the other multi-head attention layer is output to the deserialization module after addition and normalization operations and a feedforward layer.

4. The system according to any of the preceding claims, wherein the convolutional neural network employs a VGG network model.

5. A method for medical image segmentation using the system as described in any of the preceding claims, comprising: Step S1: Extract features of different dimensions from the input medical image via a convolutional neural network acting as an encoder; Step S2) Capture long-range global contextual information via a transformer based on at least a portion of the extracted features of different dimensions; Step S3) The output from the transformer is converted into features suitable for image segmentation by the classifier via a convolutional neural network acting as a decoder; Step S4) The medical image is segmented by a classifier based on features from the decoder, and the corresponding segmentation results are output. The step S2) includes: The features from the encoder are converted into a series of feature sequences with the same feature dimension via a serialization module; A feature sequence reflecting the relationship between the feature sequences is generated via a self-attention module; The feature sequence output from the self-attention module is modified by a vector constructed across attention modules based on the relationship between medical image categories; The deserialization module is used to transform the feature sequences from the cross-attention module into a feature format suitable for convolutional neural networks.

6. The method of claim 5, wherein generating a feature sequence reflecting the relationship between the feature sequences via the self-attention module comprises: The feature sequences from the serialization module are mapped into input pairs consisting of query vectors, key vectors, and value vectors through different parameter matrices. The score between pairwise features in the feature sequence is calculated by performing a dot product operation between the query vector and the key vector. The score is normalized to the range of 0-1, weighted with the value vector, and then added to the input feature sequence. The calculation results obtained based on each feature sequence are aggregated and then output.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method of any one of claims 5-6.

Images