Small sample medical image segmentation method and system based on multi-scale jump connection and comparative learning
By adopting a dual-branch network model with multi-scale jump connection and contrast learning in medical image segmentation, the scarcity of labeled data and image complexity are solved, the accuracy and robustness of medical image segmentation are improved, and higher segmentation is achieved Accuracy and practicality.
Patent Information
- Application Number
- CN202410806496.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-06-21
AI Technical Summary
Medical image segmentation technology faces the scarcity, complexity and diversity of labeled data, which makes it difficult to effectively apply small sample learning in actual scenarios.
Using a dual-branch network model based on multi-scale jump connection and contrast learning, the support set feature vector is extracted through conditional branches, combined with multi-scale jump connection and contrast learning loss function with sequential splicing, to improve the model's segmentation ability and generalization ability of small targets. .
It improves the accuracy and robustness of medical image segmentation, enhances the semantic correlation of the model to finite data, and improves the segmentation accuracy and practicality.
Smart Images

Figure CN119048525A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a small sample medical image segmentation method and system based on multi-scale skip connection and contrastive learning. Background Art
[0002] Medical image segmentation is a key task in medical image analysis, which aims to accurately distinguish the region of interest (such as organs, tumors, etc.) in medical images from the background. This process plays an important role in clinical diagnosis, treatment planning, and efficacy evaluation. However, the acquisition of medical image annotation data often faces many challenges, such as the complexity of the annotation process, high professionalism, high cost, and patient privacy protection. These problems have seriously restricted the development and application of medical image segmentation technology.
[0003] In response to the problem of scarce annotated data for medical images, Few-Shot Learning (FSL) does not require a large amount of annotated data and can reduce the consumption of computing resources during training. Although FSL has potential application value in the field of medical image segmentation, existing technologies still face many challenges. First, the complexity and diversity of medical images make it difficult to directly apply FSL technology in practical scenarios. Medical images have a high diversity, including different anatomical structures, pathological changes, and imaging modes. FSL needs to be able to handle this diversity in order to learn generalized features from limited data. Summary of the invention
[0004] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and provide a small sample medical image segmentation method and system based on multi-scale skip connections and contrastive learning.
[0005] The technical solution adopted by the present invention to solve the technical problem is:
[0006] The present invention provides a small sample medical image segmentation method based on multi-scale skip connection and contrastive learning, comprising:
[0007] Acquire and process MRI image data of different organs, use the MRI image data of non-cardiac organs as training sets and validation sets, and use the cardiac MRI image data as a test set, wherein the training set, validation set, and test set include a support set and a query set, respectively;
[0008] The MRI image data of the non-cardiac organ is input into a dual-branch network small sample learning segmentation model, and a support set feature vector is extracted by using a conditional branch, and the support set feature vector is passed to the segmentation branch after being stimulated and compressed to segment the query set image;
[0009] The traditional skip connection is replaced by sequentially spliced multi-scale skip connections to fuse encoder feature maps of different scales, and then pass the feature maps to the corresponding decoder through skip connections;
[0010] The loss function calculation module based on multi-scale features is used to calculate the contrastive learning loss, and combine the contrastive learning loss function result with the cross entropy loss function to obtain the final hybrid loss function;
[0011] Use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results.
[0012] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, wherein: processing MRI image data of different organs includes:
[0013] The slice data that does not contain the segmentation target in the MRI image data of the different organs are filtered, and the MRI image data of the different organs are subjected to geometric transformation and brightness transformation.
[0014] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, wherein: a support set feature vector is extracted by a conditional branch, and the support set feature vector is passed to a segmentation branch to segment the query set image after being stimulated and compressed, including:
[0015] Both the conditional branch and the segmentation branch contain four encoders, a bottleneck block, and four decoders;
[0016] The support set feature vector is stimulated and compressed as:
[0017]
[0018] Among them, u con is the feature vector output by the conditional branch convolution block, u seg is the feature vector output by the segmentation branch convolution block, P(·) is a 1*1*1 convolution operation, σ(·) is the sigmoid function, is the output after excitation and compression.
[0019] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, wherein: the segmentation ability of the dual-branch network small sample learning segmentation model for small targets is improved by using a sequentially spliced multi-scale skip connection module, including:
[0020] The feature maps e1, e2, e3, and e4 of different scales output by the four encoders are sequentially spliced; the feature map e4 is reduced in dimension through a convolution layer, then passed through a ReLU activation layer, and finally spliced on the channel with e3 through upsampling as the jump connection input feature map d4 of the decoder D4. d4 contains the information of the feature maps of both scales e4 and e3; then, d4 is upsampled to the same scale as e2 through the convolution layer and the activation layer, and finally spliced on the channel with e2 as the jump connection input feature map d3 of the decoder D3. d3 contains the feature map information of the three scales e4, e3, and e2; and so on. After being processed by three network layers, d3 is spliced with e1 as the jump connection input feature map d2 of the decoder D2. d2 contains the feature information of four different scales e1, e2, e3, and e4.
[0021] Considering the network complexity and the fact that the feature map of the first shallow layer of the network contains rich detail information but is relatively simple in semantics and does not use sequential concatenation operations, the jump connection input feature map of the decoder D1 uses traditional jump connections.
[0022] It should also be noted that the network model adopts a dual-branch structure, so the multi-scale skip connection module is used in both the conditional branch and the segmentation branch, and is exactly the same.
[0023] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, a contrastive learning module based on multi-scale features is proposed, which also includes:
[0024] The present invention obtains three pairs of three-scale feature maps d4, d3, and d2 of the conditional branch and the segmentation branch.
[0025] The present invention inputs the d4 image features of the conditional branch and the segmentation branch into the contrastive learning model to obtain the contrastive learning loss.
[0026] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, wherein: calculating the contrastive learning loss function includes:
[0027] The positive sample contrast learning loss function is used.
[0028] The positive sample contrast loss function is expressed as:
[0029]
[0030] Among them, L MFCL is a positive sample contrast loss function based on sequential concatenation, N is the number of feature vectors, and d i It is the sequential concatenation feature map of the segmentation branch And the feature graph of the conditional branch The Euclidean distance between the i-th pair of eigenvectors can be specifically expressed as:
[0031]
[0032] Where d(a, b) represents the sequential concatenation feature map of the segmentation branch The vector a and the feature map of the conditional branch The Euclidean distance between vectors b, n is the vector dimension, a j , b j Represents the values of vectors a and b in the jth dimension.
[0033] As a preferred solution of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in the present invention, the contrastive learning loss function result and the cross entropy loss function are combined to obtain the final hybrid loss function, including:
[0034] The cross entropy loss function is expressed as:
[0035]
[0036] Among them, y i is the cardiac MRI label map, p(y i ) is the cardiac MRI prediction image, N is the number of labels, and L is the cross entropy loss function value;
[0037] Combining the contrastive learning function with the cross entropy loss function, the hybrid loss function is expressed as:
[0038] L M =L+λL MFCL
[0039] Among them, L M is the mixed loss function, L is the cross entropy loss function value, L MFCL is the contrast loss function value, λ is the weight coefficient, which is a constant.
[0040] The present invention provides a system for small sample medical image segmentation based on multi-scale skip connection and contrastive learning, comprising:
[0041] A data preprocessing module is used to acquire and process MRI image data of different organs, using the MRI image data of non-cardiac organs as a training set and a validation set, and using the cardiac MRI image data as a test set. The training set, validation set, and test set include a support set and a query set, respectively.
[0042] A dual-branch network model module, used for inputting the MRI image data of the non-cardiac organ into the dual-branch network small sample learning segmentation model, extracting the support set feature vector by using the conditional branch, and passing the support set feature vector to the segmentation branch after excitation and compression to segment the query set image;
[0043] A sequentially spliced multi-scale skip connection module fuses encoder feature maps of different scales and then passes the feature maps to the decoder through skip connections, so as to improve the segmentation ability of the dual-branch network small sample learning segmentation model for small objects;
[0044] The loss function calculation module based on multi-scale features is used to calculate the contrastive learning loss function, combine the contrastive learning loss function result with the cross entropy loss function, and obtain the final hybrid loss function;
[0045] The prediction module is used to use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results.
[0046] The present invention provides a computing device, comprising:
[0047] Memory and processor;
[0048] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning are implemented.
[0049] The present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the small sample medical image segmentation method based on multi-scale skip connection and contrast learning.
[0050] Beneficial effects:
[0051] 1. The multi-scale skip connection based on sequential splicing and the contrastive learning method proposed in the present invention first use multi-scale skip connection to help the network capture small-size features, ensure that important detail information is not lost during the segmentation process, and can better adapt to various changes in the image, such as noise, contrast difference and occlusion, thereby not only improving the accuracy of segmentation, but also improving the robustness of segmentation.
[0052] 2. The present invention utilizes the dual-branch structure of the model and introduces a contrastive learning module based on multi-scale features. It effectively utilizes the data information extracted by the conditional branch and takes it as the contrastive learning target of the segmentation branch. This strategy enables the model to extract more generalized feature representations from limited data and enhance the semantic correlation between samples, thereby further improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0054] in:
[0055] Figure 1 A schematic diagram of the overall process of a small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to an embodiment of the present invention;
[0056] Figure 2 A schematic diagram of a dual-branch network small sample learning network model of a small sample medical image segmentation method based on multi-scale skip connections and contrastive learning according to an embodiment of the present invention;
[0057] Figure 3 A schematic diagram of a multi-scale skip connection module for sequential splicing based on multi-scale skip connection and contrastive learning according to an embodiment of the present invention;
[0058] Figure 4 A schematic diagram of a contrastive learning module of a multi-scale skip connection based on multi-scale skip connection and contrastive learning according to an embodiment of the present invention;
[0059] Figure 5 This is a schematic diagram of a portion of cardiac MRI prediction results of a small sample medical image segmentation method based on multi-scale skip connection and contrastive learning described in an embodiment of the present invention, wherein (a) is the segmentation result of LV (left ventricle), (b) is the segmentation result of RV (right ventricle), and (c) is the segmentation result of LV-MYO (left ventricular myocardium), and the red line in each figure is the segmentation contour. DETAILED DESCRIPTION
[0060] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0062] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0063] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.
[0064] At the same time, in the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.
[0065] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0066] Embodiment 1
[0067] Reference Figure 5 , which is an embodiment of the present invention, provides a small sample medical image segmentation method based on multi-scale skip connection and contrastive learning, comprising:
[0068] Acquire and process MRI image data of different organs, use the MRI image data of non-cardiac organs as training sets and validation sets, and use the cardiac MRI image data as a test set, wherein the training set, validation set, and test set include a support set and a query set, respectively;
[0069] Further, the slice data that does not contain the segmentation target in the MRI image data of different organs are filtered, and the MRI image data of different organs are subjected to geometric transformation and brightness transformation;
[0070] It should be noted that two medical image datasets, CHAOST2 and CMR, are input, both of which are MRI but have different segmentation target locations. The CHAOST2 dataset is used as the training set and validation set, and the CMR dataset is used as the test set. For the test set, 5 data (5-shot) are sampled as the support set, and the rest are used as the query set.
[0071] Specifically, 80% of the MRI images in the CHAOST2 dataset are used as training sets and 20% as validation sets. After the training sets and validation sets are equally divided into support sets and query sets, a segmentation target (label) is randomly selected each time, and a query image and a support image are input in pairs each time. This pair of images first has a segmentation target, and secondly, they come from two slice samples with the same volume number in the data;
[0072] The CMR dataset is used as the test set. After selecting a segmentation target, 5 sample slices and their labels are taken as the support set data. These 5 samples come from the same volume. The selection method is to divide the volume (3D) containing the segmentation target into 5 small volumes on average, and then take the central slices of these 5 volumes as 5 support samples.
[0073] In contrast, the query set of the test set comes from other volumes. These volumes containing segmentation targets are also divided into 5 parts, each of which corresponds to 5 support samples. Each time a support sample is input, the query sample is a query sample in the corresponding part.
[0074] It should also be noted that the data input method during testing is similar to that during training. Each time a segmentation target is selected, a query image and a support image are input in pairs.
[0075] The MRI image data of non-cardiac organs is input into the dual-branch network small sample learning segmentation model, and the support set feature vector is extracted by the conditional branch. The support set feature vector is then passed to the segmentation branch to segment the query set image after excitation and compression; the dual-branch network small sample learning segmentation model is as follows Figure 3 As shown;
[0076] Furthermore, both the conditional branch and the segmentation branch contain four encoders, a bottleneck block, and four decoders;
[0077] The support set feature vector is stimulated and compressed as:
[0078]
[0079] Among them, u con is the feature vector output by the conditional branch convolution block, u seg is the feature vector output by the segmentation branch convolution block, P(·) is a 1*1*1 convolution operation, σ(·) is the sigmoid function, is the output after excitation and compression;
[0080] It should be noted that each decoder, bottleneck block and encoder are composed of convolution blocks. Each layer of convolution blocks is composed of two repeated 3*3 convolution blocks, followed by ReLU activation function, and skip connection is used between encoding block and decoding block; four times downsampling uses 2*2 maximum pooling operation with a step size of 2, and four times upsampling uses 4*4 deconvolution to restore image resolution. Finally, after 1*1 convolution operation, Sigmoid activation function is used to output the segmentation map of the network;
[0081] It should also be noted that this scheme achieves efficient segmentation of non-cardiac organ MRI image data by introducing a dual-branch network small sample learning segmentation model; the conditional branch in this architecture is specifically used to extract support set feature vectors, which are stimulated and compressed to effectively enhance key information and suppress irrelevant noise, thereby significantly improving the segmentation accuracy of the segmentation branch for the query set image; the dual-branch network design strategy ensures that the model can still maintain good generalization capabilities in an environment where labeled data is scarce. This design not only reduces the labeling cost, but also greatly improves the practicality and popularity of medical image segmentation technology, allowing the technology to be more widely used in clinical practice and provide more accurate support for medical diagnosis and treatment.
[0082] The multi-scale skip connection module of sequential splicing is used to improve the segmentation ability of the dual-branch network small sample learning segmentation model for small objects, including:
[0083] The feature maps e1, e2, e3, and e4 of different scales output from the four encoders are sequentially concatenated. It should be noted that the resolution of e1 is 128*128, the resolution of e2 is 64*64, the resolution of e3 is 32*32, and the resolution of e4 is 16*16;
[0084] The feature map e4 with a dimension of 512 is reduced to 256 channels through a 3D convolution, which is equivalent to the channel number of the feature map e3. It then passes through a ReLU activation layer and is finally upsampled to the same scale as the feature map e3. It is then concatenated with e3 on the channel as the jump connection input feature map d4 of the decoder D4. The number of channels of d4 is 512, and it contains the information of the feature maps of both scales e3 and e4.
[0085] Then, the d4 obtained in the first step is reduced to 128 dimensions through a 3D convolutional layer and an activation layer, upsampled to a resolution of 64*64 and concatenated with e2 on the channel as the jump connection input feature map d3 of the decoder D3. The number of channels of d3 is 256, and d3 contains feature map information of three scales: e4, e3, and e2.
[0086] Similarly, after being processed by three network layers, d3 is concatenated with e1 as the jump connection input feature map d2 of decoder D2. The number of channels of d2 is 128, and d2 contains information of features of four different scales at the same time.
[0087] The skip connection input feature map of the last layer decoder D1 uses traditional skip connection.
[0088] It should also be noted that the network model adopts a dual-branch structure, so the multi-scale jump connection module is used in both the conditional branch and the segmentation branch, and is completely consistent.
[0089] In order to further explore the potential of the dual-branch structure of the model, a contrastive learning module based on multi-scale features is introduced. The three pairs of three-scale feature maps d4, d3, and d2 of the conditional branch and the segmentation branch are obtained by the sequential concatenation module, and d4 is selected as the input of the contrastive learning model.
[0090] The image features of the conditional branch and the segmentation branch are input into the contrastive learning model to obtain the contrastive learning loss.
[0091] The positive sample contrast learning loss function is used.
[0092] The positive sample contrast loss function is expressed as:
[0093]
[0094] Among them, L MFCL is a positive sample contrast loss function based on sequential concatenation, N is the number of feature vectors, and d i It is the sequential concatenation feature map of the segmentation branch And the feature graph of the conditional branch The Euclidean distance between the i-th pair of eigenvectors can be specifically expressed as:
[0095]
[0096] Where d(a, b) represents the sequential concatenation feature map of the segmentation branch The vector a and the feature map of the conditional branch The Euclidean distance between vectors b, n is the vector dimension, a j , b j Represents the values of vectors a and b in the jth dimension.
[0097] Combine the contrastive learning loss function result with the cross entropy loss function to obtain the final mixed loss function. The cross entropy loss function is expressed as:
[0098]
[0099] Among them, y i is the cardiac MRI label map, p(y i ) is the cardiac MRI prediction image, N is the number of labels, and L is the cross entropy loss function value;
[0100] Combining the contrastive learning function with the cross entropy loss function, the hybrid loss function is expressed as:
[0101] L M =L+λL MFCL
[0102] Among them, L M is the mixed loss function, L is the cross entropy loss function value, L MFCL is the contrast loss function value, λ is the weight coefficient, and its value is 0.8.
[0103] It should be noted that by calculating the contrastive learning loss function and combining it with the cross-entropy loss function, the final hybrid loss function is obtained, thereby more effectively guiding the training process of the model. It not only makes full use of the semantic information between pixels, but also combines the label data, so that the model can simultaneously consider the similarity between pixels and the accuracy of the labels during the training process, thereby improving the segmentation accuracy of the model; the contrastive learning loss function based on multi-scale jump connection introduced in this technical solution effectively utilizes the data information extracted by the conditional branch and uses it as the contrastive learning target of the segmentation branch, which significantly improves the performance of the dual-branch network small sample learning segmentation model, and provides strong support for the development and application of medical image segmentation technology.
[0104] Use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results;
[0105] Furthermore, the preprocessed CMR dataset is input into the trained dual-branch network small sample learning segmentation model to obtain the cardiac MRI image segmentation map.
[0106] The above is a schematic scheme of a small sample medical image segmentation method based on multi-scale jump connection and contrastive learning in this embodiment. It should be noted that the technical scheme of the system for small sample medical image segmentation based on multi-scale jump connection and contrastive learning belongs to the same concept as the technical scheme of the small sample medical image segmentation method based on multi-scale jump connection and contrastive learning. For details not described in detail in the technical scheme of the small sample medical image segmentation system based on multi-scale jump connection and contrastive learning in this embodiment, please refer to the description of the technical scheme of the small sample medical image segmentation method based on multi-scale jump connection and contrastive learning.
[0107] The small sample medical image segmentation system based on multi-scale skip connection and contrastive learning in this embodiment includes:
[0108] A data preprocessing module is used to acquire and process MRI image data of different organs, using the MRI image data of non-cardiac organs as a training set and a validation set, and using the cardiac MRI image data as a test set. The training set, validation set, and test set include a support set and a query set, respectively.
[0109] A dual-branch network model module, used for inputting the MRI image data of the non-cardiac organ into the dual-branch network small sample learning segmentation model, extracting the support set feature vector by using the conditional branch, and passing the support set feature vector to the segmentation branch after excitation and compression to segment the query set image;
[0110] A sequentially spliced multi-scale skip connection module fuses encoder feature maps of different scales and then passes the feature maps to the decoder through skip connections, so as to improve the segmentation ability of the dual-branch network small sample learning segmentation model for small objects;
[0111] The loss function calculation module based on multi-scale features is used to calculate the contrastive learning loss function, combine the contrastive learning loss function result with the cross entropy loss function, and obtain the final hybrid loss function;
[0112] The prediction module is used to use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results.
[0113] This embodiment further provides a computing device, which is applicable to the case of small sample medical image segmentation based on multi-scale skip connection and contrastive learning, including:
[0114] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning as proposed in the above embodiment.
[0115] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning proposed in the above embodiment is implemented.
[0116] The storage medium proposed in this embodiment and the small sample medical image segmentation method based on multi-scale jump connection and contrastive learning proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0117] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform the methods of various embodiments of the present invention.
[0118] Embodiment 2
[0119] Reference Figure 5 , as an embodiment of the present invention, provides a small sample medical image segmentation method based on multi-scale skip connections and contrastive learning, and provides experimental test results to verify its beneficial effects.
[0120] First, we input two preprocessed medical image datasets: abdominal MRI data CHAOST2 and cardiac MRI data CMR. The resolution of both datasets is 256*256. CHAOST2 has 20 volumes with a total of about 3000 slices, and CMR has 35 volumes with a total of about 380 slices.
[0121] There are four parts to be segmented in CHAOST2: liver, right kidney, left kidney, and spleen; the parts to be segmented in CMR include two parts: left ventricle (LV), right ventricle (RV), and left ventricular myocardium (LV-MYO);
[0122] CHAOST2 is divided into a training set and a validation set in a ratio of 4:1. The training set and the validation set are further divided into a support set and a query set. First, a segmentation target (label) is randomly selected. During training, a query image and a support image are input in pairs each time. This pair of images first has a segmentation target, which comes from two slice samples with connected numbers in the same volume (3D voxel of medical images) in the data.
[0123] Secondly, the CMR dataset is used as the test set. After selecting a segmentation target, 5 sample slices and their labels are taken as the support set data. These 5 samples come from the same volume. The slice selection method is to divide the volume containing the segmentation target into 5 small volumes on average and select the central slices of these 5 volumes as 5 support samples. The corresponding query set comes from other volumes. These volumes are also divided into 5 parts. The 5 support samples correspond to each part one by one. Each time a support sample is input, the query sample is a query sample of the corresponding part.
[0124] Then, the abdominal MRI data CHAOST2 is input into a dual-branch network small sample learning segmentation model with multi-scale sequential splicing jump connections for training and verification, positive sample contrast learning is implemented based on multi-scale connections, and the hybrid loss function is calculated and optimized;
[0125] Finally, after obtaining the trained dual-branch network small sample learning segmentation model, the test set data (cardiac MRI) is input into the saved model for prediction. The results are as follows: Figure 5 As shown, it is a segmentation result of a part of the cardiac MRI image;
[0126] Through experimental statistics, it is found that according to the method of the present invention, after adding multi-scale jump connections, the Dice coefficient (Dice similarity coefficient) and IoU (Intersection over Union) of LV reach 66.14% / 77.77%, the Dice / IoU of RV reaches 60.74% / 74.85%, and the Dice / IoU of LV-MYO reaches 15.20% / 25.44%;
[0127] After adding the contrastive learning module on this basis, the Dice / IoU of LV reaches 66.44% / 78.30%, the Dice / IoU of RV reaches 62.75% / 76.43%, and the Dice / IoU of LV-MYO reaches 17.28% / 28.37%;
[0128] Therefore, the present invention effectively improves the accuracy and performance of medical image segmentation by introducing a dual-branch network small sample learning segmentation model and a comparative learning method; whether in terms of the Dice coefficient or the IoU indicator, the present invention has achieved satisfactory results, fully demonstrating its effectiveness in practical applications. These results not only provide strong support for the development of medical image segmentation technology, but also provide a more accurate and reliable auxiliary means for clinical diagnosis and treatment.
[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A small sample medical image segmentation method based on multi-scale skip connections and contrastive learning, characterized in that: include: Collect and preprocess MRI image data of different organs. MRI image data of non-cardiac organs are used as training sets and validation sets, while cardiac MRI image data are used as test sets. Each dataset consists of a support set and a query set to meet the needs of the small sample learning framework. The MRI image data of the non-cardiac organ is input into a dual-branch network small sample learning segmentation model, in which, in the conditional branch, the model focuses on extracting feature vectors of the support set, and the support set feature vectors are passed to the segmentation branch after excitation and compression to segment the query set image; The Sequential Concatenation of Multiscale SkipConnections (SC-MSC) method replaces the traditional skip connection and uses a sequential concatenation method to effectively integrate features of different scales from the encoder and pass them to the corresponding decoder. On the basis of adopting multi-scale skip connection, the dual-branch structure of the model is used to introduce the Multiscale Feature Contrastive Learning module (MFCL), which combines the contrastive learning loss function with the cross entropy loss function to obtain the final hybrid loss function. Use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results.
2. The small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 1, characterized in that: Processing MRI image data of different organs, including, The slice data that does not contain the segmentation target in the MRI image data of the different organs are filtered, and the MRI image data of the different organs are subjected to data enhancement by geometric transformation and brightness transformation.
3. The small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 1, characterized in that: The conditional branch is used to extract the support set feature vector, which is then passed to the segmentation branch to segment the query set image after being stimulated and compressed, including: Both the conditional branch and the segmentation branch contain four encoders, a bottleneck block, and four decoders; The support set feature vector is stimulated and compressed as: Among them, u con is the feature vector output by the conditional branch convolution block, u seg is the feature vector output by the segmentation branch convolution block, P(·) is a 1*1*1 convolution operation, σ(·) is the sigmoid function, is the output after excitation and compression.
4. The small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 1, characterized in that: The multi-scale skip connection module of sequential splicing is used to improve the segmentation ability of the dual-branch network small sample learning segmentation model for small objects, including: The feature maps e1, e2, e3, and e4 of different scales output from the four encoders are sequentially spliced; the feature map e4 is reduced in number of channels through a convolution layer to make the number of channels equal to that of the feature map e3, and then passes through a ReLU activation layer, and is finally upsampled to the same scale as the feature map e3 and spliced with e3 on the channel as the jump connection input feature map d4 of the decoder D4, d4 contains the information of the two scale feature maps of e3 and e4; d4 is then upsampled to the same scale as e2 through the convolution layer and the activation layer, and finally spliced with e2 on the channel as the jump connection input feature map d3 of the decoder D3, d3 contains the three scale feature map information of e2, e3, and e4; and so on, d3 is spliced with e1 after being processed by three network layers as the jump connection input feature map d2 of the decoder D2, d2 contains the information of the four scale feature maps of e1, e2, e3, and e4; The jump connection input feature map of decoder D1 uses traditional jump connection; The network model adopts a dual-branch structure, so the multi-scale skip connection module is used in both the conditional branch and the segmentation branch, and is exactly the same.
5. The small sample size medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 4, characterized in that: Contrastive learning module based on multi-scale features, including: In order to further explore the potential of the dual-branch structure of the small sample model, a contrastive learning module based on multi-scale features was introduced. Three pairs of three-scale feature maps d4, d3, and d2 of the conditional branch and the segmentation branch were obtained, and d4 was selected as the input of the contrastive learning model. The d4 image features of the conditional branch and the segmentation branch are input into the contrastive learning model to obtain the contrastive learning loss.
6. The small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 1, characterized in that: Compute the contrastive learning loss function, including: A positive sample contrast learning loss function is used, which is expressed as: Among them, L MFCL is a contrast loss function based on sequential concatenation, N is the number of feature vectors, and d i is the Euclidean distance between the i-th pair of eigenvectors: Where d(a, b) represents the sequential concatenation feature map of the segmentation branch The vector a and the feature map of the conditional branch The Euclidean distance between vectors b, n is the vector dimension, a j , b j Represents the values of vectors a and b in the jth dimension.
7. The small sample medical image segmentation method based on multi-scale skip connection and contrastive learning according to claim 1, characterized in that: Combine the contrastive learning loss function result with the cross entropy loss function to obtain the final mixed loss function, including, The cross entropy loss function is expressed as: Among them, y i is the cardiac MRI label map, p(y i ) is the cardiac MRI prediction image, N is the number of labels, and L is the cross entropy loss function value; Combining the contrastive learning function with the cross entropy loss function, the hybrid loss function is expressed as: THE M =L+λL MFCL Among them, L M is the mixed loss function, L is the cross entropy loss function value, L MFCL is the contrast loss function value, λ is the weight coefficient, which is a constant.
8. A system for small sample size medical image segmentation based on multi-scale skip connections and contrastive learning, characterized in that: include: A data preprocessing module is used to acquire and process MRI image data of different organs, using the MRI image data of non-cardiac organs as a training set and a validation set, and using the cardiac MRI image data as a test set. The training set, validation set, and test set include a support set and a query set, respectively. A dual-branch network model module is used to input the MRI image data of the non-cardiac organ into the dual-branch network small sample learning segmentation model, and extract the support set feature vector by using the conditional branch. The support set feature vector is then passed to the segmentation branch to segment the query set image after being stimulated and compressed; The sequentially spliced multi-scale skip connection module replaces the traditional skip connection, fuses the encoder feature maps of different scales, and then passes the feature maps to the decoder through the skip connection, which is applied to improve the segmentation ability of the dual-branch network small sample learning segmentation model for small objects; The loss function calculation module based on multi-scale features is used to calculate the contrastive learning loss function, and the contrastive learning loss function result is combined with the cross entropy loss function to obtain the final hybrid loss function; The prediction module is used to use the trained dual-branch network small sample learning segmentation model to predict the test set data and obtain the segmentation prediction results.
9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the small sample medical image segmentation method based on multi-scale jump connection and contrastive learning as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the small sample medical image segmentation method based on multi-scale skip connection and contrastive learning as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Cross-modal double-branch complementary fusion image segmentation method and device
CN115482241A
Human body behavior recognition method and system based on multi-channel directed graph convolution
CN116895097A
Crack segmentation model training method, crack image segmentation method and related assembly
CN118014050A
Small sample medical image segmentation method and system based on pixel-level contrast learning
CN118212249A
Medical image segmentation method based on u-shaped network
WO2022199143A1
