A blood vessel segmentation method, device and equipment based on large model fine tuning and medium

Abstract

The application discloses a blood vessel segmentation method and device based on large model fine tuning, equipment and medium, wherein the method comprises: acquiring a blood vessel image, preprocessing the obtained blood vessel image, and obtaining a training set; fine tuning the pre-trained image segmentation model using the training set, so that the image segmentation model learns the related features of blood vessel segmentation; wherein the low-rank adaptive fine tuning technology is used to fine tune the image segmentation model: two low-rank matrices are inserted into the Transformer layer in the image encoder in parallel, the input is processed through the two low-rank matrices in turn, and the processing result is added to the original output to obtain the final output. The application solves the deficiency of large models in the blood vessel segmentation task through efficient parameter fine tuning and multi-scale feature fusion module. The application can realize accurate blood vessel segmentation under limited labeled data, greatly reduces the difficulty and cost of medical image segmentation task, and can be widely applied to the technical field of medical image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a blood vessel segmentation method and device based on large model fine-tuning, equipment and medium. BACKGROUND

[0002] Blood vessel segmentation is an important research direction in the field of medical image processing, mainly used for extracting tubular blood vessel structures from medical images. This technology is of great significance in many medical scenarios, such as segmenting fundus blood vessels from fundus color photographs, fluorescent fundus angiography, or fundus optical coherence tomography blood flow imaging, for the diagnosis of eye diseases (such as diabetic retinopathy, hypertensive retinopathy, etc.); segmenting coronary blood vessels from coronary X-ray angiography images, computed tomography images, for preoperative diagnosis, surgical planning, and postoperative evaluation of heart diseases (such as coronary artery stenosis, coronary artery calcification, etc.).

[0003] Current commonly used blood vessel segmentation methods are mainly based on convolutional neural network architectures, such as UNet, ResUNet, nnUNet, etc. These neural network architectures can achieve blood vessel segmentation tasks to some extent, but due to the relatively small number of model parameters, they cannot fully capture the complex features and context information in the image, so their performance in processing multi-modal medical image segmentation tasks is not satisfactory. A mainstream view is that multi-modal tasks should be completed by large models. The introduction of the SegmentAnything Model (SAM) fills the gap in the field of large model segmentation. Although the original SAM has not been specifically trained for medical images, researchers have successively proposed SAM variants for the medical field, such as SAMMed2d, MedSAMAdapter. However, these large models have poor segmentation performance at object edges, resulting in their poor performance in blood vessel segmentation tasks. Moreover, the training process of fine-tuning large models in the above methods requires a large amount of segmentation annotation data, while blood vessel segmentation tasks are extremely scarce in terms of manually annotated data. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a blood vessel segmentation method, device, equipment and medium based on large model fine-tuning.

[0005] The first technical solution adopted by the present application is:

[0006] A blood vessel segmentation method based on large model fine-tuning, comprising the following steps:

[0007] Obtaining a blood vessel image, preprocessing the obtained blood vessel image to obtain a training set;

[0008] The pre-trained image segmentation model is fine-tuned by using the training set, so that the image segmentation model learns the related features of blood vessel segmentation; the image segmentation model comprises an image encoder, a multi-scale feature fusion module and a high-quality decoder;

[0009] The image segmentation model is fine-tuned by using a low-rank adaptive fine-tuning technique: two low-rank matrices are inserted into the Transformer layer in the image encoder in parallel, the input is processed by the two low-rank matrices in turn, and the processing results are added to the original output to obtain the final output.

[0010] Further, the blood vessel image is obtained, and the obtained blood vessel image is preprocessed to obtain a training set, comprising:

[0011] The fundus color photograph FIVES dataset is used as training data, and the fundus color photograph FIVES dataset is composed of 800 images with a resolution of 2048x2048;

[0012] The images in the fundus color photograph FIVES dataset are non-overlappingly cropped into 64 images with a resolution of 256x256, and the images with a blood vessel area smaller than a preset value in the mask are deleted.

[0013] Further, the image segmentation model works as follows:

[0014] The image encoder is used for feature extraction of the input image, and the extracted image features are input into the high-quality decoder; the image encoder comprises a plurality of Transformer layers;

[0015] The multi-scale feature fusion module is used to input the output F1 of the first layer of the Transformer layer of the image encoder and the output F n As v and k in attention, respectively, the image embedding F Emb The two calculation results are added after being down-sampled to obtain the fusion features, and the fusion features are input into the high-quality decoder;

[0016] The high-quality decoder is used to generate an output mask according to the image features and the fusion features.

[0017] Further, the image encoder comprises an image embedding layer and 12 Transformer layers, and each Transformer layer is fine-tuned by using the low-rank adaptive fine-tuning technique; finally, the image encoder encodes the image into a set of image features.

[0018] Further, the expressions of the two attention calculations in the multi-scale feature fusion module are as follows:

[0019]

[0020]

[0021] In the formula, d k is F Emb The length of the dimension.

[0022] Further, the specific working mode of the high-quality decoder is as follows:

[0023] First, two tokens are generated by the image coding through the self-attention mechanism, one of which is matrix multiplied with the fusion feature and then deconvolved to obtain a detail mask, and the other is matrix multiplied with the image feature and then deconvolved to obtain a coarse mask. The detail mask and the coarse mask are added to obtain the final output mask.

[0024] Finally, the loss is calculated according to the obtained output mask and the preset standard mask, the gradient is calculated by back propagation, and the weight of the image segmentation model is updated.

[0025] Further, the calculation method of the loss function for training the image segmentation model is as follows:

[0026] Loss=Dice Loss+clDice Loss

[0027] Dice Loss=1-Dice(A,B)

[0028]

[0029] clDice Loss=1-clDice(V P ,V L )

[0030]

[0031] In the formula, A is the predicted segmentation region, B is the real segmentation region; V L is the true value mask, V P is the predicted mask, S L is the skeleton extracted from V L , S P is the skeleton extracted from V P ; T prec (S P , V L ) is the score of the skeleton S P , and T sens (S L , V P ) is the score of the skeleton S L .

[0032] Further, the inference process of the image segmentation model is as follows:

[0033] Obtain a medical image, input the medical image into the fine-tuned image segmentation model;

[0034] First, the zero pixel dimension of the input medical image is padded to the nearest multiple of 256; then, a sliding window prediction is performed with a window size of 256x256 and a step size of 128; in each prediction, only the center 128x128 region is retained for splicing, thereby generating a final prediction mask.

[0035] The second technical solution adopted by the present application is:

[0036] A blood vessel segmentation device based on large model fine-tuning, comprising

[0037] A data acquisition module is configured to acquire a blood vessel image, pre-process the obtained blood vessel image, and obtain a training set;

[0038] A model fine-tuning module is configured to fine-tune a pre-trained image segmentation model using the training set, so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model comprises an image encoder, a multi-scale feature fusion module, and a high-quality decoder;

[0039] The image segmentation model is fine-tuned using a low-rank adaptive fine-tuning technique: a parallel two low-rank matrix is inserted into the Transformer layer in the image encoder, the input is processed through the two low-rank matrices in turn, and the processing results are added to the original output to obtain the final output.

[0040] The third technical solution adopted by the present application is:

[0041] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement a blood vessel segmentation method based on large model fine-tuning as described above.

[0042] The fourth technical solution adopted by the present application is:

[0043] A computer-readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement a blood vessel segmentation method based on large model fine-tuning as described above.

[0044] The fifth technical solution adopted by the present application is:

[0045] A computer program product or computer program comprises computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned blood vessel segmentation method based on large model fine-tuning.

[0046] The beneficial effects of the present application are: the present application solves the deficiency of large models in the blood vessel segmentation task through efficient parameter fine-tuning and multi-scale feature fusion module. The present application not only can effectively adapt to the input of new modal images, but also can realize accurate blood vessel segmentation under limited labeled data, greatly reducing the difficulty and cost of medical image segmentation task. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise.

[0048] Figure 1 is a structural schematic diagram of an image segmentation model in the embodiments of the present application;

[0049] Figure 2 is a schematic diagram of a multi-scale feature fusion module MSFF in the embodiments of the present application;

[0050] Figure 3 is a training and working flowchart of an image segmentation model in the embodiments of the present application;

[0051] Figure 4 is a step flowchart of a blood vessel segmentation method based on large model fine-tuning in the embodiments of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right, etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0054] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0055] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0056] Term explanation:

[0057] SAM: The abbreviation of Segment Anything Model, SAM is an advanced image segmentation model. SAM aims to achieve efficient and accurate image segmentation through simple input such as points, boxes or masks, and is suitable for various computer vision tasks. With its high efficiency and high-quality segmentation capability, it is suitable for a variety of computer vision tasks, and has promoted the development of image segmentation technology.

[0058] LoRA: The abbreviation of Low-Rank Adaptation, low-rank adaptation fine-tuning. LoRA is an efficient parameter fine-tuning technology, which aims to achieve efficient adaptation and fine-tuning by performing low-rank decomposition on the weights of pre-trained models. LoRA can significantly reduce the number of parameters and computing resources required for fine-tuning while preserving the original performance of the model.

[0059] HQ Decoder: The abbreviation of High Quality Decoder, high-quality decoder. HQ Decoder can capture more rich context information and details by processing the feature maps from the encoder. This enables the model to more accurately segment the boundaries of target objects when dealing with complex scenes, which is particularly important in blood vessel segmentation.

[0060] MSFF: the abbreviation of Multi-Scale Feature Fusion, multi-scale feature fusion. MSFF uses a cross-attention mechanism to fuse shallow and deep features, capturing vascular information at multiple scales, and then feeds it into the HQ Decoder module. This module can improve the model's overall performance in segmenting coarse and fine blood vessels.

[0061] Sliding Window Prediction Module: In many image segmentation and detection tasks, directly scaling the image to a smaller size can result in loss of detail. Especially when dealing with complex scenes or objects with complex structures such as blood vessels. The Sliding Window Prediction Module ensures that details within each window are preserved by processing the image locally, thereby improving the accuracy of segmentation or detection.

[0062] To address the shortcomings of the prior art, the present application proposes a fine-tuning method for applying large models to blood vessel segmentation, aiming to address the shortcomings of large models in blood vessel segmentation tasks through Parameter-Efficient Fine-Tuning and Multi-Scale Feature Fusion Module. Compared with existing methods, the method of the present application not only effectively adapts to the input of new modal images, but also realizes accurate blood vessel segmentation under limited labeled data, greatly reducing the difficulty and cost of medical image segmentation tasks.

[0063] Embodiment 1

[0064] As shown in Figure 3 and 4 , the present embodiment provides a blood vessel segmentation method based on fine-tuning of large models, comprising the following steps:

[0065] S1, obtaining a blood vessel image, pre-processing the obtained blood vessel image to obtain a training set.

[0066] Before training, we crop the image into several small images with a resolution of 256x256 to adapt to the image input size of SAM, and delete images with blood vessel areas smaller than the preset value in the mask.

[0067] As an optional implementation, the fundus color photograph FIVES dataset is used as training data, which consists of 800 images with a resolution of 2048x2048. The images are cropped into 64 images with a resolution of 256x256 without overlap, and images with blood vessel areas smaller than 1% in the mask are deleted.

[0068] S2, fine-tuning the pre-trained image segmentation model using the training set to enable the image segmentation model to learn the relevant features of blood vessel segmentation; the image segmentation model comprises an image encoder, a multi-scale feature fusion module and a high-quality decoder.

[0069] The embodiment selects a SAM benchmark large model for subsequent fine-tuning. The SAM has strong image segmentation capability and is particularly suitable for processing multi-modal images. In the embodiment, the pre-training weight of the SAM is trained on natural images. In order to enable the model to learn the relevant features of blood vessel segmentation, the embodiment uses the LoRA fine-tuning method to fine-tune the pre-trained model, and the specific steps are as follows: two low-rank matrices are inserted in parallel in the Transformer layer of the image encoder, and the input is sequentially passed through the two low-rank matrices, and the result is added to the original output to obtain the final output. Through the LoRA fine-tuning method, the training of the model is realized under the condition of updating only a small number of parameters, while the performance of the model is maintained.

[0070] As an optional implementation, we freeze all parameters in the original image encoder and deploy LoRA matrices for the attention matrix of the Transformer layer in the SAM. During the training process, only the weight of the LoRA matrix in the SAM encoder will be changed. We use the AdamW optimizer with an initial learning rate of 1e-4, and a total of 30 iteration cycles, the learning rate is divided by 2 at the 5th and 10th iteration cycles, and the batchsize is 32.

[0071] The image segmentation model is described in detail below in conjunction with the accompanying drawings and exemplary embodiments.

[0072] Participation Figure 1 , Figure 1 The image segmentation model includes an image encoder, a high-quality decoder and an MSFF module, as shown in the overall structure and training schematic diagram of the image segmentation model. The image encoder includes an image embedding layer and 12 Transformer layers, and each Transformer layer is fine-tuned by LoRA fine-tuning technology. Finally, the image encoder encodes the image into a set of image features. The image features extracted by the image encoder are input into the high-quality decoder, as shown in Figure 1 .

[0073] The uneven distribution of thick and thin blood vessels may cause the model to over-learn the thick / thin blood vessel features and ignore the thin / thick blood vessel features. Based on this, the embodiment uses the MSFF module to fuse the shallow and deep features. Referring to Figure 2 , the specific steps are as follows: the output F1 of the first layer Transformer layer and the output F nAs attention in v and k, respectively, and as q the original graph Embedding(F Emb ) is calculated. That is:

[0074]

[0075]

[0076] where d k is the dimension length of F Emb . The first layer of the Transformer layer output and the image embedding layer output are calculated by cross attention, extracting the detailed features of the first layer output. Similarly, the cross attention calculation is performed between the last layer of the Transformer layer output and the image embedding layer output, extracting the detailed features of the last layer output. Both are down-sampled by convolution, and finally the down-sampled results are added to obtain the final fusion feature, which is output to the high-quality decoder. The fusion feature is input into the high-quality decoder, as shown in Figure 1 .

[0077] The high-quality decoder will generate an output mask from the fusion feature obtained by the MSFF module and the image feature output by the image encoder. The decoder will first generate two tokens from the image encoding by self-attention mechanism, one of which will be multiplied by the fusion feature to obtain a detailed mask after deconvolution, and the other will be multiplied by the image feature to obtain a coarse mask after deconvolution. Finally, the two masks are added to obtain the final output mask.

[0078] Finally, the output mask of the high-quality decoder will be calculated with the standard mask by the loss function, and the gradient will be calculated by back propagation and the model weight will be updated. In this embodiment, the standard mask refers to the correct blood vessel mask labeled by the doctor in the blood vessel segmentation data set.

[0079] In some embodiments, we use the following loss function suitable for supervised blood vessel segmentation task to make the model achieve our training purpose:

[0080] 1) Dice Loss

[0081] DiceLoss is a loss function commonly used in image segmentation tasks, especially in the field of medical image analysis. It is derived from the Dice coefficient, which is an index for measuring the similarity of two samples, especially for unbalanced class problems.

[0082] The calculation formula of the Dice coefficient is:

[0083]

[0084] Where (A) is the predicted segmentation region, (B) is the real segmentation region. The value of the coefficient is between 0 and 1, 1 represents complete overlap, 0 represents no overlap.

[0085] DiceLoss is defined based on the Dice coefficient, usually represented as:

[0086] Dice Loss = 1-Dice(A, B)

[0087] 2) clDice Loss

[0088] In the task of similar blood vessel topology extraction, the correct pixel prediction sequence in space is more meaningful than the correct prediction of spatial region.

[0089] The formula of clDice operator is as follows:

[0090]

[0091] Where, V L is the true mask, V P is the predicted mask, S L is the skeleton extracted from V L , S P is the skeleton extracted from V P . T prec (S P , V L ) is the fraction of skeleton S P , that is, the topology precision. T sens (S L , V P ) is the fraction of skeleton S L , that is, the topology sensitivity. The corresponding operator formula is as follows:

[0092]

[0093]

[0094] clDice Loss is defined based on the clDice coefficient, usually represented as:

[0095] clDice Loss = 1-clDice(V P , V L )

[0096] 3) Total loss function

[0097] The final loss function is obtained by adding Dice Loss and clDice Loss, that is:

[0098] Loss = Dice Loss + clDice Loss

[0099] S3, acquire a medical image to be segmented, input the medical image into the fine-tuned image segmentation model, and output a predicted mask.

[0100] participate Figure 3 After the fine-tuning of the image segmentation model is completed, the image segmentation model can be applied to the segmentation of medical images. Due to the block training method adopted in the embodiment, an inflation prediction method is adopted during prediction. Specifically, during the inference stage, the zero pixel dimension of the image is first padded to the nearest multiple of 256. Then, a sliding window prediction is performed with a window size of 256x256 and a step size of 128. In each prediction, only the central 128x128 region is retained for splicing, thereby generating a final predicted mask.

[0101] In summary, compared with the prior art, the present application has at least the following advantages and beneficial effects:

[0102] (1) effectively improve the cross-modal segmentation performance: the large model itself has strong generalization ability, compared with the traditional convolutional segmentation network, the large model can share and utilize the features between different modalities. This feature sharing helps the model to extract more representative features when processing cross-modal data, thereby improving the segmentation effect.

[0103] (2) accurate generation of coarse and fine blood vessel boundaries: the present application introduces a cross-attention method to guide the fusion of feature layers within the image itself. By establishing an attention link between these layers, the utilization rate of local and global information in the image is improved, thereby improving the accuracy and robustness of blood vessel segmentation.

[0104] (3) efficiency of transfer learning: by utilizing the rich knowledge of large-scale pre-trained models, excellent performance can be achieved on limited labeled data. With the help of the features obtained on large-scale datasets in the past, the model significantly reduces the training time and computational resource consumption when fine-tuning for specific tasks, while maintaining high-precision prediction results. This efficiency not only speeds up the training process of the model, but also enables reliable performance in data-scarce situations, providing a practical solution for clinical applications.

[0105] (4) the present application first applies a segmentation large model to the field of blood vessel segmentation. Blood vessel segmentation can help doctors more accurately identify and locate blood vessel structures, thereby assisting in the diagnosis of various cardiovascular diseases, eye blood vessel diseases, brain blood vessel diseases, and tumors and other lesions.

[0106] (5) The application proposes a multi-scale feature fusion module MSFF, which can effectively capture the correlation between different features of thick and thin blood vessels, and realize more accurate feature fusion by focusing on important features and suppressing irrelevant information. This attention mechanism enables the HQ Decoder to have better decoding performance.

[0107] (6) The application can effectively capture complex blood vessel structures and context information with only limited labeled samples, and still achieve high-precision blood vessel segmentation. Through fine-tuning, the model can quickly adapt to specific tasks, fully utilize existing knowledge, and significantly improve performance in data-scarce scenarios, thereby providing a reliable solution for clinical applications.

[0108] (7) The application can adapt to different modal feature distributions, capture more comprehensive and detailed blood vessel structure information, and thus improve segmentation accuracy and robustness. It enhances the recognition ability of complex lesions and provides more accurate and reliable blood vessel segmentation results for clinical applications.

[0109] Embodiment 2

[0110] The embodiment provides a blood vessel segmentation device based on large model fine-tuning, comprising:

[0111] A data acquisition module is configured to acquire a blood vessel image, pre-process the obtained blood vessel image, and obtain a training set.

[0112] A model fine-tuning module is configured to fine-tune a pre-trained image segmentation model using the training set, so that the image segmentation model learns relevant features of blood vessel segmentation. The image segmentation model comprises an image encoder, a multi-scale feature fusion module, and a high-quality decoder.

[0113] The image segmentation model is fine-tuned using a low-rank adaptive fine-tuning technique: a parallel two low-rank matrix is inserted into the Transformer layer in the image encoder, the input is processed through the two low-rank matrices in turn, and the processing results are added to the original output to obtain the final output.

[0114] Since the device is a blood vessel segmentation device based on large model fine-tuning according to the embodiment of the application, and the principle of solving the problem is similar to the method, the implementation of the device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0115] Embodiment 3

[0116] The embodiment of the application further provides an electronic device, which comprises a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method asFigure 4 A blood vessel segmentation method based on large model fine-tuning is shown.

[0117] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; and the data storage area can store data created according to the use of the server, etc.

[0118] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0119] Since the electronic device is an electronic device corresponding to the blood vessel segmentation method based on large model fine-tuning of the embodiment of the application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described herein.

[0120] Embodiment 4

[0121] The embodiment of the application further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method asFigure 4 A blood vessel segmentation method based on large model fine-tuning is shown.

[0122] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0123] Since the storage medium is a storage medium corresponding to the blood vessel segmentation method based on large model fine-tuning of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0124] Embodiment 5

[0125] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the blood vessel segmentation method based on large model fine-tuning according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Among them, the executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (for example, Transact-SQL), Perl, or in various other programming languages.

[0126] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0127] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0128] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A blood vessel segmentation method based on large model fine-tuning, characterized in that, The method comprises the following steps: obtaining a blood vessel image, pre-processing the obtained blood vessel image, and obtaining a training set; using the training set to fine-tune a pre-trained image segmentation model, so that the image segmentation model learns related features of blood vessel segmentation; the image segmentation model comprises an image encoder, a multi-scale feature fusion module, and a high-quality decoder; wherein the image segmentation model is fine-tuned using a low-rank adaptive fine-tuning technique: two low-rank matrices are inserted in parallel in the Transformer layer of the image encoder, the input is processed through the two low-rank matrices in turn, and the processing results are added to the original output to obtain the final output; the working mode of the image segmentation model is as follows: the image encoder is used for feature extraction of the input image, and the extracted image features are input into the high-quality decoder; the image encoder comprises a plurality of Transformer layers; The multi-scale feature fusion module is configured to fuse outputs of the first layer of the Transformer layer of the image encoder and the output of the last layer of the Transformer layer as v and k in attention, respectively, and the image embedding of the original image as q perform attention calculation, add the two calculation results after downsampling, obtain the fusion feature, and input the fusion feature into the high-quality decoder; the high-quality decoder is used for generating an output mask according to the image features and the fused features.

2. The blood vessel segmentation method based on large model fine tuning according to claim 1, characterized in that, The method comprises the following steps: The fundus color photograph FIVES dataset, which consists of a plurality of images with a resolution of 2048 2048 is used as training data. The images in the fundus color photograph FIVES dataset were cropped non-overlappingly into 64 images with a resolution of 256 by 256 pixels and the images with the blood vessel area in the mask smaller than a preset value were deleted.

3. The blood vessel segmentation method based on large model fine tuning according to claim 1, characterized in that, The image encoder comprises an image embedding layer and 12 Transformer layers, and each Transformer layer is fine-tuned by the low-rank adaptive fine-tuning technique; finally, the image encoder encodes the image into a set of image features.

4. The blood vessel segmentation method based on large model fine tuning according to claim 1, characterized in that, The specific working mode of the high-quality decoder is as follows: first, two tokens are generated from the image encoding through the self-attention mechanism, one of which is multiplied by the fused features to obtain a detailed mask after deconvolution, and the other is multiplied by the image features to obtain a coarse mask after deconvolution, and the detailed mask and the coarse mask are added to obtain the final output mask; finally, the loss is calculated according to the obtained output mask and the preset standard mask, the gradient is calculated through back propagation, and the weight of the image segmentation model is updated.

5. The blood vessel segmentation method based on large model fine tuning according to claim 1, characterized in that, The loss function of the image segmentation model is calculated as follows: wherein, is the predicted segmentation region, is the true segmentation region; is the ground truth mask, is the predicted mask, is the skeleton extracted from is the skeleton extracted from is the skeleton extracted from is the skeleton extracted from is the score of skeleton is the score of skeleton is the score of skeleton is the score of skeleton 6. The blood vessel segmentation method based on large model fine tuning according to claim 1, characterized in that, The inference process of the image segmentation model is as follows: obtaining a medical image and inputting the medical image into the fine-tuned image segmentation model; first, the zero pixel dimension of the input medical image is padded to the nearest multiple of 256; then, a sliding window prediction is performed using a window size of 256x256 and a step size of 128; in each prediction, only the center 128x128 region is retained for splicing, thereby generating a final prediction mask.

7. A blood vessel segmentation device based on large model fine-tuning, characterized by, The method comprises the following steps: a data acquisition module is configured to obtain a blood vessel image, pre-process the obtained blood vessel image, and obtain a training set; a model fine-tuning module is configured to fine-tune a pre-trained image segmentation model using the training set, so that the image segmentation model learns related features of blood vessel segmentation; the image segmentation model comprises an image encoder, a multi-scale feature fusion module, and a high-quality decoder; wherein the image segmentation model is fine-tuned using a low-rank adaptive fine-tuning technique: two low-rank matrices are inserted in parallel in the Transformer layer of the image encoder, the input is processed through the two low-rank matrices in turn, and the processing results are added to the original output to obtain the final output; the working mode of the image segmentation model is as follows: The image encoder is configured to extract features of an input image, and input the extracted image features into a high-quality decoder; the image encoder comprises a plurality of Transformer layers; The multi-scale feature fusion module is configured to fuse outputs of the first layer of the Transformer layer of the image encoder and the output of the last layer of the Transformer layer as v and k in attention, respectively, and the image embedding of the original image as q perform attention calculation, add the two calculation results after downsampling, obtain the fusion feature, and input the fusion feature into the high-quality decoder; The high-quality decoder is configured to generate an output mask according to the image features and the fusion features.

8. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 6.

Images