Blood vessel segmentation method, device and equipment based on large model fine tuning and medium

Abstract

The invention discloses a blood vessel segmentation method and device based on large model fine tuning, equipment and a medium, and the method comprises the steps: obtaining a blood vessel image, carrying out the preprocessing of the obtained blood vessel image, and obtaining a training set; performing fine tuning on a pre-trained image segmentation model by adopting the training set, so that the image segmentation model learns related features of blood vessel segmentation; wherein the low-rank adaptive fine tuning technology is adopted to perform fine tuning on the image segmentation model: two parallel low-rank matrixes are inserted into a Transform layer in the image encoder, the input is processed by the two low-rank matrixes in sequence, and the processing result and the original output are added to obtain the final output. According to the method, the defects of a large model in a blood vessel segmentation task are overcome through efficient parameter fine tuning and a multi-scale feature fusion module. Accurate blood vessel segmentation can be realized under limited annotation data, the difficulty and cost of medical image segmentation tasks are greatly reduced, and the method can be widely applied to the technical field of medical image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a blood vessel segmentation method, device, equipment and medium based on large model fine-tuning. Background Art

[0002] Blood vessel segmentation is an important research direction in the field of medical image processing, mainly used to extract tubular vascular structures from medical images. This technology is of great significance in many medical scenarios, such as segmenting fundus blood vessels from fundus color photographs, fluorescein fundus angiography or fundus optical coherence tomography blood flow imaging for the diagnosis of eye diseases (such as diabetic retinopathy, hypertensive retinopathy, etc.); segmenting coronary vessels from coronary artery X-ray angiography images and computed tomography images for preoperative diagnosis, surgical planning and postoperative evaluation of heart diseases (such as coronary artery stenosis, coronary artery calcification, etc.).

[0003] The commonly used vascular segmentation methods are mainly based on convolutional neural network architectures, such as UNet, ResUNet, nnUNet, etc. These neural network architectures can achieve vascular segmentation tasks to a certain extent, but due to the relatively small number of model parameters, they cannot fully capture the complex features and contextual information in the image. Therefore, their performance in processing multimodal medical image segmentation tasks is not satisfactory. A mainstream view is that multimodal tasks should be completed by large models. The introduction of SegmentAnything Model (SAM) fills the gap in the field of large segmentation models. Although the original SAM has not been specially trained for medical images, researchers have successively proposed SAM variants for the medical field, such as SAMMed2d and MedSAMAdapter. However, these large models have poor segmentation effects on the edges of objects, resulting in their unsatisfactory performance in vascular segmentation tasks. Moreover, the training process of fine-tuning the large model in the above methods requires a large amount of segmentation annotation data, while the vascular segmentation task is extremely scarce in terms of manually annotated data. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide a blood vessel segmentation method, device, equipment and medium based on large model fine-tuning.

[0005] The first technical solution adopted by the present invention is:

[0006] A blood vessel segmentation method based on large model fine-tuning includes the following steps:

[0007] Acquire a vascular image, pre-process the acquired vascular image, and obtain a training set;

[0008] The pre-trained image segmentation model is fine-tuned using the training set so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder;

[0009] Among them, the low-rank adaptive fine-tuning technology is used to fine-tune the image segmentation model: two parallel low-rank matrices are inserted into the Transformer layer in the image encoder, the input is processed by the two low-rank matrices successively, and the processing result is added to the original output to obtain the final output.

[0010] Furthermore, the acquiring of the vascular image and preprocessing of the acquired vascular image to obtain a training set includes:

[0011] The fundus color photo FIVES dataset is used as training data, and the fundus color photo FIVES dataset consists of 800 images with a resolution of 2048×2048;

[0012] The images in the FIVES fundus color photo dataset were cropped into 64 images with a resolution of 256×256 without overlap, and the images with vascular areas smaller than the preset value in the mask were deleted.

[0013] Furthermore, the image segmentation model works as follows:

[0014] The image encoder is used to extract features from the input image and input the extracted image features into a high-quality decoder; the image encoder includes multiple Transformer layers;

[0015] The multi-scale feature fusion module is used to transform the output F of the first layer Transformer layer of the image encoder 1 And the output F of the last Transformer layer n As v and k in the attention, respectively, with the image embedding F of the original image as q Emb Perform attention calculation, downsample the two calculation results and add them together to obtain fused features, and input the fused features into the high-quality decoder;

[0016] The high-quality decoder is used to generate an output mask according to the image features and the fused features.

[0017] Furthermore, the image encoder includes an image embedding layer and 12 Transformer layers, each Transformer layer is fine-tuned by a low-rank adaptive fine-tuning technique; finally, the image encoder encodes the image into a set of image features.

[0018] Furthermore, the expressions of the two attention calculations in the multi-scale feature fusion module are as follows:

[0019]

[0020]

[0021] Where, d k F Emb The length of the dimension.

[0022] Furthermore, the specific working mode of the high-quality decoder is as follows:

[0023] First, through the self-attention mechanism, two tokens are generated from the image code. One token is matrix-multiplied with the fusion feature and then deconvolved to obtain a detail mask. The other token is matrix-multiplied with the image feature and then deconvolved to obtain a coarse mask. The detail mask and the coarse mask are added to obtain the final output mask.

[0024] Finally, the loss is calculated based on the obtained output mask and the preset standard mask, and the gradient is calculated through back propagation to update the weights of the image segmentation model.

[0025] Furthermore, the loss function for training the image segmentation model is calculated as follows:

[0026] Loss = Dice Loss + clDice Loss

[0027] Dice Loss = 1-Dice(A, B)

[0028]

[0029] clDice Loss=1-clDice(V P , V L )

[0030]

[0031] Where A is the predicted segmentation area, B is the actual segmentation area; V L is the truth mask, V P is the predicted mask, S L From V L Extract the skeleton from S P From V P Extract the skeleton from T prec (S P , V L ) is the skeleton S P The score, T sens (S L , V P ) is the skeleton S L score.

[0032] Furthermore, the reasoning process of the image segmentation model is as follows:

[0033] Acquire a medical image, and input the medical image into the fine-tuned image segmentation model;

[0034] First, the zero-pixel dimension of the input medical image is padded to the nearest multiple of 256; then, a sliding window prediction is performed with a window size of 256×256 and a step size of 128; in each prediction, only the central 128×128 area is retained for stitching to generate the final prediction mask.

[0035] The second technical solution adopted by the present invention is:

[0036] A blood vessel segmentation device based on large model fine-tuning, comprising:

[0037] A data acquisition module, used for acquiring blood vessel images, preprocessing the acquired blood vessel images, and obtaining a training set;

[0038] A model fine-tuning module, used to fine-tune the pre-trained image segmentation model using a training set so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder;

[0039] Among them, the low-rank adaptive fine-tuning technology is used to fine-tune the image segmentation model: two parallel low-rank matrices are inserted into the Transformer layer in the image encoder, the input is processed by the two low-rank matrices successively, and the processing result is added to the original output to obtain the final output.

[0040] The third technical solution adopted by the present invention is:

[0041] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a blood vessel segmentation method based on large model fine-tuning as described above.

[0042] The fourth technical solution adopted by the present invention is:

[0043] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a blood vessel segmentation method based on large model fine-tuning as described above.

[0044] The fifth technical solution adopted by the present invention is:

[0045] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned blood vessel segmentation method based on large model fine-tuning.

[0046] The beneficial effects of the present invention are as follows: the present invention solves the shortcomings of large models in the task of blood vessel segmentation through efficient parameter fine-tuning and multi-scale feature fusion modules. The present invention can not only effectively adapt to the input of new modality images, but also achieve accurate blood vessel segmentation under limited annotated data, greatly reducing the difficulty and cost of medical image segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a structural schematic diagram of an image segmentation model in an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of a multi-scale feature fusion module MSFF in an embodiment of the present invention;

[0050] Figure 3 is a training and workflow diagram of an image segmentation model in an embodiment of the present invention;

[0051] Figure 4 It is a flowchart of the steps of a blood vessel segmentation method based on large model fine-tuning in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0054] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0055] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0056] Terminology explanation:

[0057] SAM: Segment Anything Model, SAM is an advanced image segmentation model. SAM is designed to achieve efficient and accurate image segmentation with simple inputs (such as points, boxes, or masks), and is applicable to various computer vision tasks. With its high efficiency and high-quality segmentation capabilities, it is applicable to a variety of computer vision tasks and has promoted the development of image segmentation technology.

[0058] LoRA: short for Low-Rank Adaptation. LoRA is an efficient parameter fine-tuning technology that aims to achieve efficient adaptation and fine-tuning by performing low-rank decomposition on the weights of pre-trained models. LoRA can significantly reduce the number of parameters and computing resources required for fine-tuning while retaining the original performance of the model.

[0059] HQ Decoder: High Quality Decoder. HQ Decoder can capture richer contextual information and details by processing the feature map from the encoder. This enables the model to more accurately segment the boundaries of the target object when processing complex scenes, which is especially important in blood vessel segmentation.

[0060] MSFF: Multi-Scale Feature Fusion. MSFF uses a cross-attention mechanism to fuse shallow and deep features, captures vascular information at multiple scales, and then feeds it into the HQ Decoder module. This module can improve the model's comprehensive segmentation performance in thick and thin blood vessels.

[0061] Sliding Window Prediction Module: In many image segmentation and detection tasks, directly scaling the image to a smaller size may result in loss of details. This is especially true when dealing with complex scenes or objects with complex structures such as blood vessels. The Sliding Window Prediction Module performs local processing on the image to ensure that the details within each window are preserved, thereby improving the accuracy of segmentation or detection.

[0062] In view of the shortcomings of the existing technology, the present invention proposes a fine-tuning method for applying a large model to blood vessel segmentation, aiming to solve the shortcomings of the large model in the blood vessel segmentation task through efficient parameter fine-tuning (Parameter-Efficient Fine-Tuning) and multi-scale feature fusion module. Compared with the existing methods, the method of the present invention can not only effectively adapt to the input of new modality images, but also achieve accurate blood vessel segmentation under limited annotated data, greatly reducing the difficulty and cost of medical image segmentation tasks.

[0063] Example 1

[0064] like Figure 3 and 4 As shown, this embodiment provides a blood vessel segmentation method based on large model fine-tuning, comprising the following steps:

[0065] S1. Acquire a vascular image, preprocess the acquired vascular image, and obtain a training set.

[0066] Before training, we cropped the images into several small images with a resolution of 256×256 to fit the image input size of SAM, and deleted the images whose blood vessel area in the mask was smaller than a preset value.

[0067] As an optional implementation, the fundus color photo FIVES dataset is used as training data, which consists of 800 images with a resolution of 2048 × 2048. The images are cropped into 64 images with a resolution of 256 × 256 without overlap, and images with a vascular area of ​​less than 1% in the mask are deleted.

[0068] S2. Fine-tune the pre-trained image segmentation model using the training set so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder.

[0069] This embodiment selects the SAM benchmark large model for subsequent fine-tuning. SAM has powerful image segmentation capabilities and is particularly suitable for processing multimodal images. In this embodiment, the pre-trained weights of SAM are trained on natural images. In order to allow the model to learn the relevant features of vascular segmentation, this embodiment uses the LoRA fine-tuning method to fine-tune the pre-trained model. The specific steps are as follows: insert two parallel low-rank matrices into the Transformer layer in the image encoder, and pass the input through the two low-rank matrices in turn, and add the result to the original output to obtain the final output. Through the LoRA fine-tuning method, the model is trained while only updating a small number of parameters, while maintaining the performance of the model.

[0070] As an optional implementation, we freeze all parameters in the original image encoder and deploy the LoRA matrix for the attention matrix of the Transformer layer in SAM. During the training process, only the weights of the LoRA matrix in the SAM encoder will be changed. We use the AdamW optimizer with an initial learning rate of 1e-4 and train for a total of 30 epochs. The learning rate is divided by 2 at the 5th and 10th epochs, and the batch size is 32.

[0071] The image segmentation model is described in detail below with reference to the accompanying drawings and exemplary embodiments.

[0072] join Figure 1 , Figure 1 The overall structure and training diagram of the image segmentation model are shown in Figure 2. The image segmentation model includes an image encoder, a high-quality decoder, and an MSFF module. The image encoder contains an image embedding layer and 12 Transformer layers. Each Transformer layer is fine-tuned using the LoRA fine-tuning technology. Finally, the image encoder encodes the image into a set of image features. The image features extracted by the image encoder are input into the high-quality decoder, such as Figure 1 shown.

[0073] The unbalanced distribution of thick and thin blood vessels may cause the model to over-learn the thick / thin blood vessel features and ignore the thin / thick blood vessel features. Based on this, this embodiment uses the MSFF module to fuse the shallow features and deep features. Figure 2 The specific steps are as follows: Output F of the first Transformer layer 1 And the output F of the last Transformer layer nAs v and k in the attention, respectively, they are embedded in the original image as q (F Emb ) to perform attention calculation. That is:

[0074]

[0075]

[0076] where d k F Emb The output of the first Transformer layer and the output of the image embedding layer perform cross-attention calculations to extract the detailed features of the first layer output. Similarly, the output of the last Transformer layer and the output of the image embedding layer perform cross-attention calculations to extract the detailed features of the last layer output. Both are downsampled through convolution, and the downsampled results are finally added to obtain the final fusion feature, which is output to the high-quality decoder. The fusion feature is input to the high-quality decoder, such as Figure 1 shown.

[0077] The high-quality decoder will generate an output mask from the fusion features obtained by the MSFF module and the image features output by the image encoder. The decoder will first generate two tokens from the image code through the self-attention mechanism. One token will be matrix-multiplied with the fusion features and then deconvolved to obtain a detail mask. The other token will be matrix-multiplied with the image features and then deconvolved to obtain a coarse mask. Finally, the two masks are added to obtain the final output mask.

[0078] Finally, the output mask of the high-quality decoder will be compared with the standard mask to calculate the loss function, and the gradient will be calculated and the model weight will be updated through back propagation. In this embodiment, the standard mask refers to the correct blood vessel mask annotated by the doctor in the blood vessel segmentation dataset.

[0079] In some embodiments, we use the following loss function suitable for supervised vessel segmentation tasks to enable the model to achieve our training objectives:

[0080] 1) Dice Loss

[0081] DiceLoss is a loss function commonly used in image segmentation tasks, especially in fields such as medical image analysis. It is derived from the Dice coefficient, which is an indicator used to measure the similarity between two samples, and is particularly suitable for unbalanced category problems.

[0082] The calculation formula of Dice coefficient is:

[0083]

[0084] Among them, (A) is the predicted segmentation area, (B) is the actual segmentation area. The value of this coefficient is between 0 and 1, 1 means complete overlap, and 0 means no overlap.

[0085] DiceLoss is defined based on the Dice coefficient, usually expressed as:

[0086] Dice Loss = 1-Dice(A, B)

[0087] 2)clDice Loss

[0088] In tasks like vascular topology extraction, a spatially continuous sequence of correct pixel predictions is more meaningful than a correct prediction of a spatial region.

[0089] The clDice operator formula is as follows:

[0090]

[0091] Among them, V L is the truth mask, V P is the predicted mask, S L From V L Extract the skeleton from S P From V P Extract the skeleton from T prec (S P , V L ) is the skeleton S P The fraction of T is the topological accuracy. sens (S L , V P ) is the skeleton S L The fraction of is the topological sensitivity. The corresponding operator formula is as follows:

[0092]

[0093]

[0094] The clDice Loss is defined based on the clDice coefficient and is usually expressed as:

[0095] clDice Loss=1-clDice(V P , V L )

[0096] 3) Total loss function

[0097] The final loss function is obtained by adding Dice Loss and clDice Loss, that is:

[0098] Loss = Dice Loss + clDice Loss

[0099] S3. Obtain a medical image to be segmented, input the medical image into the fine-tuned image segmentation model, and output a predicted mask.

[0100] join Figure 3 After fine-tuning, the image segmentation model can be applied to the segmentation of medical images. Since the method of this embodiment adopts block training, we use the expansion prediction method in prediction. Specifically, in the inference stage, the zero pixel dimension of the image is first padded to the nearest multiple of 256. Then, a sliding window prediction is performed with a window size of 256×256 and a step size of 128. In each prediction, only the central 128×128 area is retained for splicing to generate the final prediction mask.

[0101] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0102] (1) Effectively improve cross-modal segmentation performance: Large models have strong generalization capabilities. Compared with traditional convolutional segmentation networks, large models can share and utilize features between different modalities. This feature sharing helps the model extract more representative features when processing cross-modal data, thereby improving segmentation results.

[0103] (2) Accurate generation of thick and thin blood vessel boundaries: This paper introduces a cross-attention method to guide the fusion of feature layers within the image itself. By establishing attention connections between these layers, the utilization of local and global information in the image is improved, thereby improving the accuracy and robustness of blood vessel segmentation.

[0104] (3) High efficiency of transfer learning: By leveraging the rich knowledge of large-scale pre-trained models, it is possible to achieve excellent performance on limited labeled data. With the help of features previously obtained on large-scale datasets, the model significantly reduces the consumption of training time and computing resources when fine-tuning specific tasks, while maintaining high-precision prediction results. This high efficiency not only accelerates the model training process, but also enables reliable performance to be achieved even when data is scarce, providing a practical solution for clinical applications.

[0105] (4) The present invention is the first to apply the segmentation model to the field of vascular segmentation. Vascular segmentation can help doctors more accurately identify and locate vascular structures, thereby assisting in the diagnosis of various cardiovascular diseases, ocular vascular diseases, cerebrovascular diseases, as well as tumors and other lesions.

[0106] (5) The present invention proposes a multi-scale feature fusion module MSFF, which can effectively capture the correlation between different features of thick and thin blood vessels, and achieve more accurate feature fusion by focusing on important features and suppressing irrelevant information. This attention mechanism enables the HQ Decoder to have better decoding performance.

[0107] (6) The present invention can effectively capture complex vascular structures and contextual information and still achieve high-precision vascular segmentation when there are only limited annotated samples. Through fine-tuning, the model can quickly adapt to specific tasks, make full use of existing knowledge, and significantly improve performance in data-scarce scenarios, thereby providing a reliable solution for clinical applications.

[0108] (7) The present invention can adapt to the feature distribution of different modalities, capture more comprehensive and detailed vascular structure information, thereby improving segmentation accuracy and robustness, enhancing the ability to identify complex lesions, and providing more accurate and reliable vascular segmentation results for clinical use.

[0109] Example 2

[0110] This embodiment provides a blood vessel segmentation device based on large model fine-tuning, including:

[0111] A data acquisition module, used for acquiring blood vessel images, preprocessing the acquired blood vessel images, and obtaining a training set;

[0112] A model fine-tuning module, used to fine-tune the pre-trained image segmentation model using a training set so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder;

[0113] Among them, the low-rank adaptive fine-tuning technology is used to fine-tune the image segmentation model: two parallel low-rank matrices are inserted into the Transformer layer in the image encoder, the input is processed by the two low-rank matrices successively, and the processing result is added to the original output to obtain the final output.

[0114] Since the device is a blood vessel segmentation device based on large model fine-tuning according to an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0115] Example 3

[0116] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 4 A blood vessel segmentation method based on large model fine-tuning is shown.

[0117] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0118] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0119] Since the electronic device is an electronic device corresponding to a blood vessel segmentation method based on large model fine-tuning in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0120] Example 4

[0121] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 4 A blood vessel segmentation method based on large model fine-tuning is shown.

[0122] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0123] Since the storage medium is a storage medium corresponding to a blood vessel segmentation method based on large model fine-tuning in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0124] Example 5

[0125] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a blood vessel segmentation method based on large model fine-tuning according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0126] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0128] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A blood vessel segmentation method based on large model fine-tuning, characterized in that: The following steps are involved: Acquire a vascular image, pre-process the acquired vascular image, and obtain a training set; The pre-trained image segmentation model is fine-tuned using the training set so that the image segmentation model learns relevant features of blood vessel segmentation; the image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder; Among them, the low-rank adaptive fine-tuning technology is used to fine-tune the image segmentation model: two parallel low-rank matrices are inserted into the Transformer layer in the image encoder, the input is processed by the two low-rank matrices successively, and the processing result is added to the original output to obtain the final output.

2. The blood vessel segmentation method based on large model fine-tuning according to claim 1, characterized in that: The step of acquiring a vascular image and preprocessing the acquired vascular image to obtain a training set includes: The fundus color photo FIVES dataset is used as training data, and the fundus color photo FIVES dataset consists of multiple images with a resolution of 2048×2048; The images in the FIVES fundus color photo dataset were cropped into 64 images with a resolution of 256×256 without overlap, and the images with vascular areas smaller than the preset value in the mask were deleted.

3. The blood vessel segmentation method based on large model fine-tuning according to claim 1, characterized in that: The image segmentation model works as follows: The image encoder is used to extract features from the input image and input the extracted image features into a high-quality decoder; The image encoder comprises a plurality of Transformer layers; The multi-scale feature fusion module is used to combine the output F1 of the first Transformer layer of the image encoder and the output F n As v and k in the attention, respectively, with the image embedding F of the original image as q Emb Perform attention calculation, downsample the two calculation results and add them together to obtain fused features, and input the fused features into the high-quality decoder; The high-quality decoder is used to generate an output mask according to the image features and the fused features.

4. The blood vessel segmentation method based on large model fine-tuning according to claim 3, characterized in that: The image encoder includes an image embedding layer and 12 Transformer layers, each of which is fine-tuned by a low-rank adaptive fine-tuning technique; ultimately, the image encoder encodes the image into a set of image features.

5. The blood vessel segmentation method based on large model fine-tuning according to claim 3, characterized in that: The specific working method of the high-quality decoder is as follows: First, through the self-attention mechanism, two tokens are generated from the image code. One token is matrix-multiplied with the fusion feature and then deconvolved to obtain a detail mask. The other token is matrix-multiplied with the image feature and then deconvolved to obtain a coarse mask. The detail mask and the coarse mask are added together to obtain the final output mask. Finally, the loss is calculated based on the obtained output mask and the preset standard mask, and the gradient is calculated through back propagation to update the weights of the image segmentation model.

6. The blood vessel segmentation method based on large model fine-tuning according to claim 1, characterized in that: The loss function for training the image segmentation model is calculated as follows: Loss = Dice Loss + clDice Loss Dice Loss = 1-Dice(A,B) clDice Loss=1-clDice(V P ,V L ) Where A is the predicted segmentation area, B is the actual segmentation area; V L is the truth mask, V P is the predicted mask, S L From V L Extract the skeleton from S P From V P Extract the skeleton from T prec (S P ,V L ) is the skeleton S P The score, T sens (S L ,V P ) is the skeleton S L score.

7. The blood vessel segmentation method based on large model fine-tuning according to claim 1, characterized in that: The reasoning process of the image segmentation model is as follows: Acquire a medical image, and input the medical image into the fine-tuned image segmentation model; First, the zero-pixel dimension of the input medical image is padded to the nearest multiple of 256; then, a sliding window prediction is performed with a window size of 256×256 and a step size of 128; in each prediction, only the central 128×128 area is retained for stitching to generate the final prediction mask.

8. A blood vessel segmentation device based on large model fine-tuning, characterized in that: include A data acquisition module, used for acquiring blood vessel images, preprocessing the acquired blood vessel images, and obtaining a training set; A model fine-tuning module, used to fine-tune the pre-trained image segmentation model using a training set, so that the image segmentation model learns relevant features of blood vessel segmentation; The image segmentation model includes an image encoder, a multi-scale feature fusion module and a high-quality decoder; Among them, the low-rank adaptive fine-tuning technology is used to fine-tune the image segmentation model: two parallel low-rank matrices are inserted into the Transformer layer in the image encoder, the input is processed by the two low-rank matrices successively, and the processing result is added to the original output to obtain the final output.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images