Guidewire segmentation method, device and medium based on pixel topological coupling

Abstract

The present invention relates to a technical solution of a guidewire segmentation method, device and medium based on pixel topological coupling, including: topological expansion model processing and attention focus segmentation model processing; the topological expansion model processing includes: inputting a segmented first blood vessel image, performing guidewire expansion training through a first U-Net neural network, and obtaining a corresponding expansion map; the attention focus segmentation model processing includes: inputting the expansion map and the original image into a second U-Net neural network with a loss function for attention focus segmentation prediction, and obtaining a segmented second blood vessel image; taking the second blood vessel image as input, iteratively executing the topological expansion model processing and the attention focus segmentation model processing until the optimal segmentation result is obtained. The beneficial effects of the present invention are: realizing the combination of pixel segmentation and topological segmentation processing, and obtaining a better segmentation result.

Description

Technical Field

[0001] The present invention relates to the field of computer and medical image processing, and in particular to a guidewire segmentation method, device and medium based on pixel topological coupling. Background Art

[0002] Guidewire end segmentation technology can visualize the guidewire and its tip shape in real time, and can help identify unusual or potentially risky guidewire shapes and issue warnings to doctors when interventional doctors feel fatigued due to long surgical processes. In addition, virtual vascular roaming technology can reconstruct 3D vascular tree diagrams and has many advantages such as helping to reduce the duration of interventional surgery, improve surgical success rates, fewer recurrences, and lower contrast agent usage. Reliable guidewire segmentation technology is necessary for the smooth operation of the system. Moreover, for the next generation of surgical robots that can assist inexperienced surgeons in precisely manipulating guidewires in difficult-to-pass vascular areas, guidewire segmentation technology is also indispensable as an important part of the visual feedback of the robot system operation.

[0003] Pixel-level high-precision guidewire segmentation technology can provide accurate and reliable information on the position of the guidewire end point to assist doctors or surgical robots in the next surgical operation. However, segmentation technology that only has high-precision recognition without considering the guidewire morphology is still insufficient in practical application. The guidewire ring structure identified by the guidewire segmentation technology can be used to alert doctors or surgical robots to re-manipulate the guidewire, thereby avoiding damage to the blood vessel wall caused by the guidewire end ring structure. Therefore, a guidewire segmentation technology that comprehensively considers the pixel level and topological level is crucial for the above clinical applications.

[0004] The existing guidewire segmentation methods that take this problem into consideration still have defects in the processing method: B-spline or other manually selected topological models are used to optimize the segmentation results. However, B-spline or other optimized manually selected topological models are all established from simple mathematical models, but due to the flexibility and softness of the guidewire itself, it will produce a variety of complex shapes in the blood vessel, and these shapes are difficult to be fitted by simple mathematical topological models.

[0005] In contrast, models based on deep neural network learning of guidewire posture from massive amounts of real intraoperative fluorescence imaging images can achieve better segmentation results than manual topological models. However, these deep learning models also have some problems: most of them use neural networks to perform segmentation first and then perform post-processing of topological improvement. This processing process treats pixel segmentation and topological segmentation as two relatively independent tasks. Summary of the invention

[0006] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art, and to provide a guidewire segmentation method, device and medium based on pixel topological coupling, which achieves better segmentation results of pixel segmentation and topological segmentation through two complementary models of topological expansion and attention focus segmentation.

[0007] The technical solution of the present invention includes a guidewire segmentation method based on pixel topological coupling, characterized in that the method includes: topological expansion model processing and attention focus segmentation model processing; the topological expansion model processing includes: inputting a segmented first blood vessel image, performing guidewire expansion training through a first U-Net neural network, and obtaining a corresponding expansion map; the attention focus segmentation model processing includes: inputting the expansion map and the original image into a second U-Net neural network with a loss function to perform attention focus segmentation prediction, and obtain a segmented second blood vessel image; using the second blood vessel image as input, iteratively executing the topological expansion model processing and the attention focus segmentation model processing until the optimal segmentation result is obtained.

[0008] According to the guidewire segmentation method based on pixel topology coupling, the first U-net neural network includes: taking each of the first blood vessel images that have undergone fluorescence imaging and manual annotation in the offline guidewire data set as input training data, performing random cutting, and dilating the cut first blood vessel images.

[0009] According to the guidewire segmentation method based on pixel topology coupling, the expansion of the severed first blood vessel image includes: the first blood vessel image is expanded within the guidewire range based on the original annotated segmentation result.

[0010] According to the guidewire segmentation method based on pixel topological coupling, the loss function includes: a left channel and a right channel, the right channel is used to absorb the input image using the trained TE function to generate the corresponding expansion map and the labeled true value; the left channel is used to receive the expansion map and the labeled true value, and calculate the loss value at the topological level, and the loss function of the left channel is DiceLoss.

[0011] According to the guidewire segmentation method based on pixel topology coupling, the optimal segmentation result also includes: when any segmented image obtained after the attention-focused segmentation process is an under-segmented image, and the overlap rate between the dilated image and the actual guidewire exceeds a set value, the attention of the next attention-focused segmentation process performed on the dilated image is narrowed from the entire image to the corresponding guidewire dilation area, and this step is repeated until the optimal segmentation result is reached.

[0012] According to the guidewire segmentation method based on pixel topology coupling, the topology expansion model processing and the attention focus segmentation model processing are coupled equivalent relationships.

[0013] The technical solution of the present invention also includes a guidewire segmentation device based on pixel topological coupling, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that the processor implements any of the above method steps when executing the computer program.

[0014] The technical solution of the present invention also includes a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above methods is implemented.

[0015] The beneficial effects of the present invention are: realizing the combined processing of pixel segmentation and topological segmentation, and obtaining a better segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is further described below in conjunction with the accompanying drawings and embodiments;

[0017] Figure 1 Shown is a flow chart according to an embodiment of the present invention;

[0018] Figure 2 It is shown as an iterative segmentation framework according to an embodiment of the present invention;

[0019] Figure 3 The figure shows the composition of the topological expansion model and the attention focus segmentation model in the iterative segmentation framework according to the embodiment of the present invention;

[0020] Figure 4 Shown are examples of topological expansion and attention-focused segmentation input and output according to an embodiment of the present invention;

[0021] Figure 5 The figure shows the offline generation process of training data for topological expansion and attention focus segmentation according to an embodiment of the present invention;

[0022] Figure 6 An example of loss calculation of a pixel topology coupling loss function according to an embodiment of the present invention;

[0023] Figure 7 Shown is a diagram of an apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0025] In the description of the present invention, “several” means one or more, “more” means more than two, “greater than”, “less than”, “exceed”, etc. are understood as not including the number itself, and “above”, “below”, “within”, etc. are understood as including the number itself.

[0026] In the description of the present invention, the consecutive numbering of the method steps is for the convenience of review and understanding. Combined with the overall technical scheme of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical scheme of the present invention.

[0027] In the description of the present invention, unless otherwise clearly defined, words such as "setting" should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above words in the present invention in combination with the specific content of the technical solution.

[0028] Figure 1 The flowchart shown is according to an embodiment of the present invention, which process includes: topological expansion model processing and attention focus segmentation model processing; the topological expansion model processing includes: inputting a segmented first blood vessel image, performing guidewire expansion training through a first U-Net neural network, and obtaining a corresponding expansion map; the attention focus segmentation model processing includes: inputting the expansion map and the original image into a second U-Net neural network with a loss function to perform attention focus segmentation prediction, and obtain a segmented second blood vessel image; using the second blood vessel image as input, iteratively executing the topological expansion model processing and the attention focus segmentation model processing until the optimal segmentation result is obtained.

[0029] Figure 2 The iterative segmentation framework according to an embodiment of the present invention is shown, and the framework includes two complementary models: topological expansion (TE) and attention focused segmentation (FS). The technical solution of the present invention designs a pixel topological coupling loss function, which enables the neural network to place pixel segmentation and topological segmentation in an equally important position during the learning process.

[0030] like Figure 3 The topological expansion (TE) model presented is used to explain the first U-Net neural network and the second U-Net neural network. Figure 3 It includes (a) iterative segmentation framework; (b) topological expansion model composition; (c) attention focus segmentation model composition. In the test phase, the k-th segmentation result S k As input, the wire is expanded in a topological manner to generate an expansion graph E k If the k-th segmentation result S k is under-segmented (ref. Figure 4 , Figure 4Including (a) topological expansion, (b) attention focus segmentation input and output examples), that is, there are some areas of the guide wire that should be segmented but are not segmented. The experimental results can prove that the topological expansion (TE) model can generate an expansion graph E k The under-segmented areas are compensated by coinciding with the real guide wire with a high probability, thus helping the attention focused segmentation (FS) to produce a more complete segmentation result S k+1 .

[0031] In order to train the U-Net neural network architecture in the topological expansion (TE) model so that it can expand in a topological manner according to the guidewire, a new database was established offline based on the most basic guidewire dataset (ref. Figure 5 , Figure 5 Including (a) topological expansion, (b) attention focus segmentation training data offline generation process). For each intraoperative fluorescence image and the corresponding radiologist-annotated segmentation result, the segmentation result is randomly chopped (the chopped method can be Figure 2 Each cut result, called cut (model i), is used as TE training data to simulate the under-segmented input that the TE model may encounter during testing. Considering the diversity in guidewire morphology, it is impossible to completely restore these cut results without knowing the true value. Therefore, the training label of TE is not the original annotated segmentation result, but a slight expansion based on the cut, called expansion (model i). Of course, this slight expansion is based on the expansion of the original annotated segmentation result, so as to ensure that the expanded area falls completely in the true area.

[0032] Focused Segmentation (FS) Model Figure 3 ), in the testing phase, the k-th expansion graph E k and the original image I as input to produce a more complete segmentation result S k+1 The experimental results show that if the k-th segmentation result S k is under-segmented ( Figure 4 ), that is, some areas where the guidewire exists should be segmented but are not segmented, and the expansion map E k If the dilated image overlaps with the real guide wire with a high probability, then this dilated image can reduce the attention of the FS model from the whole image to these subtle (dilated) areas, thereby guiding the FS model to produce more accurate images than S. k Better segmentation results S in both pixel and topology k+1 .

[0033] refer to Figure 6 An example of the loss calculation of the pixel topology coupling loss function. Figure 6Including topology-pixel joint loss, two examples of topology-level loss of different guidewires (b) and (c). For the training of the attention focused segmentation (FS) model, there is a problem with the loss function. At present, the commonly used loss function in the field of biomedical imaging is DiceLoss, which cannot well reflect the quality of the segmentation results in the topological sense. For this reason, the technical solution of the present invention designs a guidewire pixel topology coupling loss function ( Figure 6 ). refer to Figure 6 (a), (b) and (c) of the function. The function contains two channels. The left channel still uses the commonly used DiceLoss, while the right channel is the topological loss value. The topological loss value channel accepts the predicted segmentation image as input. The predicted segmentation image will first be absorbed by a trained TE function to generate an expansion map. The expansion map will be sent to the DiceLoss channel together with the true value of the joint annotation to calculate the topological level loss value.

[0034] Figure 7 The figure shows a device diagram according to an embodiment of the present invention. The device includes a memory 100 and a processor 200, wherein the processor 200 stores a computer program, and the computer program is used to execute: topological expansion model processing and attention focus segmentation model processing; the topological expansion model processing includes: inputting a segmented first blood vessel image, performing guidewire expansion training through a first U-Net neural network, and obtaining a corresponding expansion map; the attention focus segmentation model processing includes: inputting the expansion map and the original image into a second U-Net neural network with a loss function to perform attention focus segmentation prediction, and obtain a segmented second blood vessel image; using the second blood vessel image as input, iteratively executing the topological expansion model processing and the attention focus segmentation model processing until the optimal segmentation result is obtained. The memory 100 is used to store data.

[0035] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0036] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0037] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0038] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0039] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present invention.

Claims

1. A guidewire segmentation method based on pixel topology coupling, characterized in that: The method includes: Topological expansion model processing and attention focus segmentation model processing; The topological expansion model processing includes: inputting a segmented first blood vessel image, performing guidewire expansion training through a first U-Net neural network, and obtaining a corresponding expansion map; The attention focus segmentation model processing includes: inputting the dilation map and the original image into a second U-Net neural network with a loss function to perform attention focus segmentation prediction to obtain a segmented second blood vessel image; The second blood vessel image is used as input, and the topological expansion model processing and the attention focus segmentation model processing are iteratively performed until an optimal segmentation result is obtained.

2. The guidewire segmentation method based on pixel topological coupling according to claim 1, characterized in that: The first U-net neural network includes: Each of the first blood vessel images that has been fluorescently imaged and manually annotated in the offline guidewire data set is used as input training data, random cutting is performed, and the cut first blood vessel images are expanded.

3. The guidewire segmentation method based on pixel topology coupling according to claim 2, characterized in that: The expanding the cut first blood vessel image comprises: The first blood vessel image is expanded within the guidewire range based on the original annotated segmentation result.

4. The guidewire segmentation method based on pixel topology coupling according to claim 1, characterized in that: The loss function includes: The left channel and the right channel are used to absorb the input image using the trained TE function to generate the corresponding expansion map and the labeled true value; the left channel is used to receive the expansion map and the labeled true value, and calculate the loss value at the topological level. The loss function of the left channel is DiceLoss; wherein the TE function is a topological expansion function.

5. The guidewire segmentation method based on pixel topology coupling according to claim 1, characterized in that: The optimal segmentation result also includes: Any segmented image obtained after the attention-focused segmentation processing is an under-segmented image, and when the overlap rate between the expansion image and the actual guidewire exceeds a set value, the next attention-focused segmentation processing is performed on the expansion image, and the attention is narrowed from the entire image to the corresponding guidewire expansion area, and this step is repeated until the optimal segmentation result is achieved.

6. The guidewire segmentation method based on pixel topology coupling according to claim 1, characterized in that: The topological expansion model processing and the attention focus segmentation model processing are coupled and equivalent.

7. A guidewire segmentation device based on pixel topological coupling, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method steps described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

Images