Lacrimal passage segmentation and report intelligent writing method and system for lacrimal passage radiography CT

Abstract

The invention relates to a medical image processing technology, in particular to a lacrimal passage segmentation and report intelligent writing method and system for lacrimal passage radiography CT. According to the method, a CT image is used as input, a lacrimal passage and a key area of lacrimal passage blockage are recognized firstly, a multi-connected expert model is adopted to be divided into multiple fields according to the four angles of a lacrimal sac, a nasolacrimal duct, eyeballs and a nasal cavity, and finally a report conforming to professional specifications is generated through a text writing expert model.

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis, and in particular to a method and system for tear duct segmentation and intelligent report writing for tear duct CT angiography. Background Art

[0002] Tear duct obstruction is one of the common eye diseases. In clinical diagnosis, tear duct angiography CT is often used to identify the site of tear duct obstruction and select the appropriate surgical plan. The use of medical image analysis technology and report generation can improve the work efficiency of doctors and improve the accuracy of diagnosis and treatment. Patent CN117292783A extracts medical history information based on historical examination reports and generates structured labels, which are used as input and combined with the current diagnosis description text to generate the current examination report, but this method skips the image analysis process and is a text-to-text process, which has limited efficiency improvement. Patent CN117352121A first uses a classifier to judge the image representation, and then uses the image representation to generate an image report text, but this method lacks the anatomical position relationship information between the tissues or representations in the image, which will affect the integrity of the generated report information. Patent CN109961834A also uses image analysis to obtain the first and second image information, and then obtains the first and second report information, and finally uses the report label to obtain the report text. This method from image to label information omits too much information in the image and lacks the information required to generate the report. Summary of the invention

[0003] In order to solve the above defects in the prior art, the present invention provides the following technical solutions:

[0004] The first aspect of the present invention is to provide a method for tear duct segmentation and intelligent report writing for tear duct CT, characterized in that the method comprises:

[0005] Step 1. Construct a 3D segmentation model of the lacrimal duct system, segment the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, and nasal bone tissues, including nasal bones, nasal septum cartilages, small alar cartilages, large alar cartilages, and accessory nasal cartilages from the original CT images, and extract the anatomical framework lines;

[0006] Step 2. Assume that the contrast agent development threshold is t, perform threshold segmentation on the image data to obtain a binary segmentation result of the contrast agent; and based on the segmentation results of other regions in step 1, obtain three-dimensional image data after removing irrelevant regions;

[0007] Step 3. Build a multi-link expert model, which consists of a routing network M R and 5 expert networks.

[0008] Furthermore, the specific operations of step 1 are:

[0009] Step 1.1 Manually annotate N cases of eyeball, lacrimal sac, nasolacrimal duct, and nasal bone tissue as segmentation training set, build ViT3D model and train to obtain lacrimal duct system 3D segmentation model M seg ; Original image input M seg Get tissue segmentation result I seg ;

[0010] Step 1.2: The segmentation result of the nasolacrimal duct is de-noised by anisotropic diffusion filtering, and the maximum sphere tracking algorithm is used to refine the segmentation result to obtain the single-pixel centerline, and the Lagrange polynomial is used for interpolation to obtain the centerline of the nasolacrimal duct;

[0011] Step 1.3 confirms whether the centerline of the nasolacrimal duct is connected to the lacrimal sac. If not, use the marching cubes algorithm to extract the three-dimensional contour line of the lacrimal sac, calculate the Euclidean distance with the highest coordinate point of the centerline of the nasolacrimal duct in turn, select the point on the contour with the closest distance and connect it with the highest coordinate point of the centerline of the nasolacrimal duct to obtain the centerline image I of the nasolacrimal duct. duct ;

[0012] Step 1.4: Based on the nasal bone tissue segmentation result (i.e., the value at the nasal bone tissue coordinate is 1, and the others are 0), set the axial coordinate to z, the coronal coordinate to y, and the sagittal coordinate to x, take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1);

[0013] Step 1.5: Traverse from z0 to z1 to obtain each axial view A; for each axial view, traverse from y0 to y1 to obtain the coronal line C. In each coronal line C, take the minimum and maximum coordinate points (x0, x1) with a value of 1; set all points from x0 to x1 to 1, and then set the points that were originally 1 to 0. Repeat until all traversals are completed to obtain the nasal cavity segmentation result I. nasal ;

[0014] Step 1.6 I seg Remove the pixel value of the nasal bone and add it to the nasal cavity segmentation result to get I seg2 .

[0015] Furthermore, the operation of step 2 is:

[0016] Step 2.1: Combine the contrast agent segmentation result and the organ segmentation result obtained in step 1.1 seg2 Add together to get I seg3 , and set the pixels with a median value greater than 1 to 1, and then multiply it with the original image to obtain the tissue image data I without irrelevant areas. org ;

[0017] Furthermore, the operation of step 3 is:

[0018] Step 3.1 Calculate tissue image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct The minimum bounding box of the area where the median value is not 0 pixels, and the largest bounding box among the three is used as the standard to crop the three data;

[0019] Step 3.2, the multi-linked expert model is used to organize the image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct As input, output the report text for tear duct obstruction;

[0020] Step 3.3: All the features f output by the expert model i,4 After weighted addition and normalization according to the ratio of the probability array P, it is input into the transformer decoder and then passed through softmax to obtain the text output, which is the report text for tear duct obstruction.

[0021] Further, in step 3,

[0022] The routing network is divided into two modules: cross-image weighting module and routing decision module;

[0023] The cross-image weighting module consists of three branches, each of which receives an input and organizes the image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct The three inputs are transformed into corresponding features F after passing through the transformer encoder of each branch. org 、F seg3 、F duct ; Then after the cross-branch weight layer, the weighted feature f is obtained org 、f seg3 、f duct ;

[0024] The routing decision module consists of a multi-head attention layer. The three weighted features are used as the three inputs of the multi-head attention layer to obtain the final feature f. After passing through a fully connected layer and performing a softmax function, f obtains a probability array P of length 5, where the probability value of the i-th dimension (i∈{0,1,2,3,4}) represents the token weight of inputting the final feature f into the i-th expert network.

[0025] Furthermore, the cross-branch weight layer shares weights in the three branches, taking two features of the same dimension f1 and f2 as input, downsampling the two input features and then subtracting them to obtain the residual f0, and inputting f0, f1, and f2 into a multi-head attention layer to obtain the output feature. The cross-branch weight layer uses F org、F seg3 、F duct Calculate each pair of combinations once and get f org 、f seg3 、f duct .

[0026] Furthermore, the basic network structure of the expert network is a feedforward neural network. Each expert network represents a model that is good at one field, including five fields: lacrimal sac abnormality, nasolacrimal duct abnormality, nasal cavity abnormality, eye abnormality, and conjunctival sac abnormality. The five expert networks do not share weights. The number of tokens of feature f is input into the expert model through the component value of the probability array P. The expert model outputs feature f i,4 , where the token length of the input to the i-th expert model is round(len(f)×p i ), where len(f) represents the length of feature f, p i It is the i-th dimension component of the probability array P, and round means rounding to the nearest integer.

[0027] Furthermore, in order to ensure that all expert networks maintain load balance during training and avoid overtraining of a few expert networks, the present invention proposes an expert routing balance loss function L = L equ +L c ,

[0028]

[0029] Among them, batch_num represents the number of samples in each batch during the training process, x is the output of the expert model, and β is the weight coefficient of the loss;

[0030]

[0031] Among them, p i is the probability of inputting the i-th expert model, var is the variance, and mean is the mean.

[0032] A second aspect of the present invention is to provide an electronic device, characterized in that it includes:

[0033] A memory for storing executable instructions;

[0034] The processor is used to implement the tear duct segmentation and intelligent report writing method for tear duct angiography CT described in the first aspect when executing the executable instructions stored in the memory.

[0035] The third aspect of the present invention is to provide the application of the electronic device described in the second aspect in the tear duct segmentation and intelligent report writing of tear duct CT.

[0036] The fourth aspect of the present invention is to provide a computer-readable storage medium, characterized in that executable instructions are stored therein for causing a processor to execute and implement the method for tear duct segmentation and intelligent report writing for tear duct angiography CT described in the first aspect.

[0037] The fifth aspect of the present invention is to provide an application of the computer-readable storage medium described in the fourth aspect in the tear duct segmentation and intelligent report writing of tear duct CT.

[0038] The sixth aspect of the present invention is to provide a computer program product, including a computer program or a computer executable instruction, characterized in that when the computer program or the computer executable instruction is executed by a processor, the method for tear duct segmentation and intelligent report writing for tear duct angiography CT described in the first aspect is implemented.

[0039] The seventh aspect of the present invention provides the application of the computer program product described in the sixth aspect in the tear duct segmentation and intelligent report writing of tear duct CT.

[0040] The beneficial effects of the present invention include: the present invention discloses an algorithm for tear duct segmentation, blocked area identification and intelligent report generation for tear duct perfusion CT images. Specifically, the CT image is used as input, the key areas of the tear duct and tear duct blockage are first identified, and a multi-link expert model is used to divide them into multiple areas according to four angles: lacrimal sac, nasolacrimal duct, eyeball, and nasal cavity. Finally, a report that meets professional standards is generated through a text writing expert model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a multi-expert model of the nasolacrimal duct;

[0042] Figure 2 It is a routing network structure;

[0043] Figure 3 is the cross-branch weight layer;

[0044] Figure 4 It is the result of the division of the nasolacrimal duct, lacrimal sac, and eyeball;

[0045] Figure 5 It is the actual result that the center line of the nasolacrimal duct is connected to the lacrimal sac and the eyeball; DETAILED DESCRIPTION

[0046] The following is a further description of the concept of the present invention and the technical effects produced in conjunction with specific embodiments, so as to fully understand the purpose, features and effects of the present invention. The methods are conventional methods unless otherwise specified. The materials can be obtained from public commercial channels unless otherwise specified. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] Example 1

[0048] Step 1. Construct a 3D segmentation model of the lacrimal duct system, segment the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, and nasal bone tissue (including nasal bones, nasal septum cartilages, small alar cartilages, large alar cartilages, and accessory nasal cartilages) from the original CT images, and extract the anatomical framework lines.

[0049] Step 1.1 Manually annotate N cases of eyeball, lacrimal sac, nasolacrimal duct, and nasal bone tissue as segmentation training set, build ViT3D model and train to obtain lacrimal duct system 3D segmentation model M seg The original image input M seg Get tissue segmentation result I seg .

[0050] Step 1.2: Segmentation results of the nasolacrimal duct (such as Figure 4 ) Anisotropic diffusion filtering was used to remove noise, and the maximum sphere tracking algorithm was used to refine the segmentation result to obtain the single-pixel centerline, which was then interpolated using Lagrangian polynomials to obtain the nasolacrimal duct centerline.

[0051] Step 1.3 confirms whether the centerline of the nasolacrimal duct is connected to the lacrimal sac. If not, use the marching cubes algorithm to extract the three-dimensional contour line of the lacrimal sac, calculate the Euclidean distance with the highest coordinate point of the centerline of the nasolacrimal duct in turn, select the point on the contour with the closest distance and connect it with the highest coordinate point of the centerline of the nasolacrimal duct to obtain the centerline image I of the nasolacrimal duct. duct ,like Figure 5 shown.

[0052] Step 1.4: Based on the segmentation result of the nasal bone tissue (i.e., the value at the nasal bone tissue coordinate is 1, and the others are 0), set the axial coordinate to z, the coronal coordinate to y, and the sagittal coordinate to x. Take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1).

[0053] Step 1.5: Traverse from z0 to z1 to obtain each layer of axial view A; for each axial view, traverse from y0 to y1 to obtain the coronal line C. In each coronal line C, take the minimum and maximum coordinate points (x0, x1) with a value of 1; set all points from x0 to x1 to 1, and then set the points that were originally 1 to 0. Repeat until all traversals are completed to obtain the nasal cavity segmentation result I nasal .

[0054] Step 1.6 I seg Remove the pixel value of the nasal bone and add it to the nasal cavity segmentation result to get I seg2

[0055] Step 2: Assume that the contrast agent development threshold is t, perform threshold segmentation on the image data, and obtain the binary segmentation result of the contrast agent. And based on the segmentation results of other regions in step 1, obtain the three-dimensional image data with irrelevant regions removed.

[0056] Step 2.1: Combine the contrast agent segmentation result and the organ segmentation result obtained in step 1.1 seg2 Add together to get I seg3 The pixels with values ​​greater than 1 are set to 1, and then multiplied by the original image to obtain the tissue image data I without irrelevant areas. org .

[0057] Step 3. Build a multi-link expert model, which consists of a routing network M R and 5 expert networks such as Figure 1 shown.

[0058] Step 3.1 Calculate tissue image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct The minimum bounding box of the area whose median value is not 0 pixel is used as the standard, and the three data are cropped based on the largest bounding box among the three.

[0059] Step 3.2, the multi-linked expert model is used to organize the image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct Takes as input and outputs report text for tear duct obstruction.

[0060] The routing network structure is as follows Figure 2 The cross-image weighting module contains three branches, each receiving an input. org , Segmentation Result I seg3 、Nasolacrimal duct centerline I ductThe three inputs are transformed into corresponding features F after passing through the transformer encoder of each branch. org 、F seg3 、F duct Then, after the cross-branch weight layer, the weighted feature f is obtained org 、f seg3 、f duct ,The routing decision module consists of a multi-head attention layer. The three weighted features are used as the three inputs of the multi-head attention layer to obtain the final feature f. After a fully connected layer, f is subjected to a softmax function to obtain a probability array P of length 5, where the probability value of the i-th dimension (i∈{0,1,2,3,4}) represents the token weight of inputting the final feature f into the i-th expert network;

[0061] The cross-branch weight layer shares weights among the three branches. Figure 3 As shown in Figure 1, two features of the same dimension, f1 and f2, are taken as input, the two input features are downsampled and then subtracted to obtain the residual f0, and f0, f1, and f2 are input into a multi-head attention layer to obtain the output feature. org 、F seg3 、F duct Calculate each pair of combinations once and get f org 、f seg3 、f duct .

[0062] The basic network structure of the expert network is a feedforward neural network. Each expert network represents a model that is good at one field, including five fields: lacrimal sac abnormality, nasolacrimal duct abnormality, nasal cavity abnormality, eye abnormality, and conjunctival sac abnormality. The five expert networks do not share weights. The number of tokens of feature f is input into the expert model through the component value of the probability array P. The expert model outputs feature f i,4 , where the token length of the input to the i-th expert model is round(len(f)×p i ), where len(f) represents the length of feature f, p i It is the i-th dimension component of the probability array P, and round means rounding to the nearest integer.

[0063] Step 3.3, all the features f output by the expert model i,4 After weighted addition and normalization according to the ratio of the probability array P, it is input into the transformer decoder and then passed through softmax to obtain the text output, which is the report text for tear duct obstruction.

[0064] In order to ensure that all expert networks maintain load balance during training and avoid overtraining of a few expert networks, the present invention proposes an expert routing balance loss function L = Lequ +L c ,

[0065]

[0066] Among them, batch_num represents the number of samples in each batch during the training process, x is the output of the expert model, and β is the weight coefficient of the loss.

[0067]

[0068] Among them, p i is the probability of inputting the i-th expert model, var is the variance, and mean is the mean.

[0069] The final report results were "enlarged right lacrimal sac, contrast retention in the lacrimal sac and nasolacrimal duct, a small amount of contrast in the inferior nasal meatus and naso-oropharynx, partial canaliculus display, and conjunctival sac overflow."

Claims

1. A method for tear duct segmentation and intelligent report writing for tear duct CT, characterized in that: The method includes: Step 1. Construct a 3D segmentation model of the lacrimal duct system, segment the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, and nasal bone tissues, including nasal bones, nasal septum cartilages, small alar cartilages, large alar cartilages, and accessory nasal cartilages from the original CT images, and extract the anatomical framework lines; Step 2. Assume that the contrast agent development threshold is t, perform threshold segmentation on the image data to obtain a binary segmentation result of the contrast agent; and based on the segmentation results of other regions in step 1, obtain three-dimensional image data after removing irrelevant regions; Step 3. Build a multi-link expert model, which consists of a routing network M R and 5 expert networks.

2. The method according to claim 1, characterized in that The specific operations of step 1 are: Step 1.1 Manually annotate N cases of eyeball, lacrimal sac, nasolacrimal duct, and nasal bone tissue as segmentation training set, build ViT3D model and train to obtain lacrimal duct system 3D segmentation model M seg ; Original image input M seg Get tissue segmentation result I seg ; Step 1.2: The segmentation result of the nasolacrimal duct is de-noised by anisotropic diffusion filtering, and the maximum sphere tracking algorithm is used to refine the segmentation result to obtain the single-pixel centerline, and the Lagrange polynomial is used for interpolation to obtain the centerline of the nasolacrimal duct; Step 1.3 confirms whether the centerline of the nasolacrimal duct is connected to the lacrimal sac. If not, use the marching cubes algorithm to extract the three-dimensional contour line of the lacrimal sac, calculate the Euclidean distance with the highest coordinate point of the centerline of the nasolacrimal duct in turn, select the point on the contour with the closest distance and connect it with the highest coordinate point of the centerline of the nasolacrimal duct to obtain the centerline image I of the nasolacrimal duct. duct ; Step 1.4: Based on the nasal bone tissue segmentation result (i.e., the value at the nasal bone tissue coordinate is 1, and the others are 0), set the axial coordinate to z, the coronal coordinate to y, and the sagittal coordinate to x, take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1); Step 1.5: Traverse from z0 to z1 to obtain each axial view A; for each axial view, traverse from y0 to y1 to obtain the coronal line C. In each coronal line C, take the minimum and maximum coordinate points (x0, x1) with a value of 1; set all points from x0 to x1 to 1, and then set the points that were originally 1 to 0. Repeat until all traversals are completed to obtain the nasal cavity segmentation result I. nasal ; Step 1.6 I seg Remove the pixel value of the nasal bone and add it to the nasal cavity segmentation result to get I seg2 .

3. The method according to claim 2, characterized in that The operation of step 2 is: Step 2.1: Combine the contrast agent segmentation result and the organ segmentation result obtained in step 1.1 seg2 Add together to get I seg3 , and set the pixels with a median value greater than 1 to 1, and then multiply it with the original image to obtain the tissue image data I without irrelevant areas. org .

4. The method according to claim 3, characterized in that The operation of step 3 is: Step 3.1 Calculate tissue image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct The minimum bounding box of the area where the median value is not 0 pixels, and the largest bounding box among the three is used as the standard to crop the three data; Step 3.2 The multi-linked expert model organizes the image data I org , Segmentation Result I seg3 、Nasolacrimal duct centerline I duct As input, output the report text for tear duct obstruction; Step 3.3: All the features f output by the expert model i,4 After weighted addition and normalization according to the ratio of the probability array P, it is input into the transformer decoder and then passed through softmax to obtain the text output, which is the report text for tear duct obstruction.

5. The method according to claim 4, characterized in that In step 3, The routing network is divided into two modules: cross-image weighting module and routing decision module; The cross-image weighting module contains three branches, each of which receives an input. The three inputs, the organizational image data Iorg, the segmentation result Iseg3, and the nasolacrimal duct centerline Iduct, are respectively obtained after passing through the transformer encoder of each branch to obtain the corresponding features Forg, Fseg3, and Fduct; then, after passing through the cross-branch weight layer, the weighted features forg, fseg3, and fduct are obtained; The routing decision module consists of a multi-head attention layer. The three weighted features are used as the three inputs of the multi-head attention layer to obtain the final feature f. After passing through a fully connected layer and performing a softmax function, f obtains a probability array P of length 5, where the probability value of the i-th dimension (i∈{0,1,2,3,4}) represents the token weight of inputting the final feature f into the i-th expert network.

6. The method according to claim 5, characterized in that in, The cross-branch weight layer shares weights in the three branches, taking two features of the same dimension f1 and f2 as input. The two input features are downsampled and then subtracted to obtain the residual f0, and f0, f1, and f2 are input into a multi-head attention layer to obtain the output feature. The cross-branch weight layer uses F org 、F seg3 、F duct Calculate each pair of combinations once and get f org 、f seg3 、f duct .

7. The method according to claim 5, characterized in that The basic network structure of the expert network is a feedforward neural network. Each expert network represents a model that is good at one field, including five fields: lacrimal sac abnormality, nasolacrimal duct abnormality, nasal cavity abnormality, eye abnormality, and conjunctival sac abnormality. The five expert networks do not share weights. The number of tokens of feature f is input into the expert model through the component value of the probability array P. The expert model outputs feature f i,4 , where the token length of the input to the i-th expert model is round(len(f)×p i ), where len(f) represents the length of feature f, p i It is the i-th dimension component of the probability array P, and round means rounding to the nearest integer.

8. The method according to claim 5, characterized in that In order to ensure that all expert networks maintain load balance during training and avoid overtraining of a few expert networks, the present invention proposes an expert routing balance loss function L = L equ +L c , Among them, batch_num represents the number of samples in each batch during the training process, x is the output of the expert model, and β is the weight coefficient of the loss; Among them, p i is the probability of inputting the i-th expert model, var is the variance, and mean is the mean.

9. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the tear duct segmentation and intelligent report writing method for tear duct angiography CT as described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute and implement the tear duct segmentation and intelligent report writing method for tear duct angiography CT as described in any one of claims 1 to 8.

Images