A lacrimal passage segmentation and report intelligent writing method and system for lacrimal passage radiography CT

Abstract

The present application relates to medical image processing technology, a kind of tear duct segmentation and report intelligent writing method and system for lacrimal passage radiography CT.The present application takes CT image as input, first, identify the key area of tear duct and tear duct obstruction, and adopt multiple expert model according to the four angles of lacrimal sac, nasolacrimal duct, eyeball, nasal cavity, divided into multiple fields, finally, through text writing expert model, generate the report that conforms to professional specification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image analysis, and in particular to a lacrimal passage segmentation and report intelligent writing method and system for lacrimal passage imaging CT. BACKGROUND

[0002] Lacrimal passage obstruction is one of the common eye diseases. In clinical diagnosis, lacrimal passage imaging CT is often used to determine the site of lacrimal passage obstruction and select appropriate surgical plan. Using medical image analysis technology and generating reports can improve the work efficiency of doctors and improve the accuracy of diagnosis and treatment. Patent CN117292783A extracts medical history information based on historical examination reports and generates structured labels, which are used as input combined with the current diagnosis description text to generate the report of the current examination, but this method skips the image analysis process, which is a text-to-text process, and the degree of efficiency improvement is limited. Patent CN117352121A first uses a classifier to determine the image representation, and then uses the image representation to generate an image report text, but this method lacks the anatomical positional relationship information between the various tissues or representations in the image, which affects the completeness of the generated report information. Patent CN109961834A also uses image analysis to obtain first and second image information, and then obtains first and second report information, and finally uses report labels to obtain a report text. This type of method from image to label type information omits too much information in the image and lacks the information needed to generate a report. SUMMARY

[0003] To solve the above-mentioned defects existing in the prior art, the present application provides the following technical scheme:

[0004] The first aspect of the present application provides a lacrimal passage segmentation and report intelligent writing method for lacrimal passage imaging CT, characterized in that the method comprises:

[0005] Step 1. Construct a three-dimensional segmentation model of the lacrimal passage system, segment the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, nasal bone tissues including nasal bone, nasal septum cartilage, alar cartilage, alar cartilage, and nasal accessory cartilage from the original CT image, and extract the anatomical framework line;

[0006] Step 2. Set the developing threshold of the contrast agent as t, perform threshold segmentation on the image data to obtain the binary segmentation result of the contrast agent; and based on the segmentation results of other regions in step 1, obtain three-dimensional image data excluding irrelevant regions;

[0007] Step 3. Construct a multi-connection expert model, which is composed of a routing network M R and five expert networks.

[0008] Further, the specific operation of step 1 is:

[0009] Step 1.1. 1 human eye, lacrimal sac, nasolacrimal duct, nasal bone tissue N is marked as a segmentation training set, a ViT3D model is constructed and trained to obtain a lacrimal duct system three-dimensional segmentation model M seg ; the original image input M seg obtain the tissue segmentation result I seg ;

[0010] Step 1.2. The segmentation result of the nasolacrimal duct is filtered by anisotropic diffusion filtering, and a maximum sphere tracking algorithm is used to refine the segmentation result to obtain a single-pixel center line. The center line of the nasolacrimal duct is obtained by combining the Lagrange polynomial for interpolation.

[0011] Step 1.3. Confirm whether the nasolacrimal duct center line is connected to the lacrimal sac. If not, use the marching cubes algorithm to extract the three-dimensional contour line of the lacrimal sac, and calculate the Euclidean distance between the highest coordinate point of the nasolacrimal duct center line and the contour line in sequence. The point on the contour line closest to the highest coordinate point of the nasolacrimal duct center line is connected to obtain the nasolacrimal duct center line image I duct ;

[0012] Step 1.4. Based on the segmentation result of the nasal bone tissue (i.e. the value of the nasal bone tissue coordinate is 1, and the others are 0), the axial coordinate is set as z, the coronal coordinate is set as y, and the sagittal coordinate is set as x. Take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1).

[0013] Step 1.5. From z0 to z1, take each layer axial view A; for each axial view, from y0 to y1, take the coronal line C, and in each coronal line C, take the minimum and maximum coordinate points (x0, x1) whose value is 1; set all points from x0 to x1 to 1, and set the original 1 to 0. Repeat until all traversals are completed, i.e. the nasal cavity segmentation result I nasal ;

[0014] Step 1.6. I seg remove the pixel value of the nasal bone and add it to the nasal cavity segmentation result to obtain I seg2 .

[0015] Further, the operation of step 2 is:

[0016] Step 2.1. Add the contrast agent segmentation result and the organ segmentation result I seg2 obtained in step 1.1 to obtain I seg3 , and set the pixel points with a value greater than 1 to 1, and then multiply it with the original image to obtain the tissue image data I org ;

[0017] Further, the operation of step 3 is:

[0018] Step 3.1 Calculate the tissue image data I org , the segmentation result I seg3 , the nasolacrimal duct centerline I duct The minimum bounding box of the region with a median of 0 pixels is obtained, and the three data are cropped according to the maximum bounding box among the three;

[0019] Step 3.2, the multi-specialist model takes the tissue image data I org , the segmentation result I seg3 , the nasolacrimal duct centerline I duct as input, and outputs the report text for the lacrimal passage obstruction;

[0020] Step 3.3, the features f i,4 After being weighted and added according to the ratio of the probability array P and being normalized, the transformer decoder is inputted, and the text output obtained after softmax is the report text for the lacrimal passage obstruction.

[0021] Further, in step 3,

[0022] The routing network is divided into two modules: cross-image weighting module and routing decision module;

[0023] The cross-image weighting module includes 3 branches, each branch receiving one input, the tissue image data I org , the segmentation result I seg3 , the nasolacrimal duct centerline I duct The three inputs are respectively obtained after the transformer encoder of each branch, and the corresponding features F org , F seg3 , F duct are obtained; then the weighted features f org , f seg3 , f duct are obtained after the cross-branch weight layer.

[0024] The routing decision module is composed of a multi-head attention layer, which takes the three weighted features as inputs of the multi-head attention layer to obtain the final feature f, and f is inputted into a fully connected layer and then a softmax function to obtain a probability array P with a length of 5, wherein the probability value of the i-th dimension (i∈{0,1,2,3,4}) represents the token weight of the i-th specialist network inputted into the final feature f.

[0025] Further, the cross-branch weight layer shares weights among the three branches, takes two features f1 and f2 with the same dimension as input, subtracts the two input features after downsampling to obtain a residual f0, and inputs f0, f1 and f2 into a multi-head attention layer to obtain an output feature, and the cross-branch weight layer obtains F org, F seg3 , F duct Each of the two combinations is calculated once to obtain f org , f seg3 , f duct .

[0026] Further, the basic network structure of the expert network is a feedforward neural network, each expert network represents a model skilled in one field, and a total of 5 fields of lacrimal sac abnormalities, nasolacrimal duct abnormalities, nasal cavity abnormalities, eye abnormalities and conjunctival sac abnormalities are included, the weights are not shared between the 5 expert networks, and the number of tokens of the feature f is input into the expert model through the component value calculation of the probability array P, and the expert model outputs the feature f i,4 , wherein the token length input into the i-th expert model is round(len(f) x p i ), wherein len(f) represents the length of the feature f, p i is the i-th component of the probability array P, and round is rounding.

[0027] Further, in order to ensure that all expert networks maintain load balancing during training and avoid overtraining of a small number of expert networks, the present application proposes an expert routing balanced loss function L = L equ + L c ,

[0028]

[0029] Wherein, batch_num represents the number of samples in each batch in the training process, is the output of the j-th sample in the i-th expert model, and β is the weight coefficient of the loss;

[0030]

[0031] Wherein, p i is the probability of inputting the i-th expert model, var is the variance, and mean is the mean.

[0032] The second aspect of the present application provides an electronic device, characterized in that it comprises:

[0033] A memory for storing executable instructions;

[0034] A processor for executing the executable instructions stored in the memory to implement the lacrimal passage segmentation and report intelligent writing method for lacrimal passage imaging CT of the first aspect.

[0035] The third aspect of the present application provides the application of the electronic device of the second aspect in the lacrimal passage segmentation and report intelligent writing of lacrimal passage imaging CT.

[0036] The fourth aspect of the present application provides a computer readable storage medium, characterized in that, storing executable instructions for causing the processor to execute the tear duct segmentation and report intelligent writing method for darcycystography CT of the first aspect.

[0037] The fifth aspect of the present application provides the application of the computer readable storage medium of the fourth aspect in the tear duct segmentation and report intelligent writing of darcycystography CT.

[0038] The sixth aspect of the present application provides a computer program product, including computer program or computer executable instructions, characterized in that, the computer program or computer executable instructions are executed by the processor to realize the tear duct segmentation and report intelligent writing method for darcycystography CT of the first aspect.

[0039] The seventh aspect of the present application provides the application of the computer program product of the sixth aspect in the tear duct segmentation and report intelligent writing of darcycystography CT.

[0040] The beneficial effects of the present application include: the present application discloses a kind of tear duct segmentation, obstruction area identification and intelligent report generation algorithm for darcycystography CT image, specifically, CT image is input, first, identify the key area of tear duct and tear duct obstruction, and adopt multiple expert model according to four angles of lacrimal sac, nasolacrimal duct, eyeball, nasal cavity and divide into multiple fields, finally, text writing expert model is generated to meet the report of professional specification. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 It is nasolacrimal duct multiple expert model;

[0042] Figure 2 It is routing network structure;

[0043] Figure 3 It is cross branch weight layer;

[0044] Figure 4 It is the segmentation result of nasolacrimal duct and lacrimal sac, eyeball;

[0045] Figure 5 It is the real result of nasolacrimal duct center line and lacrimal sac, eyeball; DETAILED DESCRIPTION

[0046] The concept and technical effects of the present application are further described below in combination with specific embodiments, so as to fully understand the purposes, features and effects of the present application. The methods are all conventional methods unless otherwise specified. The materials can be obtained from public commercial channels unless otherwise specified. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute undue limitations on the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] Embodiment 1

[0048] Step 1. Construct a three-dimensional segmentation model of the lacrimal passage system, segment the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, and nasal bone tissues (including nasal bone, nasal septum cartilage, alar cartilage, alar cartilage, and nasal accessory cartilage) from the original CT image, and extract the anatomical framework line.

[0049] Step 1.1 Manually label N examples of eyeballs, lacrimal sacs, nasolacrimal ducts, and nasal bone tissues as a segmentation training set, construct a ViT3D model, and train to obtain a three-dimensional segmentation model M of the lacrimal passage system. seg Original image input M seg Obtain tissue segmentation result I seg .

[0050] Step 1.2 Adopt anisotropic diffusion filtering to de-noise the segmentation result of the nasolacrimal duct (such as Figure 4 ), and adopt a maximum sphere tracking algorithm to refine the segmentation result to obtain a single-pixel center line, and combine a Lagrange polynomial to obtain a nasolacrimal duct center line.

[0051] Step 1.3 Confirm whether the nasolacrimal duct center line is connected to the lacrimal sac. If not, adopt a marching cubes algorithm to extract a three-dimensional contour line of the lacrimal sac, sequentially calculate the Euclidean distance between the highest coordinate point of the nasolacrimal duct center line and the contour points with the shortest distance, and connect the highest coordinate point of the nasolacrimal duct center line to the point on the contour with the shortest distance to obtain a nasolacrimal duct center line image I duct , as shown in Figure 5 .

[0052] Step 1.4 Based on the segmentation result of the nasal bone tissue (i.e., the value of the nasal bone tissue coordinate is 1, and the others are 0), set the axial coordinate as z, the coronal coordinate as y, and the sagittal coordinate as x. Take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1).

[0053] Step 1.5. Take each axial view A from z0 to z1; for each axial view, take coronal line C from y0 to y1. In each coronal line C, take the minimum and maximum coordinate points (x0, x1) with value 1; set all points from x0 to x1 to 1, and set the points originally with value 1 to 0. Repeat until all traversals are completed, and the nasal cavity segmentation result I is obtained nasal .

[0054] Step 1.6. Add I seg to the pixel value of the removed nasal bone to obtain I seg2

[0055] Step 2. Set the development threshold of the contrast agent as t, and perform threshold segmentation on the image data to obtain the binary segmentation result of the contrast agent. Based on the other region segmentation results in step 1, obtain the three-dimensional image data excluding irrelevant regions.

[0056] Step 2.1. Add the contrast agent segmentation result and the organ segmentation result I seg2 obtained in step 1.1 to obtain I seg3 . Set the pixel points with a median value greater than 1 to 1, and multiply them with the original image to obtain the tissue image data I org excluding irrelevant regions.

[0057] Step 3. Construct a multi-connection expert model composed of a routing network M R and five expert networks as shown in Figure 1 .

[0058] Step 3.1. Calculate the minimum bounding box of the non-zero pixel region in the tissue image data I org , the segmentation result I seg3 , and the nasolacrimal duct centerline I duct , and take the largest bounding box among the three as the standard to crop the three data.

[0059] Step 3.2. The multi-connection expert model takes the tissue image data I org , the segmentation result I seg3 , and the nasolacrimal duct centerline I duct as input, and outputs a report text for the lacrimal passage obstruction.

[0060] The routing network structure is shown in Figure 2 and is divided into two modules: a cross-image weighting module and a routing decision module. The cross-image weighting module includes 3 branches, and each branch receives one input. The tissue image data I org , the segmentation result I seg3 , and the nasolacrimal duct centerline I ductThe three inputs respectively obtain corresponding features F after passing through the branch transformer encoder org seg3 duct Then, the weight after feature f is obtained after passing through the cross-branch weight layer org seg3 duct The routing decision module is composed of a multi-head attention layer, and the three weight after features are taken as three inputs of the multi-head attention layer to obtain the final feature f. After the final feature f passes through a fully connected layer and a softmax function, a probability array P with a length of 5 is obtained, wherein the probability value of the i-th dimension (i e {0, 1, 2, 3, 4}) represents the token weight of inputting the final feature f into the i-th expert network;

[0061] The cross-branch weight layer shares weights among the three branches. As shown in Figure 3 , two input features f1 and f2 are inputted, the two input features are down-sampled and then subtracted to obtain a residual f0, and f0, f1 and f2 are inputted into a multi-head attention layer to obtain an output feature. The cross-branch weight layer calculates once for each combination of F org seg3 duct to obtain f org , f seg3 and f duct .

[0062] The basic network structure of the expert network is a feedforward neural network. Each expert network represents a model skilled in one field, and a total of 5 expert networks are included, including dacryocyst abnormality, nasolacrimal duct abnormality, nasal cavity abnormality, eye abnormality and conjunctival sac abnormality. The 5 expert networks do not share weights. Through component value calculation of the probability array P, how many tokens of the feature f are inputted into the expert model, and the expert model outputs a feature f i,4 , wherein the length of the token inputted into the i-th expert model is round(len(f)xp i ), wherein len(f) represents the length of the feature f, p i is the i-th component of the probability array P, and round is rounding.

[0063] Step 3.3, the features f i,4 outputted by all the expert models are weighted and added according to the ratio of the probability array P and then normalized, inputted into the transformer decoder and then outputted as a text after passing through a softmax function, that is, a report text for the lacrimal passage obstruction.

[0064] To ensure that all expert networks maintain load balancing during training and avoid overtraining of a small number of expert networks, the present application proposes an expert routing balancing loss function L = L​​​​​​equ +L c ,

[0065]

[0066] where batch_num represents the number of samples in each batch in the training process, is the output of the jth sample in the ith expert model, and β is the weight coefficient of the loss.

[0067]

[0068] where p i is the probability of inputting the ith expert model, var is the variance, and mean is the mean.

[0069] The final report result is "right lacrimal sac enlargement, contrast agent retention in lacrimal sac and nasolacrimal duct, a little contrast agent in lower nasal cavity and naso-oral cavity, partial lacrimal canaliculus display, conjunctival sac overflow."

Claims

1. A method for intelligent writing of a lacrimal passage segmentation and report for dacryocystography CT, characterized in that, The method comprises: Step 1. Constructing a three-dimensional segmentation model of the lacrimal passage system, segmenting the left and right eyeballs, left and right lacrimal sacs, left and right nasolacrimal ducts, nasal bone tissues including nasal bones, nasal septum cartilage, alar cartilages, alar cartilages, and extracting an anatomical framework line; The specific operation of step 1 is: Step 1.1 Manually label 10 cases of eyeball, lacrimal sac, nasolacrimal duct and nasal bone tissue as a segmentation training set, construct a ViT3D model and train to obtain a lacrimal passage system three-dimensional segmentation model M seg ; original image input M seg Obtain tissue segmentation result I seg ; Step 1.2 The segmentation result of nasolacrimal duct is filtered by anisotropic diffusion filter, and the segmentation result is refined by maximum sphere tracking algorithm to obtain single-pixel center line, and the Lagrange polynomial is used for interpolation to obtain the nasolacrimal duct center line image I duct ; Step 1.3 confirms whether the nasolacrimal duct center line is connected with the lacrimal sac, if connected, directly uses the nasolacrimal duct center line image I obtained in step 1.2 duct , if not connected, extracts the three-dimensional contour line of the lacrimal sac by using the marching cubes algorithm, in turn calculates the Euclidean distance with the highest coordinate point of the nasolacrimal duct center line, selects the nearest point on the contour to connect with the highest coordinate point of the nasolacrimal duct center line, and obtains the corrected nasolacrimal duct center line image I duct ; Step 1.4 Based on the segmentation result of the nasal bone tissue, that is, the value of the pixel point at the coordinate of the nasal bone tissue is 1, and the others are 0, the axial coordinate is set as z, the coronal coordinate is set as y, and the sagittal coordinate is set as x. Take the minimum and maximum coordinate points of the coronal coordinate (y0, y1), and take the minimum and maximum coordinate points of the axial coordinate (z0, z1); Step 1.5: Take each axial view A from z0 to z1; for each axial view, take the coronal line C from y0 to y1, and in each coronal line C, take the minimum and maximum coordinate points (x0, x1) of the pixel points with value 1; set all points from x0 to x1 to 1, and set the points originally with value 1 to 0; repeat until all traversals are completed, and the nasal cavity segmentation result I is obtained nasal ; Step 1.6 I seg The pixel value of the removed nasal bone is added to the nasal cavity segmentation result to obtain I seg2 ; Step 2. Set the developing threshold of the contrast agent as t, and perform threshold segmentation on the image data to obtain the binary segmentation result of the contrast agent; combine the contrast agent segmentation result with the I seg2 obtained in step 1.6 to obtain the image data of the tissue I seg3 , and set the pixel points with a median value greater than 1 as 1, and then multiply the original image to obtain the tissue image data I org without the irrelevant region. Step 3. Constructing the multi-chain expert model, which consists of one routing network M R and five expert networks; The operation of step 3 is: Step 3.1 Calculate I org , I seg3 , I duct The minimum bounding box of the non-0 pixel region of the median is calculated, and the three data are cropped according to the largest bounding box among the three. Step 3.2 The multi-union expert model takes I org , seg3 , duct as input; the features f i,4 output by all expert models are weighted and added according to the ratio of the probability array P and normalized, and then input into the transformer decoder after softmax to obtain the text output, i.e. the report text for the lacrimal passage obstruction.

2. The method of claim 1, wherein, In step 3, The routing network is divided into two modules: an inter-image weighting module and a routing decision module. The cross-image weighting module includes 3 branches, each branch receives one input, I org 、 seg3 、 duct The three inputs respectively obtain corresponding features F org 、 F seg3 、 F duct after passing through the transformer encoder of each branch; then obtain the weighted feature f org 、 f seg3 、 f duct after passing through the cross-branch weight layer; The routing decision module is composed of a multi-head attention layer, which takes the three weighted feature maps as inputs of the multi-head attention layer to obtain the final feature f. After f passes through a fully connected layer and a softmax function, a probability array P with a length of 5 is obtained, where the probability value of the i-th dimension represents the token weight of the i-th expert network input by the final feature f, i∈{0,1,2,3,4}.

3. The method of claim 2, wherein, Wherein, The cross-branch weight layer shares weights among three branches, takes two same-dimension features f1 and f2 as input, subtracts the two input features after down-sampling to obtain a residual f0, inputs f0, f1 and f2 into a multi-head attention layer to obtain output features, and the cross-branch weight layer obtains F org , F seg3 , F duct , f org , f seg3 , f duct by calculating once respectively in each combination.

4. The method of claim 2, wherein, The basic network structure of the expert network is a feedforward neural network, each expert network represents a model skilled in one field, and a total of five fields including lacrimal sac abnormalities, nasolacrimal duct abnormalities, nasal cavity abnormalities, eye abnormalities, and conjunctival sac abnormalities are included. The five expert networks do not share weights, and the number of tokens of the feature f is input into the expert model through the component value calculation of the probability array P, and the expert model outputs the feature f i,4 , wherein the token length input into the i-th expert model is round(len(f)×p i ), wherein len(f) represents the length of the feature f, p i is the i-th component of the probability array P, and round is the rounding integer.

5. The method of claim 2, wherein, To ensure that all expert networks remain load balanced during training and avoid overtraining a few of them, an expert routing balanced loss function L = L equ +L c is proposed. wherein batch_num represents the number of samples in each batch in the training process, is the output of the jth sample in the ith expert model, and β is the weight coefficient of the loss. where p i is the probability of inputting the i-th expert model, var is the variance, and mean is the mean.

6. An electronic device, comprising: It comprises: A memory for storing executable instructions; A processor for executing the executable instructions stored in the memory to implement the lacrimal passage segmentation and report intelligent writing method for lacrimal passage CT according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, Executable instructions are stored, and the executable instructions are executed by the processor to implement the lacrimal passage segmentation and report intelligent writing method for lacrimal passage CT according to any one of claims 1 to 5.

Images