Glaucoma optic nerve damage report generation system based on multi-modal characteristics
Through a multimodal feature-based glaucoma optic nerve damage report generation system, combining fundus photography and OCT images with personalized characteristics and using the FPN-transform network to generate reports, the accurate judgment of glaucoma optic nerve damage in the prior art is solved. The problem of low rate achieves higher diagnostic accuracy and reporting efficiency.
Patent Information
- Application Number
- CN202410973189.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-12
AI Technical Summary
The current glaucoma optic nerve damage reports have low accuracy, mainly because they rely on doctors' experience to judge and are easily affected by subjective factors, resulting in inaccurate diagnosis results.
Design a glaucoma optic nerve damage report generation system based on multimodal features. Through medical image acquisition module, feature vector acquisition module and personalized feature acquisition module, fundus photography and OCT images are preprocessed and input into the ResNet network, combining patient personality , use FPN-transform network to generate reports, avoid subjective influence and improve diagnostic accuracy.
It realizes the deep fusion of images and personalized data, captures richer diagnostic information, improves the accuracy of glaucoma's optic nerve damage diagnosis results and report generation efficiency, and the generated reports are highly complete, referenceable and credible.
Smart Images

Figure CN118942613A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical report generation, and in particular to a glaucomatous optic nerve damage report generation system based on multimodal features. Background Art
[0002] Glaucoma is a relatively common disease in ophthalmology, mainly caused by pathological increase in intraocular pressure. This leads to impaired visual function. Glaucoma can be divided into primary glaucoma, secondary glaucoma and developmental glaucoma. There are no obvious symptoms in the early stage of glaucomatous optic neuropathy, but the visual function damage in the late stage is irreversible, so early screening of glaucoma has become the focus of this field.
[0003] Currently, the screening method for glaucoma is mainly through medical imaging. Medical imaging plays an important role in glaucoma examination, among which fundus photography and OCT images (optical coherence tomography images) are the most widely used. However, the current judgment of glaucomatous optic nerve damage through medical imaging is mainly based on the doctor's experience to generate a medical report. However, this method requires a large number of experienced glaucoma experts or well-trained technicians. It is not only time-consuming and labor-intensive, but the judgment results are also easily affected by subjective factors, which ultimately leads to the problem of low accuracy in the judgment of glaucomatous optic nerve damage in the glaucomatous optic nerve damage report. Summary of the invention
[0004] The purpose of the present invention is to solve the problem of low accuracy in judging glaucomatous optic nerve damage in existing glaucomatous optic nerve damage reports, and propose a glaucomatous optic nerve damage report generation system based on multimodal features.
[0005] A glaucomatous optic nerve damage report generation system based on multimodal features, comprising: a medical image acquisition module, a feature vector acquisition module, a personalized feature acquisition module, and a glaucomatous optic nerve damage report generation module;
[0006] The medical image acquisition module is used to acquire the medical image to be diagnosed, preprocess the medical image to be diagnosed, and input the preprocessed medical image to be diagnosed into the feature vector acquisition module;
[0007] The medical images to be diagnosed include: fundus photography and fundus OCT scanning images;
[0008] The feature vector acquisition module is used to input the preprocessed medical image to be diagnosed into the trained ResNet network to obtain the feature vector of the medical image to be diagnosed;
[0009] The personalized feature acquisition module is used to acquire the personalized features of the patient to be diagnosed, and input the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation module;
[0010] The personalized characteristics of the patient to be diagnosed include: gender, age, and family medical history;
[0011] The glaucomatous optic nerve damage report generation module inputs the feature vector of the medical image to be diagnosed and the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation network to obtain the glaucomatous optic nerve damage report.
[0012] Furthermore, the medical image to be diagnosed is preprocessed, specifically:
[0013] The medical images to be diagnosed are adjusted to a uniform size, and then the fundus photography is downsampled by 50% of the original resolution, and the fundus OCT scanning images are downsampled by 25% of the original resolution.
[0014] Furthermore, the trained ResNet network is obtained by:
[0015] A1. The dataset is composed of glaucoma medical images and corresponding medical reports:
[0016] The glaucoma medical images include: fundus OCT scanning images at different depths, fundus photography at different depths;
[0017] A2. Preprocess the data set and use the preprocessed data set to obtain a first training set;
[0018] A3. Use the first training set to train the ResNet network to obtain a trained ResNet network.
[0019] Furthermore, the step A2 of preprocessing the data set and using the preprocessed data set to obtain the first training set is specifically:
[0020] First, the glaucoma medical images and medical reports that did not correspond to the dataset were deleted;
[0021] Then, the image feature description and the text diagnosis result in the medical report are divided, and the image feature description and the text diagnosis result in the medical report are respectively converted into vector representations to obtain an image feature description vector and a text diagnosis result vector;
[0022] The image feature description includes: ONH analysis results, RNFL thickness, and blood flow density;
[0023] The ONH analysis results include: optic cup volume CV, optic disc area DA, disc rim area RA, cup-disc area ratio CDAR, cup-disc vertical ratio CDVR, cup-disc horizontal ratio CDHR;
[0024] The text diagnosis results include: glaucoma diagnosis results, glaucoma staging and classification diagnosis results;
[0025] Then, the glaucoma medical images are adjusted to a uniform size, the fundus photography at different depths in the data set is downsampled by 50% of the original resolution, and the fundus OCT scanning images at different depths are downsampled by 25% of the original resolution, so as to obtain the preprocessed data set;
[0026] Finally, the image feature description vector in the medical report is used as the preprocessed glaucoma medical image label, and the preprocessed glaucoma medical image and the corresponding label form the first training set.
[0027] Furthermore, the glaucoma optic nerve damage report generation network is specifically:
[0028] B1. Obtain the personality characteristics of the patient corresponding to the medical report;
[0029] The patient's personality characteristics include: gender, age, family history;
[0030] B2, using the patient's personality characteristics, the image feature description vector in the medical report and the corresponding text diagnosis result vector form a second training set;
[0031] B3. Use the second training set to train the FPN-transform network, and use the trained FPN-transform network as a glaucoma optic nerve damage report generation network.
[0032] Furthermore, the FPN-transform network includes: a comprehensive feature vector acquisition subnetwork and a Transformer subnetwork;
[0033] The comprehensive feature vector acquisition subnetwork includes: an FPN unit, a personalized feature acquisition unit, and a comprehensive feature fusion unit;
[0034] The FPN unit is used to fuse the image feature description vectors in the medical report to obtain the fused image feature description vectors;
[0035] The personalized feature acquisition unit includes: a first embedding layer and a first linear layer;
[0036] The first embedding layer is used to obtain the individual characteristics of the patient after discretization;
[0037] The first linear layer is used to convert the discretized patient personality characteristics into a continuous personality characteristic vector;
[0038] The comprehensive feature fusion unit includes: a feature connection layer and a second linear layer;
[0039] The feature connection layer is used to connect the fused image feature description vector and the continuous individual feature vector to obtain a connection result;
[0040] The second linear layer is used to convert the connection result into a comprehensive feature vector;
[0041] The Transformer sub-network includes: an encoder and a decoder;
[0042] The encoder is an xLSTM network; the decoder is a Transformer decoder;
[0043] The encoder is used to encode the comprehensive feature vector to obtain the encoded comprehensive feature vector;
[0044] The decoder is used to decode the encoded comprehensive feature vector to obtain a text diagnosis result vector.
[0045] Furthermore, the xLSTM network includes: an sLSTM memory structure, an mLSTM memory structure, and a vector concatenation structure;
[0046] The input feature of the sLSTM memory structure is a comprehensive feature vector;
[0047] The input feature of the mLSTM memory structure is a comprehensive feature vector;
[0048] The vector concatenation structure is used to concatenate the output of the sLSTM memory structure with the output of the mLSTM memory structure. Further, the second training is used to train the FPN-transform network using the following loss function:
[0049] L=αL CE +βL MT +γL BLEU +δL Metrica
[0050]
[0051] Among them, L is the loss function of the FPN-transform network, α is the weight of the cross entropy loss function, β is the weight of the academic term loss function, γ is the weight of the structured loss function, δ is the weight of the medical indicator loss function, and L CE is the cross entropy loss function, L MT is the academic term loss function, L BLEU is the structured loss function, L Metrica is the medical index loss function, N is the number of glaucoma medical images, j is the label of glaucoma medical images, P(y j,t |y j,<t ) is to predict the tth word as y given t-1 words j,t The probability of ω is the medical terminology, V medical is a glossary of medical terms, |Vmedical | is the total number of words in the medical term vocabulary, and P(ω) is the probability of the medical term labeled ω in the text diagnosis result.
[0052] Furthermore, the structured loss function is specifically:
[0053]
[0054] in, Generate a glaucoma medical report BLEU score between the real glaucoma medical report y.
[0055] Furthermore, the medical indicator loss function L Metrica , specifically:
[0056]
[0057] in, is the true result of the optic cup volume corresponding to the jth glaucoma medical image, is the optic cup volume prediction result corresponding to the jth glaucoma medical image, is the true result of the optic disc area corresponding to the jth glaucoma medical image, is the optic disc area prediction result corresponding to the jth glaucoma medical image, is the true result of the disc edge area corresponding to the jth glaucoma medical image, is the disc edge area prediction result corresponding to the jth glaucoma medical image, is the true result of the cup-to-disc area ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc area ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-to-disc vertical ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc vertical ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-disc level ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the prediction result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the true result of blood flow density corresponding to the jth glaucoma medical image, is the blood flow density prediction result corresponding to the jth glaucoma medical image.
[0058] The beneficial effects of the present invention are:
[0059] The present invention combines the medical images related to glaucomatous optic nerve damage with the patient's personalized data, achieving a deep fusion of images and personalized data. The present invention can capture richer diagnostic information by extracting features of glaucoma medical images of different resolutions. The present invention uses Resnet network, xLSTM encoder, and Transformer decoder to process medical images of glaucomatous optic nerve damage and finally automatically generates medical images of glaucomatous optic nerve damage, avoiding the subjective influence of human judgment of glaucomatous optic nerve damage, improving the accuracy of the diagnosis results of glaucomatous optic nerve damage, and improving the efficiency of generating medical reports on glaucomatous optic nerve damage. The medical report on glaucomatous optic nerve damage generated by the present invention has the characteristics of high integrity, high reference value, and high credibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 This is a module diagram of the present invention. DETAILED DESCRIPTION
[0061] Specific implementation method 1: Figure 1 As shown, this embodiment is a glaucomatous optic nerve damage report generation system based on multimodal features, comprising: a medical image acquisition module, a feature vector acquisition module, a personalized feature acquisition module, and a glaucomatous optic nerve damage report generation module;
[0062] The medical image acquisition module is used to acquire the medical image to be diagnosed, preprocess the medical image to be diagnosed, and input the preprocessed medical image to be diagnosed into the feature vector acquisition module;
[0063] The medical images to be diagnosed include: fundus photography and fundus OCT scanning images;
[0064] The feature vector acquisition module is used to input the preprocessed medical image to be diagnosed into the trained ResNet network to obtain the feature vector of the medical image to be diagnosed, and input the feature vector of the medical image to be diagnosed into the glaucomatous optic nerve damage report generation module;
[0065] The personalized feature acquisition module is used to acquire the personalized features of the patient to be diagnosed, and input the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation module;
[0066] The personalized characteristics of the patient to be diagnosed include: gender, age, and family medical history;
[0067] The glaucomatous optic nerve damage report generation module inputs the feature vector of the medical image to be diagnosed and the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation network to obtain the glaucomatous optic nerve damage report.
[0068] Specific implementation method 2: The preprocessing of the medical image to be diagnosed is specifically as follows:
[0069] The medical images to be diagnosed are adjusted to a uniform size of 224×224, and the fundus photography is downsampled by 50% of the original resolution, and the fundus OCT scanning image is downsampled by 25% of the original resolution to obtain the preprocessed medical images to be diagnosed.
[0070] Specific implementation method three: The trained ResNet network is obtained by the following method:
[0071] A1. Obtain glaucoma medical images and corresponding medical reports to form a data set;
[0072] The glaucoma medical images include: fundus OCT scanning (Optical Coherence Tomography) images at different depths (layers), fundus photography (nerve / vascular images) at different depths;
[0073] A2. Preprocess the data set to obtain a preprocessed data set, and use the preprocessed data set to obtain the first training set:
[0074] First, the glaucoma medical images and medical reports that did not have corresponding consistency in the dataset were deleted;
[0075] Then, the image feature description and text diagnosis results in the medical report are divided, and the image feature description and text diagnosis results in the medical report are respectively converted into vector representations through an embedding process;
[0076] The image feature description includes: ONH analysis results, RNFL thickness (nerve fiber layer thickness RNFLT), blood flow density BFD;
[0077] The ONH analysis results include: optic cup volume CV, optic disc area DA, disc rim area RA, cup-disc area ratio CDAR, cup-disc vertical ratio CDVR, cup-disc horizontal ratio CDHR;
[0078] The text diagnosis results include: whether glaucoma exists, glaucoma stage and type;
[0079] Then, the glaucoma medical images were adjusted to a uniform size of 224×224, and the fundus photography was downsampled by 50% of the original resolution, and the fundus OCT scanning images were downsampled by 25% of the original resolution to obtain the preprocessed glaucoma medical images;
[0080] Finally, the image feature description vector in the medical report is used as the preprocessed glaucoma medical image label, and the preprocessed glaucoma medical image and the corresponding label form the first training set;
[0081] A3. Use the first training set to train the ResNet network to obtain a trained ResNet network.
[0082] The ResNet network includes: an input submodule, a convolution submodule, a pooling submodule, a residual submodule, a dynamic attention submodule, and a global average pooling submodule;
[0083] The input submodule is used to input glaucoma medical images; the size of each input glaucoma medical image is 224×224×3;
[0084] The convolution submodule Conv1 is a convolution layer with a convolution kernel size of 7×7, a stride of 2, and an output channel number of 64, which is used to obtain a preliminary feature map of a glaucoma medical image; the size of the preliminary feature map of the glaucoma medical image output by the convolution submodule is 112×112×64;
[0085] Conv1=Conv2d(7×7,64, stride=2)
[0086] The pooling submodule is a maximum pooling layer MaxPool with a pooling kernel size of 3×3 and a stride of 2, which is used to reduce the resolution of the preliminary feature map of the glaucoma medical image and reduce the size of the preliminary feature map of the glaucoma medical image; the output feature map size of the pooling submodule is 56×56×64;
[0087] MaxPool=MaxPool2d(3×3, stride=2)
[0088] The residual submodule is used to obtain high-level features of the glaucoma medical image according to the feature map output by the pooling submodule;
[0089] The residual submodule includes four groups of residual blocks, including: a first group of residual blocks, a second group of residual blocks, a third group of residual blocks, and a fourth group of residual blocks; the output of the first group of residual blocks is the input of the second group of residual blocks, the output of the second group of residual blocks is the input of the third group of residual blocks, and the output of the third group of residual blocks is the input of the fourth group of residual blocks;
[0090] The first group of residual blocks Conv2_x includes a plurality of first residual blocks Conv2_Block1;
[0091] The output feature map size of the first group of residual blocks Conv2_x is 56×56×64;
[0092] The first residual block Conv2_Block1 includes: a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer;
[0093] The convolution kernel size of the first convolution layer and the second convolution layer is 3×3, and the number of output channels is 64;
[0094] The second group of residual blocks Conv3_x includes a plurality of second residual blocks Conv2_Block2; the output feature map size of the second residual block is 28×28×128;
[0095] The second residual block Conv2_Block2 includes: a third convolutional layer, a third BN layer, a third ReLU layer, a fourth convolutional layer, a fourth BN layer, and a fourth ReLU layer;
[0096] The convolution kernel size of the third convolution layer and the fourth convolution layer is 3×3, and the number of output channels is 128;
[0097] The third group of residual blocks Conv4_x includes a plurality of third residual blocks Conv2_Block3;
[0098] The output feature map size of the third group of residual blocks Conv4_x is 14×14×256;
[0099] The third residual block Conv2_Block3 includes: a fifth convolutional layer, a fifth BN layer, a fifth ReLU layer, a sixth convolutional layer, a sixth BN layer, and a sixth ReLU layer;
[0100] The convolution kernel size of the fifth convolution layer and the sixth convolution layer is 3×3, and the number of output channels is 256;
[0101] The fourth group of residual blocks Conv5_x includes a plurality of fourth residual blocks Conv2_Block4;
[0102] The output feature map size of the fourth group of residual blocks Conv5_x is 7×7×512;
[0103] The fourth residual block Conv2_Block4 includes: a seventh convolutional layer, a seventh BN layer, a seventh ReLU layer, an eighth convolutional layer, an eighth BN layer, and an eighth ReLU layer;
[0104] The convolution kernel size of the seventh convolution layer and the eighth convolution layer is 3×3, and the number of output channels is 512;
[0105] The dynamic attention submodule is used to obtain an image feature description vector in a glaucoma report based on high-level features of glaucoma medical images;
[0106] The dynamic attention submodule includes: a feature flattening layer, an attention weight acquisition layer, and a context vector generation layer;
[0107] The feature flattening layer is used to flatten the high-level features of the glaucoma medical image into a one-dimensional vector FlattenedFeature, specifically: Flattened Feature = Flattened (high-level features of the glaucoma medical image)
[0108] The attention weight acquisition layer includes: a first fully connected layer FullyConnected, a first softmax function, and the attention weight acquisition layer uses the one-dimensional vector Flattened Feature obtained by the feature flattening layer to obtain the attention weights Attention Weights, specifically:
[0109] Attention Weights=Softmax(FullyConnected(Flattened Feature))
[0110] The context vector generation layer is used to obtain the context vector Context Vector according to the attention weights Attention Weights and the one-dimensional vector Flattened Feature, specifically:
[0111] Context Vector=∑(Attention Weights?Feature Map)
[0112] The context vector Context Vector dimension is 512 dimensions;
[0113] The global average pooling submodule is used to perform global average pooling on the context vector, reduce the size of each feature map to 1*1, and obtain the image feature description vector in the medical report, specifically:
[0114] The image feature description vector in the medical report = GlobalAveragePooling2D (Context Vector).
[0115] Specific implementation method 4: The glaucoma optic nerve damage report generation network is specifically:
[0116] B1. Obtain the personality characteristics of the patient corresponding to the medical report;
[0117] The personality characteristics include: gender, age, family medical history;
[0118] B2, using the patient's personality characteristics, the image feature description vector in the medical report and the corresponding text diagnosis result vector to form a second training set;
[0119] B3, using the second training set to train the FPN-transform network, and using the trained FPN-transform network as a glaucoma optic nerve damage report generation network;
[0120] The FPN-Transformer-CNN network includes: a comprehensive feature vector acquisition subnetwork and a Transformer subnetwork;
[0121] The comprehensive feature vector acquisition subnetwork includes: an FPN unit, a personalized feature acquisition unit, and a comprehensive feature fusion unit;
[0122] The FPN layer is a feature pyramid, which is used to fuse the image feature description vectors in the medical report to obtain the fused image feature vectors;
[0123] The personalized feature acquisition unit includes: a first embedding layer and a first linear layer;
[0124] The first embedding layer is used to obtain discretized personality features;
[0125] The first linear layer is used to convert the discrete personality features into a continuous personality feature vector;
[0126] The comprehensive feature fusion unit includes: a feature connection layer and a second linear layer;
[0127] The feature connection layer is used to connect the fused image feature vector and the continuous individual feature vector to obtain a connection result;
[0128] The second linear layer is used to adjust the dimension of the connection result to obtain the features adapted to the Transformer sub-network, i.e., the comprehensive feature vector;
[0129] The Transformer sub-network includes: an encoder and a decoder;
[0130] The encoder is an xLSTM network; the decoder is a Transformer decoder;
[0131] The encoder is used to encode the comprehensive feature vector to obtain the encoded comprehensive feature vector;
[0132] The encoder includes: an sLSTM memory structure, an mLSTM memory structure, and a vector splicing structure;
[0133] The input feature of the sLSTM memory structure is a comprehensive feature vector;
[0134] The vector concatenation structure is used to concatenate the sLSTM memory structure output vector and the mLSTM memory structure output vector to obtain the encoded comprehensive feature vector Feature Vector;
[0135] The decoder is used to decode the encoded comprehensive feature vector to obtain a text diagnosis result.
[0136] Specific implementation method five: the sLSTM memory structure includes: an input gate layer, a forget gate layer, a stabilization layer, an output gate layer, a new information layer, a memory state update layer, a normalized state update layer, and a hidden state update layer;
[0137] The input gate layer is used to combine the current comprehensive feature vector and the hidden state of the previous time step. After linear transformation of the weight matrix and the bias vector, the exponential activation function is applied to obtain the value of the input gate, thereby determining which information is added to the memory unit. Specifically:
[0138] i t =σ(W i x t +U i h t-1 +b i )
[0139] Among them, i t is the input gate value, W i is the input gate weight matrix, b i is the input gate bias term, x t is the comprehensive eigenvector, h t-1 is the hidden state of the previous time step, U i is the input gate recursive weight matrix;
[0140] Among them, the initial value of the hidden state h0=0, the initial value of the memory state c0=0, and the initial value of the normalized state n0=0;
[0141] The forget gate layer combines the current comprehensive feature vector and the hidden state of the previous time step, and after linear transformation of the weight matrix and the bias vector, applies the exponential activation function to obtain the value of the forget gate, thereby determining which information is forgotten, specifically:
[0142] f t =σ(W f x t +U f h t-1 +b f )
[0143] W f ~N(0,σ' 2 )
[0144] U f~N(0,σ' 2 )
[0145] Among them, f t is the forget gate value, W f is the forget gate weight matrix, b f is the bias term of the forget gate, x t is the comprehensive eigenvector, h t-1 is the hidden state of the previous time step, σ' 2 is a constant related to the dimensions of the input and hidden state, U f Input forget gate recursive weight matrix;
[0146] The stabilization layer is used to stabilize the output gate value and the forget gate value to obtain the stabilized input gate value i t ′ and the forget gate value f t ′:
[0147] i t ′=exp(log(i t )-m t )
[0148] f t ′=exp(log(f t )+m t-1 -m t )
[0149] m t =max(log(f t )+m t-1 ,log(i t ))
[0150] Among them, m t is the stabilized state of the current time step, m t-1 It is the stabilized state of the previous time step, and the initial value of the stabilized state is m0 = 0;
[0151] The output gate layer is used to determine which information is output based on the current input and the hidden state of the previous step:
[0152] o t =σ(W o x t +U o h t-1 +b o )
[0153] Among them, t is the output gate value of the current time step, W o is the output gate weight, U o is the output gate recursive weight matrix, b o is the output gate bias term;
[0154] The new information layer is used to generate candidate memory information:
[0155]
[0156] in, is the candidate memory information, W c is the weight of the new information layer, U c is the recursive weight matrix of the new information layer, b c is the recursive weight bias of the new information layer;
[0157] The memory state update layer is used to update the memory state by combining the memory state of the previous time step, the input gate and forget gate values of the current time step, and the candidate memory information:
[0158]
[0159] Among them, c t is the updated memory state, c t-1 is the memory state of the previous time step;
[0160] The normalized state update layer is used to normalize the updated memory state to obtain a normalized state, specifically:
[0161] n t =f′ t ⊙n t-1 +i′ t
[0162] Among them, n t is the normalized state of the current time step, n t-1 is the normalized state of the previous time step;
[0163] The hidden state update layer uses the updated memory state and normalized state to obtain the hidden state of the current time step:
[0164]
[0165] In this implementation, the stochastic gradient descent method is used to update the weights:
[0166]
[0167] Where η is the learning rate.
[0168] Specific implementation method six: The mLSTM memory structure includes: an input gate layer, a forget gate layer, a stabilization layer, an output gate layer, a new information layer, a memory unit state update layer, and a hidden state update layer;
[0169] The input gate layer is used to combine the current comprehensive feature vector and the hidden state of the previous time step, and after linear transformation of the weight matrix and the bias vector, an exponential activation function is applied to obtain the value of the input gate, thereby determining which information is added to the matrix memory unit;
[0170] I t =σ(W I x t +U I H t-1 +b I )
[0171] Among them, I t is the input gate value, W I is the input gate weight matrix, b I is the input gate bias term, x t is the comprehensive eigenvector, H t-1 is the hidden state of the previous time step, U I is the input gate recursive weight matrix;
[0172] Among them, the initial value of the hidden state matrix H0=0 and the initial value of the matrix memory state C0=0;
[0173] The forget gate layer combines the current comprehensive feature vector and the hidden state of the previous time step to determine which information is forgotten:
[0174] F t =σ(W F x t +U F H t-1 +b F )
[0175] Among them, F t is the forget gate value, W F is the forget gate weight matrix, b F is the bias term of the forget gate, U F is the recursive weight matrix of the input forget gate;
[0176] The stabilization layer is used to obtain the stabilized input gate value I t ′ and the forget gate value F t ′:
[0177] I t ′=exp(log(I t )-M t )
[0178] F t ′=exp(log(F t )+M t-1 -M t )
[0179] M t =max(log(F t )+M t-1 ,log(I t ))
[0180] Among them, M t is the stabilized state of the current time step, M t-1 It is the stable state of the previous time step, and the initial value of the stable state is M0 = 0;
[0181] The output gate layer is used to determine which information is output based on the current input and the hidden state of the previous step:
[0182] O t =σ(W o x t +U O H t-1 +b O )
[0183] Among them, O t is the output gate value of the current time step, W O is the output gate weight, U O is the output gate recursive weight matrix, b O is the output gate bias term;
[0184] The new information layer is used to generate candidate memory information:
[0185]
[0186] in, is the candidate memory information, W C is the weight of the new information layer, U C is the recursive weight matrix of the new information layer, b C is the recursive weight bias of the new information layer;
[0187] The memory unit state update layer is used to update the memory unit state by combining the memory state of the previous time step, the input gate and forget gate values of the current time step, and the candidate memory information:
[0188]
[0189] Among them, C t is the updated memory cell state, C t-1 is the memory cell state at the previous time step;
[0190] The hidden state update layer uses the updated memory unit state and normalized state to obtain the hidden state of the current time step:
[0191] H t =Ot ⊙tanh(C t )
[0192] Specific implementation method seven: The Transformer decoder includes: an input embedding layer, a position encoding layer, a masked multi-head self-attention layer, a feature fusion attention layer, a feedforward network layer, and an output layer;
[0193] The input embedding layer is used to convert the encoded comprehensive feature vector into a comprehensive feature embedding vector:
[0194] Embedding(y t )=W e y t +b e
[0195] Among them, Embedding(y t ) is the comprehensive feature embedding vector, y t is the word in the text diagnosis result sequence generated at the current time step, W e is the embedding matrix, b e is the embedding bias;
[0196] The position encoding layer is used to convert the comprehensive feature embedding vector into an embedding vector Input(y t ), specifically:
[0197] Input(y t )=Embedding(y t )+PE(pos)
[0198]
[0199] Among them, pos is the position index, i * is the dimension index, d modle is the dimension of the comprehensive feature embedding vector, PE(pos) is the position encoding matrix, PE(pos,2i * ) is the element of the even-numbered column of the position encoding matrix pos, PE(pos,2i * +1) is the element of the odd column of the position encoding matrix pos,
[0200] The masked multi-head self-attention layer uses the embedding vector Input(y t )Get the attention matrix composed of the attention representation of each position in the text diagnosis result sequence:
[0201] Z=Attetion(Q,K,V)*V
[0202]
[0203] Q=Input(y t )W Q
[0204] K=Input(y t )W K
[0205] V=Input(y t )W V
[0206] Among them, d k is the key vector dimension, M is the masking matrix, W Q is the query weight matrix of the attention head, W K is the attention head key weight matrix, W Q is the attention head value weight matrix, Q is the query value of the attention head, K is the key of the attention head, V is the value of the attention head, and Z is the attention matrix;
[0207] The feature fusion attention layer uses the attention representation of each position in the text diagnosis result sequence and the comprehensive feature vector to obtain the attention representation x of the feature fusion information:
[0208] x=Attention(Q dec ,K fusion ,V fusion )*V fusion
[0209]
[0210] Q dec =W Q *Z
[0211] K fusion =W K *Feature Vector
[0212] V fusion =W V *Feature Vector
[0213] Among them, x is the attention representation of feature fusion information, Attentlon(Q dec ,K fusion ,V fusion ) is the intermediate representation, Q dec is the decoder self-attention output (query), K fusion is the key of the comprehensive feature vector, V fusion is the value of the comprehensive feature vector, and Feature Vector is the encoded comprehensive feature vector;
[0214] The feedforward network layer FFN uses the attention representation of feature fusion information to obtain the feature FFN(x):
[0215] FFN(x)=max(0,xW1+b1)W2+b2
[0216] Among them, W1 and W2 are the weight matrices of the layers of the feedforward network, and b1 and b2 are the bias vectors of the feedforward network layers;
[0217] The output layer uses the feature FFN(x) to obtain the probability distribution of words in the vocabulary, and obtains the words with the highest probability to form the medical report text.
[0218] The probability distribution of words in the vocabulary is:
[0219] P(y t |y <t ,FeatureVector)=softmox(W3FFN(x)+b3
[0220] Among them, FeatureVector is the comprehensive feature vector, t is the time step, y <t is the sequence of text diagnosis results generated before time step t, W3 is the output layer weight matrix, and b3 is the output layer bias vector.
[0221] Specific implementation eight: The loss function of the FPN-transform network is as follows:
[0222] L=αL CE +βL MT +γL BLEU +δL Metrica
[0223]
[0224] Among them, L is the loss function of the FPN-transform network, α is the weight of the cross entropy loss function, β is the weight of the academic term loss function, γ is the weight of the structured loss function, δ is the weight of the medical indicator loss function, and L CE is the cross entropy loss function, L MT is the academic term loss function, L BLEU is the structured loss function, L Metrica is the medical indicator loss function, N is the number of glaucoma medical images, j is the glaucoma medical image label, T is the total number of time steps, P(y j,t |y j,<t ) is to obtain the report corresponding to the jth glaucoma medical image, and predict the tth word as y given t-1 words j,tω is the medical term label, P(ω) is the probability of the medical term labeled ω in the text diagnosis result, V medical is a glossary of medical terms, |V medical | is the total number of words in the medical term vocabulary, is the BLEU score between the generated glaucoma medical report y and the real glaucoma medical report y, is the true result of the optic cup volume corresponding to the jth glaucoma medical image, is the optic cup volume prediction result corresponding to the jth glaucoma medical image, is the true result of the optic disc area corresponding to the jth glaucoma medical image, is the optic disc area prediction result corresponding to the jth glaucoma medical image, is the true result of the disc edge area corresponding to the jth glaucoma medical image, is the disc edge area prediction result corresponding to the jth glaucoma medical image, is the true result of the cup-to-disc area ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc area ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-to-disc vertical ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc vertical ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-disc level ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the prediction result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the true result of blood flow density corresponding to the jth glaucoma medical image, is the blood flow density prediction result corresponding to the jth glaucoma medical image.
[0225] The cross-entropy loss in this implementation is used to measure the difference between the generated text and the real text. Medical Term Loss: Ensures that the generated report contains necessary medical terms. Structured Loss: Ensures the structural integrity of the report. Medical Metrics Loss: Ensures the accuracy of specific medical metrics included in the report.
Claims
1. A glaucomatous optic nerve damage report generation system based on multimodal features, characterized in that The system comprises: a medical image acquisition module, a feature vector acquisition module, a personalized feature acquisition module, and a glaucomatous optic nerve damage report generation module; The medical image acquisition module is used to acquire the medical image to be diagnosed, preprocess the medical image to be diagnosed, and input the preprocessed medical image to be diagnosed into the feature vector acquisition module; The medical images to be diagnosed include: fundus photography and fundus OCT scanning images; The feature vector acquisition module is used to input the preprocessed medical image to be diagnosed into the trained ResNet network to obtain the feature vector of the medical image to be diagnosed; The personalized feature acquisition module is used to acquire the personalized features of the patient to be diagnosed, and input the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation module; The personalized characteristics of the patient to be diagnosed include: gender, age, and family medical history; The glaucomatous optic nerve damage report generation module inputs the feature vector of the medical image to be diagnosed and the personalized features of the patient to be diagnosed into the glaucomatous optic nerve damage report generation network to obtain the glaucomatous optic nerve damage report.
2. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 1, characterized in that: Preprocess the medical images to be diagnosed, specifically: The medical images to be diagnosed are adjusted to a uniform size, and then the fundus photography is downsampled by 50% of the original resolution, and the fundus OCT scanning images are downsampled by 25% of the original resolution.
3. The system for generating a report of glaucomatous optic nerve damage based on multimodal features according to claim 2, characterized in that: The trained ResNet network is obtained in the following way: A1. The dataset is composed of glaucoma medical images and corresponding medical reports: The glaucoma medical images include: fundus OCT scanning images at different depths, fundus photography at different depths; A2. Preprocessing the data set, and using the preprocessed data set to obtain a first training set; A3. Use the first training set to train the ResNet network to obtain a trained ResNet network.
4. The system for generating a report of glaucomatous optic nerve damage based on multimodal features according to claim 3, characterized in that: The step A2 of preprocessing the data set and using the preprocessed data set to obtain the first training set is specifically as follows: First, the glaucoma medical images and medical reports that did not correspond to the dataset were deleted; Then, the image feature description and the text diagnosis result in the medical report are divided, and the image feature description and the text diagnosis result in the medical report are respectively converted into vector representations to obtain an image feature description vector and a text diagnosis result vector; The image feature description includes: ONH analysis results, RNFL thickness, and blood flow density; The ONH analysis results include: optic cup volume CV, optic disc area DA, disc rim area RA, cup-disc area ratio CDAR, cup-disc vertical ratio CDVR, cup-disc horizontal ratio CDHR; The text diagnosis results include: glaucoma diagnosis results, glaucoma staging and classification diagnosis results; Then, the glaucoma medical images are adjusted to a uniform size, the fundus photography at different depths in the data set is downsampled by 50% of the original resolution, and the fundus OCT scanning images at different depths are downsampled by 25% of the original resolution, so as to obtain the preprocessed data set; Finally, the image feature description vector in the medical report is used as the preprocessed glaucoma medical image label, and the preprocessed glaucoma medical image and the corresponding label form the first training set.
5. The system for generating a report of glaucomatous optic nerve damage based on multimodal features according to claim 4, characterized in that: The glaucoma optic nerve damage report generation network is specifically: B1. Obtain the personality characteristics of the patient corresponding to the medical report; The patient's personality characteristics include: gender, age, family history; B2, using the patient's personality characteristics, the image feature description vector in the medical report and the corresponding text diagnosis result vector form a second training set; B3. Use the second training set to train the FPN-transform network, and use the trained FPN-transform network as a glaucoma optic nerve damage report generation network.
6. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 5, characterized in that: The FPN-transform network includes: a comprehensive feature vector acquisition subnetwork and a Transformer subnetwork; The comprehensive feature vector acquisition subnetwork includes: an FPN unit, a personalized feature acquisition unit, and a comprehensive feature fusion unit; The FPN unit is used to fuse the image feature description vectors in the medical report to obtain the fused image feature description vectors; The personalized feature acquisition unit includes: a first embedding layer and a first linear layer; The first embedding layer is used to obtain the individual characteristics of the patient after discretization; The first linear layer is used to convert the discretized patient personality characteristics into a continuous personality characteristic vector; The comprehensive feature fusion unit includes: a feature connection layer and a second linear layer; The feature connection layer is used to connect the fused image feature description vector and the continuous individual feature vector to obtain a connection result; The second linear layer is used to convert the connection result into a comprehensive feature vector; The Transformer sub-network includes: an encoder and a decoder; The encoder is an xLSTM network; the decoder is a Transformer decoder; The encoder is used to encode the comprehensive feature vector to obtain the encoded comprehensive feature vector; The decoder is used to decode the encoded comprehensive feature vector to obtain a text diagnosis result vector.
7. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 6, characterized in that: The xLSTM network includes: an sLSTM memory structure, an mLSTM memory structure, and a vector splicing structure; The input feature of the sLSTM memory structure is a comprehensive feature vector; The input feature of the mLSTM memory structure is a comprehensive feature vector; The vector concatenation structure is used to concatenate the output of the sLSTM memory structure with the output of the mLSTM memory structure.
8. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 7, characterized in that: The second training is used to train the FPN-transform network using the following loss function: L=αL CE +βL MT +γL BLEU +δL Metrica Among them, L is the loss function of the FPN-transform network, α is the weight of the cross entropy loss function, β is the weight of the academic term loss function, γ is the weight of the structured loss function, δ is the weight of the medical indicator loss function, and L CE is the cross entropy loss function, L MT is the academic term loss function, L BLEU is the structured loss function, L Metrica is the medical indicator loss function, T is the total number of time steps, N is the number of glaucoma medical images, j is the glaucoma medical image label, P(y j,t |y j,<t ) is to predict the tth word as y given t-1 words j,t The probability of ω is the medical terminology, V medical is a glossary of medical terms, |V medical | is the total number of words in the medical term vocabulary, and P(ω) is the probability of the medical term labeled ω in the text diagnosis result.
9. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 8, characterized in that: The structured loss function is: in, Generate a glaucoma medical report BLEU score between the real glaucoma medical report y.
10. A glaucomatous optic nerve damage report generation system based on multimodal features according to claim 9, characterized in that: Medical indicator loss function L Metrica , specifically: in, is the true result of the optic cup volume corresponding to the jth glaucoma medical image, is the optic cup volume prediction result corresponding to the jth glaucoma medical image, is the true result of the optic disc area corresponding to the jth glaucoma medical image, is the optic disc area prediction result corresponding to the jth glaucoma medical image, is the true result of the disc edge area corresponding to the jth glaucoma medical image, is the disc edge area prediction result corresponding to the jth glaucoma medical image, is the true result of the cup-to-disc area ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc area ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-to-disc vertical ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc vertical ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the cup-disc level ratio corresponding to the j-th glaucoma medical image, is the cup-to-disc ratio prediction result corresponding to the j-th glaucoma medical image, is the true result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the prediction result of the optic nerve fiber layer thickness corresponding to the jth glaucoma medical image, is the true result of blood flow density corresponding to the jth glaucoma medical image, is the blood flow density prediction result corresponding to the jth glaucoma medical image.
Citation Information
Patent Citations
Glaucoma multi-mode intelligent identification system based on transfer learning
CN114724231A
Medical report generation method based on cross-modal comparative learning
CN114743630A
Medical examination pre-training model construction method and identification method
CN115759073A
Devices and methods employing optical-based machine learning using diffractive deep neural networks
WO2019200289A1
