A laryngoscope image multi-attribute classification method based on multi-modal information fusion

Abstract

The present application belongs to the field of medical image classification, and particularly relates to a laryngoscope image multi-attribute classification method based on multi-modal information fusion. The method of the present application comprises a text feature extraction module, an image feature extraction module, an inline feature mining module and an inverse mapping output module. First, a paired laryngoscope image and report are used to establish a multi-modal data set of one's own, and the target attribute label is automatically obtained from the diagnosis report through the text feature extraction module; then, the feature representation of the image is obtained through the image feature extraction module; thereafter, the inline feature mining module is proposed, which fuses the text features and image features, and mines the internal correlation of each attribute, thereby realizing automatic classification of diseases; finally, the inverse mapping output module realizes automatic generation of a report containing multiple representation attributes and a final diagnosis conclusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image classification, and particularly relates to a laryngoscope image multi-attribute classification method based on multi-modal information fusion. BACKGROUND

[0002] As a non-invasive examination, the fiber electronic nasopharyngolaryngoscope can clearly show the anatomic structure of the throat and lesions, and has been widely used in clinical practice. Common laryngoscope images include white light (WL) images and narrow band imaging (NBI) images of standard endoscopes. The laryngeal morphology presented by the laryngoscope image is an important basis for diagnosing laryngeal diseases. However, the image performance of the lesion tissue such as polyps, leukoplakia and early laryngeal cancer has great similarity under the laryngoscope, and the diagnosis mainly depends on the subjective judgment of doctors. In recent years, the intelligent medical technology combining deep learning technology with clinical medicine has attracted widespread attention and research. However, most of the current methods mainly focus on simple disease type classification using WL images. The smoothness of the mucosa of the throat, necrosis, color and the like are important bases for doctors to give a diagnosis and develop a follow-up treatment plan. However, there is currently no method for automatically judging the state of the lesion site, and a diagnosis model that can adapt to both WL and NBI images is needed to adapt to the combined examination method of the two. In addition, the existing methods usually rely on experienced doctors to manually annotate the images, which is undoubtedly a time-consuming and laborious process. If automatic image annotation, automatic disease recognition, positioning and qualitative analysis of the lesion site, and automatic generation of a diagnosis report can be achieved, it will help to improve accuracy and save labor costs. SUMMARY

[0003] To solve the above problems, the application provides a laryngoscope image multi-attribute classification and report automatic generation method, which can automatically obtain multiple attribute labels of the lesion site in the image through text mapping, realize multi-modal feature extraction of text and image, feature internal information mining, and finally realize accurate laryngeal lesion recognition and automatic generation of a diagnosis report.

[0004] The application takes the intelligent classification diagnosis of laryngeal diseases as the background, and in view of the problems of time-consuming and laborious artificial annotation of images, attribute discrimination of lesion states and imbalance in the number of laryngoscope images of different lesion types, a laryngoscope image multi-attribute classification and report automatic generation method based on multi-modal information fusion is proposed. The method includes a text feature extraction module, an image feature extraction module, an inline feature mining module and an inverse mapping output module. First, a paired laryngoscope image and report are used to establish their own multi-modal data set, and the target attribute label is automatically obtained from the diagnostic report through the text feature extraction module; then, the image feature extraction module is used to obtain the feature representation of the image; then, the inline feature mining module is proposed, which fuses the text features and image features, and mines the internal correlation of each attribute, so as to realize automatic classification of diseases; finally, the inverse mapping output module realizes automatic generation of the report containing multiple attribute labels and the final diagnostic conclusion.

[0005] The technical scheme of the application is:

[0006] A laryngoscope image multi-attribute classification method based on multi-modal information fusion automatically obtains multiple attribute labels of lesion sites in images from text reports, realizes multi-modal feature extraction of text and image, feature inline information mining, finally realizes accurate laryngeal lesion recognition and multiple attribute recognition, and automatically generates a diagnostic report. The method comprises the following steps:

[0007] S1, obtaining original laryngoscope images denoted as i0=1,2,...,a,a is the number of original laryngoscope images obtained, h0, w0 is the height and width of the original laryngoscope image; the original laryngoscope image is processed to obtain a laryngoscope image with uniform size and format, denoted as x i ∈R 3×h×w , i=1,2,...a, h, w are the height and width of the original laryngoscope image after resizing;

[0008] At the same time, the text format diagnostic report corresponding to the original laryngoscope image is obtained, denoted as x t , t=1,2,...,a, wherein a represents the number of diagnostic report samples, the diagnostic report and the laryngoscope image are one-to-one corresponding matching, and the training data [x i , x t ] is obtained;

[0009] S2, constructing a neural network, including a text feature extraction module, an image feature extraction module, an inline feature mining module and an inverse mapping output module;

[0010] The input of the text feature extraction module is the diagnostic report, which is used to extract the information of interest from the diagnostic report as the multi-attribute label of the image. For x t , a text mapping function f(I,xt Inference is performed, and the inference function acts on the input x. t And the list of information of interest I, where I is the attribute information to be acquired, which consists of n medical perception results determined by the endoscopist, denoted as I = [I1, I2, I3, ..., I...]. n Output x t Matching information with I: L = f(I, x) t = [l1,l2,l3,...,l n ], where l i This indicates that the i-th piece of information of interest is related to x. t The similarity was calculated; finally, the text features of the medical perception results were obtained through the word-level embedding function E(x). Where channels represent the feature dimension;

[0011] The image feature extraction module is used to process the laryngoscope image x i Feature extraction is performed specifically using a ResNet50 network. The output of layer C5 (C1, C2, C3, C4, C5) is selected as the output of the ResNet50 network, resulting in the extracted image features. Where channels is the number of channels of the output feature, which is the same as the feature dimension channels mentioned above, and h′ and w′ are the height and width of the output feature, respectively;

[0012] The inline feature mining module consists of M enhanced Transformer encoders. It concatenates and fuses the text features output by the text feature extraction module with the image features output by the image feature extraction module. Specifically, it first reduces the dimensionality of the image features using the view function, then flattens the second and third dimensions to obtain a new dimension.

[0013]

[0014] Then obtained through splicing. As the input set for the enhanced Transformer encoder, the input set elements s are computed in the encoder through a self-attention mechanism. i ∈S and s j Attention coefficients between ∈S:

[0015]

[0016] Among them, W q and W k These are the query and key weight matrices, respectively. It is a scaling factor; s is weighted by a summation operation. i Updated to si :

[0017]

[0018] where W v is a value weight matrix, and then introduce transformation matrix W t , W f and bias vector b1, b2, and after ReLu activation function, the updated is obtained:

[0019]

[0020] Finally, through the convolution layer to enhance the learning of local information, repeat the update process M times, so as to fuse the enhanced local information with the global information of the multi-head self-attention mechanism, reduce the correlation coefficient between irrelevant attributes, and increase the correlation coefficient between relevant attributes;

[0021] The encoder outputs the learned fusion feature representation: cat = m + n, where channels is the number of channels of the fused features, and cat is the size of the fused features;

[0022] The inverse mapping output module is used to predict the category and attribute of the laryngoscope image through a multi-branch output structure, and then output the final diagnosis report through the inverse mapping function, specifically using the fusion features output by the inline feature mining module, and using multiple feedforward neural networks FFN to realize attribute prediction, which is represented as:

[0023]

[0024] where f i represents the FFN used to predict the i-th attribute and its parameters, T represents the transpose operation, n represents the total number of attributes, s i ' is an element in S'; for mutually exclusive attributes, the attribute with a larger prediction probability will be the final predicted attribute; for non-mutually exclusive attributes, a threshold th i is determined, and attributes below the threshold will be discarded; according to the predicted attribute result, a diagnosis report is generated through the inverse mapping function The diagnosis report contains the prediction results of disease diagnosis and lesion attribute state.

[0025] S3, using the training data [x i , x t ] obtained by S1 to train the neural network of S2, wherein the image feature extraction module part is pre-trained on the ImageNet dataset, and the parameters obtained through pre-training are used as the initial parameters of ResNet50, and the loss function used in training is an asymmetric loss:

[0026]

[0027] where a + , a - is a focusing parameter, y is the output probability of the network, the probability parameter z>0 is an adjustable hyperparameter, the parameters of the network are corrected through the back propagation mechanism of the neural network until the network is basically converged to obtain the trained neural network.

[0028] S4, using the trained network to perform attribute prediction on the laryngoscope image and generate a diagnosis report:

[0029] Selecting a laryngoscope image that needs to be classified, feeding it into the image feature extraction module, and obtaining the feature map through the forward feedback network to obtain the lesion category prediction and the corresponding attribute prediction about the laryngoscope image, and finally obtaining the predicted diagnosis report through the inverse mapping function.

[0030] The beneficial effects of the present application are: the present application initiatively proposes to obtain multiple attribute information of the lesion part in the image, and then uses the internal correlation between different attributes to realize more accurate classification of the disease type, and finally generates a diagnosis report based on the multiple attributes and the disease type recognition result. The method has strong practicability and reliability, and will serve as auxiliary information to reduce the burden of professional doctors. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a schematic diagram of the overall model of the network.

[0032] Figure 2 is a schematic diagram of the detailed structure of the network. DETAILED DESCRIPTION

[0033] The present application will be described in detail below with reference to the accompanying drawings.

[0034] As Figure 1 described, the present application includes the following steps:

[0035] S1: data preprocessing:

[0036] 1867 WL and NBI laryngoscope images were collected from multiple electronic laryngoscope devices in multiple medical centers, and high-quality images were randomly grabbed from laryngoscope videos of hundreds of patients, i.e. the original images with the vocal cords in an open state and approximately located in the center of the picture, clearly showing the structure and lesions of the larynx. 1867 text format laryngoscope diagnosis reports were collected, and the 1867 laryngoscope images were paired. The images and paired diagnosis reports were randomly assigned to the training set, the validation set and the test set according to the ratio of 6:2:2.

[0037] For the input laryngoscope image, denoted as i = 1, 2, ..., a, where a represents the number of image samples, i.e., 1867, and h0 and w0 are the height and width of the original laryngoscope image. Since the sizes of laryngoscope images are not uniform, the resize function is used to scale the images to a size of 640×640 to fit the input of the feature extraction network. Then, since the vocal cords are usually located in the middle of the laryngoscope image, the image is randomly cropped to 576×576 to reduce interference from edge-irrelevant regions. The final image input to the feature extraction network is represented as: x i ∈R 3×h×w , i = 1, 2, ..., n, h, w are the height and width of the laryngoscope image after resizing, which are 576 and 576 respectively.

[0038] For diagnostic reports in text format, denoted as x t ,t=1,2,...,a, where a represents the number of report samples. Before being input into the text feature extraction module, n medical perception results need to be determined by a professional endoscopist as the attribute information we need to obtain, denoted as the list of information of interest: I=[I1,I2,I3,...,I... n The key attribute information focused on in this invention includes disease category, smoothness, whether the color is dirty, whether there is necrosis, and lesion location information, determining n=13 medical perception results, namely [leukoplakia, polyp, cancer, normal, smooth, rough, dirty color, normal color, necrosis, no necrosis, left vocal cord, right vocal cord, both vocal cords].

[0039] S2: Construct a neural network, including a text feature extraction module, an image feature extraction module, an inline feature mining module, and an inverse mapping output module.

[0040] S2.1: Text Feature Extraction Module:

[0041] This module is for reports in text format. t Extract information of interest as multi-attribute labels for the image. For the input text report x... t ,like Figure 2 As shown, the text mapping function f(I,x) is used. t This involves reasoning on a report consisting of multiple sentences. The reasoning function operates on the input x. t Given a list of information of interest I, output x t Matching information with I: L = f(I, x) t = [l1,l2,l3,...,l n ]. Among them, l i This indicates that the i-th piece of information of interest is related to x. t The similarity is then determined. Next, the feature vector of the medical perception result is obtained through the word-level embedding function E(x). where channels represents the number of channels consistent with the image feature extraction module output.

[0042] S2.2: Image feature extraction module:

[0043] The image feature extraction module is to extract features from the laryngoscope image. The preprocessed laryngoscope image x i ∈R 3 ×h×w is input into the image feature extraction module to learn the feature representation of the image, as shown in Figure 2 The module is composed of ResNet50, which has 5 output layers C1, C2, C3, C4, C5, which gradually excavate deep features, and finally take the output of C5 as the output of ResNet50. Due to the problem of insufficient number of samples, the ResNet model is pre-trained on the ImageNet dataset first, and the parameters obtained by model pre-training are used as the initial parameters of ResNet50, and then the laryngoscope image data collected is used to fine-tune the parameters of ResNet50. The parameters of the 5 output layers of ResNet50 are shown in Table 1. Finally, the output of the image feature extraction module is obtained as where channels is the number of channels of the image feature extraction module output feature, which is 2048, h' and w' are the height and width of the output feature, respectively, which are 18 and 18.

[0044] Table 1 Parameters of ResNet50 output layer

[0045]

[0046] S2.3: Inline feature mining module:

[0047] The inline feature mining module is mainly composed of M enhanced Transformer encoders, which mine the relationship between different attributes. Based on the feature representation obtained by the paired input [x i ,x t ] through the feature extraction module, in order to be able to splice and fuse the text features and image features , first reduce the dimension of the image features through the view function, and flatten the second and third dimensions to obtain a new dimension, i.e.

[0048]

[0049] Through splicing, we get as the input set of the enhanced Transformer encoder. In the encoder, first calculate s i ∈S and si Attention coefficients between S:

[0050]

[0051] where W q and W k are query and key weight matrices respectively, is a scaling factor, taking d = 64. Then the weighted sum operation will update s i to

[0052]

[0053] where W v is a value weight matrix. Further, introduce transformation matrices W t , W f and bias vectors b1, b2, and after the ReLu activation function, the updated s

[0054]

[0055] Finally, a convolutional layer is used to enhance the learning of local information. Repeat the above update process M = 3 times, so as to fuse the enhanced local information with the global information of the multi-head self-attention mechanism, and the multi-head attention is 4. This process reduces the correlation coefficient between unrelated attributes, and increases the correlation coefficient between related attributes. For example, in the laryngoscope image with cancerous lesions, the features of "necrosis" and "color dirty" will be given higher attention weights. This means that these two attributes will be highlighted in the feature map, so that the module pays attention to the association between different attributes.

[0056] The learned feature representation output by the encoder: cat = m + n, where channels is the number of channels of the fused features, which is 2048, and cat is the size of the fused features, which is 337.

[0057] S2.4: Inverse mapping output module:

[0058] This module mainly predicts the category and attributes of the laryngoscope image through a multi-branch output structure, and then outputs the final diagnosis report through an inverse mapping function. First, based on the fusion learning of image features and attribute information by the inline feature mining module, multiple feedforward neural networks (FFN) are used to realize the prediction of attributes, that is, through a linear layer f, a set of attributes corresponding to the objects and attributes existing in the image are predicted:

[0059]

[0060] where f idenotes the FFN used to predict the i-th attribute and its parameters, T denotes the transpose operation, n denotes the total number of attributes, s i is an element in the feature representation S' output by the inline feature mining module.

[0061] Then, for mutually exclusive attributes, the attribute with the higher prediction probability will be the final prediction. For non-mutually exclusive attributes, a threshold th i is set to 0.5, and attributes below the threshold will be discarded.

[0062] Finally, according to the predicted attribute results, the inverse mapping function is used to generate a diagnostic report containing the disease diagnosis result and the four lesion attributes (smoothness, color contamination, necrosis, and location). For example, assuming represents an attribute that has necrosis, if then the text description "the lesion site of this laryngoscope image shows signs of necrosis" is obtained.

[0063] S3: Train the constructed neural network:

[0064] The input data is the paired diagnostic report and laryngoscope image [x i ,x t ], the pre-trained weights of the network model are loaded, and the model is fine-tuned using laryngoscope data. Due to the imbalance in the number of image categories, the loss function uses asymmetric loss (Asymmetric Loss):

[0065]

[0066] where a + , a - are focus parameters, y is the output probability of the network, and the probability parameter z > 0 is an adjustable hyperparameter. Through the backpropagation mechanism of the neural network, the parameters of the network are corrected until the network converges. The experimental parameter settings during training are shown in Table 2.

[0067] Table 2: Experimental parameter settings during training

[0068]

[0069] S4: Attribute prediction and diagnostic report generation for laryngoscope images using the trained network:

[0070] Select the laryngoscope image to be tested and input it into the image feature extraction module. The obtained feature map is fed into the forward feedback network to obtain the lesion category prediction and the corresponding attribute prediction for the laryngoscope image. Finally, the inverse mapping function is used to obtain the predicted diagnostic report. The performance of the test set on the model is shown in Table 3.

[0071] Table 3 mAP, sensitivity, specificity, report accuracy metrics on test set

[0072]

Claims

1. A laryngoscope image multi-attribute classification method based on multi-modal information fusion, characterized in that, The method comprises the following steps: S1, obtaining an original laryngoscope image is denoted as , is the number of original laryngoscope images obtained, , is the height and width of the original laryngoscope image; the original laryngoscope image is processed to obtain a laryngoscope image with uniform size and format, denoted as , , is the height and width of the original laryngoscope image after resizing; Meanwhile, a text format diagnosis report corresponding to the original laryngoscope image is acquired, denoted as Wherein, a represents the number of diagnosis report samples, the diagnosis report and the laryngoscope image are one-to-one matched, and the training data is obtained ; S2, constructing a neural network, comprising a text feature extraction module, an image feature extraction module, an inline feature mining module, and an inverse mapping output module; The input of the text feature extraction module is the diagnosis report, which is used to extract the information of interest from the diagnosis report as the multi-attribute label of the image, and the information of interest is represented as , and the inference function is used for inference , the inference function acts on the input and the list of information of interest I, , n medical perception results determined by the endoscope physician are used as the attribute information to be obtained, represented as , the output matches the information of I: , wherein represents the similarity between the i-th information of interest and ; finally, the text features of the medical perception results are obtained through the word-level embedding function E(x) : , , wherein channels represent the feature dimension; The image feature extraction module is used for laryngoscope images. Feature extraction is performed using a ResNet50 network, specifically from the five output layers of the ResNet50 network. Selected from The output of is used as the output of ResNet50 to obtain the extracted image features. Here, channels is the number of output feature channels, which is the same as the feature dimension channels mentioned above. and These are the height and width of the output feature, respectively; The inline feature mining module is composed of M enhanced Transformer encoders, which splice and fuse the text features output by the text feature extraction module and the image features output by the image feature extraction module, specifically by first reducing the image features through a view function, flattening the second and third dimensions to obtain a new dimension, i.e. ， Then by concatenation we obtain As an input set for an enhanced Transformer encoder, in the encoder, the attention coefficients between the elements of the input set are computed by a self-attention mechanism: and ​ ， where and are the query and key weight matrices, respectively, is a scaling factor; and is updated by a weighted sum operation : ， wherein is the value weight matrix, and introducing a transformation matrix , and a bias vector , , through the ReLu activation function to obtain the updated ， Finally, the local information is enhanced through a convolution layer to enhance the learning of local information, and the updating process is repeated M times to fuse the enhanced local information with the global information of the multi-head self-attention mechanism, reduce the correlation coefficient between irrelevant attributes, and increase the correlation coefficient between relevant attributes. The encoder outputs the learned fused feature representation: where channels is the number of channels of the fused feature, and cat is the size of the fused feature. The inverse mapping output module is used to predict the category and attributes of the laryngoscope image through a multi-branch output structure, and then output the final diagnosis report through an inverse mapping function, specifically by using the fused features output by the inline feature mining module and using multiple feedforward neural networks FFN to realize attribute prediction, represented as: ， wherein, represents the FFN used to predict the i-th attribute and its parameters, T represents the transpose operation, and n represents the total number of attributes, is the element in ; for mutually exclusive attributes, the attribute with the greater predicted probability will be the final predicted attribute; for non-mutually exclusive attributes, a threshold value is determined, and attributes below the threshold value will be discarded, and a diagnostic report is generated by the inverse mapping function based on the predicted attribute results, and the diagnostic report contains the prediction results of the lesion attribute states; S3, training data obtained by S1 The neural network constructed in S2 is trained, wherein the image feature extraction module part is pre-trained on the ImageNet dataset, the parameters obtained by the pre-training are used as the initial parameters of ResNet50, and the loss function used in the training is an asymmetric loss: ， wherein, , is a focusing parameter, y is the output probability of the network, the probability parameter z>0 is an adjustable hyperparameter, and the parameters of the network are corrected through a back propagation mechanism of the neural network until the network is basically converged to obtain a trained neural network; S4, using the trained network to predict the attributes of the laryngoscope image and generate a diagnosis report: Select the laryngoscope image that needs to be classified, and send it to the image feature extraction module to obtain the feature map, which is fed back through the forward network to obtain the lesion category prediction and the corresponding attribute prediction of the laryngoscope image, and finally obtain the predicted diagnosis report through the inverse mapping function.

Images