Intestinal polyp identification method based on image and multi-modal data

Abstract

The invention discloses an intestinal polyp identification method based on images and multi-modal data. The method comprises the following steps: acquiring pathological images of a plurality of microscopes, objective magnification information corresponding to the pathological images, and text information corresponding to enteroscopes; performing data preprocessing on each frame of acquired pathological image and text information to obtain a training data set; a multi-scale multi-modal feature classification network is constructed, the training data set is used for training, and a trained multi-scale multi-modal feature fusion classification network is obtained; the method comprises the following steps: collecting a microscope pathological image of a patient, obtaining corresponding enteroscope text information, respectively carrying out data preprocessing, and inputting into a multi-scale multi-modal feature fusion classification network to obtain disease category prediction of a model. According to the method, pathological images and enteroscope text information are fully fused, accurate classification and prediction of disease types are finally achieved, the effect and efficiency of multi-modal diagnosis of intestinal diseases are improved, and therefore the accuracy of patient diagnosis is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human tissue pathological image analysis, and in particular to an intestinal polyp identification method based on images and multi-modal data. BACKGROUND

[0002] In the existing tissue pathological image analysis technology, the intestinal polyps can be effectively classified as benign or malignant based on pathological image features. However, most of the existing technologies only focus on the coarse-grained classification of polyps, such as simply dividing them into benign or malignant, or only identifying whether they are adenomatous polyps, which cannot meet the needs of more detailed classification of polyps in clinical practice. In the actual clinical diagnosis process, doctors need to consider the multi-dimensional feature information of the lesion and give a classification result with higher discrimination and interpretability. The existing pathological image feature analysis technology cannot achieve this analysis purpose. For example, doctors need to make judgments based on the details of the lesion morphology observed during the colonoscopy operation. SUMMARY

[0003] The present application provides an intestinal polyp identification method based on images and multi-modal data to solve the above-mentioned technical problems. The technical solution adopted is as follows:

[0004] An intestinal polyp identification method based on images and multi-modal data includes the following steps:

[0005] Obtaining pathological images of multiple microscopes, objective magnification information corresponding to the pathological images, and text information corresponding to the colonoscopy.

[0006] Data preprocessing is performed on each frame of the obtained pathological images and text information to obtain a training data set.

[0007] A multi-scale multi-modal feature classification network is constructed, and the training data set is used for training to obtain a trained multi-scale multi-modal feature fusion classification network.

[0008] The microscope pathological images of the patient are collected, the corresponding colonoscopy text information is obtained, and after data preprocessing, they are input into the multi-scale multi-modal feature fusion classification network to obtain the class prediction of the disease by the model.

[0009] Further, the method for data preprocessing of each frame of the obtained pathological images and text information includes:

[0010] The pathological images are windowed, filtered, and the features after the filtering are extracted according to different objective magnifications to obtain image features.

[0011] The text information is structured, and the structured text information is vectorized to obtain text features.

[0012] Further, the method for windowing the pathological images according to different objective lens magnifications is as follows:

[0013] According to different objective lens magnifications to which the pathological images belong, different sizes of small images are cut by the windowing method;

[0014] The resolution size of the small images is determined according to different objective lens magnifications are ;

[0015] The method for effectively screening the pathological images according to different objective lens magnifications is as follows:

[0016] The average value of the image RGB data on the red channel is calculated and the maximum value , and the ratio of the two is calculated:

[0017] ,

[0018] The effective threshold value is set , and is considered as an effective image;

[0019] The method for feature extraction after the pathological images are screened according to different objective lens magnifications is as follows:

[0020] The small images of the same scale are stacked and resampled to the specified size required by the pre-trained network, and the number of small images of each scale is recorded as:

[0021] ,

[0022] Then, any network pre-trained on the ImageNet dataset or pathological images is used for image feature extraction, and all small images are extracted into one-dimensional vectors with a length of , so that the image features of each scale are obtained as: wherein represents the image features, and the shape is represents the number of small images under the magnification of m, represents the number of neurons in the previous layer of the pre-trained network classification layer, that is, the length of the output feature vector.

[0023] Further, the method for structuring the text information is as follows:

[0024] The original text information is segmented according to the sentence level unit, and impurity content including invalid characters, special punctuation and redundant spaces is removed to form a text collection;

[0025] For each sentence of text information in the text collection, an entity extraction strategy based on rules and entity dictionaries is used to identify key information units and obtain structured text data in a unified format.

[0026] The method for vectorizing the text information after structured processing is as follows:

[0027] For category variables including lesion types and part names, one-hot encoding or learnable feature vectors are used for representation;

[0028] For numerical variables including lesion size and location distance, normalization processing is performed; for original text information, a pre-trained language model is used for encoding, the above features are spliced, and further projection is performed to fuse the above text information, thereby obtaining the text features of the pathological image of each polyp:

[0029] ,

[0030] wherein represents a feature vector obtained by projecting the spliced text features, and the shape is is the number of output neurons of the projection layer.

[0031] Further, a multi-scale multi-modal feature classification network is constructed, wherein the network comprises:

[0032] a single-scale feature fusion module for generating single-scale representative features from image features at the same magnification; a multi-modal semantic fusion module for deep fusion of image features and text features through cross-attention mechanism to obtain fusion features; and a classification module for outputting intestinal polyp categories based on the fusion features.

[0033] Further, the method for generating single-scale representative features from image features at the same magnification is as follows:

[0034] The image features are input into the single-scale feature fusion classification module to obtain multi-scale representative features;

[0035] The single-scale feature fusion classification module comprises a linear projection layer and a self-attention module.

[0036] The linear projection layer first reduces the dimension of the input feature , and outputs a feature with a dimension of , which is then input into the self-attention module to obtain the representative feature at this scale, and the calculation formula is as follows:

[0037] ,

[0038] wherein , , , i.e. the reduced image features at m scales, , are respectively learnable query, key, and value mapping matrices, let and equal; Refers to the normalized exponential function, which is used to convert the similarity score into an attention weight whose sum is 1; after weighted summation of V, the representative feature vector of scale m is obtained ;

[0039] The representative features of each scale are stacked to obtain multi-scale features:

[0040]

[0041] in Represents the vector after multi-scale feature stacking, the shape is , is the scale quantity.

[0042] Furthermore, the text features are projected and mapped into representations of the same dimension as the image features through a linear projection layer to obtain text features that are spatially aligned with the image features.

[0043] Furthermore, the multimodal semantic fusion module includes a preliminary semantic fusion module and a deep semantic fusion module;

[0044] The text features and image features are input into the multimodal semantic fusion module to obtain the fusion features as follows:

[0045] The preliminary semantic fusion module first concatenates image features and text features, and then uses a multi-layer perceptron to obtain preliminary fusion features. The formula is as follows:

[0046] ,

[0047] ,

[0048] in is the number of neurons in the last layer of MLP, indicating the length of the output vector;

[0049] The deep semantic fusion module uses a cross-attention network to obtain deep fusion features, as follows:

[0050] The text features are used as query vectors and the image features are used as key-value pairs to perform cross attention calculation. The formula is as follows:

[0051] ,

[0052] in, , , ,in respectively are learnable query, key, value mapping matrix; using After converting the similarity score into attention weights with the sum of weights being 1, the V is weighted and summed to obtain the deep fusion feature ;

[0053] Finally, the preliminary fusion feature and the deep fusion feature are spliced to obtain the fusion feature:

[0054] .

[0055] Further, the method for training the multi-scale multi-modal feature fusion classification network is as follows:

[0056] The multi-scale multi-modal feature fusion classification network is trained using the training set data divided in step S1);

[0057] After one round of training of all data, the slice prediction results of the network are verified on the validation set;

[0058] The cross-entropy of the final prediction result of the network and the slice label, and the contrastive loss of the text feature and the image feature are used as the joint loss function for verification, and the specific calculation formula is as follows:

[0059] ,

[0060] Wherein, λ is the weight hyperparameter of the contrastive loss, which is set to 0.5 in the embodiment, and The cross-entropy loss of the final prediction result of the network and the slice label, and the contrastive loss of the text feature and the image feature are respectively

[0061] ,

[0062] Wherein, C represents the number of categories, is the real label of the i-th category, which is encoded by one-hot, is the logits value output by the model;

[0063] ,

[0064] Wherein represents the calculation and Cosine similarity, wherein , is the feature vector after average pooling of M scale image features , wherein wherein, is the text vector .

[0065] Further, the method for dividing the training data set is as follows:

[0066] In a patient unit, all data is evenly divided into n parts in a class-balanced manner, 1 part is taken as a test set each time, and the remaining data is divided into a training set and a validation set in a (n-1):1 ratio, thereby obtaining n data set division methods;

[0067] The sum of the results of the n test sets or the mean of the training results on the n test sets is taken as the model result;

[0068] When the joint loss function of the validation results of the n rounds of validation training is unchanged, the training is stopped.

[0069] The beneficial effect of the present application is that the provided image and multi-modal data based intestinal polyp recognition method collects clear pathological images containing current objective magnification information from a microscope, and combines the text information of the corresponding colonoscope of the corresponding patient, respectively pre-processes to obtain multi-modal features, then constructs a multi-scale multi-modal feature classification network, multi-scale modeling is performed on the pathological images under different objective magnifications, and then the collected microscope pathological images of the patient and the corresponding colonoscope text information features are input into the multi-scale multi-modal feature fusion classification network, deep semantic interaction and fusion are realized, and the class prediction of the disease by the model is obtained. This method fully fuses pathological images and colonoscope text information, and finally realizes accurate classification and prediction of disease types, improves the effect and efficiency of multi-modal diagnosis of intestinal diseases, and improves the accuracy of diagnosis of patients. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0071] Fig. 1 is a schematic diagram of the image and multi-modal data based intestinal polyp recognition method of the present application;

[0072] Fig. 2 is a schematic diagram of the pathological image and text information data processing flow in the image and multi-modal data based intestinal polyp recognition method of the present application;

[0073] Fig. 3 is a schematic diagram of the multi-scale multi-modal feature fusion classification network structure in the image and multi-modal data based intestinal polyp recognition method of the present application. DETAILED DESCRIPTION

[0074] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but are not to be construed as limiting the present invention.

[0075] like Figs. 1 to 3 The present invention shows a method for identifying intestinal polyps based on images and multimodal data, comprising the following steps:

[0076] S1) Obtain pathological images from multiple microscopes, objective lens magnification information corresponding to the pathological images, and text information corresponding to the colonoscopy. During this step, the pathologist operates the microscope to observe the slices and simultaneously obtains clear pathological images with known current objective lens magnification information from the microscope. When the pathologist finishes the observation, he stops collecting the pathological images of the slice and obtains the true intestinal polyp classification of the slice. Each slice sample contains ≥1 images. The image pixel size obtained in this implementation is 1824*1216, and the specific image size in use is not limited to this. Images of different sizes can be obtained according to different acquisition equipment. At the same time, the microscopic descriptive text information corresponding to the patient and intestinal part is obtained. This implementation specifically divides colon polyps into six categories: adenocarcinoma, tubular adenoma, sessile serrated adenomatous polyps, hyperplastic polyps, inflammatory polyps and normal tissue.

[0077] S2) Data preprocessing is performed on each acquired pathological image and text information to obtain a training dataset. During this step, the training dataset is divided as follows: All data is equally divided into n equal parts, balanced in a class-balanced manner, with one part taken at a time as the test set. The remaining data is then divided into a training set and a validation set in a ratio of (n-1):1, resulting in n different dataset division methods. The sum of the results of the n test sets or the mean of the training results on the n test sets is used as the model result. Training is terminated when the joint loss function of the validation results remains unchanged after n rounds of validation training.

[0078] Specifically, this scheme divides all data into 10 equal parts in a class-balanced manner based on patients, takes one part each time as the test set, and divides the remaining data into training and validation sets in an 8:1 ratio. This results in ten ways of dividing the data set, and the sum of the results of the ten test sets is used as the model result.

[0079] In step S2), the method for data preprocessing of each acquired frame of pathological image and text information includes: windowing the pathological image according to different objective lens magnifications, validity screening, and feature extraction after validity screening to obtain image features; structuring the text information, and vectorizing the text information after structured processing to obtain text features.

[0080] Specifically, the method of windowing the pathological image according to different objective lens magnifications is as follows:

[0081] According to different objective lens magnifications to which the pathological image belongs, different sizes of small images are cut by using the windowing method;

[0082] The resolution size of the small image is determined according to The different objective lens magnifications are respectively .

[0083] In the specific implementation, the small image size of different objective lens magnifications can be set by the user. In the present embodiment, five objective lens magnifications, 2x, 4x, 10x, 20x and 40x, are taken as examples, and the resolution of the small image cut by each objective lens is respectively 5.59 , 2.78 , 1.39 , 0.56 , 0.28 . When cutting the image, starting from the coordinate , the window length is 453 pixels and the interval is 227 pixels, and then 28 small images can be cut from the image with a size of 1824*1216 pixels. The actual object size corresponding to the five different objective lens magnifications, 40x, 20x, 10x, 4x and 2x, is respectively 81 , 163 , 326 , 814 , 1637 .

[0084] Specifically, the method of effectively screening the pathological image according to different objective lens magnifications is as follows:

[0085] The average value of the image RGB data on the red channel is calculated , and the maximum value is calculated , and the ratio of the two is calculated:

[0086] ,

[0087] The effective threshold value is set , and is considered as an effective image.

[0088] In the present embodiment, the threshold value is taken as an example, and when the calculated of the small image is greater than 0.01, the small image is considered as an effective small image containing pathological tissue.

[0089] The method of feature extraction after effectively screening the pathological image according to different objective lens magnifications is as follows:

[0090] Stack small images of the same scale and resample them to the specified size required by the pre-trained network. The number of small images of each scale is recorded as:

[0091] ,

[0092] Then, any network pre-trained on the ImageNet dataset or pathological images is used to extract image features, and all small images are extracted into The one-dimensional vector of , thus obtaining the image features of each scale is: ,in Represents image features, the shape is Indicates the number of small images at m magnification, Indicates the number of neurons in the previous layer of the pre-trained network classification layer, that is, the length of the output feature vector.

[0093] This embodiment takes the ResNet50 pre-trained on ImageNet as an example. First, the small image needs to be resampled to the (256, 256) size required by the ResNet50 network, and then input into the ResNet50 network to obtain a one-dimensional vector of length 1024.

[0094] Specifically, the method for structuring text information is as follows:

[0095] The original text information is segmented into sentence-level units, and impurity content including invalid characters, special punctuation marks, and redundant spaces is removed to form a text collection;

[0096] For each sentence in the text collection, an entity extraction strategy based on rules and entity dictionaries is used to identify key information units and obtain structured text data in a unified format.

[0097] In this example, the full-width / half-width symbols, Chinese / English punctuation, uppercase / lowercase letters, and units (e.g., "cm" vs. "cm") in the text are first standardized. Multi-entity recognition is then performed within a sentence, using a regularization method to extract lesion entity types such as "polyp," "erosion," "protuberance," and "lump," as well as location information such as "20 cm from the anus" and "mid-sigmoid colon," and polyp size information. This structured information for each polyp is then broken down into lesion type, location, location distance, size, and the original text.

[0098] Specifically, the method for vectorizing the text information after structured processing is as follows:

[0099] For categorical variables including lesion type and site name, one-hot encoding or learnable feature vectors are used for representation;

[0100] For numerical variables including lesion size, location distance, normalization processing is performed; for original text information, a pre-trained language model is used for encoding, the above features are spliced, and further projection is performed to fuse the above text information, thereby obtaining the text features of the pathological image of each polyp:

[0101] ,

[0102] wherein represents a feature vector obtained by projecting the spliced text features, and the shape is is the number of output neurons of the projection layer.

[0103] In this embodiment, the lesion type and the position name are represented by a learnable feature vector, and the length is set to 32; the lesion size and distance are normalized to obtain a feature with a length of 1; the original text information is input into the pre-trained bert model, and the output CLS is taken as the sentence feature, thereby obtaining a sentence feature with a length of 768. The above features are spliced, and a linear projection layer is used to project the vector to fuse the above text features, thereby obtaining a text feature vector with a length of 512 .

[0104] S3) Construct a multi-scale multi-modal feature classification network, train using the training data set, and obtain the trained multi-scale multi-modal feature fusion classification network. In step S3), a multi-scale multi-modal feature classification network is constructed, wherein the network includes:

[0105] a single-scale feature fusion module for generating single-scale representative features from image features at the same magnification; a multi-modal semantic fusion module for deeply fusing image features and text features through a cross-attention mechanism to obtain fusion features; and a classification module for outputting intestinal polyp categories based on the fusion features.

[0106] As a further method of step S3), the method of generating single-scale representative features from image features at the same magnification is as follows:

[0107] The image features are input into the single-scale feature fusion classification module to obtain multi-scale representative features;

[0108] The single-scale feature fusion classification module includes a linear projection layer and a self-attention module.

[0109] The linear projection layer first reduces the dimension of the input feature , and outputs a feature with a dimension of , which is then input into the self-attention module to obtain the representative feature at this scale, and the calculation formula is as follows:

[0110] ,

[0111] wherein, , , , i.e. the image feature in m scale after dimension reduction, , are learnable query, key, value mapping matrices respectively, let be equal to ; denotes a normalized exponential function, which is used to convert the similarity score into attention weight whose sum is 1; the representative feature vector in m scale is obtained by weighted sum of V .

[0112] The representative features in each scale are stacked to obtain multi-scale features:

[0113] ,

[0114] wherein denotes the vector after stacking of multi-scale features, and the shape is . is the number of scales.

[0115] In the embodiment, it is set that , i.e. the feature with input dimension of 1024 is first reduced to 512, and then the feature fusion is performed through the self-attention module to obtain multi-scale features with feature dimension of .

[0116] As a further method of step S3), the text feature is projected, and the text feature is mapped to a representation with the same dimension as the image feature through a linear projection layer to obtain a text feature aligned with the image feature in space.

[0117] In the embodiment, the aforementioned text feature , and the image feature , so the projection layer maps the text feature to .

[0118] As a further method of step S3), the multi-modal semantic fusion module includes a preliminary semantic fusion module and a deep semantic fusion module.

[0119] The method for inputting the text feature and the image feature into the multi-modal semantic fusion module to obtain the fused feature is as follows:

[0120] The preliminary semantic fusion module first splices the image feature and the text feature, and then uses a multi-layer perceptron to obtain a preliminary fused feature, and the formula is as follows:

[0121] ,

[0122] ,

[0123] wherein is the number of neurons in the last layer of the MLP, representing the length of the output vector.

[0124] In the present embodiment, the output features .

[0125] The deep semantic fusion module uses a cross-attention network to obtain deep fusion features, as follows:

[0126] The text features are taken as query vectors, and the image features are taken as key-value pairs, and cross-attention calculation is performed, with the formula as follows:

[0127] ,

[0128] wherein , , wherein are learnable query, key, and value mapping matrices, respectively; and is used to convert the similarity scores into attention weights whose sum is 1, and then the V is weighted and summed to obtain the deep fusion features .

[0129] Finally, the preliminary fusion features and the deep fusion features are spliced to obtain the fusion features:

[0130] .

[0131] In the present embodiment, the number of fusion features is set to be , .

[0132] The fused features are classified to output the final diagnosis result. Any classification network can be used for classification, such as a support vector machine, a multilayer perceptron, etc. In the present embodiment, a multilayer perceptron is used for classification, with the input and the output of prediction scores of 6 categories.

[0133] As a further method of step S3), the method of training the multi-scale multi-modal feature fusion classification network is as follows:

[0134] The multi-scale multi-modal feature fusion classification network is trained using the training set data divided in step S1);

[0135] After one round of training of all data, the slice prediction results of the network are verified on the validation set;

[0136] The cross-entropy of the network final prediction result and the slice label, and the contrast loss of the text feature and the image feature are used as the joint loss function for verification, and the specific calculation formula is as follows:

[0137] ,

[0138] Where λ is the weight hyperparameter of the contrast loss, which is set to 0.5 in this implementation. and The cross-entropy loss of the network final prediction result and the slice label, and the contrast loss of the text feature and the image feature are respectively, and the calculation formula is as follows:

[0139] ,

[0140] Where C represents the number of categories, is the true label of the i-th category, which is encoded by one-hot, is the logits value output by the model.

[0141] ,

[0142] Where represents the calculation and of the cosine similarity, where , is the feature vector after average pooling of M scale image features , where is the text vector .

[0143] In the above steps, the training set data divided by step S1) is used to train the above-mentioned multi-scale multi-modal feature fusion classification network, the Adam optimization method is used, the learning rate is 0.0001, and the data batch size for model training is 1. After training all the data for one round, the slice prediction result of the network is verified on the verification set. In order to prevent model overfitting, the early stopping strategy is adopted, that is, if the verification result of 20 rounds does not improve, the training is stopped. Run on the ten divided data sets, and take the average of the results on the ten test sets as the model result.

[0144] S4) Collect the microscopic pathological images of patients, obtain the corresponding colonoscopy text information, and input them into the multi-scale multi-modal feature fusion classification network after data preprocessing, to obtain the class prediction of the model. In the operation process of this step, when the pathologist actually uses the microscope, the model synchronously obtains the clear pathological image and the corresponding colonoscopy text description information of the patient and the part, and outputs the model prediction structure to provide diagnosis suggestions for the doctor.

[0145] The image and multi-modal data based intestinal polyp recognition method of the scheme first collects a clear pathological image containing current objective magnification information from a microscope, and combines the corresponding patient and site text description information under the colonoscope to obtain multi-modal features after respective preprocessing, then a multi-scale multi-modal feature classification network is constructed to perform multi-scale modeling on images under different objective magnifications, and image feature fusion is completed in combination with an attention mechanism; then the image and text features are input into a multi-modal fusion module to realize deep semantic interaction and fusion. The method fully fuses the pathological image and colonoscopy text information, finally realizes accurate classification and prediction of the disease type, improves the effect of multi-modal diagnosis of intestinal diseases, and the diagnosis efficiency and diagnosis accuracy are obviously improved compared with the image processing method in the prior art.

[0146] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the above examples do not limit the present application in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the protection scope of the present application.

Claims

1. A method for identifying intestinal polyps based on images and multimodal data, characterized in that: The following steps are involved: Obtain pathological images from multiple microscopes, objective lens magnification information corresponding to the pathological images, and text information corresponding to the colonoscopy; Performing data preprocessing on each frame of the acquired pathological image and the acquired text information to obtain a training data set; Constructing a multi-scale multi-modal feature classification network, and training it using the training data set to obtain a trained multi-scale multi-modal feature fusion classification network; The patient's microscopic pathology images are collected, and the corresponding colonoscopy text information is obtained. After data preprocessing, they are input into the multi-scale and multi-modal feature fusion classification network to obtain the model's prediction of the disease category.

2. The intestinal polyp identification method based on images and multimodal data according to claim 1, characterized in that: The method for performing data preprocessing on each frame of the acquired pathological image and the acquired text information includes: performing window cutting, effectiveness screening, and feature extraction on the pathological image according to different objective lens magnifications to obtain image features; The text information is structured, and the structured text information is vectorized to obtain text features.

3. The intestinal polyp identification method based on images and multimodal data according to claim 2, characterized in that: The method for performing window cutting on the pathological image according to different objective lens magnifications is as follows: According to the different objective magnifications of the pathological images, small images of different sizes were cut out using the windowing method; The resolution of the thumbnail is as follows The different objective lens magnifications are ; The method for screening the effectiveness of the pathological images according to different objective lens magnifications is as follows: Calculate the average value of the image RGB data on the red channel With the maximum value , calculate the ratio of the two: ， Setting the effective threshold ,think is a valid image; The method for extracting features after effectiveness screening of the pathological images according to different objective lens magnifications is as follows: Stack small images of the same scale and resample them to the specified size required by the pre-trained network. The number of small images of each scale is recorded as: ， Then, any network pre-trained on the ImageNet dataset or pathological images is used to extract image features, and all small images are extracted into The one-dimensional vector of , thus obtaining the image features of each scale is: ， in Represents image features, the shape is Indicates the number of small images at m magnification, Indicates the number of neurons in the previous layer of the pre-trained network classification layer, that is, the length of the output feature vector.

4. The intestinal polyp identification method based on images and multimodal data according to claim 2, characterized in that: The method for structuring the text information is as follows: Segment the original text information into sentence-level units, remove impurity content including invalid characters, special punctuation marks, and redundant spaces, and form a text collection; For each sentence of text information in the text collection, an entity extraction strategy based on rules and entity dictionaries is adopted to identify key information units and obtain structured text data in a unified format; The method for vectorizing the text information after structured processing is as follows: For categorical variables including lesion type and site name, one-hot encoding or learnable feature vectors are used for representation; For the numerical variables including lesion size and location distance, normalization processing is performed; for the original text information, a pre-trained language model is used for encoding, the above features are spliced, and further projected and fused to obtain the text features of each polyp pathological image: ， in Represents a feature vector obtained by projecting text features, with a shape of is the number of output neurons in the projection layer.

5. The intestinal polyp identification method based on images and multimodal data according to claim 1, characterized in that: Construct a multi-scale multi-modal feature classification network, wherein the network includes: A single-scale feature fusion module is used to generate single-scale representative features from image features at the same magnification; a multimodal semantic fusion module is used to deeply fuse the image features and the text features through a cross-attention mechanism to obtain fused features; and a classification module is used to output the intestinal polyp category based on the fused features.

6. The intestinal polyp identification method based on images and multimodal data according to claim 5, characterized in that: The method of generating single-scale representative features from image features at the same magnification is as follows: Inputting the image features into a single-scale feature fusion classification module to obtain multi-scale representative features; The single-scale feature fusion classification module includes a linear projection layer and a self-attention module; The linear projection layer first transforms the input features Perform dimensionality reduction and output feature dimension is , and then input the self-attention module to obtain the representative features of this scale , the calculation formula is as follows: ， in, , , , That is, the image features after dimensionality reduction at scale m, , are respectively learnable query, key, and value mapping matrices, let and equal; Refers to the normalized exponential function used to convert similarity scores into attention weights whose sum is 1; After weighted summation of V, the representative eigenvector of scale m is obtained , The representative features of each scale are stacked to obtain multi-scale features: ， in Represents the vector after multi-scale feature stacking, the shape is , is the scale quantity.

7. The intestinal polyp identification method based on images and multimodal data according to claim 6, characterized in that: The text features are projected and mapped to representations of the same dimension as the image features through a linear projection layer to obtain text features that are spatially aligned with the image features.

8. The intestinal polyp identification method based on images and multimodal data according to claim 7, characterized in that: The multimodal semantic fusion module includes a preliminary semantic fusion module and a deep semantic fusion module; The text features and image features are input into the multimodal semantic fusion module to obtain the fusion features as follows: The preliminary semantic fusion module first concatenates image features and text features, and then uses a multi-layer perceptron to obtain preliminary fusion features. The formula is as follows: ， ， in is the number of neurons in the last layer of MLP, indicating the length of the output vector, The deep semantic fusion module uses a cross-attention network to obtain deep fusion features, as follows: The text features are used as query vectors and the image features are used as key-value pairs to perform cross attention calculation. The formula is as follows: ， in, , , ,in are respectively learnable query, key, and value mapping matrices; use After converting the similarity scores into attention weights whose sum is 1, we perform weighted summation on V to obtain the deep fusion feature. ; Finally, the preliminary fusion features and the deep fusion features are spliced ​​to obtain the fusion features: 。 9. The intestinal polyp identification method based on images and multimodal data according to claim 1, characterized in that: The method for training the multi-scale multi-modal feature fusion classification network is as follows: Using the training set data divided in step S1) to train a multi-scale multi-modal feature fusion classification network; After one round of training on all data, the network's slice prediction results are verified on the validation set; The cross entropy between the final prediction result of the network and the slice label, as well as the contrast loss between text features and image features, are used as the joint loss function for verification. The specific calculation formula is as follows: ， Where λ is the weight hyperparameter of the contrast loss, which is set to 0.5 in this implementation. and They are the cross entropy loss between the network's final prediction result and the slice label, and the contrast loss between text features and image features. The calculation formulas are as follows: ， Where C represents the number of categories, is the true label of the i-th category, using one-hot encoding, is the logits value output by the model, ， in Represents calculation and The cosine similarity of , is the feature vector after average pooling of M scale image features, ,in, For text vector .

10. The intestinal polyp identification method based on images and multimodal data according to claim 1, characterized in that: The obtained training data set is divided into the following ways: Taking patients as units, all data are divided into n equal parts in a class-balanced manner. One part is taken as the test set each time, and the remaining data is divided into a training set and a validation set in a ratio of (n-1):

1. Thus, n data set division methods are obtained; The sum of the results of n test sets or the mean of the training results on n test sets is used as the model result; When the joint loss function of the verification results of n rounds of verification training remains unchanged, stop training.

Images