Intestinal polyp recognition method based on image and multi-modal data

Abstract

The application discloses an intestinal polypus identification method based on images and multi-modal data, comprising the following steps: acquiring a plurality of microscope pathological images, objective magnification information corresponding to the pathological images, and text information corresponding to the colonoscopy; performing data preprocessing on each frame of acquired pathological images and text information to obtain a training data set; constructing a multi-scale multi-modal feature classification network, training the network using the training data set, and obtaining a trained multi-scale multi-modal feature fusion classification network; collecting microscope pathological images of a patient, acquiring corresponding colonoscopy text information, inputting the data after data preprocessing into the multi-scale multi-modal feature fusion classification network, and obtaining a class prediction of the disease by the model. The application fully fuses pathological images and colonoscopy text information, finally realizes accurate classification and prediction of the disease type, improves the effect and efficiency of multi-modal diagnosis of intestinal diseases, and thus improves the accuracy of diagnosis of the patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human tissue pathological image analysis, and in particular to an intestinal polyp identification method based on images and multi-modal data. BACKGROUND

[0002] In the existing tissue pathological image analysis technology, the intestinal polyps can be effectively classified as benign or malignant based on pathological image features. However, most of the existing technologies only focus on the coarse-grained classification of polyps, such as simply dividing them into benign or malignant, or only identifying whether they are adenomatous polyps, which cannot meet the needs of more detailed classification of polyps in clinical practice. In the actual clinical diagnosis process, doctors need to consider the multi-dimensional feature information of the lesion and give a classification result with higher discrimination and interpretability. The existing pathological image feature analysis technology cannot achieve this analysis purpose. For example, doctors need to make judgments based on the details of the lesion morphology observed during the colonoscopy operation. SUMMARY

[0003] The present application provides an intestinal polyp identification method based on images and multi-modal data to solve the above-mentioned technical problems. The technical solution adopted is as follows:

[0004] An intestinal polyp identification method based on images and multi-modal data includes the following steps:

[0005] Obtaining pathological images of multiple microscopes, objective magnification information corresponding to the pathological images, and text information corresponding to the colonoscopy.

[0006] Data preprocessing is performed on each frame of the obtained pathological images and text information to obtain a training data set.

[0007] A multi-scale multi-modal feature classification network is constructed, and the training data set is used for training to obtain a trained multi-scale multi-modal feature fusion classification network.

[0008] The microscope pathological images of the patient are collected, the corresponding colonoscopy text information is obtained, and after data preprocessing, they are input into the multi-scale multi-modal feature fusion classification network to obtain the class prediction of the disease by the model.

[0009] Further, the method for data preprocessing of each frame of the obtained pathological images and text information includes:

[0010] The pathological images are windowed, filtered, and the features after the filtering are extracted according to different objective magnifications to obtain image features.

[0011] The text information is structured, and the structured text information is vectorized to obtain text features.

[0012] Further, the method for windowing the pathological images according to different objective lens magnifications is as follows:

[0013] According to different objective lens magnifications to which the pathological images belong, different sizes of small images are cut by the windowing method;

[0014] The resolution size of the small images is determined according to different objective lens magnifications are ;

[0015] The method for effectively screening the pathological images according to different objective lens magnifications is as follows:

[0016] The average value of the image RGB data on the red channel is calculated and the maximum value , and the ratio of the two is calculated:

[0017] ,

[0018] The effective threshold value is set , and is considered as an effective image;

[0019] The method for feature extraction after the pathological images are screened according to different objective lens magnifications is as follows:

[0020] The small images of the same scale are stacked and resampled to the specified size required by the pre-trained network, and the number of small images of each scale is recorded as:

[0021] ,

[0022] Then, any network pre-trained on the ImageNet dataset or pathological images is used for image feature extraction, and all small images are extracted into one-dimensional vectors with a length of , so that the image features of each scale are obtained as: wherein represents the image features, and the shape is represents the number of small images under the magnification of m, represents the number of neurons in the previous layer of the pre-trained network classification layer, that is, the length of the output feature vector.

[0023] Further, the method for structuring the text information is as follows:

[0024] The original text information is segmented according to the sentence level unit, and impurity content including invalid characters, special punctuation and redundant spaces is removed to form a text collection;

[0025] For each sentence of text information in the text collection, an entity extraction strategy based on rules and entity dictionaries is used to identify key information units and obtain structured text data in a unified format.

[0026] The method for vectorizing the text information after structured processing is as follows:

[0027] For category variables including lesion types and part names, one-hot encoding or learnable feature vectors are used for representation;

[0028] For numerical variables including lesion size and location distance, normalization processing is performed; for original text information, a pre-trained language model is used for encoding, the above features are spliced, and further projection is performed to fuse the above text information, thereby obtaining the text features of the pathological image of each polyp:

[0029] ,

[0030] wherein represents a feature vector obtained by projecting the spliced text features, and the shape is is the number of output neurons of the projection layer.

[0031] Further, a multi-scale multi-modal feature classification network is constructed, wherein the network comprises:

[0032] a single-scale feature fusion module for generating single-scale representative features from image features at the same magnification; a multi-modal semantic fusion module for deep fusion of image features and text features through cross-attention mechanism to obtain fusion features; and a classification module for outputting intestinal polyp categories based on the fusion features.

[0033] Further, the method for generating single-scale representative features from image features at the same magnification is as follows:

[0034] The image features are input into the single-scale feature fusion classification module to obtain multi-scale representative features;

[0035] The single-scale feature fusion classification module comprises a linear projection layer and a self-attention module.

[0036] The linear projection layer first reduces the dimension of the input feature , and outputs a feature with a dimension of , which is then input into the self-attention module to obtain the representative feature at this scale, and the calculation formula is as follows:

[0037] ,

[0038] wherein , , , i.e. the reduced image features at m scales, , Let be the learnable query, key, and value mapping matrices, respectively. and equal; The normalized exponential function is used to convert similarity scores into attention weights whose weights sum to 1; a representative feature vector at scale m is obtained by weighted summation of V. ;

[0039] Multi-scale features are obtained by stacking representative features from various scales:

[0040]

[0041] in This represents a vector resulting from the stacking of multi-scale features, with the shape being... , For scale quantity.

[0042] Furthermore, the text features are projected, and a linear projection layer is used to map the text features into a representation with the same dimension as the image features, thereby obtaining text features that are spatially aligned with the image features.

[0043] Furthermore, the multimodal semantic fusion module includes a preliminary semantic fusion module and a deep semantic fusion module;

[0044] The method for obtaining fused features by inputting text features and image features into the multimodal semantic fusion module is as follows:

[0045] The initial semantic fusion module first concatenates image features and text features, and then uses a multilayer perceptron to obtain preliminary fused features, as shown in the following formula:

[0046] ,

[0047] ,

[0048] in The number of neurons in the last layer of the MLP represents the length of the output vector.

[0049] The deep semantic fusion module uses a cross-attention network to obtain deep fusion features, as detailed below:

[0050] Using text features as the query vector and image features as key-value pairs, cross-attention is calculated using the following formula:

[0051] ,

[0052] in, , , ,in respectively are learnable query, key, value mapping matrix; using After converting the similarity score into attention weights with the sum of weights being 1, the V is weighted and summed to obtain the deep fusion feature ;

[0053] Finally, the preliminary fusion feature and the deep fusion feature are spliced to obtain the fusion feature:

[0054] .

[0055] Further, the method for training the multi-scale multi-modal feature fusion classification network is as follows:

[0056] The multi-scale multi-modal feature fusion classification network is trained using the training set data divided in step S1);

[0057] After one round of training of all data, the slice prediction results of the network are verified on the validation set;

[0058] The cross-entropy of the final prediction result of the network and the slice label, and the contrastive loss of the text feature and the image feature are used as the joint loss function for verification, and the specific calculation formula is as follows:

[0059] ,

[0060] Wherein, λ is the weight hyperparameter of the contrastive loss, which is set to 0.5 in the embodiment, and The cross-entropy loss of the final prediction result of the network and the slice label, and the contrastive loss of the text feature and the image feature are respectively

[0061] ,

[0062] Wherein, C represents the number of categories, is the real label of the i-th category, which is encoded by one-hot, is the logits value output by the model;

[0063] ,

[0064] Wherein represents the calculation and Cosine similarity, wherein , is the feature vector after average pooling of M scale image features , wherein wherein, is the text vector .

[0065] Further, the method for dividing the training data set is as follows:

[0066] Taking patients as the unit, all data are divided into n equal parts in a class-balanced manner. Each time, one part is taken as the test set, and the remaining data is divided into training set and validation set according to the ratio of (n-1):1. This results in n ways of dividing the dataset.

[0067] The sum of the results from the n test sets or the average of the training results from the n test sets is used as the model result;

[0068] Training stops when the joint loss function of the validation results remains unchanged after n rounds of validation training.

[0069] The advantage of this invention lies in the provided method for identifying intestinal polyps based on images and multimodal data. It acquires clear pathological images containing information about the current objective magnification from a microscope and combines this with textual information from the corresponding patient's colonoscopy. After preprocessing, multimodal features are obtained. A multi-scale multimodal feature classification network is then constructed to model pathological images at different objective magnifications at multiple scales. Finally, the acquired microscopic pathological images and corresponding colonoscopy textual information features are input into the multi-scale multimodal feature fusion classification network to achieve deep semantic interaction and fusion, obtaining the model's prediction of the disease category. This method fully integrates pathological images and colonoscopy textual information, ultimately achieving accurate classification and prediction of disease types, improving the effectiveness and efficiency of multimodal diagnosis of intestinal diseases, and thus enhancing the accuracy of patient diagnosis. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Fig. 1 This is a schematic diagram of the intestinal polyp identification method based on image and multimodal data according to this application;

[0072] Fig. 2 This is a schematic diagram of the pathological image and text information data processing flow in the intestinal polyp identification method based on image and multimodal data of this application;

[0073] Fig. 3 This is a schematic diagram of the multi-scale, multi-modal feature fusion classification network structure in the intestinal polyp identification method based on image and multimodal data of this application. Detailed Implementation

[0074] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0075] like Figs. 1 to 3 The image shown illustrates a method for identifying intestinal polyps based on images and multimodal data, comprising the following steps:

[0076] S1) Acquire pathological images from multiple microscopes, the corresponding objective magnification information for each pathological image, and the corresponding text information from the colonoscope. During this step, the pathologist operates the microscope to observe the slide, simultaneously acquiring clear pathological images from the microscope with known current objective magnification information. When the pathologist finishes observation, the collection of pathological images for that slide is stopped, and the actual polyp classification for that slide is obtained. Each slide sample contains ≥1 image. The image size acquired in this implementation is 1824*1216 pixels, but the actual image size used is not limited to this; different image sizes can be obtained depending on the acquisition device. Simultaneously, the microscopic descriptive text information for the corresponding patient and intestinal location is acquired. This implementation specifically classifies colonic polyps into six categories: adenocarcinoma, tubular adenoma, sessile serrated adenoma polyp, hyperplastic polyp, inflammatory polyp, and normal tissue.

[0077] S2) Perform data preprocessing on each acquired frame of pathological image and text information to obtain the training dataset. The method for dividing the training dataset in this step is as follows: Using patients as units, divide all data into n equal parts in a class-balanced manner. Take one part as the test set each time, and divide the remaining data into training and validation sets in a ratio of (n-1):1, thus obtaining n possible dataset partitioning methods. Use the sum of the results from the n test sets or the mean of the training results from the n test sets as the model result. Stop training when the joint loss function of the validation results remains unchanged after n rounds of validation training.

[0078] Specifically, this scheme divides all data into 10 equal parts based on the patient, using a class-balanced approach. One part is taken as the test set each time, and the remaining data is divided into training and validation sets in an 8:1 ratio, resulting in ten different dataset partitioning methods. The sum of the results from the ten test sets is used as the model result.

[0079] In step S2), the method for data preprocessing of each acquired frame of pathological image and text information includes: windowing the pathological image according to different objective magnifications, filtering for validity and extracting features after filtering for validity to obtain image features; performing structured processing on the text information, and vectorizing the structured text information to obtain text features.

[0080] Specifically, the method for windowing pathological images according to different objective magnifications is as follows:

[0081] Based on the different objective magnifications of the pathological images, small images of different sizes were cut out using the windowing method;

[0082] The resolution of the small image is based on The different objective lens magnifications are respectively .

[0083] In practice, the size of the thumbnail can be set for different objective lens magnifications. In this implementation, taking 2×, 4×, 10×, 20×, and 40× objective lens magnifications as examples, the resolution of the thumbnail for each objective lens is 5.59. 2.78 1.39 0.56 0.28 When slicing the image, start from the coordinates. Starting with a window length of 453 pixels and an interval of 227 pixels, the image of size 1824*1216 pixels can be sliced ​​into 28 smaller images. These smaller images, with five different objective magnifications (40×, 20×, 10×, 4×, and 2×), correspond to actual object sizes of 81 pixels and 100 pixels respectively. 163 326 814 1637 .

[0084] Specifically, the method for screening the effectiveness of pathological images according to different objective magnifications is as follows:

[0085] Calculate the average value of the RGB data of an image in the red channel. With the maximum value Calculate the ratio between the two:

[0086] ,

[0087] Set effective threshold ,think This is a valid image.

[0088] This embodiment uses a threshold. For example, when the small image is calculated... A value greater than 0.01 is considered a valid small image containing pathological tissue.

[0089] The method for feature extraction after effective screening of pathological images according to different objective magnifications is as follows:

[0090] Small images of the same scale are stacked together and resampled to the specified size required by the pre-trained network. The number of small images at each scale is denoted as:

[0091] ,

[0092] Subsequently, an arbitrary pre-trained network on the ImageNet dataset or pathological images is used for image feature extraction, extracting all small images into units of length [length missing]. A one-dimensional vector, from which the image features at each scale are obtained: ,in Represents image features, with shape as This indicates the number of minimaps at a scaling factor of m. This indicates the number of neurons in the previous layer of the pre-trained network's classification layer, i.e., the length of the output feature vector.

[0093] In this embodiment, a ResNet50 pre-trained on ImageNet is used as an example. First, the small image needs to be resampled to the size (256, 256) required by the ResNet50 network. After being input into the ResNet50 network, a one-dimensional vector with a length of 1024 is obtained.

[0094] Specifically, the methods for structuring text information are as follows:

[0095] The original text information is segmented into sentence-level units, and impurities including invalid characters, special punctuation marks, and redundant spaces are removed to form a text set;

[0096] For each sentence in the text set, a rule-based and entity dictionary-based entity extraction strategy is used to identify key information units and obtain structured text data in a unified format.

[0097] In this embodiment, firstly, the full-width / half-width characters, Chinese / English punctuation, capitalization, and units (such as "cm" and "centimeter") in the text are standardized. Then, multi-entity recognition is performed from a sentence, and regularization methods are used to extract lesion entity types such as "polyp," "erosion," "protrusion," and "mass," as well as location information such as "20cm from the anus" and "middle segment of the sigmoid colon," and polyp size information. Thus, each polyp information is structured into lesion type, location, location distance, size, and original text information.

[0098] Specifically, the method for vectorizing structured text information is as follows:

[0099] For categorical variables including lesion type and location name, one-hot encoding or learnable feature vectors are used for representation;

[0100] For numerical variables including lesion size and location distance, normalization is performed; for the original text information, a pre-trained language model is used for encoding, the above features are concatenated, and further projected to fuse the text information, thereby obtaining the text features of the pathological image of each polyp:

[0101] ,

[0102] in This represents a feature vector obtained by projecting concatenated text features, with the shape being... This represents the number of output neurons in the projection layer.

[0103] In this embodiment, the lesion type and location name are represented by a learnable feature vector with a length of 32. The lesion size and distance are normalized to obtain features of length 1. The original text information is input into a pre-trained BERT model, and the output CLS is taken as the sentence feature, thus obtaining a sentence feature of length 768. These features are concatenated, and a linear projection layer is used to project this vector, fusing the text features to obtain a text feature vector of length 512. .

[0104] S3) Construct a multi-scale, multi-modal feature classification network, train it using the training dataset, and obtain the trained multi-scale, multi-modal feature fusion classification network. In step S3), the multi-scale, multi-modal feature classification network is constructed, wherein the network includes:

[0105] The single-scale feature fusion module is used to generate single-scale representative features from image features at the same magnification; the multimodal semantic fusion module is used to deeply fuse image features and text features through a cross-attention mechanism to obtain fused features; and the classification module is used to output the intestinal polyp category based on the fused features.

[0106] As a further method of step S3), the method for generating single-scale representative features from image features at the same magnification is as follows:

[0107] Image features are input into a single-scale feature fusion classification module to obtain representative features at multiple scales.

[0108] The single-scale feature fusion classification module includes a linear projection layer and a self-attention module;

[0109] The linear projection layer first processes the input features Dimensionality reduction is performed, and the output feature dimension is... The data is then input into a self-attention module to obtain representative features at that scale. The calculation formula is as follows:

[0110] ,

[0111] in, , , , That is, the image features after dimensionality reduction at scale m. , Let be the learnable query, key, and value mapping matrices, respectively. and equal; The normalized exponential function is used to convert similarity scores into attention weights whose weights sum to 1; a representative feature vector at scale m is obtained by weighted summation of V. .

[0112] Multi-scale features are obtained by stacking representative features from various scales:

[0113] ,

[0114] in This represents a vector resulting from the stacking of multi-scale features, with the shape being... . For scale quantity.

[0115] In this embodiment, the following settings are provided: , The input features with a dimension of 1024 are first reduced to 512, and then fused using a self-attention module to obtain features with a dimension of 512. Multiscale features.

[0116] As a further method of step S3), the text features are projected and mapped to a representation with the same dimension as the image features through a linear projection layer, thereby obtaining text features that are spatially aligned with the image features.

[0117] In this implementation, the aforementioned text features Image features Therefore, the projection layer maps text features to .

[0118] As a further method of step S3), the multimodal semantic fusion module includes a preliminary semantic fusion module and a deep semantic fusion module;

[0119] The method for obtaining fused features by inputting text features and image features into the multimodal semantic fusion module is as follows:

[0120] The initial semantic fusion module first concatenates image features and text features, and then uses a multilayer perceptron to obtain preliminary fused features, as shown in the following formula:

[0121] ,

[0122] ,

[0123] in represents the number of neurons in the last layer of the MLP, and represents the length of the output vector.

[0124] In this embodiment, the output features .

[0125] The deep semantic fusion module uses a cross-attention network to obtain deep fusion features, as detailed below:

[0126] Using text features as the query vector and image features as key-value pairs, cross-attention is calculated using the following formula:

[0127] ,

[0128] in, , , ,in These are learnable query, key, and value mapping matrices, respectively; using After converting the similarity scores into attention weights with a sum of 1, a weighted sum is applied to V to obtain the deep fusion features. ;

[0129] Finally, the preliminary fusion features and the deep fusion features are concatenated to obtain the fusion features:

[0130] .

[0131] In this embodiment, the following settings are provided: Thus, the fusion characteristics are obtained. , .

[0132] The fused features are classified, and the final diagnostic result is output. Any classification network can be used, such as support vector machines or multilayer perceptrons. In this embodiment, a multilayer perceptron is used for classification, and the input... Output the predicted scores for 6 categories.

[0133] As a further method of step S3), the method for training the multi-scale, multi-modal feature fusion classification network is as follows:

[0134] The multi-scale, multi-modal feature fusion classification network is trained using the training set data divided in step S1).

[0135] After training all the data for one round, the network's slice prediction results are validated on the validation set.

[0136] The cross-entropy between the network's final prediction result and the slice label, along with the contrastive loss of text features and image features, are used as the joint loss function for validation. The specific calculation formula is as follows:

[0137] ,

[0138] Wherein, λ is the weight hyperparameter of the contrastive loss, which is set to 0.5 in this implementation. and The cross-entropy loss for the final network prediction result and the slice label, and the contrast loss for text features and image features are respectively calculated using the following formulas:

[0139] ,

[0140] Where C represents the number of categories, For the true label of class i, use one-hot encoding. The logits value output by the model.

[0141] ,

[0142] in Indicates calculation and The cosine similarity, where , The feature vector obtained by average pooling of M scale image features , among which, For text vectors .

[0143] In the above steps, the multi-scale, multi-modal feature fusion classification network is trained using the training set data divided in step S1), employing the Adam optimization method with a learning rate of 0.0001 and a batch size of 1. After one round of training on all data, the network's slice prediction results are validated on the validation set. To prevent overfitting, an early stopping strategy is adopted, i.e., training stops if the validation results do not improve after 20 rounds. The network is run on all ten divided datasets, and the average of the results on the ten test sets is used as the model result.

[0144] S4) Acquire microscopic pathological images of the patient and obtain corresponding colonoscopy text information. After data preprocessing, input the data into a multi-scale, multi-modal feature fusion classification network to obtain the model's prediction of the disease category. During this step, while the pathologist is actually using the microscope, the model simultaneously acquires clear pathological images and corresponding patient and site colonoscopy text descriptions, outputting the model's predicted structure to provide diagnostic suggestions to the doctor.

[0145] This proposed method for identifying intestinal polyps based on images and multimodal data first acquires clear pathological images containing information about the current objective magnification from a microscope. This is combined with textual descriptions of the corresponding patient and location from the colonoscopy, and both are preprocessed to obtain multimodal features. Subsequently, a multi-scale multimodal feature classification network is constructed to model images at different objective magnifications at multiple scales, and an attention mechanism is used to fuse image features. Finally, the image and textual features are input into a multimodal fusion module to achieve deep semantic interaction and fusion. This method fully integrates pathological images and colonoscopy textual information, ultimately achieving accurate classification and prediction of disease types, improving the effectiveness of multimodal diagnosis of intestinal diseases. Compared with existing image processing methods, both diagnostic efficiency and accuracy are significantly improved.

[0146] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for intestinal polyp identification based on image and multi-modal data, characterized in that, The method comprises the following steps: Obtaining a plurality of pathological images of a microscope, objective magnification information corresponding to the pathological images, and text information corresponding to the pathological images under an enteroscope; Data preprocessing is performed on each frame of the obtained pathological images and the text information to obtain a training data set; A multi-scale multi-modal feature classification network is constructed, and the training data set is used for training to obtain a trained multi-scale multi-modal feature fusion classification network; Microscope pathological images of a patient are collected, text information corresponding to the enteroscope is obtained, and after data preprocessing, the text information is input into the multi-scale multi-modal feature fusion classification network to obtain a class prediction of the disease by the model; The method for data preprocessing on each frame of the obtained pathological images and the text information comprises: Windowing, effectiveness screening and feature extraction after effectiveness screening are performed on the pathological images according to different objective magnifications to obtain image features; The text information is structured, and the structured text information is vectorized to obtain text features; The windowing method for the pathological images according to different objective magnifications is as follows: According to different objective magnifications of the pathological images, different sizes of small images are cut by the windowing method; The resolution size of the small pictures is W m pixel / μm, m ∈ [1, M]; The effectiveness screening method for the pathological images according to different objective magnifications is as follows: Compute the average value R of the image RGB data on the red channel mean with the maximum value R max , compute the ratio of both: Setting an effective threshold C threshold R > C is considered an effective image; The feature extraction method for the pathological images after effectiveness screening according to different objective magnifications is as follows: Small images of the same scale are stacked and resampled to a specified size required by a pre-training network, and the number of small images of each scale is denoted as n i ; N m m e [0, M - 1], Subsequently, any network pre-trained on ImageNet dataset or pathology images is used for image feature extraction, and all small images are extracted as one-dimensional vectors of length D input , thus obtaining image features of each scale as: where f image represents image features, shape N m × D input , N m represents the number of small images at m times, D input represents the number of neurons in the previous layer of the pre-trained network classification layer, that is, the length of the output feature vector.

2. The intestinal polyp recognition method based on image and multi-modal data according to claim 1, wherein, The structured text information is vectorized to obtain text features; For category variables including lesion types and part names, one-hot encoding or a learnable feature vector is used for representation; For numerical variables including lesion sizes and position distances, normalization is performed; for the original text information, a pre-trained language model is used for encoding, the features are spliced, and further projection is performed to fuse the text information, thereby obtaining text features of each polyp pathological image:

3. The intestinal polyp recognition method based on image and multi-modal data according to claim 1, wherein, A multi-scale multi-modal feature classification network is constructed, wherein the network comprises: A single-scale feature fusion module is configured to generate single-scale representative features from image features under the same magnification; where f text represents a feature vector obtained by projecting the spliced text features, with a shape of 1 x D text , D text is the number of output neurons of the projection layer. A multi-modal semantic fusion module is configured to perform deep fusion on the image features and the text features through a cross-attention mechanism to obtain fusion features; A classification module is configured to output an intestinal polyp class based on the fusion features. ​ ​ ​ 4. The image and multi-modal data based intestinal polyp recognition method according to claim 3, characterized in that, the method for generating single-scale representative features from image features at the same magnification is as follows: the image features are input into a single-scale feature fusion classification module to obtain multi-scale representative features; the single-scale feature fusion classification module comprises a linear projection layer and a self-attention module; Linear projection layer first reduces the dimension of the input feature (N m ,D input ), and outputs the feature dimension (N m ,D mid ), and then inputs the self-attention module to obtain the representative feature x m of this scale, and the calculation formula is as follows: where Q=XW Q , K=XW K , V=XW V , X is the image feature after dimensionality reduction in m dimensions, W Q ,W K , are learnable query, key, value mapping matrices respectively, let d k and D text be equal; Softmax refers to a normalization exponential function, which is used to convert the similarity score into attention weight with the sum of weights being 1; a representative feature vector of the m scale is obtained by weighted summing of V the representative features of each scale are stacked to obtain multi-scale features: where f image represents the vector after multi-scale feature stacking, with the shape of Mxd k , M is the number of scales.

5. The image and multi-modal data based intestinal polyp recognition method according to claim 4, characterized in that, the text features are projected, and the text features are mapped into the same dimension as the image features by the linear projection layer to obtain text features aligned with the image features in space.

6. The image and multi-modal data based intestinal polyp recognition method according to claim 5, characterized in that, the multi-modal semantic fusion module comprises a preliminary semantic fusion module and a deep semantic fusion module; the method for inputting the text features and the image features into the multi-modal semantic fusion module to obtain the fusion features is as follows: the preliminary semantic fusion module first splices the image features and the text features, and then uses a multi-layer perception machine to obtain preliminary fusion features, and the formula is as follows: where D o is the number of neurons in the last layer of the MLP, denoting the output vector length, the deep semantic fusion module uses a cross-attention network to obtain deep fusion features, and the details are as follows: the text features are taken as query vectors, and the image features are taken as key-value pairs to perform cross-attention calculation, and the formula is as follows: where Q = f text W Q , K = f image W K , V = f image W V where W Q , W K , are learnable query, key, value mapping matrices respectively; after converting the similarity scores into attention weights with sum of weights being 1 using Softmax, V is weighted summed to obtain the deep fusion feature F2; finally, the preliminary fusion features and the deep fusion features are spliced to obtain the fusion features:

7. The image and multi-modal data based intestinal polyp recognition method according to claim 1, characterized in that, the method for training the multi-scale multi-modal feature fusion classification network is as follows: the multi-scale multi-modal feature fusion classification network is trained using the training set data divided in step S1); after one round of training of all data, the slice prediction results of the network are verified on the validation set; the cross-entropy of the final prediction results of the network and the slice labels, and the contrastive loss of the text features and the image features are taken as the joint loss function for verification, and the specific calculation formula is as follows: Loss = Loss CE + λLoss CL where λ is a weight hyper-parameter of the contrastive loss, which is set to 0.5 in the present implementation, Loss CE and Loss CL are the cross-entropy losses of the final prediction results of the network and the slice labels, respectively, and the contrastive losses of the text features and the image features, and the calculation formulas are as follows: where C denotes the number of classes, y i is the true label of the ith class, encoded using one-hot encoding, z i is the logits value output by the model. where cos(f img ,f text ) represents the cosine similarity of f img and f text , where f img is the feature vector after average pooling of M scale image features, where f text is the text vector, D text = d k .

8. The image and multi-modal data based intestinal polyp recognition method according to claim 1, characterized in that, the obtained training data set is divided, and the division method of the training data set is as follows: all data are divided into n parts in a class-balanced manner, and each time 1 part is taken as a test set, and the remaining data are divided into a training set and a validation set in a ratio of (n-1):1, thereby obtaining n kinds of data set division methods; the sum of the n test set results or the average of the training results on the n test sets is taken as the model result; when the joint loss function of the verification results of n rounds of verification training does not change, the training is stopped.

Images