Rare disease image classification method, device and storage medium based on small sample learning

Abstract

The present invention provides a rare disease image classification method, device and medium based on small sample learning. By cleverly constructing a feature backtracking fusion encoder, the next layer of features is used to generate an attention mask for the current layer to reduce useless noise information in the low-level features, thereby better integrating the spatial detail information in the low-level features into the high-level features, effectively improving the model's classification performance for gastrointestinal disease areas; secondly, the multi-level prototype reconstruction network further captures the semantic relevance between the support set and query set samples to enhance the distinguishing areas on the support image representation, and generates a calibration class center suitable for the current query sample for each query sample. The classifier based on Euclidean distance outputs the classification result of the query sample, and the cross-entropy function is used to guide model optimization to ensure the accuracy of the classification result. Ultimately, the rare disease image classification model based on small sample learning is able to output the classification result of each image after rigorous training and testing.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a rare disease image classification method, device and storage medium based on small sample learning. Background Art

[0002] Endoscopy is the gold standard for gastrointestinal disease screening. It can help clinicians accurately identify gastrointestinal diseases during early gastrointestinal cancer screening, thereby developing more precise treatment plans and reducing patient mortality. Therefore, accurate identification of gastrointestinal diseases, including rare diseases, is crucial for promoting early diagnosis and improving patient survival rates in gastrointestinal cancer patients.

[0003] In recent years, with the rapid development of artificial intelligence technologies, represented by deep learning, some intelligent methods for endoscopic-assisted diagnosis have been initially explored in the gastrointestinal tract, demonstrating the feasibility of artificial intelligence in promoting endoscopic examinations. However, these technologies still face challenges in clinical application. In medical endoscopic images, gastrointestinal diseases often appear similar in color and texture to the surrounding intestinal wall, and the boundaries and morphological features of the lesions are not clear, making it difficult to accurately identify lesion features. At the same time, the randomness of the shooting position and posture of medical endoscopic equipment leads to significant differences in the distribution and size of similar lesions in gastrointestinal images. Although existing deep learning models can achieve end-to-end fully automatic gastrointestinal image classification, their success relies heavily on sufficient training with large amounts of labeled data, which is difficult to meet for the diagnosis of rare diseases in the gastrointestinal tract.

[0004] In summary, developing rare disease image classification methods based on small-sample learning has significant practical significance and important scientific research value for the development of medical artificial intelligence. However, current technologies still face challenges in accuracy and have not yet been widely adopted in clinical practice. Summary of the Invention

[0005] The present invention proposes a rare disease image classification method, device and storage medium based on small sample learning, which can solve the technical problems of how to solve the problem of difficulty in identifying lesion characteristics due to low contrast between lesions and intestinal walls, and the inaccurate classification of gastrointestinal diseases including rare diseases due to large intra-class differences caused by differences in distribution and size between similar lesions.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A rare disease image classification method based on small sample learning includes the following steps:

[0008] S1. Collect an endoscopic image dataset containing rare diseases in the gastrointestinal tract and divide the dataset into a support set and a query set.

[0009] S2. Constructing a feature backtracking fusion encoder to extract support image features in the support set and query image features in the query set, and passing these feature information to a multi-stage prototype reconstruction network;

[0010] S3. Use a multi-level prototype reconstruction network to capture the semantic correlation between support set and query set samples, and use the semantic correlation to enhance the distinguishing regions on the support image representation, and generate the corresponding support set class center for each query sample;

[0011] S4. Construct a classifier based on Euclidean distance to calculate the Euclidean distance between the query image features and all support set features output by the multi-level prototype reconstruction network, and use the inverse of the Euclidean distance as the classification score, thus forming a rare disease image classification model based on small sample learning;

[0012] S5. Using the collected training samples to train the rare disease image classification model based on small-sample learning, and using the validation set samples to verify and select the optimal network parameters, thereby forming a final rare disease image classification model based on small-sample learning;

[0013] S6. When applied, endoscopic images of the gastrointestinal tract containing rare diseases are collected and input into the above-mentioned rare disease image classification model based on small sample learning, and the classification results of each query sample are output after calculation.

[0014] Furthermore, the step S1 includes:

[0015] S11. Collect an endoscopic image dataset containing rare diseases in the gastrointestinal tract and divide it into a training set , validation set and test set , and there is no overlapping category between the three, in the training set , validation set and test set The set is further divided into support set and query set;

[0016] S12, using bilinear interpolation method to adjust the resolution of all images to 224×224;

[0017] S13, perform data augmentation by randomly flipping the images and their labels in the training dataset horizontally, rotating them 90 degrees clockwise, and rotating them 180 degrees clockwise;

[0018] S14. Using the support image, support label, and query image as inputs of a neural network, and using the query label as supervisory information output by the rare disease image classification model based on small sample learning.

[0019] Furthermore, the step S2 includes:

[0020] S21, construct a feature backtracking fusion encoder, the input support set image and the query set image share the feature backtracking fusion encoder, and obtain the support image feature and the query image features ;

[0021] S22, the feature backtracking fusion encoder consists of a ResNet18 encoder with initialized pre-trained weights, a backtracking attention learning module, and a multi-scale fusion module;

[0022] S221, initialize the ResNet18 encoder with pre-trained weights and input the support set image and query set images , extract the feature representation of each image respectively; the ResNet18 encoder consists of 1 convolutional layer, 1 maximum pooling layer and 4 residual blocks, where each residual block contains 2 convolutional layers and 1 residual connection;

[0023] The ResNet18 encoder converts the 224×224 original data into a 7×7 feature code through a series of convolution and pooling operations;

[0024] Assume the encoder input is First, a 7×7 convolutional layer is used to extract the 112×112 low-level features of the image, and a 56×56 intermediate layer feature is obtained through the maximum pooling layer. , then In the 4 residual blocks of the input series, the intermediate layer features of the 4 residual blocks output are recorded as ,in Indicates the The output features of the residual blocks and , the residual block is expressed as ;

[0025] S222, the retrospective attention learning module receives and The output features of the residual block and , and output a two-dimensional attention mask , where C is the number of channels of the feature map, H is the height of the feature map, W is the width of the feature map, and at this time middle , that is, for the features of each layer of residual blocks, in addition to the features And the last layer of features , which uses the features of the next layer of residual blocks to calculate a two-dimensional attention mask , and is used to refine the current residual block features;

[0026] The retrospective attention learning module first uses convolutional layers with three convolution kernel sizes of 3×3, 5×5 and 7×7 to Perform convolution operations to obtain information features with different receptive fields ,in represents the convolution kernel size, Indicates the The residual block features are convolved with a kernel size of k×k, and then two consecutive point-by-point convolutions are performed to ensure better nonlinear learning ability and capture the complex correlation between spatial positions; the first 1×1 convolution layer is to convert the features The channel from Compress to , is a reduction ratio, set to , then through Layer and After the layer, the second 1×1 convolution layer is used to convert the channel from Converted to 1, then the features are upsampled by a factor of 2 and passed through The function gets a two-dimensional attention mask , and finally the three two-dimensional attention masks The average obtained ;

[0027] The above process is expressed as:

[0028] ;

[0029] ;

[0030] in yes function, It is the upsampling process; finally, the features of each layer of residual block are And the last layer of features , the refined features are ;

[0031] S223, the multi-scale fusion module combines the features of each layer of residual blocks, except for the features And the last layer of features , the refined features are As input, we use the refined low-level features Layer-by-layer fusion is used to supplement low-level features at different levels, and the final features are output after the fusion of low-level features and high-level features. ; The multi-scale fusion module first transforms the input features The features of three different receptive fields are obtained through three pooling operations of 3×3, 5×5 and 7×7. , and then sum the corresponding elements of the three to get the feature ,in , and then use global average pooling Get a channel-level embedding vector , and after three fully connected layers and normalization operations, three weight vectors are obtained , multiply it by the features of the corresponding three different receptive fields Get each layer feature , and finally merge layer by layer, As input features, we get , and so on to get the final feature .

[0032] Furthermore, step S3 includes:

[0033] S31, Multi-level prototype reconstruction network to feature back-fusion encoder transfer of the final feature As input, it includes support image features and query image features ;

[0034] S32, the multi-level prototype reconstruction network consists of a contrast level relationship module, a saliency level relationship module and an attention level relationship module in parallel

[0035] S321, the contrast level relationship module first compares the query image features Perform global average pooling to obtain the embedding vector , and then use the subtraction operation to get the embedding vector of the query image and all supported image features Compare each position of to output the relationship between the instance-level objects ;

[0036] The output characteristic relationship formula is expressed as ,in Indicates a broadcast operation, Repeated expansion to , Indicates absolute value operation, and finally ;

[0037] S322, the saliency level relationship module first performs the query image feature Perform global average pooling to obtain the embedding vector , then As a convolution kernel, it supports image features in a deep separation convolution manner. Perform convolution operation to output the feature relationship between the two ; The output characteristic relationship formula is expressed as ,in Represents the depth separation convolution operation, and finally ;

[0038] S323, the attention-level relationship module first obtains the two embedded features of the support image and the query image through 1×1 convolution, and then calculates the spatial attention weight matrix of the two based on the matrix multiplication between the two embedded features , then query the image features and the attention weight matrix Perform matrix multiplication to obtain spatial perception query image features , and extract spatial perception query image features through subtraction operation and supporting image features Local semantic relationship between , output Expressed as ,at last ;

[0039] S324, the multi-level prototype reconstruction network will finally and 、 、 Perform channel-level splicing and compress the channel through 1×1 convolution , get the reconstructed support image features ,at last .

[0040] Furthermore, the step S4 includes:

[0041] S41. The classifier based on Euclidean distance reconstructs the supporting image features transmitted in the multi-level prototype reconstruction network and query image features As input, output is the score of each category that the query image belongs to vector;

[0042] S411, the classifier based on Euclidean distance will first support image features Flatten and average the embedding vectors of the same category to get the prototype center of the category in Euclidean space , similarly query the image features Flattened to embedding vector ;

[0043] S412, Euclidean distance-based classifier calculates the query image embedding vector Prototype center for each category The Euclidean distance to obtain the query image Belong to each category Score , select the category with the highest score as the prediction result of the query image category.

[0044] Furthermore, the step S5 includes:

[0045] S51. Build a rare disease image classification model based on small sample learning, use the Adam optimizer, and set the initial learning rate to , the training epoch is set to 400, and each epoch consists of 200 episodes;

[0046] S52, during the training process, using the training set The rare disease image classification model based on small sample learning is trained and optimized using the cross entropy loss function, which is expressed as:

[0047] ,in Indicates the number of supported image categories;

[0048] S53. Train the model through the loss function, adjust the parameters through backpropagation, and select the model parameters with the highest accuracy in the validation set for saving.

[0049] Furthermore, the step S5 further includes:

[0050] S54. Select evaluation indicators including accuracy to evaluate model performance.

[0051] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0052] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0053] It can be seen from the above technical solutions that the rare disease image classification method and system based on small sample learning of the present invention is intended to accurately classify various diseases in the gastrointestinal tract, including rare diseases. This method first cleverly constructs a feature backtracking fusion encoder and uses the next layer of features to generate an attention mask for the current layer to reduce useless noise information in the low-level features, thereby better integrating the spatial detail information in the low-level features into the high-level features, effectively improving the model's classification performance for gastrointestinal disease areas. Secondly, the multi-level prototype reconstruction network further captures the semantic relevance between the support set and query set samples to enhance the distinguishing areas on the support image representation, and generates a calibration class center suitable for the current query sample for each query sample. At the same time, the classifier based on Euclidean distance outputs the classification result of the query sample, and the cross-entropy function is used to guide model optimization to ensure the accuracy of the classification results. Finally, the rare disease image classification model based on small sample learning is rigorously trained and tested and can output the classification results of each image. The present invention not only effectively solves the problem of difficulty in identifying lesion characteristics due to low contrast between lesions and intestinal walls, but also reduces intra-class differences and significantly improves classification accuracy, providing strong technical support for the diagnosis and treatment of gastrointestinal diseases, including rare diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of the steps of the classification method according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the structure of the classification model according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of a feature backtracking fusion encoder according to an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a multi-stage prototype reconstruction network structure according to an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the application effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0060] like Figure 1 As shown, the rare disease image classification method based on small sample learning described in this embodiment includes the following steps:

[0061] Step 1: Collect a dataset of endoscopic images of the gastrointestinal tract, including rare diseases. These datasets contain endoscopic images of different gastrointestinal tissues and corresponding tissue and lesion category annotations. Divide the dataset into a training set, a validation set, and a test set. The training samples, validation samples, and test samples are further divided into a support set and a query set, respectively.

[0062] Specifically, the step 1 includes:

[0063] Step 1.1: Collect a dataset of endoscopic images of the gastrointestinal tract containing rare diseases and divide it into a training set , validation set and test set , and there is no overlapping category between the three, in the training set , validation set and test set The set is further divided into support set and query set;

[0064] Step 1.2: Use bilinear interpolation to adjust the resolution of all images to 224×224;

[0065] Step 1.3: Perform data augmentation by randomly flipping the images and their labels in the training dataset horizontally, rotating them 90 degrees clockwise, and rotating them 180 degrees clockwise.

[0066] Step 1.4: Using the support images, support labels, and query images as inputs to the neural network, and using the query labels as supervisory information output by the rare disease image classification model based on small sample learning.

[0067] Step 2: Figure 2 As shown, a feature backtracking fusion encoder is constructed to extract support image features in the support set, extract query image features in the query set, and pass these feature information to a multi-stage prototype reconstruction network;

[0068] The step S2 comprises:

[0069] Step 2.1: Construct a feature back-fusion encoder. The input support set image and query set image share the feature back-fusion encoder to obtain the support image feature. and the query image features ;

[0070] Step 2.2: The feature backtracking fusion encoder consists of a ResNet18 encoder with initialized pre-trained weights, a backtracking attention learning module, and a multi-scale fusion module. The backtracking attention learning module refines shallow features and cleverly fuses shallow features with spatial details and deep features with rich semantic context layer by layer. This utilizes feature information at different levels to address the problem of gastrointestinal diseases with low contrast between the intestines and the intestines, making it difficult to identify lesions.

[0071] Step 2.2.1. Initialize the ResNet18 encoder with pre-trained weights and input the support set image and query set images , extract the feature representation of each image separately. The ResNet18 encoder consists of 1 convolution layer, 1 maximum pooling layer and 4 residual blocks, where each residual block contains 2 convolution layers and 1 residual connection. After a series of convolution and pooling operations, the ResNet18 encoder can convert the 224×224 original data into 7×7 feature encoding. Assume that the encoder input is First, a 7×7 convolutional layer is used to extract the 112×112 low-level features of the image, and a 56×56 intermediate layer feature is obtained through the maximum pooling layer. , then In the 4 residual blocks of the input series, the intermediate layer features of the 4 residual blocks output are recorded as ,in Indicates the The output features of the residual blocks and , the residual block is expressed as ;

[0072] Step 2.2.2, the backtracking attention learning module receives the and The output features of the residual block and , and output a two-dimensional attention mask , where C is the number of channels of the feature map, H is the height of the feature map, W is the width of the feature map, and at this time middle , that is, for the features of each layer of residual block (except for the features And the last layer of features ), which uses the features of the next layer of residual blocks to calculate a two-dimensional attention mask , and is used to refine the features of the current residual block. The retrospective attention learning module first uses convolutional layers with three convolution kernel sizes of 3×3, 5×5, and 7×7 to Perform convolution operations to obtain information features with different receptive fields ,in represents the convolution kernel size, Indicates the The features of the residual block are convolved with a kernel size of k×k, and then two consecutive point-by-point convolutions are performed to ensure better nonlinear learning ability and capture the complex correlation between spatial positions. The first 1×1 convolution layer is to convert the features The channel from Compress to ( is a reduction ratio, set to ), then through Layer and After the layer, the second 1×1 convolution layer is used to convert the channel from Converted to 1, then the features are upsampled by a factor of 2 and passed through The function gets a two-dimensional attention mask , and finally the three two-dimensional attention masks The average obtained The above process can be expressed as:

[0073] ;

[0074] ;

[0075] in yes function, It is the upsampling process. Finally, the features of each layer of residual block (except the feature And the last layer of features ) The refined features are .

[0076] Step 2.2.3, the multi-scale fusion module combines the features of each layer of residual blocks (except for the features And the last layer of features ) The refined features are As input, we use the refined low-level features Layer-by-layer fusion is used to supplement low-level features at different levels, and the final features are output after the fusion of low-level features and high-level features. The multi-scale fusion module first transforms the input features The features of three different receptive fields are obtained through three pooling operations of 3×3, 5×5 and 7×7. , and then sum the corresponding elements of the three to get the feature ,in , and then use global average pooling Get a channel-level embedding vector , and after three fully connected layers and normalization operations, three weight vectors are obtained , multiply it by the features of the corresponding three different receptive fields Get each layer feature , and finally merge layer by layer, As input features, we get , and so on to get the final feature .

[0077] Step 3: Figure 3 As shown, a multi-level prototype reconstruction network is constructed and used to capture the semantic correlation between support set and query set samples, and the semantic correlation is used to enhance the distinguishing regions on the support image representation, generating the corresponding support set class center for each query sample;

[0078] The step S3 specifically includes:

[0079] Step 3.1: Multi-level prototype reconstruction network uses feature back-fusion to fuse the final features delivered by the encoder As input, it includes support image features and query image features ;

[0080] Step 3.2: The multi-level prototype reconstruction network consists of a contrast-level relation module, a saliency-level relation module, and an attention-level relation module in parallel.

[0081] Step 3.2.1: The contrast-level relation module first compares the query image features Perform global average pooling to obtain the embedding vector , and then use the subtraction operation to get the embedding vector of the query image and all supported image features Compare each position of to output the relationship between the instance-level objects , to better represent the global features such as the contour boundary of the query object. The output feature relationship formula is expressed as ,in Indicates a broadcast operation, Repeated expansion to , Indicates absolute value operation, and finally .

[0082] Step 3.2.2: The salient level relationship module first analyzes the query image features Perform global average pooling to obtain the embedding vector , then As a convolution kernel, it supports image features in a deep separation convolution manner. Perform convolution operation to output the feature relationship between the two , in order to better highlight the main area features of the query object. The output feature relationship formula is expressed as ,in Represents the depth separation convolution operation, and finally .

[0083] Step 3.2.3: The attention-level relationship module first obtains the two embedded features of the support image and the query image through 1×1 convolution, and then calculates the spatial attention weight matrix of the two based on the matrix multiplication between the two embedded features. , then query the image features and the attention weight matrix Perform matrix multiplication to obtain spatial perception query image features , and extract spatial perception query image features through subtraction operation and supporting image features Local semantic relationship between , output Expressed as ,at last ;

[0084] Step 3.2.4, the multi-level prototype reconstruction network will finally and 、 、 Perform channel-level splicing and compress the channel through 1×1 convolution , get the reconstructed support image features ,Right now , thereby making full use of the complementary advantages to comprehensively extract the semantic information generated by the three different relationship modules and enhance the query feature area possessed by the support features to solve the problem of large differences within the gastrointestinal image class.

[0085] Step 4: Figure 4 As shown, a classifier based on Euclidean distance is constructed to calculate the Euclidean distance between the query image features and all support set features output by the multi-level prototype reconstruction network, and the opposite of the Euclidean distance is used as the classification score;

[0086] The step S4 specifically includes:

[0087] Step 4.1: The classifier based on Euclidean distance reconstructs the supporting image features transmitted in the multi-level prototype reconstruction network and query image features As input, output is the score of each category that the query image belongs to vector;

[0088] Step 4.1.1: The classifier based on Euclidean distance first supports image features Flatten and average the embedding vectors of the same category to get the prototype center of the category in Euclidean space , similarly query the image features Flattened to embedding vector ;

[0089] Step 4.1.2: Calculate the query image embedding vector using the Euclidean distance classifier Prototype center for each category The Euclidean distance to obtain the query image Belong to each category Score , select the category with the highest score as the prediction result of the query image category.

[0090] Step 5: Use the collected training samples to train the gastrointestinal disease classification model composed of the feature backtracking fusion encoder, the multi-level prototype reconstruction network, and the Euclidean distance-based classifier, and use the validation set samples to verify and select the optimal network parameters to form the final model;

[0091] The step S5 specifically includes:

[0092] Step 5.1: Build a rare disease image classification model based on small sample learning, use the Adam optimizer, and set the initial learning rate to , the training epoch is set to 400, and each epoch consists of 200 episodes.

[0093] Step 5.2: During the training process, use the training set The rare disease image classification model based on small sample learning is trained and optimized using the cross entropy loss function, which is expressed as:

[0094] ,in Indicates the number of supported image categories.

[0095] Step 5.3: Train the model using the loss function, adjust the parameters through backpropagation, and select the model parameters with the highest accuracy in the validation set and save them;

[0096] Step 5.4: Select evaluation indicators including accuracy to evaluate the model performance.

[0097] Step 6: When applying, collect endoscopic images of the gastrointestinal tract containing rare diseases and input them into the rare disease image classification model based on small sample learning. After calculation, output the classification result of each query sample.

[0098] like Figure 5As shown in the figure, the input image is an image taken by an endoscope in the gastrointestinal tract, the label is the image classification result manually annotated by professional gastrointestinal disease experts, and the model output result is the classification accuracy result automatically output by the rare disease image classification model based on small sample learning. The average classification accuracy of the five diseases reached 83.56%.

[0099] In summary, the embodiment of the present invention cleverly constructs a feature backtracking fusion encoder and uses the next layer features to generate an attention mask for the current layer to reduce useless noise information in the low-level features, thereby better integrating the spatial detail information in the low-level features into the high-level features, effectively improving the model's classification performance for gastrointestinal disease areas including rare diseases. Secondly, the introduction of the multi-level prototype reconstruction network further enhances the distinguishing areas on the support image representation by capturing the semantic relevance between the support set and the query set samples, and generates a calibration class center suitable for the current query sample for each query sample. At the same time, the classifier based on Euclidean distance outputs the classification result of the query sample, and the cross-entropy function is used to guide model optimization to ensure the accuracy of the classification results. Ultimately, the rare disease image classification model based on small sample learning has shown excellent classification performance after rigorous training and testing. It not only solves the problem of difficulty in identifying lesion features due to low contrast between lesions and intestinal walls, but also narrows the intra-class differences, providing strong technical support for the diagnosis and treatment of gastrointestinal diseases including rare diseases.

[0100] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0101] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0102] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the rare disease image classification methods based on small sample learning in the above embodiments.

[0103] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0104] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0105] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0106] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0107] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A rare disease image classification method based on small sample learning, characterized by: The following steps are included: S1. Collect an endoscopic image dataset containing rare diseases in the gastrointestinal tract and divide the dataset into a support set and a query set. S2. Constructing a feature backtracking fusion encoder to extract support image features in the support set and query image features in the query set, and passing these feature information to a multi-stage prototype reconstruction network; S3. Use a multi-level prototype reconstruction network to capture the semantic correlation between support set and query set samples, and use the semantic correlation to enhance the distinguishing regions on the support image representation, and generate the corresponding support set class center for each query sample; S4. Construct a classifier based on Euclidean distance to calculate the Euclidean distance between the query image features and all support set features output by the multi-level prototype reconstruction network, and use the inverse of the Euclidean distance as the classification score, thus forming a rare disease image classification model based on small sample learning; S5. Using the collected training samples to train the rare disease image classification model based on small-sample learning, and using the validation set samples to verify and select the optimal network parameters, thereby forming a final rare disease image classification model based on small-sample learning; S6. During application, endoscopic images of the gastrointestinal tract containing rare diseases are collected and input into the rare disease image classification model based on small sample learning, and the classification result of each query sample is output after calculation; The step S3 comprises: S31, Multi-level prototype reconstruction network to feature back-fusion encoder transfer of the final feature As input, it includes support image features and query image features ; S32, the multi-level prototype reconstruction network consists of a contrast-level relation module, a saliency-level relation module, and an attention-level relation module in parallel; S321, the contrast level relationship module first compares the query image features Perform global average pooling to obtain the embedding vector , and then use the subtraction operation to get the embedding vector of the query image and all supported image features Compare each position of to output the relationship between the instance-level objects ; S322, the saliency level relationship module first performs the query image feature Perform global average pooling to obtain the embedding vector , then As a convolution kernel, it supports image features in a deep separation convolution manner. Perform convolution operation to output the feature relationship between the two ; S323, the attention-level relationship module first obtains the two embedded features of the support image and the query image through 1×1 convolution, and then calculates the spatial attention weight matrix of the two based on the matrix multiplication between the two embedded features , then query the image features and the attention weight matrix Perform matrix multiplication to obtain spatial perception query image features , and extract spatial perception query image features through subtraction operation and supporting image features Local semantic relationship between ; S324, the multi-level prototype reconstruction network will finally and 、 、 Perform channel-level splicing and compress the channel through 1×1 convolution , get the reconstructed support image features .

2. The rare disease image classification method based on small sample learning according to claim 1, characterized in that: The step S1 comprises: S11. Collect an endoscopic image dataset containing rare diseases in the gastrointestinal tract and divide it into a training set , validation set and test set , and there is no overlapping category between the three, in the training set , validation set and test set The set is further divided into support set and query set; S12, using bilinear interpolation method to adjust the resolution of all images to 224×224; S13, perform data augmentation by randomly flipping the images and their labels in the training dataset horizontally, rotating them 90 degrees clockwise, and rotating them 180 degrees clockwise; S14. Using the support image, support label, and query image as inputs of a neural network, and using the query label as supervisory information output by the rare disease image classification model based on small sample learning.

3. The rare disease image classification method based on small sample learning according to claim 1, characterized in that: The step S2 comprises: S21, construct a feature backtracking fusion encoder, the input support set image and the query set image share the feature backtracking fusion encoder, and obtain the support image feature and the query image features ; S22, the feature backtracking fusion encoder consists of a ResNet18 encoder with initialized pre-trained weights, a backtracking attention learning module, and a multi-scale fusion module; S221, initialize the ResNet18 encoder with pre-trained weights and input the support set image and query set images , extract the feature representation of each image respectively; the ResNet18 encoder consists of 1 convolutional layer, 1 maximum pooling layer and 4 residual blocks, where each residual block contains 2 convolutional layers and 1 residual connection; The ResNet18 encoder converts the 224×224 original data into a 7×7 feature code through a series of convolution and pooling operations; Assume the encoder input is First, a 7×7 convolutional layer is used to extract the 112×112 low-level features of the image, and a 56×56 intermediate layer feature is obtained through the maximum pooling layer. , then In the 4 residual blocks of the input series, the intermediate layer features of the 4 residual blocks output are recorded as ,in Indicates the The output features of the residual blocks and , the residual block is expressed as ; S222, the retrospective attention learning module receives and The output features of the residual block and , and output a two-dimensional attention mask , where C is the number of channels of the feature map, H is the height of the feature map, W is the width of the feature map, and at this time middle , that is, for the features of each layer of residual blocks, in addition to the features And the last layer of features , which uses the features of the next layer of residual blocks to calculate a two-dimensional attention mask , and is used to refine the current residual block features; The retrospective attention learning module first uses convolutional layers with three convolution kernel sizes of 3×3, 5×5 and 7×7 to Perform convolution operations to obtain information features with different receptive fields ,in represents the convolution kernel size, Indicates the The residual block features are convolved with a kernel size of k×k, and then two consecutive point-by-point convolutions are performed to ensure better nonlinear learning ability and capture the complex correlation between spatial positions; the first 1×1 convolution layer is to convert the features The channel from Compress to , is a reduction ratio, set to , then through Layer and After the layer, the second 1×1 convolution layer is used to convert the channel from Converted to 1, then the features are upsampled by a factor of 2 and passed through The function gets a two-dimensional attention mask , and finally the three two-dimensional attention masks The average obtained ; The above process is expressed as: ； ； in yes function, It is the upsampling process; finally, the features of each layer of residual block are And the last layer of features , the refined features are ; S223, the multi-scale fusion module combines the features of each layer of residual blocks, except for the features And the last layer of features , the refined features are As input, we use the refined low-level features Layer-by-layer fusion is used to supplement low-level features at different levels, and the final features are output after the fusion of low-level features and high-level features. ; The multi-scale fusion module first transforms the input features The features of three different receptive fields are obtained through three pooling operations of 3×3, 5×5 and 7×7. , and then sum the corresponding elements of the three to get the feature ,in , and then use global average pooling Get a channel-level embedding vector , and after three fully connected layers and normalization operations, three weight vectors are obtained , multiply it by the features of the corresponding three different receptive fields Get each layer feature , and finally merge layer by layer, As input features, we get , and so on to get the final feature .

4. The rare disease image classification method based on small sample learning according to claim 3, characterized in that: The step S4 comprises: S41. The classifier based on Euclidean distance reconstructs the supporting image features transmitted in the multi-level prototype reconstruction network and query image features As input, output is the score of each category that the query image belongs to vector; S411, the classifier based on Euclidean distance will first support image features Flatten and average the embedding vectors of the same category to get the prototype center of the category in Euclidean space , similarly query the image features Flattened to embedding vector ; S412, Euclidean distance-based classifier calculates the query image embedding vector Prototype center for each category The Euclidean distance to obtain the query image Belong to each category Score , select the category with the highest score as the prediction result of the query image category.

5. The rare disease image classification method based on small sample learning according to claim 2, characterized in that: The step S5 comprises: S51. Build a rare disease image classification model based on small sample learning, use the Adam optimizer, and set the initial learning rate to , the training epoch is set to 400, and each epoch consists of 200 episodes; S52, during the training process, using the training set The rare disease image classification model based on small sample learning is trained and optimized using the cross entropy loss function, which is expressed as: ,in Indicates the number of supported image categories; S53. Train the model through the loss function, adjust the parameters through backpropagation, and select the model parameters with the highest accuracy in the validation set for saving.

6. The rare disease image classification method based on small sample learning according to claim 5, characterized in that: The step S5 further includes: S55. Select evaluation indicators including accuracy to evaluate model performance.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

Images