Model training method, digestive tract endoscopy image recognition method, device and medium

By preprocessing the digestive tract endoscopic image sample dataset and training in multi-category context analysis model, combined with convolutional neural network and vision transformer network, the problem of inefficient image recognition of digestive tract endoscopic image is solved, achieving efficient and accurate automated identification.

CN118470453BActive Publication Date: 2024-12-03NANFANG HOSPITAL OF SOUTHERN MEDICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410388384.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-01
Publication Date
2024-12-03
Estimated Expiration
2044-04-01

AI Technical Summary

Technical Problem

The existing digestive tract endoscopic image recognition methods are inefficient, traditional image processing technology is time-consuming and labor-intensive, deep learning models lack interpretability and limited recognition accuracy, which cannot meet the current image recognition analysis needs.

Method used

A model training method is proposed, including obtaining the digestive tract endoscopic image sample data set for data preprocessing and quality screening, building a multi-category context analysis model, using the training group data set to train the model, and using a convolutional neural network and a visual transformer The network-combined recognition method outputs image prediction results and regional attention scores to achieve automated recognition.

Benefits of technology

It improves the efficiency and accuracy of gastrointestinal endoscopic image recognition, meets the recognition and analysis needs of gastrointestinal endoscopic images, realizes automatic recognition, and improves diagnostic efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470453B_ABST
    Figure CN118470453B_ABST
Patent Text Reader

Abstract

The present application discloses a model training method, a method, device and medium for gastrointestinal endoscopic image recognition, which obtains a sample data set of gastrointestinal endoscopic images, performs data preprocessing, data quality screening and data grouping on the sample set of gastrointestinal endoscopic images, obtains multiple training group data sets, uses the training group data sets to train a multi-category context analysis model, obtains a gastrointestinal endoscopic image recognition model, inputs the gastrointestinal endoscopic image into the gastrointestinal endoscopic image recognition model, and uses the gastrointestinal endoscopic image recognition model to output the image prediction results corresponding to the gastrointestinal endoscopic image and the attention scores of each area of ​​the image. The present application can establish a high-quality training data set, train the gastrointestinal endoscopic image recognition model, use the gastrointestinal endoscopic image recognition model to realize automatic recognition of gastrointestinal endoscopic images, improve image recognition efficiency and accuracy, meet the recognition and analysis needs of gastrointestinal endoscopic images, and is widely used in the field of medical image analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of medical image analysis, and in particular to a model training method, a gastrointestinal endoscopic image recognition method, a device and a medium. Background Art

[0002] Gastrointestinal endoscopy is a diagnostic and therapeutic procedure that allows one to image, evaluate, and treat gastrointestinal diseases. Each diagnosis of gastrointestinal endoscopy generates a large number of digestive tract endoscopic images. Before clinicians can make a diagnosis, they need to spend a lot of effort and a long time to find the lesion image from the huge image data set, which is inefficient.

[0003] At present, in terms of the recognition of digestive tract endoscopy images, the existing methods can be roughly divided into two categories. The first category is based on the combination of traditional image processing technology and machine learning, and the second category is based on deep learning.

[0004] The first type of method requires artificial design to extract image features when using traditional image processing technology, which is time-consuming, labor-intensive and inefficient. The second type of method is widely used in the current field of medical image analysis. The most common method is to use convolutional neural network models for image analysis, but the convolutional neural network model itself is not interpretable. Therefore, its recognition accuracy is limited, the detection efficiency is not high, and it cannot meet the current image recognition and analysis needs. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the above-mentioned related technologies, the embodiments of the present application propose a model training method, a gastrointestinal endoscopic image recognition method, a device and a medium, which can realize the automatic recognition of gastrointestinal endoscopic images and improve the image recognition efficiency and accuracy.

[0006] On the one hand, an embodiment of the present application proposes a model training method, the method comprising the following steps:

[0007] Obtain a sample dataset of digestive tract endoscopy images;

[0008] Performing data preprocessing and data quality screening on each sample image in the digestive tract endoscopy image sample data set to obtain a digestive tract endoscopy image sample processed data set;

[0009] According to a plurality of preset data categories, the digestive tract endoscopy image sample processing data set is grouped to obtain a training group data set corresponding to each of the data categories;

[0010] Build a multi-category contextual analysis model;

[0011] Using each of the training group data sets to train the multi-category context analysis model, and iteratively optimizing the multi-category context analysis model in combination with a classification loss function;

[0012] Obtain a test data set, perform model evaluation on the multi-category context analysis model using the test data set according to preset model evaluation indicators, determine the model evaluation result, and determine whether to continue training the multi-category context analysis model based on the model evaluation result.

[0013] In some embodiments, the step of performing data preprocessing and data quality screening on each sample image in the digestive tract endoscopy image sample data set to obtain the digestive tract endoscopy image sample processed data set specifically includes:

[0014] Get image quality conditions;

[0015] According to the image quality condition and the image quality of each of the sample images, the sample images in the digestive tract endoscopy image sample data set that do not meet the image quality condition are eliminated;

[0016] The redundant information of each sample image in the digestive tract endoscopy image sample data set after the image is eliminated is deleted to obtain the digestive tract endoscopy image sample processed data set.

[0017] In some embodiments, the step of grouping the digestive tract endoscopy image sample processing data set according to a plurality of preset data categories to obtain a training group data set corresponding to each of the data categories specifically includes:

[0018] According to each of the data categories, classify and label each of the sample images in the digestive tract endoscopy image sample processing data set to determine an image category label corresponding to each of the sample images, wherein the image category label corresponds to the data category;

[0019] Determining a number of the sample images corresponding to each of the data categories according to the image category labels corresponding to each of the sample images;

[0020] A plurality of the sample images belonging to the same data category in the digestive tract endoscopy image sample processing data set are merged to obtain the training group data set corresponding to each data category.

[0021] On the other hand, an embodiment of the present application proposes a method for digestive tract endoscopy image recognition, the method comprising the following steps:

[0022] Acquire a number of digestive tract endoscopy images to be identified;

[0023] Each of the digestive tract endoscopic images is input into a digestive tract endoscopic image recognition model, and the digestive tract endoscopic image recognition model is used to output image prediction results corresponding to each of the digestive tract endoscopic images and attention scores for each area of ​​the image; the digestive tract endoscopic image recognition model is trained using the model training method described above.

[0024] In some embodiments, the method further comprises the following steps:

[0025] According to the attention scores of each area of ​​the image corresponding to each digestive tract endoscopy image, the average attention score of each area of ​​the image is calculated, and the average attention score of each area of ​​the image is normalized and converted into a color value of each area of ​​the image;

[0026] A target digestive tract endoscopic image is randomly selected from the plurality of digestive tract endoscopic images, and the color values ​​of each region of the image are mapped to corresponding image regions in the target digestive tract endoscopic image to obtain an image feature hotspot map.

[0027] In some embodiments, the step of inputting each of the digestive tract endoscopic images into a digestive tract endoscopic image recognition model, and using the digestive tract endoscopic image recognition model to output image prediction results corresponding to each of the digestive tract endoscopic images and attention scores for each region of the image specifically includes:

[0028] Inputting the digestive tract endoscopy image into a convolutional neural network branch in the digestive tract endoscopy image recognition model, extracting image features, and determining a first image feature;

[0029] Inputting the digestive tract endoscopic image into the visual transformer network branch in the digestive tract endoscopic image recognition model, segmenting the digestive tract endoscopic image to obtain a plurality of image blocks, aggregating background information of each of the image blocks, and determining a second image feature of each of the image blocks;

[0030] spatially concatenating the first image feature with the second image feature of each of the image blocks to obtain feature space information;

[0031] Inputting the feature space information into the feature selection module in the digestive tract endoscopy image recognition model, filtering each spatial cascade feature in the feature space information, and obtaining corresponding feature space processing information;

[0032] Using the digestive tract endoscopy image recognition model to perform global average pooling and probability distribution mapping on the feature space processing information, and determine the prediction category and attention score of each feature space in the feature space processing information;

[0033] Each of the feature spaces is located on the digestive tract endoscopic image, and each of the feature spaces is mapped to the digestive tract endoscopic image according to the predicted category and attention score of each of the feature spaces to obtain a model predicted image.

[0034] In some embodiments, the step of spatially concatenating the first image feature and the second image feature of each image block to obtain feature space information specifically includes:

[0035] spatially cascading the first image feature and the second image feature of each image block to obtain an image block cascade feature corresponding to each image block;

[0036] Each of the image blocks is located in the digestive tract endoscopic image, and the image block cascade features corresponding to each of the image blocks are mapped to the digestive tract endoscopic image to determine the feature space information corresponding to the digestive tract endoscopic image.

[0037] On the other hand, an embodiment of the present application provides a digestive tract endoscopy image recognition device, the device comprising:

[0038] The first module is used to obtain a number of digestive tract endoscopy images to be identified;

[0039] The second module is used to input each of the gastrointestinal endoscopic images into a gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction results corresponding to each of the gastrointestinal endoscopic images and the attention scores of each area of ​​the image; the gastrointestinal endoscopic image recognition model is trained using the model training method described above.

[0040] On the other hand, an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the model training method or the gastrointestinal endoscopic image recognition method described above.

[0041] On another aspect, an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model training method or the gastrointestinal endoscopic image recognition method described above.

[0042] The present application provides a model training method, a gastrointestinal endoscopic image recognition method, a device and a medium, which obtain a gastrointestinal endoscopic image sample data set, perform data preprocessing, data quality screening and data grouping on the gastrointestinal endoscopic image sample set, obtain multiple training group data sets, use the training group data sets to train a multi-category context analysis model, obtain a gastrointestinal endoscopic image recognition model, input each gastrointestinal endoscopic image into the gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction result corresponding to the gastrointestinal endoscopic image and the attention score of each area of ​​the image. The present application can establish a high-quality training data set, train the gastrointestinal endoscopic image recognition model, use the gastrointestinal endoscopic image recognition model to realize the automatic recognition of gastrointestinal endoscopic images, improve the efficiency and accuracy of image recognition, and meet the recognition and analysis needs of gastrointestinal endoscopic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a model training method provided in an embodiment of the present application;

[0044] Figure 2 is a flow chart of a method for gastrointestinal endoscopy image recognition provided by an embodiment of the present application;

[0045] Figure 3 It is a structural schematic diagram of a digestive tract endoscopy image recognition device provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0051] Reference Figure 1 , Figure 1 This is an optional flowchart of a model training method provided in an embodiment of the present application. The method may include but is not limited to steps S101 to S106:

[0052] Step S101, obtaining a digestive tract endoscopy image sample data set;

[0053] Step S102, performing data preprocessing and data quality screening on each sample image in the digestive tract endoscopy image sample data set to obtain a digestive tract endoscopy image sample processed data set;

[0054] Step S103, grouping the digestive tract endoscopy image sample processing data set according to a plurality of preset data categories to obtain a training group data set corresponding to each data category;

[0055] Step S104, constructing a multi-category context analysis model;

[0056] Step S105, using each training group data set to train a multi-category context analysis model, and iteratively optimizing the multi-category context analysis model in combination with a classification loss function;

[0057] Step S106, obtain a test data set, perform model evaluation on the multi-category context analysis model using the test data set according to preset model evaluation indicators, determine the model evaluation result, and determine whether to continue training the multi-category context analysis model based on the model evaluation result.

[0058] In step S101 of some embodiments, a large number of digestive tract endoscopy sample images are obtained through the hospital case data system, medical cooperation related units and medical case database to form a digestive tract endoscopy image sample data set.

[0059] In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0060] Step S201, obtaining image quality conditions;

[0061] Step S202, according to the image quality condition and the image quality of each sample image, the sample images in the digestive tract endoscopy image sample data set that do not meet the image quality condition are eliminated;

[0062] Step S203, deleting redundant image information of each sample image in the digestive tract endoscopy image sample data set after the images are eliminated, to obtain a digestive tract endoscopy image sample processed data set.

[0063] In step S201 of some embodiments, the image quality condition may include but is not limited to an image clarity condition, an image information completeness condition, and an image size condition. Optionally, the image clarity condition is to eliminate images whose image clarity is less than a given clarity threshold, the image information completeness condition is to eliminate images that do not include all preset image information, and the image size condition is to eliminate images whose image size is not a given size or cannot be adjusted to a given size.

[0064] In step S202 of some embodiments, sample images that do not meet image quality requirements are eliminated from the digestive tract endoscopy image sample data set to establish a high-quality digestive tract endoscopy image sample data set.

[0065] In step S203 of some embodiments, the image redundant information includes the patient privacy information corresponding to the image and the privacy information of the doctor operating the endoscope, etc. The image redundant information involved in the image is deleted to reduce the interference from the image redundant information and provide a good data foundation for the subsequent training of the multi-category context analysis model.

[0066] In some embodiments, step S103 may include but is not limited to steps S301 to S303:

[0067] Step S301, classifying and labeling each sample image in the digestive tract endoscopy image sample processing data set according to each data category, and determining an image category label corresponding to each sample image, wherein the image category label corresponds to the data category;

[0068] Step S302, determining a number of sample images corresponding to each data category according to the image category label corresponding to each sample image;

[0069] Step S303: merge several sample images belonging to the same data category in the digestive tract endoscopy image sample processing data set to obtain a training group data set corresponding to each data category.

[0070] In steps S301 to S302 of some embodiments, multiple data categories of gastrointestinal endoscopy images are preset, such as data category A, data category B, and data category C, each data category corresponds to an image category label, data category A corresponds to image category label A, data category B corresponds to image category label B, and data category C corresponds to image category label C. Each sample image is classified and labeled, and the image category label corresponding to each sample image is determined, thereby achieving integrated classification of the sample images and further improving the quality of the training data set.

[0071] In step S303 of some embodiments, sample images are merged according to the image category labels corresponding to each sample image to obtain multiple training group data sets. Assume that there are sample images 1 to 15, the image category labels corresponding to sample images 1 to 5 are all image category labels A, the image category labels corresponding to sample images 8 to 12 are all image category labels B, and the image category labels corresponding to the remaining sample images are all image category labels B. The sample images are merged, and sample images 1 to 5 form training group data set A. Training group data set A is the sample image data set corresponding to data category A, and so on.

[0072] In step S104 of some embodiments, a multi-category context analysis model is constructed. The multi-category context analysis model is an artificial intelligence model for identifying gastrointestinal endoscopic images by combining a vision transformer model (ViT) with a convolutional neural network (CNN) model for context analysis.

[0073] The multi-category context analysis model may also include, but is not limited to, a channel attention (SE, squeeze and excitation block) module and a feature selection module, wherein the main components of the SE module include:

[0074] Squeeze submodule: This part compresses the spatial information of each channel of the input feature map into a global feature through global average pooling, and its output dimension is 1x1xC;

[0075] Excitation submodule: This part predicts the importance of each channel through a fully connected layer, and then weights the original feature map according to these importances to highlight important features and suppress unimportant features;

[0076] Scaling submodule: Multiply the weight values ​​of each channel calculated by the SE module by the two-dimensional matrix of the corresponding channel of the original feature map to obtain the enhanced feature map.

[0077] In step S105 of some embodiments, each training group data set is input into a multi-category context analysis model, and the recognition results of each sample image are output accordingly. The recognition results of each sample image and the classification loss function based on the true label are combined to iteratively optimize the various parameters of the above-mentioned multi-category context analysis model.

[0078] Optionally, in order to optimize the learning process of the multi-category context analysis model to predict the lesion type, each training group data set is randomly divided into a training set and a validation set, wherein, during the training process of the multi-category context analysis model, the validation set is used to train the attention pooling network in the multi-category context analysis model to prevent overfitting, and the validation set is also used to optimize hyperparameters, such as the training period, learning rate and image preprocessing options of the multi-category context analysis model. The training set and the corresponding classification loss function based on the true label are used to train the image-level model in the multi-category context analysis model as a feature extraction network.

[0079] In step S106 of some embodiments, in order to evaluate the performance of the model on an independent test data set, a receiver operating characteristic (AUROC) curve is plotted and a confusion matrix is ​​generated to visualize the performance of the multi-category context analysis model. In addition to the AUROC curve, the model evaluation value corresponding to each model evaluation indicator is calculated based on model evaluation indicators such as the recall rate, precision rate, F1 score, and overall and balanced accuracy of the model.

[0080] Optionally, obtain the model evaluation indicator threshold corresponding to each model evaluation indicator. When the model evaluation values ​​corresponding to the above-mentioned model evaluation indicators exceed the model evaluation indicator threshold corresponding to each model evaluation indicator, stop training the multi-category context analysis model; otherwise, continue training the multi-category context analysis model.

[0081] At the same time, in order to compare the performance of the MCC model relative to each single backbone, single backbone training of the CNN model and the VIT model was performed. The classification head of these two models consists of a fully connected layer, which is used to directly predict the multi-class probability distribution of each disease type. In the single backbone training, the transfer learning strategy was adopted, and image enhancement and other techniques similar to those used in MCC model training were used in combination with the classification loss function to train the CNN model and the VIT model. Similar to the evaluation of the MCC model, the AUROC curve was used to measure the model performance of each trained single backbone model (CNN model and VIT model).

[0082] In some embodiments, illustratively, the above model training method is applied to the training of a colorectal benign ulcerative lesion image recognition model, and the specific process is as follows:

[0083] The first step is to collect a large number of images of benign ulcerative lesions of the large intestine to form a training sample image set based on the hospital case data system and the high-quality, large-scale endoscopic diagnosis and treatment cases (more than 100,000 cases / year) of medical cooperation units, as well as clinical databases of ulcerative colitis, Crohn's disease, etc. The training sample image set is screened to exclude case images with colorectal tumors or other colorectal lesions in the training sample image set, endoscopic biopsy and treatment-related images, and case images with unclear diagnosis and incomplete medical records.

[0084] The second step is to perform data preprocessing and data quality screening on the screened training sample image set, and according to the preset image quality conditions, remove the sample images that do not meet the image quality conditions, delete the image redundant information of each sample image, and obtain the training sample processed image set;

[0085] The third step is to divide the training sample processing image set into data. The preset data categories are normal group, ulcerative colitis group (UC), Crohn's disease group (CD), intestinal tuberculosis group (ITB), and ischemic enteritis group (IC). According to these five data categories, the sample images in the training sample processing image set are classified and labeled to determine the image category label corresponding to each sample image. According to the image category label corresponding to each sample image, the sample images of the same data category are merged to obtain five training group data sets: normal group training set, UC group training set, CD group training set, ITB group training set, and IC group training set.

[0086] The fourth step is to build a multi-category context analysis model, use the training group data sets to train the multi-category context analysis model, and iteratively optimize the multi-category context analysis model in combination with the classification loss function;

[0087] The fifth step is to obtain the test data set from the clinical database and preprocess the test data set, including image enhancement and redundant information deletion, and then input the processed test data set into the multi-category context analysis model for model evaluation. Combined with the AUROC curve and the model evaluation value corresponding to each model evaluation indicator, when the model evaluation value corresponding to each model evaluation indicator exceeds the model evaluation indicator threshold corresponding to each model evaluation indicator, stop training the multi-category context analysis model, otherwise continue training the multi-category context analysis model.

[0088] Reference Figure 2 , Figure 2is an optional flow chart of a method for recognizing a digestive tract endoscopy image provided in an embodiment of the present application. The method may include but is not limited to steps A101 to A102:

[0089] Step A101, obtaining a plurality of digestive tract endoscopy images to be identified;

[0090] Step A102, input each gastrointestinal endoscopic image into a gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction results corresponding to each gastrointestinal endoscopic image and the attention scores of each area of ​​the image; the gastrointestinal endoscopic image recognition model is trained using the above-mentioned model training method.

[0091] In some embodiments, the above-mentioned digestive tract endoscopy image recognition method may further include steps A103 to A104:

[0092] Step A103, according to the attention scores of each image region corresponding to each digestive tract endoscopy image, the average attention score of each image region is calculated, and the average attention score of each image region is normalized and converted into the color value of each image region;

[0093] Step A104, randomly select a target digestive tract endoscopy image from multiple digestive tract endoscopy images, map the color values ​​of each image area to the corresponding image areas in the target digestive tract endoscopy image, and obtain an image feature hotspot map.

[0094] In some embodiments, step A102 may include, but is not limited to, steps A201 to A206:

[0095] Step A201, inputting the digestive tract endoscopy image into a convolutional neural network branch in a digestive tract endoscopy image recognition model, extracting image features, and determining a first image feature;

[0096] Step A202, inputting the digestive tract endoscopy image into the visual transformer network branch in the digestive tract endoscopy image recognition model, segmenting the digestive tract endoscopy image to obtain a plurality of image blocks, aggregating background information of each image block, and determining a second image feature of each image block;

[0097] Step A203, spatially concatenating the first image feature and the second image feature of each image block to obtain feature space information;

[0098] Step A204, inputting the feature space information into the feature selection module in the digestive tract endoscopy image recognition model, filtering each spatial cascade feature in the feature space information, and obtaining corresponding feature space processing information;

[0099] Step A205, using the digestive tract endoscopy image recognition model to perform global average pooling and probability distribution mapping on the feature space processing information, and determine the prediction category and attention score of each feature space in the feature space processing information;

[0100] Step A206, locate each feature space on the gastrointestinal endoscopic image, and map each feature space to the gastrointestinal endoscopic image according to the predicted category and attention score of each feature space to obtain a model predicted image.

[0101] In steps A201 to A202 of some embodiments, a ViT (Visual Transformer Network) backbone is used to identify relevant features between different regions of the image from the segmented blocks of the image, and is combined with a CNN backbone that is very effective in extracting local features.

[0102] Specifically, the gastrointestinal endoscopy image input is divided into two parallel branches, which are built based on the state-of-the-art Resnet50 (residual network) and Swin Transformer (Swin Transformer) neural architectures.

[0103] For the CNN branch (convolutional neural network branch), the gastrointestinal endoscopic images are directly input into the CNN backbone, and the CNN backbone is used to extract the global features of the gastrointestinal endoscopic images.

[0104] For the ViT branch (visual transformer network branch), the gastrointestinal endoscopic image is segmented into multiple image blocks, and the sequence of these image blocks is input into the ViT model to extract the features of each image block, that is, to extract the local features of the gastrointestinal endoscopic image, and aggregate the object background extracted from each small block through the self-attention mechanism. This self-attention mechanism helps the ViT model to have better generalization capabilities in the pre-training stage and is suitable for downstream tasks that need to utilize global features.

[0105] In step A203 of some embodiments, specifically, the first image feature and the second image feature of each image block are spatially cascaded to obtain an image block cascade feature corresponding to each image block, then each image block is located in the gastrointestinal endoscopic image, the image block cascade feature corresponding to each image block is mapped to the gastrointestinal endoscopic image, and the feature space information corresponding to the gastrointestinal endoscopic image is determined.

[0106] In steps A204 to A205 of some embodiments, the feature space information provides diversified and heterogeneous information for disease classification, but also contains redundant and overlapping information. Therefore, a feature selection module M is used to weight various features, which is specifically expressed by the following formula:

[0107] F cat ∈R 7*7*(1024+2048) ;

[0108] Frw =M(F cat );

[0109]

[0110] w i ∈R 1024+2048 ;

[0111] Among them, R represents the real number space corresponding to the cascade feature, 7*7*1024 is the extraction dimension of the cascade feature, represents channel multiplication, M is the feature selection function, F rw is a feature highlighting function used to highlight the features most relevant to the predicted category, F cat represents the cascade feature, F tr represents the convolutional layer of the processing element, i represents the i-th cascade feature, and w i Select weights for the features corresponding to the i-th cascade feature.

[0112] The compression function F of the SE module se Use and activation function F ex Generate the above feature selection weights w i , where the compression function F se By matching the processed feature image dimensions with global average pooling followed by a fully connected layer, the probability distribution mapping of the feature space can be expressed by the following formula:

[0113] z i ∈R 1024+2048 ;

[0114] z i =F lr (F se (F cat ));

[0115] Among them, F tr The function represents the operator of two fully connected layers, z i is the probability distribution mapping corresponding to the i-th cascade feature, which is used to represent the predicted category and attention score of the feature space corresponding to the i-th cascade feature, using the activation function F ex The threshold-based gating mechanism will z i Mapping from probability distribution to feature selection weights w i , thereby determining the predicted category and attention score of each feature space in the feature space processing information.

[0116] In step A206 of some embodiments, feature space processing information of the last layer of the encoding layer corresponding to the digestive tract endoscopy image recognition model is collected, each feature space is located on the digestive tract endoscopy image, and it is converted into a localization map with a gradient class activation map, which displays the important image areas used to predict the disease category in the input digestive tract endoscopy image. After the last layer of features is globally averaged and pooled, it is simplified to a two-dimensional space through principal component analysis to realize the visualization of the feature space. Each point in the feature space represents an image and is labeled according to its disease category.

[0117] In steps A103 and A104 of some embodiments, the attention scores of various image regions of all gastrointestinal endoscopic images are visualized to explain their relative importance in predicting the patient's disease type.

[0118] The average attention score of each area of ​​the image is normalized in the range of 0 to 1 and converted into corresponding color values ​​(RGB values). Optionally, the higher the attention of the image area, the darker the color. For example, dark red is the darkest color, and image area X is displayed in dark red. This image area X is the key attention area. The image feature hotspot map can be used to explain whether the gastrointestinal endoscopic image recognition model pays attention to the lesion area during prediction, and can provide some medical references for diagnostic personnel.

[0119] Reference Figure 3 , Figure 3 This is an optional structural diagram of a digestive tract endoscopy image recognition device provided in an embodiment of the present application. The device may include but is not limited to:

[0120] The first module is used to obtain a number of digestive tract endoscopy images to be identified;

[0121] The second module is used to input each gastrointestinal endoscopic image into a gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction results corresponding to each gastrointestinal endoscopic image and the attention scores of each area of ​​the image; the gastrointestinal endoscopic image recognition model is trained using the above-mentioned model training method.

[0122] The specific implementation of the digestive tract endoscopy image recognition device is basically the same as the specific implementation of the above-mentioned digestive tract endoscopy image recognition method, and will not be repeated here.

[0123] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model training method or digestive tract endoscopy image recognition method when executing the computer program. The electronic device can be any smart terminal including a tablet computer.

[0124] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0125] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0126] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the model training method or the digestive tract endoscopy image recognition method of the embodiments of this application;

[0127] Input / output interface 903, used to implement information input and output;

[0128] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0129] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0130] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0131] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training method or gastrointestinal endoscopy image recognition method.

[0132] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The embodiments of the present application provide a model training method, a gastrointestinal endoscopic image recognition method, a device and a medium, which obtain a sample data set of gastrointestinal endoscopic images, perform data preprocessing, data quality screening and data grouping on the sample set of gastrointestinal endoscopic images, obtain multiple training group data sets, use the training group data sets to train a multi-category context analysis model, obtain a gastrointestinal endoscopic image recognition model, input each gastrointestinal endoscopic image into the gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction result corresponding to the gastrointestinal endoscopic image and the attention score of each area of ​​the image. The present application can establish a high-quality training data set, train the gastrointestinal endoscopic image recognition model, use the gastrointestinal endoscopic image recognition model to realize the automatic recognition of gastrointestinal endoscopic images, improve the efficiency and accuracy of image recognition, and meet the recognition and analysis needs of gastrointestinal endoscopic images.

[0134] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0135] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0136] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method, characterized in that: The method comprises the following steps: Obtain a sample dataset of digestive tract endoscopy images; Performing data preprocessing and data quality screening on each sample image in the digestive tract endoscopy image sample data set to obtain a digestive tract endoscopy image sample processed data set; According to a plurality of preset data categories, the digestive tract endoscopy image sample processing data set is grouped to obtain a training group data set corresponding to each of the data categories; Constructing a multi-category context analysis model; the multi-category context analysis model includes a visual transformer network branch and a convolutional neural network branch; The multi-category context analysis model extracts the image features of the digestive tract endoscopy image through the convolutional neural network branch to determine the first image feature; The multi-category context analysis model performs image segmentation on the digestive tract endoscopy image through the visual transformer network branch to obtain a plurality of image blocks, and aggregates the physical background extracted from each of the image blocks using a self-attention mechanism; The multi-category context analysis model spatially concatenates the first image feature and the second image feature of each image block to obtain feature space information, and uses a feature selection module to filter each spatial concatenation feature in the feature space information to obtain corresponding feature space processing information; The multi-category context analysis model performs global average pooling and probability distribution mapping on the feature space processing information, determines the predicted category and attention score of each feature space in the feature space processing information, locates each feature space on the digestive tract endoscopic image, and maps each feature space to the digestive tract endoscopic image according to the predicted category and attention score of each feature space to obtain a model predicted image; Using each of the training group data sets to train the multi-category context analysis model, and iteratively optimizing the multi-category context analysis model in combination with a classification loss function; Obtain a test data set, perform model evaluation on the multi-category context analysis model using the test data set according to preset model evaluation indicators, determine the model evaluation result, and determine whether to continue training the multi-category context analysis model based on the model evaluation result.

2. The model training method according to claim 1, characterized in that: The step of performing data preprocessing and data quality screening on each sample image in the digestive tract endoscopy image sample data set to obtain the digestive tract endoscopy image sample processed data set specifically includes: Get image quality conditions; According to the image quality condition and the image quality of each of the sample images, the sample images in the digestive tract endoscopy image sample data set that do not meet the image quality condition are eliminated; The redundant information of each sample image in the digestive tract endoscopy image sample data set after the image is eliminated is deleted to obtain the digestive tract endoscopy image sample processed data set.

3. The model training method according to claim 1, characterized in that: The step of grouping the digestive tract endoscopy image sample processing data set according to a plurality of preset data categories to obtain a training group data set corresponding to each data category specifically includes: According to each of the data categories, classify and label each of the sample images in the digestive tract endoscopy image sample processing data set to determine an image category label corresponding to each of the sample images, wherein the image category label corresponds to the data category; Determining a number of the sample images corresponding to each of the data categories according to the image category labels corresponding to each of the sample images; A plurality of the sample images belonging to the same data category in the digestive tract endoscopy image sample processing data set are merged to obtain the training group data set corresponding to each data category.

4. A method for recognizing digestive tract endoscopy images, characterized in that: The method comprises the following steps: Acquire a number of digestive tract endoscopy images to be identified; Each of the gastrointestinal endoscopic images is input into a gastrointestinal endoscopic image recognition model, and the gastrointestinal endoscopic image recognition model is used to output image prediction results corresponding to each of the gastrointestinal endoscopic images and attention scores for each area of ​​the image; the gastrointestinal endoscopic image recognition model is trained by the model training method described in any one of claims 1 to 3.

5. The digestive tract endoscopy image recognition method according to claim 4, characterized in that: The method further comprises the following steps: According to the attention scores of each area of ​​the image corresponding to each digestive tract endoscopy image, the average attention score of each area of ​​the image is calculated, and the average attention score of each area of ​​the image is normalized and converted into a color value of each area of ​​the image; A target digestive tract endoscopic image is randomly selected from the plurality of digestive tract endoscopic images, and the color values ​​of each region of the image are mapped to corresponding image regions in the target digestive tract endoscopic image to obtain an image feature hotspot map.

6. The digestive tract endoscopy image recognition method according to claim 4, characterized in that: The step of inputting each of the digestive tract endoscopic images into a digestive tract endoscopic image recognition model, and using the digestive tract endoscopic image recognition model to output image prediction results corresponding to each of the digestive tract endoscopic images and attention scores for each region of the image specifically includes: Inputting the digestive tract endoscopy image into a convolutional neural network branch in the digestive tract endoscopy image recognition model, extracting image features, and determining a first image feature; Inputting the digestive tract endoscopic image into the visual transformer network branch in the digestive tract endoscopic image recognition model, segmenting the digestive tract endoscopic image to obtain a plurality of image blocks, aggregating background information of each of the image blocks, and determining a second image feature of each of the image blocks; spatially concatenating the first image feature with the second image feature of each of the image blocks to obtain feature space information; Inputting the feature space information into the feature selection module in the digestive tract endoscopy image recognition model, filtering each spatial cascade feature in the feature space information, and obtaining corresponding feature space processing information; Using the digestive tract endoscopy image recognition model to perform global average pooling and probability distribution mapping on the feature space processing information, and determine the prediction category and attention score of each feature space in the feature space processing information; Each of the feature spaces is located on the digestive tract endoscopic image, and each of the feature spaces is mapped to the digestive tract endoscopic image according to the predicted category and attention score of each of the feature spaces to obtain a model predicted image.

7. The digestive tract endoscopy image recognition method according to claim 6, characterized in that: The step of spatially concatenating the first image feature and the second image feature of each image block to obtain feature space information specifically includes: spatially cascading the first image feature and the second image feature of each image block to obtain an image block cascade feature corresponding to each image block; Each of the image blocks is located in the digestive tract endoscopic image, and the image block cascade features corresponding to each of the image blocks are mapped to the digestive tract endoscopic image to determine the feature space information corresponding to the digestive tract endoscopic image.

8. A digestive tract endoscope image recognition device, characterized in that: The device comprises: The first module is used to obtain a number of digestive tract endoscopy images to be identified; The second module is used to input each of the gastrointestinal endoscopic images into a gastrointestinal endoscopic image recognition model, and use the gastrointestinal endoscopic image recognition model to output the image prediction results corresponding to each of the gastrointestinal endoscopic images and the attention scores of each area of ​​the image; the gastrointestinal endoscopic image recognition model is trained by the model training method described in any one of claims 1 to 3.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in any one of claims 1 to 3 or the gastrointestinal endoscopic image recognition method described in any one of claims 4 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the model training method described in any one of claims 1 to 3 or the gastrointestinal endoscopic image recognition method described in any one of claims 4 to 7.