Method and device for determining upper digestive tract lesion area based on multi-task assistance

Abstract

The present application is applicable to the field of image processing technology, and provides a method and device for determining an upper gastrointestinal lesion area based on multi-task assistance, wherein the method for determining an upper gastrointestinal lesion area comprises: inputting an upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed; using a retrieval model to retrieve the upper gastrointestinal endoscopic image to be processed, and selecting a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from a training data set; determining a final lesion category of the upper gastrointestinal endoscopic image to be processed based on the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category; if the final lesion category belongs to a preset lesion category, using a trained segmentation model to segment the lesion area in the upper gastrointestinal endoscopic image to be processed. The above method can determine the lesion area more efficiently.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method and device for determining an upper gastrointestinal lesion area based on multi-task assistance. Background Art

[0002] Upper gastrointestinal diseases are one of the most common diseases in humans, especially upper gastrointestinal cancer, which seriously endangers human health. According to the "2020 Global Cancer Statistics" published by the World Health Organization, 2 of the top ten tumors in terms of incidence are from the upper gastrointestinal tract, and 2 of the top ten tumors in terms of mortality are also from the upper gastrointestinal tract, namely gastric cancer and esophageal cancer. These diseases from the upper gastrointestinal tract not only seriously threaten the quality of life and life safety of patients, but also bring huge health burdens to the world. It is well known that the cure rate of upper gastrointestinal cancer in the early stage is much higher than that in the late stage, so it is very important for patients to be able to diagnose early symptoms based on endoscopic images.

[0003] However, when endoscopists diagnose early-stage diseases based on endoscopic images, it is very important to obtain the diseased area in the endoscopic images. Current endoscopists still face many problems when making diagnoses based on endoscopic images. For example, each endoscopic examination generates a large number of endoscopic images, and manual inspection by endoscopists is slow and inefficient. It takes a long time to determine the diseased area and is highly dependent on the endoscopist's experience and mental state. This manual detection method not only affects the speed of diagnosis, but also results in a high rate of missed diagnosis and misdiagnosis.

[0004] Therefore, how to more efficiently determine the upper gastrointestinal lesion area has become an important issue that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a method and device for determining the lesion area of ​​the upper digestive tract based on multi-task assistance, which can determine the lesion area more efficiently.

[0006] A first aspect of an embodiment of the present application provides a method for determining an upper digestive tract lesion area based on multi-task assistance, the method comprising:

[0007] Inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed, wherein the classification model is trained using a training data set;

[0008] Retrieve the upper gastrointestinal endoscopic image to be processed using a retrieval model, and select a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from the training data set;

[0009] Determining a final lesion category of the upper digestive tract endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category;

[0010] If the final lesion category belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed.

[0011] A second aspect of an embodiment of the present application provides an upper digestive tract lesion area determination device based on multi-task assistance, the lesion area determination device comprising:

[0012] A classification module, used for inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed, wherein the classification model is trained using a training data set;

[0013] A retrieval module, which uses a retrieval model to retrieve the upper gastrointestinal endoscopic image to be processed, and selects a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from the training data set;

[0014] a category determination module, configured to determine a final lesion category of the upper gastrointestinal endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category;

[0015] A segmentation module is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed using a trained segmentation model if the final lesion category belongs to a preset lesion category, so as to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed.

[0016] The third aspect of an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining the upper gastrointestinal lesion area based on multi-task assistance as described in the first aspect above is implemented.

[0017] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for determining the upper gastrointestinal lesion area based on multi-task assistance as described in the first aspect above.

[0018] A fifth aspect of an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the method for determining an upper gastrointestinal lesion area based on multi-task assistance as described in the first aspect above.

[0019] In an embodiment of the present application: by inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model, the initial lesion category of the upper gastrointestinal endoscopic image to be processed is obtained, and then the upper gastrointestinal endoscopic image to be processed is retrieved through the retrieval model, and a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed is selected from the training data set. Since the endoscopic sample images in the first sub-dataset have been correctly labeled with the lesion category, the final lesion category of the upper gastrointestinal endoscopic image to be processed can be determined according to the lesion category and the initial lesion category of the endoscopic sample images in the first sub-dataset, so as to facilitate the endoscopist to make a preliminary diagnosis of the endoscopic image. If the lesion category obtained by the preliminary diagnosis belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed, and the lesion area in the upper gastrointestinal endoscopic image to be processed is obtained. The technical solution of the present application performs classification, image retrieval and segmentation tasks on the upper gastrointestinal endoscopic image to be processed, and by performing the above three tasks, the process of endoscopists manually examining a large number of endoscopic images is reduced, saving time and energy, thereby more efficiently determining the lesion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 It is a flowchart of a method for determining an upper digestive tract lesion area based on multi-task assistance provided in Example 1 of the present application;

[0022] Figure 2 It is a flowchart of the training data set and the test data set performing different tasks;

[0023] Figure 3 is a classification and retrieval result graph of the upper gastrointestinal endoscopy image to be processed;

[0024] Figure 4 is the segmentation result image of the upper gastrointestinal endoscopy image to be processed;

[0025] Figure 5 This is a result display diagram of the method for determining the upper digestive tract lesion area based on multi-task assistance in Example 1 of the present application;

[0026] Figure 6It is a flowchart of a method for determining an upper digestive tract lesion area based on multi-task assistance provided in Example 2 of the present application;

[0027] Figure 7 This is a confusion matrix diagram of the upper gastrointestinal lesion area determination method based on multi-task assistance;

[0028] Figure 8 This is a ROC (receiver operating characteristic) curve of the upper gastrointestinal lesion area determination method based on multi-task assistance;

[0029] Fig. 9 This is a structural schematic diagram of an upper gastrointestinal lesion area determination device based on multi-task assistance provided in Example 3 of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0035] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0036] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0037] Reference Figure 1 , shows a flow chart of a method for determining an upper digestive tract lesion region based on multi-task assistance provided in Example 1 of the present application, the lesion region determination method is applied to a terminal device, such as Figure 1 The lesion area determination method shown may include the following steps:

[0038] Step 101: input the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed.

[0039] The upper gastrointestinal endoscopic image to be processed may refer to any upper gastrointestinal endoscopic image taken by an endoscope. After the endoscope takes the upper gastrointestinal image, the taken upper gastrointestinal endoscopic image may be sent to a terminal device. The terminal device may obtain the upper gastrointestinal endoscopic image to be processed by receiving the upper gastrointestinal endoscopic image sent by the endoscope. The upper gastrointestinal tract may refer to any upper gastrointestinal tract, such as the esophagus, stomach, pharynx, etc. of the human digestive tract.

[0040] In an embodiment of the present application, a classification model is trained using a training data set, and the classification model is used to obtain an initial lesion category of an upper gastrointestinal endoscopic image to be processed.

[0041] Among them, the training data set includes N endoscopic sample images with a first label, and the first label is used to indicate the lesion category of each endoscopic sample image, that is, the first label of each endoscopic sample image is obtained by manual marking. In order to ensure the correctness of the manually marked first label in each endoscopic sample image, it is first necessary to determine the lesion category of all endoscopic sample images through preliminary upper gastrointestinal endoscopy examination reports and pathological results. Finally, several endoscopists with many years (for example, more than 10 years) of experience in endoscopic surgery manually mark the lesion categories of the endoscopic sample images.

[0042] Exemplarily, if the above-mentioned upper gastrointestinal endoscopic images to be processed refer to esophageal endoscopic images, the N endoscopic sample images included in the training data set are N esophageal endoscopic images. The training data set can be obtained by collecting 1003 esophageal endoscopic images of 319 patients in a hospital from 2016 to 2019, and the above 1003 esophageal endoscopic images are randomly divided into a training data set and a test data set at a ratio of 4:1, that is, 805 of the esophageal endoscopic images that have been manually marked are used as the training data set, and 194 of the esophageal endoscopic images that have been manually marked are used as the test data set. The test data set is used to test the classification accuracy of the trained classification model. When there are no repeated images in the training data set and the test data set, the confidence of the classification accuracy tested by the test data set will be higher.

[0043] Since esophageal endoscopes are used for daily screening or preoperative examinations in clinical practice, all esophageal endoscopic images were obtained by using conventional endoscopes with standard inner diameter white light imaging and narrow band imaging, and the above esophageal endoscopic images were obtained through single-attachment channel endoscopes of different standards (for example, single-attachment models of different standards can be: GIF-Q240Z, GIF-RQ260Z, GIF-FQ260Z, GIF-H260Z, GIF-Q260J GIF-H290Z, GIF-HQ290, GIF-XP290N, Olympus, Tokyo, Japan).

[0044] It should be understood that the above-mentioned upper gastrointestinal endoscopy images to be processed correspond to the training data set used to train the classification model, that is, when the classification model is used to classify any type of endoscopy images, the classification model is trained using the training data set corresponding to any type of endoscopy images. If the above-mentioned upper gastrointestinal endoscopy images to be processed refer to esophageal endoscopy images, then the training data set used to train the classification model includes all esophageal endoscopy images with the first label; if the above-mentioned upper gastrointestinal endoscopy images to be processed refer to gastric endoscopy images, then the training data set used to train the classification model includes all gastric endoscopy images with the first label.

[0045] In a possible implementation, the training process of the classification model may include:

[0046] The endoscope sample images in the training data set are input into the classification model in sequence. At this time, the classification model is a model pre-trained on ImageNet, wherein the classification model includes a first input layer, 16 first convolutional layers, 3 first fully connected layers, 5 first maximum pooling layers and a first output layer. When the training data set is used to continue training the model pre-trained on ImageNet, the endoscope sample image first enters the first convolutional layer through the first input layer in the classification model, wherein the convolution kernel size of the first convolutional layer may be 3×3, and a Relu activation function is used in the first convolutional layer to obtain a 3×3 feature map of the endoscope sample image. Secondly, the 3×3 feature map of the endoscope sample image is input into the first maximum pooling layer, which can remove unimportant features in the above 3×3 feature map and further reduce the parameter parameters in the classification model, wherein the window size of the first maximum pooling layer may be 2×2. Finally, the first fully connected layer is used to classify the above-mentioned endoscopic sample images. Among the three fully connected layers, the input of the first fully connected layer is the 2×2 feature map extracted by the first convolutional layer and the first maximum pooling layer. The output layer connected to the last fully connected layer is a classifier. The number of neurons in the output layer (that is, the number of lesion categories of the endoscopic sample images) can be set. For example, if the number of neurons is 3, it means that the possible lesion categories of the endoscopic sample images are 3.

[0047] The output of the above classification model is the lesion category of each endoscopic sample image. The lesion category of each endoscopic sample image output by the classification model is compared with the lesion category indicated by the first label of each endoscopic sample image itself to obtain the first difference information. Based on the first difference information, the classification model is back-propagated using the focalloss function, and the endoscopic sample images in the training data set are classified again after adjusting the category weights in the classification model until the value of the focalloss function reaches the optimal value, and the classification model at this time is obtained. The classification model at this time is a trained classification model.

[0048] In a possible implementation, the upper gastrointestinal endoscopic image to be processed is input into a trained classification model, and the initial lesion category of the upper gastrointestinal endoscopic image to be processed is obtained, including:

[0049] Inputting the upper gastrointestinal endoscopic images to be processed into the trained classification model to classify the upper gastrointestinal endoscopic images to be processed;

[0050] The Softmax function is used to calculate the probability value of the upper gastrointestinal endoscopy image to be processed belonging to each lesion category;

[0051] The lesion category with the largest probability value among all the probability values ​​of the lesion categories is determined as the initial lesion category of the upper gastrointestinal endoscopy image to be processed.

[0052] In the embodiments of the present application, Figure 2 The figure shows a flow chart of different tasks performed by a training data set and a test data set, wherein the training data set performs a training task and the test data set performs a test task, and the classification accuracy of the classification model is obtained after the classification model in the upper gastrointestinal lesion area determination device is tested using the test data set. When the classification accuracy reaches the preset accuracy, the trained classification model is used to classify the upper gastrointestinal endoscopic image to be processed. If the number of neurons in the first output layer of the classification model is 3, the softmax function is used to calculate the probability values ​​of the upper gastrointestinal endoscopic image to be processed belonging to the three lesion categories respectively, and the lesion category corresponding to the largest probability value among the above-mentioned probability values ​​belonging to the three lesion categories is determined as the initial lesion category of the upper gastrointestinal endoscopic image to be processed.

[0053] Exemplarily, the esophageal endoscopy image to be processed is input into the trained classification model. The number of neurons in the first output layer of the trained classification model is 3, and the lesion categories are "esophageal cancer", "esophagitis" and "normal". The probability values ​​of the esophageal endoscopy image to be processed belonging to the above three categories are calculated using the Softmax function: the probability value of esophageal cancer is 99.22%; the probability value of esophagitis is 0.49%; the probability value of normal is 0.29%, so it can be determined that the initial lesion category of the above esophageal endoscopy image to be processed is esophageal cancer.

[0054] Step 102: Use the retrieval model to retrieve the upper gastrointestinal endoscopic image to be processed, and select a first sub-dataset from the training data set that has similar features to the upper gastrointestinal endoscopic image to be processed.

[0055] In the embodiment of the present application, the retrieval model shares the classification model, and a deep learning model is used to implement the two tasks of classification and retrieval, that is, a fully connected layer and a Sigmoid function are added before the first output layer of the above classification model to perform binary hash codes (binary hash codes). The fully connected layer is called the image retrieval layer, and the neurons of this layer are activated between [0,1] by the Sigmoid function. A coarse-to-fine search strategy is used in the image retrieval layer to quickly select the first sub-dataset with similar features to the upper gastrointestinal endoscopy image to be processed from the training data set.

[0056] Among them, image retrieval technology can be used to retrieve the upper gastrointestinal endoscopic images to be processed. In the training data set that has been manually labeled with the correct lesion category, the first sub-dataset with similar features to the upper gastrointestinal endoscopic images to be processed is searched and selected as a further basis for determining the final lesion category of the upper gastrointestinal endoscopic images to be processed. At the same time, the initial lesion category output by the above classification model can be further corrected.

[0057] In a possible implementation, using a retrieval model to retrieve the upper gastrointestinal endoscopy image to be processed, selecting a first sub-dataset having similar features to the upper gastrointestinal endoscopy image to be processed from the training data set includes:

[0058] Determine the image features of the upper gastrointestinal endoscopy image to be processed based on the image features extracted from the image features output by the image retrieval layer;

[0059] Determining a coding value of the upper digestive tract endoscopy image to be processed according to image features of the upper digestive tract endoscopy image to be processed;

[0060] Determining a coding value of each endoscope sample image in the training data set according to an image feature of each endoscope sample image in the training data set;

[0061] Calculating the Hamming distance between the encoding value of the upper gastrointestinal endoscopy image to be processed and the encoding value of each endoscopy sample image in the training data set;

[0062] Determine candidate data sets from the training data sets based on the Hamming distance;

[0063] Calculate the Euclidean distance between the upper gastrointestinal endoscopy image to be processed and each endoscopy sample image in the candidate data set;

[0064] According to the ranking from small to large of Euclidean distance, a first sub-dataset having similar features to the upper gastrointestinal endoscopy image to be processed is selected from the candidate data set, and the first sub-dataset includes the top K endoscopy sample images ranked in all Euclidean distances, where K is an integer greater than zero.

[0065] In the embodiment of the present application, since the retrieval model shares the classification model, the retrieval model contains the image features of each endoscopic sample image in the training data set, and the image features of each endoscopic sample image are feature maps output by the first maximum pooling layer of the classification model.

[0066] Exemplarily, the upper gastrointestinal endoscopy image to be processed is input into the retrieval model. First, the image features of the upper gastrointestinal endoscopy image to be processed extracted in the first maximum pooling layer and the image features of each endoscopy sample image in the training data set are obtained. For the upper gastrointestinal endoscopy image to be processed, Out(H) is used to represent the image features of the upper gastrointestinal endoscopy image to be processed extracted from the first maximum pooling layer, and then its binary code is obtained by setting a threshold. For each byte i=1,2,3,…,n (n is the number of nodes in the image retrieval layer), the threshold is set to 0.5, and the binary code of the upper gastrointestinal endoscopy image to be processed is:

[0067]

[0068] Using the above method of obtaining binary codes according to image features, according to the image features of each endoscope sample image in the training data set, the binary codes of each endoscope sample image in the training data set are obtained as follows: H1, H2, H3, ..., H m (m is the number of endoscope sample images in the training data set), where the training data set can be expressed as Γ = {I1, I2, I3, …, I m}.

[0069] Secondly, if the upper gastrointestinal endoscopy image I to be processed is given q and the binary code H of the upper gastrointestinal endoscopy image to be processed q , can be obtained according to the binary code H of the upper gastrointestinal endoscopy image to be processed q and the binary encoding H of the endoscope sample images in the training dataset m The Hamming distance between them is used to obtain the candidate image set P = {I1, I2, I3, …, I g}, wherein the Hamming distance between the encoding value of each endoscope sample image in the candidate data set and the encoding value of the upper gastrointestinal endoscope image to be processed is less than the distance threshold.

[0070] Finally, the upper gastrointestinal endoscopy image to be processed I q With the candidate image set P = {I1, I2, I3, ..., I g}, and determine the upper gastrointestinal endoscopy image I to be processed. q With the candidate image set P = {I1, I2, I3, ..., I g The similarity of each endoscope sample image in}, the smaller the distance, the greater the similarity. Among them, the above Euclidean distance can be specifically obtained by the following equation:

[0071]

[0072] Among them, Λ qThe upper gastrointestinal endoscopy image I to be processed q The characteristic vector of is the endoscope sample image I in the candidate image set g The eigenvector of ||x|| refers to the norm of x, s l The upper gastrointestinal endoscopy image I to be processed q With the candidate image set P = {I1, I2, I3, ..., I g Endoscope sample image I g The Euclidean distance of .

[0073] The smaller the Euclidean distance, the higher the similarity between the two images. After arranging each endoscope sample image in the candidate image set from small to large according to the Euclidean distance, the top K endoscope sample images can be obtained. Figure 3 As shown in the figure, it is the classification and retrieval result diagram of the upper gastrointestinal endoscopy image to be processed. According to the image, it can be concluded that Figure 3 In (a), the initial lesion category of the upper gastrointestinal endoscopy image to be processed is esophageal cancer, and the top five endoscopy sample images with similar features to the upper gastrointestinal endoscopy image to be processed and ranked in ascending order of Euclidean distance in the candidate image set are output; Figure 3 In (b), the initial lesion category of the upper gastrointestinal endoscopic image to be processed is esophagitis, and the top five endoscopic sample images with similar features to the upper gastrointestinal endoscopic image to be processed and ranked in ascending order of Euclidean distance in the candidate image set are output; Figure 3 The initial lesion category of the upper gastrointestinal endoscopic image to be processed input in (c) is normal, and the top five endoscopic sample images with similar features to the upper gastrointestinal endoscopic image to be processed and ranked in ascending order of Euclidean distance are also output.

[0074] Step 103 : determining the final lesion category of the upper gastrointestinal endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category.

[0075] In an embodiment of the present application, the endoscopic sample images in the first sub-data set have been manually labeled with the correct lesion category, and the endoscopic sample images in the first data set are the images most similar to the upper gastrointestinal endoscopic images to be processed. Therefore, the lesion category of the endoscopic sample images in the first data set can be referred to to determine the final lesion category of the upper gastrointestinal endoscopic images to be processed, which can make the confidence of the final lesion category higher.

[0076] In a possible implementation, determining the final lesion category of the upper gastrointestinal endoscopic image to be processed according to the lesion category and the initial lesion category of the endoscopic sample image in the first sub-dataset includes:

[0077] According to the lesion category of each endoscopic sample image in the first sub-dataset, obtaining the number of first images corresponding to different lesion categories;

[0078] If the lesion category corresponding to the maximum value of the number of first images corresponding to different lesion categories is the same as the initial lesion category, then the initial lesion category is determined as the final lesion category; or

[0079] If the lesion category corresponding to the maximum value in the first number of images is different from the initial lesion category, and the ratio of the number of images with the maximum value in the first number of images to K is greater than the probability value of the initial lesion category, then the lesion category corresponding to the maximum value in the first number of images is determined as the final lesion category, otherwise the initial lesion category is determined as the final lesion category.

[0080] In the embodiment of the present application, a voting algorithm may be used to obtain the number of images corresponding to different lesion categories of each endoscopic sample image in the first sub-dataset.

[0081] Taking the upper gastrointestinal endoscopic image to be processed as the esophageal endoscopic image to be processed as an example, if the first sub-dataset includes 5 endoscopic sample images, and the corresponding lesion categories are esophageal cancer, esophageal cancer, normal, esophageal cancer and esophagitis, respectively, then the number of first images corresponding to esophageal cancer is 3, the number of first images corresponding to normal is 1, and the number of first images corresponding to esophagitis is 1. If the initial lesion category of the esophageal endoscopic image to be processed is esophageal cancer, then the final lesion category of the esophageal endoscopic image to be processed is determined to be esophageal cancer. If the initial lesion category of the esophageal endoscopic image to be processed is esophagitis, and the probability value corresponding to esophagitis is calculated to be 90%, according to the lesion categories corresponding to the 5 endoscopic sample images included in the above first sub-dataset, it can be obtained that the number of images of the maximum value in the first image is 3, and the ratio of the number of images of the maximum value in the first image to K is 3 / 5. Since the ratio is less than 90%, it is determined that the initial lesion category of the esophageal endoscopic image to be processed, esophagitis, is the final lesion category.

[0082] Step 104: If the final lesion category belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed.

[0083] In an embodiment of the present application, a segmentation model is trained using a training data set, and the segmentation model is used to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed when the final lesion category belongs to a preset lesion category.

[0084] Among them, the preset lesion category may refer to a cancer category of the upper digestive tract, such as esophageal cancer. If the preset lesion category is set to esophageal cancer, the training data set for training the segmentation model may all be esophageal cancer endoscopy sample images, wherein the esophageal cancer endoscopy sample images in the training data set all have a second label, and the second label is used to indicate the lesion area in each esophageal cancer endoscopy sample image, and the second label is obtained by manual marking.

[0085] In a possible implementation, the training process of the segmentation model may include:

[0086] The training data set is input into the segmentation model, and the lesion area of ​​each endoscope sample image in the training data set is output through the second output layer;

[0087] Comparing the lesion area of ​​each endoscope sample image in the training data set with the lesion area indicated by the second label to obtain second difference information;

[0088] Based on the second difference information, the mean square error loss function is used to perform model back propagation on the segmentation model to obtain a trained segmentation model.

[0089] In an embodiment of the present application, the training data set is input into the segmentation model, which includes a second input layer, a second convolutional layer, a second maximum pooling layer, an expanded convolutional layer, and a second output layer, wherein the convolution kernel of the convolutional layer is 3×3, and the Relu activation function is used to obtain a 3×3 feature map of the upper gastrointestinal endoscopy image to be processed; secondly, a maximum pooling layer with a window size of 2×2 is used to remove unimportant features in the above 3×3 feature map, further reducing the number of parameters in the classification model. Finally, the convolution kernel of the output layer is 1×1, and the output is the most important feature map (i.e., the lesion area of ​​each endoscope sample image in the training data set).

[0090] The output of the segmentation model is the lesion area of ​​each endoscopic sample image. The lesion area of ​​each endoscopic sample image output by the segmentation model is compared with the lesion area indicated by the second label of each endoscopic sample image itself to obtain second difference information. Based on the second difference information, the segmentation model is back-propagated using the mean square error loss function, and the weight value of each layer in the segmentation model is adjusted, and the endoscopic sample images in the training data set are semantically segmented again until the value of the mean square error loss function reaches the optimal value, and the segmentation model at this time is obtained. The segmentation model at this time is a trained segmentation model.

[0091] Optionally, the 6 second convolutional layers of the segmentation model before the second output layer can be replaced by dilated convolutional layers, which can increase the receptive field of the convolution kernel while keeping the number of parameters unchanged, so that the segmentation model can increase the image resolution during upsampling, effectively prevent image loss, and improve the segmentation accuracy.

[0092] In a possible implementation, semantic segmentation is performed on the upper gastrointestinal endoscopic image to be processed using a trained segmentation model, and the lesion area in the upper gastrointestinal endoscopic image to be processed includes:

[0093] Inputting the upper gastrointestinal endoscopic images to be processed whose final lesion categories belong to the preset lesion categories into the trained segmentation model to perform semantic segmentation on the upper gastrointestinal endoscopic images to be processed;

[0094] It is determined that the lesion area output by the second output layer is the lesion area in the upper gastrointestinal endoscopy image to be processed.

[0095] In the embodiment of the present application, since the endoscopic sample images in the training data set are all manually marked with the correct lesion area, the upper gastrointestinal endoscopic images to be processed belonging to the preset lesion category are input into the trained segmentation model, and semantic segmentation is performed on them, so that the lesion area can be segmented from the upper gastrointestinal endoscopic images to be processed. Figure 4 The figure shows the segmentation result of the upper gastrointestinal endoscopy image to be processed. Figure 4 (a) is the original image of the upper gastrointestinal endoscopy image to be processed. Figure 4 (b) is the lesion area manually marked by the endoscopist, Figure 4 (c) is the segmentation result diagram output by the segmentation model. It can be concluded that the segmentation model proposed in the present application can segment the lesion area from the upper gastrointestinal endoscopic image to be processed, and the segmentation accuracy is relatively high.

[0096] like Figure 5 The result display diagram of the method for determining the upper gastrointestinal lesion area based on multi-task assistance in the first embodiment of the present application is shown. Taking the upper gastrointestinal endoscopic image to be processed as an esophageal endoscopic image as an example, firstly, three esophageal endoscopic images to be processed are input, and the three esophageal endoscopic images to be processed are classified and retrieved respectively to obtain the probability of each image belonging to a lesion category and five images with similar features. Secondly, the initial lesion category of each image is determined according to the probability of each image belonging to each lesion category. Figure 5 The lesion categories of the input images 1, 2, and 3 can be obtained, and the lesion category of the image 1 is esophageal cancer and is the preset lesion category. Finally, the image 1 is input into the segmentation model, and the semantic segmentation of the image 1 is performed to segment the lesion area from the image 1.

[0097] In an embodiment of the present application, the upper gastrointestinal endoscopic image to be processed is input into a trained classification model to obtain the initial lesion category of the upper gastrointestinal endoscopic image to be processed, and then the upper gastrointestinal endoscopic image to be processed is retrieved through the retrieval model, and a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed is selected from the training data set. Since the endoscopic sample images in the first sub-dataset have been correctly labeled with lesion categories, the final lesion category of the upper gastrointestinal endoscopic image to be processed can be determined based on the lesion category and the initial lesion category of the endoscopic sample images in the first sub-dataset, so as to facilitate the endoscopist to make a preliminary diagnosis of the endoscopic image. If the lesion category obtained by the preliminary diagnosis belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed. The technical solution of the present application performs classification, image retrieval and segmentation tasks on the upper gastrointestinal endoscopic image to be processed, and by performing the above three tasks, the process of endoscopists manually examining a large number of endoscopic images is reduced, saving time and energy, thereby more efficiently determining the lesion area.

[0098] Reference Figure 6 , shows a flow chart of a method for determining an upper gastrointestinal lesion region based on multi-task assistance provided in Example 2 of the present application, such as Figure 6 The lesion area determination method shown may include the following steps:

[0099] Step 601: collect a test data set to test the classification performance of the trained classification model.

[0100] In an embodiment of the present application, the test data set includes G endoscopic test images, where G is an integer greater than zero, and each endoscopic test image is manually labeled with the correct lesion category and the correct lesion area, and the test data set is two data sets that do not overlap with the training data set at all. The trained classification model is tested using the test data set, and the classification performance confidence is high. Among them, the classification performance of the classification model can be determined by obtaining the confusion matrix and ROC (Receiver Operating Characteristic) curve of the classification model based on the labeled lesion category of each endoscopic test image in the test data set (i.e., the correct lesion category manually labeled as indicated by the third label) and the final lesion category obtained by the classification model.

[0101] In a specific implementation, by statistically analyzing the labeled lesion categories and final lesion categories of each endoscopic test image, the following can be obtained: Figure 7The confusion matrix diagram of the upper gastrointestinal lesion area determination method based on multi-task assistance is shown. In this confusion matrix diagram, the horizontal axis is the predicted value (i.e., the final lesion category of each endoscopic test image), the vertical axis is the true value (i.e., the labeled lesion category of each endoscopic test image), and the color of the matrix block in the matrix represents the corresponding probability. For example, a matrix block with both the horizontal and vertical axes being normal represents the probability that the true value is normal and the final lesion category of the endoscopic test image obtained by the classification model is also normal. According to the probabilities corresponding to the colors in the figure, it can be seen that the probabilities of both the horizontal and vertical axes being normal, both the horizontal and vertical axes being esophageal cancer, and both the horizontal and vertical axes being esophagitis are all above 80%. According to the probabilities corresponding to each matrix block in the confusion matrix, a graph can be drawn as follows: Figure 8 The ROC curve of the upper gastrointestinal lesion area determination method based on multi-task assistance is shown. The solid line in the figure is the ROC curve, and the area of ​​the closed area enclosed by the ROC curve, the horizontal axis, and the axis with specificity equal to 1 can be used to obtain the size of the AUC (Area Under Curve), where AUC is an evaluation indicator for measuring the quality of the classification model. The larger the AUC value, the better the classification performance of the classification model. The AUC value calculated by the classification model in this application is 0.9435, which can indicate that the classification performance of the classification model is good.

[0102] In the embodiment of the present application, the accuracy, precision, sensitivity, specificity and negative predictive value of the classification model can also be obtained based on the confusion matrix and ROC curve of the classification model.

[0103] use Figure 7 The confusion matrix and Figure 8 The accuracy, precision, sensitivity, specificity and negative predictive value of the classification model can be calculated by using the data in the ROC curve. The classification results obtained by the classification model of this application and the classification results manually marked by the endoscopist are compared with the actual categories of the endoscope test images in the test data set, and the corresponding index values ​​can be obtained as shown in Table 1:

[0104] Table 1

[0105]

[0106] According to the data in Table 1, the accuracy, precision, sensitivity, specificity and negative predictive value of the classification results of the classification model of the present application are 8.75%, 13.45%, 9.58%, 6.25% and 6.69% higher than the manual classification results of endoscopists, respectively.

[0107] In order to verify the effect of the classification model of the present application on improving the manual classification efficiency of endoscopists, the classification results of endoscopists with and without reference to the classification model of the present application are compared with the actual categories of the endoscope test images in the test data set, and the corresponding index values ​​are shown in Table 2:

[0108] Table 2

[0109]

[0110]

[0111] According to the data in Table 2, after referring to the classification results of the classification model of the present application, the accuracy, precision, sensitivity, specificity and negative predictive value of the classification of the endoscopist are also improved. Therefore, the method proposed in the present invention can classify the upper gastrointestinal endoscopic images to be processed more efficiently.

[0112] Step 602: Use a test data set to test the segmentation performance of the trained segmentation model.

[0113] In an embodiment of the present application, the segmentation performance of the segmentation model can be determined by obtaining the Dice coefficient (set similarity measurement function) and the average IOU (Intersection over Union) index of the segmentation model based on the marked lesion area of ​​each endoscopic test image (i.e., the lesion area indicated by the fourth label) and the lesion area obtained by the segmentation model.

[0114] In the specific implementation, the Dice coefficient and average IOU index of the segmentation model can be obtained in the following ways:

[0115]

[0116] Among them, A is the area of ​​the marked lesion area, and B is the area of ​​the lesion area obtained by the segmentation model of each endoscopic test image.

[0117]

[0118] Here, the numerator is the area of ​​the intersection of A and B, and the denominator is the area of ​​the union of A and B.

[0119] According to the above equation, through the test of the test data set of this application, it can be calculated that the Dice coefficient and average IOU of the segmentation model are 0.7784 and 0.6563 respectively. The larger the value of the Dice coefficient and the average IOU, the better the segmentation performance of the segmentation model.

[0120] Step 603: input the upper gastrointestinal endoscopic image to be processed into the trained classification model to obtain the initial lesion category of the upper gastrointestinal endoscopic image to be processed.

[0121] Step 604: Use the retrieval model to retrieve the upper gastrointestinal endoscopic image to be processed, and select a first sub-dataset from the training data set that has similar features to the upper gastrointestinal endoscopic image to be processed.

[0122] Step 605 : determining the final lesion category of the upper gastrointestinal endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category.

[0123] Step 606: If the final lesion category belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed.

[0124] Steps 603-606 of this embodiment are the same as steps 101-104 of the aforementioned embodiment, and can refer to each other, and will not be described in detail in this embodiment.

[0125] Compared with Example 1 of the present application, in the method proposed in Example 2 of the present application, a test data set is used to test the trained classification model and segmentation model respectively, and different evaluation indicators are used to evaluate the performance of the classification model and the segmentation model respectively, which can effectively evaluate the classification performance of the classification model and the segmentation performance of the segmentation model. When the classification performance of the classification model and the segmentation performance of the segmentation model reach the corresponding indicators, the upper gastrointestinal endoscopic image to be processed is input into the classification model and the segmentation model, which can make the output lesion area more accurate.

[0126] Reference Fig. 9 , shows a structural schematic diagram of a lesion area determination device based on multi-task assistance provided in Example 3 of the present application. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0127] The upper digestive tract lesion area determination device based on multi-task assistance may specifically include the following modules:

[0128] A classification module 901 is used to input the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed, wherein the classification model is trained using a training data set;

[0129] A retrieval module 902 uses a retrieval model to retrieve the upper gastrointestinal endoscopy image to be processed, and selects a first sub-dataset having similar features to the upper gastrointestinal endoscopy image to be processed from the training data set;

[0130] A category determination module 903, configured to determine a final lesion category of the upper digestive tract endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category;

[0131] The segmentation module 904 is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed using the trained segmentation model if the final lesion category belongs to the preset lesion category, so as to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed.

[0132] In the embodiment of the present application, the classification model may specifically include the following submodules during the training process:

[0133] A category output submodule, used to input the training data set into the classification model, and output the lesion category of each endoscope sample image in the training data set through the first output layer, wherein the output layer in the classification model is a fully connected layer containing M neurons, where the M neurons represent M lesion categories, and M is an integer greater than zero;

[0134] A first comparison submodule, used to compare the lesion category of each endoscope sample image in the training data set with the lesion category indicated by the first label to obtain first difference information;

[0135] The first back-propagation submodule is used to perform model back-propagation on the classification model based on the first difference information by using a focal loss function to obtain a trained classification model.

[0136] In the embodiment of the present application, the classification module 901 may specifically include the following submodules:

[0137] A classification submodule, used for inputting the upper gastrointestinal endoscopic images to be processed into the trained classification model to classify the upper gastrointestinal endoscopic images to be processed;

[0138] A probability calculation submodule, used for calculating the probability value of the upper gastrointestinal endoscopy image to be processed belonging to each lesion category by using a Softmax function;

[0139] The maximum probability determination submodule is used to determine the lesion category with the largest probability value among the probability values ​​of all lesion categories as the initial lesion category of the upper gastrointestinal endoscopic image to be processed.

[0140] In the embodiment of the present application, the search module 902 may specifically include the following submodules:

[0141] A feature determination submodule, used for determining the image features of the upper gastrointestinal endoscopy image to be processed based on the image features extracted from the image features output by the image retrieval layer;

[0142] A first coding determination submodule, used to determine a coding value of the upper gastrointestinal endoscopic image to be processed according to image features of the upper gastrointestinal endoscopic image to be processed;

[0143] A second coding determination submodule, used to determine a coding value of each endoscope sample image in the training data set according to image features of each endoscope sample image in the training data set;

[0144] A first distance calculation submodule, used for calculating the Hamming distance between the encoding value of the upper gastrointestinal endoscope image to be processed and the encoding value of each endoscope sample image in the training data set;

[0145] A candidate determination submodule, used to determine a candidate data set from the training data set according to the Hamming distance, wherein the Hamming distance between the encoding value of each endoscope sample image in the candidate data set and the encoding value of the upper gastrointestinal endoscope image to be processed is less than a distance threshold;

[0146] A second distance calculation submodule is used to calculate the Euclidean distance between the upper gastrointestinal endoscopy image to be processed and each endoscopy sample image in the candidate data set;

[0147] The sorting submodule is used to select a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from the candidate data set according to the Euclidean distance from small to large, and the first sub-dataset includes the top K endoscopic sample images ranked in all Euclidean distances, where K is an integer greater than zero.

[0148] In the embodiment of the present application, the category determination module 903 may specifically include the following submodules:

[0149] A quantity acquisition submodule, used to acquire the quantity of first images corresponding to different lesion categories according to the lesion category of each endoscopic sample image in the first sub-dataset;

[0150] A first final determination submodule, configured to determine the initial lesion category as the final lesion category if the lesion category corresponding to the maximum value of the first number of images corresponding to different lesion categories is the same as the initial lesion category;

[0151] The second final determination submodule is used to determine the lesion category corresponding to the maximum value in the first number of images as the final lesion category if the lesion category corresponding to the maximum value in the first number of images is different from the initial lesion category and the ratio of the number of images of the maximum value in the first number of images to K is greater than the probability value of the initial lesion category, otherwise determine the initial lesion category as the final lesion category. .

[0152] In the embodiment of the present application, the segmentation model may specifically include the following submodules during the training process:

[0153] A region output submodule, used to input the training data set into the segmentation model, and output the lesion region of each endoscope sample image in the training data set through the second output layer;

[0154] A second comparison submodule, used to compare the lesion area of ​​each endoscope sample image in the training data set with the lesion area indicated by the second label to obtain second difference information;

[0155] The second back propagation submodule is used to perform model back propagation on the segmentation model based on the second difference information using a mean square error loss function to obtain a trained segmentation model.

[0156] In the embodiment of the present application, the segmentation module 904 may specifically include the following submodules:

[0157] A semantic segmentation submodule, used for inputting the upper gastrointestinal endoscopic images to be processed whose final lesion categories belong to the preset lesion categories into the trained segmentation model, and performing semantic segmentation on the upper gastrointestinal endoscopic images to be processed;

[0158] The region determination submodule is used to determine that the lesion region output by the second output layer is the lesion region in the upper gastrointestinal endoscopic image to be processed.

[0159] In the embodiment of the present application, the upper digestive tract lesion area determination device based on multi-task assistance may further include the following modules:

[0160] A classification test module is used to test the classification performance of the trained classification model using a test data set;

[0161] A segmentation test module is used to test the segmentation performance of the trained segmentation model using a test dataset;

[0162] The upper gastrointestinal lesion area determination device based on multi-task assistance provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0163] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0164] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0165] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0168] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for determining the upper digestive tract lesion area based on multi-task assistance, characterized in that: The lesion area determination method comprises: Inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed, wherein the classification model is trained using a training data set; Retrieve the upper gastrointestinal endoscopic image to be processed using a retrieval model, and select a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from the training data set; Determining a final lesion category of the upper digestive tract endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category; If the final lesion category belongs to the preset lesion category, using the trained segmentation model to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed to obtain the lesion area in the upper gastrointestinal endoscopic image to be processed; Determining the final lesion category of the upper digestive tract endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category includes: According to the lesion category of each endoscopic sample image in the first sub-dataset, obtaining the number of first images corresponding to different lesion categories; If the lesion category corresponding to the maximum value of the number of first images corresponding to different lesion categories is the same as the initial lesion category, then determining the initial lesion category as the final lesion category; or If the lesion category corresponding to the maximum value in the first number of images is different from the initial lesion category, and the ratio of the number of images with the maximum value in the first number of images to K is greater than the probability value of the initial lesion category, then the lesion category corresponding to the maximum value in the first number of images is determined to be the final lesion category, otherwise the initial lesion category is determined to be the final lesion category, where K is the total number of endoscopic sample images in the first sub-dataset.

2. The method for determining the upper digestive tract lesion area according to claim 1, characterized in that: The classification model includes a first input layer, 16 first convolutional layers, 3 first fully connected layers, 5 first maximum pooling layers and a first output layer, wherein the convolution kernel size of the convolutional layer is 3×3, the window size of the maximum pooling layer is 2×2, the training data set includes N endoscopic sample images with a first label, the first label is used to indicate the lesion category of each endoscopic sample image, N is an integer greater than zero, and the training process of the classification model includes: Inputting the training data set into the classification model, and outputting the lesion category of each endoscope sample image in the training data set through the first output layer, wherein the output layer in the classification model is a fully connected layer including M neurons, and the M neurons represent M lesion categories, where M is an integer greater than zero; Comparing the lesion category of each endoscope sample image in the training data set with the lesion category indicated by the first label to obtain first difference information; Based on the first difference information, a focal loss function is used to perform model back propagation on the classification model to obtain the trained classification model.

3. The method for determining the upper digestive tract lesion area according to claim 2, characterized in that: The upper gastrointestinal endoscopic image to be processed is input into the trained classification model to obtain the initial lesion category of the upper gastrointestinal endoscopic image to be processed, including: Inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model to classify the upper gastrointestinal endoscopic image to be processed; Using a Softmax function to calculate the probability value of the upper gastrointestinal endoscopy image to be processed belonging to each lesion category; The lesion category with the largest probability value among the probability values ​​of all lesion categories is determined as the initial lesion category of the upper gastrointestinal endoscopic image to be processed.

4. The method for determining the upper digestive tract lesion area according to claim 1, characterized in that: The retrieval model shares the classification model, wherein the retrieval model shares the classification model means that the retrieval model adds an image retrieval layer on the basis of the classification model, the image retrieval layer is located before the output layer of the classification model and is adjacent to the output layer of the classification model, the image retrieval layer includes a fully connected layer and a Sigmoid function, the retrieval model is used to retrieve the upper gastrointestinal endoscopy image to be processed, and the first sub-dataset having similar features to the upper gastrointestinal endoscopy image to be processed is selected from the training data set, including: Determining the image features of the upper gastrointestinal endoscopy image to be processed based on the image features extracted from the image features output by the image retrieval layer; Determining a coding value of the upper gastrointestinal endoscopic image to be processed according to image features of the upper gastrointestinal endoscopic image to be processed; Determining a coding value of each endoscope sample image in the training data set according to an image feature of each endoscope sample image in the training data set; Calculating the Hamming distance between the encoding value of the upper gastrointestinal endoscopy image to be processed and the encoding value of each endoscopy sample image in the training data set; Determine a candidate data set from the training data set according to the Hamming distance, wherein the Hamming distance between the encoding value of each endoscope sample image in the candidate data set and the encoding value of the upper gastrointestinal endoscope image to be processed is less than a distance threshold; Calculating the Euclidean distance between the upper gastrointestinal endoscopic image to be processed and each endoscopic sample image in the candidate data set; According to the ranking of the Euclidean distance from small to large, a first sub-dataset having similar features to the upper gastrointestinal endoscopy image to be processed is selected from the candidate data set, the first sub-dataset including the top K endoscopic sample images ranked in all Euclidean distances, where K is an integer greater than zero.

5. The method for determining the upper digestive tract lesion area according to claim 1, characterized in that: The segmentation model includes a second input layer, a second convolutional layer, a second maximum pooling layer, an expanded convolutional layer, and a second output layer, wherein the convolution kernel of the convolutional layer is 3×3, the convolution kernel of the output layer is 1×1, the window size of the maximum pooling layer is 2×2, the training data set also includes a second label for indicating a lesion area, and the training process of the segmentation model includes: Inputting the training data set into the segmentation model, and outputting the lesion area of ​​each endoscope sample image in the training data set through the second output layer; Comparing the lesion area of ​​each endoscope sample image in the training data set with the lesion area indicated by the second label to obtain second difference information; Based on the second difference information, a mean square error loss function is used to perform model back propagation on the segmentation model to obtain the trained segmentation model.

6. The method for determining the upper digestive tract lesion area according to claim 5, characterized in that: If the final lesion category belongs to the preset lesion category, the trained segmentation model is used to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed, and the lesion area in the upper gastrointestinal endoscopic image to be processed includes: Inputting the upper gastrointestinal endoscopic image to be processed, whose final lesion category belongs to the preset lesion category, into a trained segmentation model, and performing semantic segmentation on the upper gastrointestinal endoscopic image to be processed; It is determined that the lesion area output by the second output layer is the lesion area in the upper gastrointestinal endoscopic image to be processed.

7. The method for determining a lesion area according to any one of claims 1 to 6, characterized in that: Before inputting the upper digestive tract endoscopy image to be processed into the classification model, the method further includes: Using a test data set to test the classification performance of the trained classification model; Using the test data set to test the segmentation performance of the trained segmentation model; The test data set includes L endoscopic test images with a third label and a fourth label, where L is a positive integer less than N, the third label is used to indicate the lesion category of each endoscopic test image, and the fourth label is used to indicate the lesion area of ​​each endoscopic test image.

8. A device for determining upper digestive tract lesion areas based on multi-task assistance, characterized in that: The lesion area determining device comprises: A classification module, used for inputting the upper gastrointestinal endoscopic image to be processed into a trained classification model to obtain an initial lesion category of the upper gastrointestinal endoscopic image to be processed, wherein the classification model is trained using a training data set; A retrieval module, which uses a retrieval model to retrieve the upper gastrointestinal endoscopic image to be processed, and selects a first sub-dataset having similar features to the upper gastrointestinal endoscopic image to be processed from the training data set; a category determination module, configured to determine a final lesion category of the upper gastrointestinal endoscopic image to be processed according to the lesion category of the endoscopic sample image in the first sub-dataset and the initial lesion category; a segmentation module, configured to perform semantic segmentation on the upper gastrointestinal endoscopic image to be processed using a trained segmentation model to obtain a lesion area in the upper gastrointestinal endoscopic image to be processed if the final lesion category belongs to a preset lesion category; The category determination module comprises: A quantity acquisition submodule, used to acquire the quantity of first images corresponding to different lesion categories according to the lesion category of each endoscopic sample image in the first sub-dataset; A first final determination submodule, configured to determine the initial lesion category as the final lesion category if the lesion category corresponding to the maximum value of the first image quantities corresponding to different lesion categories is the same as the initial lesion category; or The second final determination submodule is used to determine that the lesion category corresponding to the maximum value in the first number of images is the final lesion category if the lesion category corresponding to the maximum value in the first number of images is different from the initial lesion category, and the ratio of the number of images with the maximum value in the first number of images to K is greater than the probability value of the initial lesion category; otherwise, determine the initial lesion category as the final lesion category, wherein K is the total number of endoscopic sample images in the first sub-dataset.

Images