Digestive tract cancer detection method, device, equipment, storage medium and program product

Abstract

The present invention provides a method, device, equipment, storage medium and program product for detecting digestive tract cancer, and relates to the field of image recognition technology. The method includes: inputting a digestive tract image into an image classification model to obtain an image classification result output by the image classification model; the image classification result is used to characterize the recognition result of cancer recognition for the digestive tract image; inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image; based on the semantic segmentation result and the image classification result, determining the detection method of the digestive tract part corresponding to the digestive tract image; when the detection method indicates that the digestive tract part corresponding to the digestive tract image needs to be detected, outputting prompt information corresponding to the detection method. The present invention can realize efficient, highly sensitive and highly accurate digestive tract cancer detection.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, equipment, storage medium and program product for detecting digestive tract cancer. Background Art

[0002] The prognosis of digestive tract tumors (such as esophageal cancer, gastric cancer and colorectal cancer) is closely related to early diagnosis and treatment. Gastroenteroscopy and endoscopic biopsy are currently the gold standard for clinical diagnosis of digestive tract tumors. Most guidelines for the diagnosis and treatment of digestive tract tumors show that the standardized diagnostic process is to detect lesions with ordinary white light endoscopy, confirm or clarify the nature of the lesions with chromoendoscopy, and perform qualitative biopsy.

[0003] With the rapid development of artificial intelligence, artificial intelligence technology is being used more and more widely in medical devices. Medical image recognition based on artificial intelligence can assist in the diagnosis of digestive tract cancer, especially early digestive tract cancer, and improve the diagnostic accuracy of digestive tract cancer.

[0004] At present, through target detection on digestive tract images, it is determined whether there is a lesion area in the digestive tract image based on the target detection result, that is, the lesion area suspected of cancer is framed with a rectangular frame to assist doctors in making further diagnosis. However, the resolution of the collected digestive tract images is different. Some are collected by close-up lens, and some are collected by telephoto lens, which leads to the reduction of the accuracy of target detection through a target detection model, which may cause missed detection. That is, the sensitivity of digestive tract cancer recognition is not high at present; and the target detection result is to select the lesion area with a rectangular frame, but the lesion area may not be a rectangular area, so the positioning accuracy of the lesion area is not high, resulting in the doctor still needing to determine the lesion area by himself in the rectangular frame, that is, the efficiency of digestive tract cancer recognition is not high at present. Summary of the invention

[0005] The present invention provides a method, device, equipment, storage medium and program product for detecting digestive tract cancer, so as to solve the defects of low sensitivity and efficiency in digestive tract cancer identification in the prior art and realize efficient and highly sensitive digestive tract cancer detection.

[0006] The present invention provides a method for detecting digestive tract cancer, comprising:

[0007] Inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model; the image classification result is used to represent a recognition result of cancer recognition for the digestive tract image;

[0008] Inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image;

[0009] Determining a detection method for a digestive tract portion corresponding to the digestive tract image based on the semantic segmentation result and the image classification result;

[0010] When the detection method indicates that the digestive tract portion corresponding to the digestive tract image needs to be detected, prompt information corresponding to the detection method is output.

[0011] According to a method for detecting digestive tract cancer provided by the present invention, if the digestive tract image is a stained image, the image classification result includes a first classification result or a second classification result, and the detection method of determining the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result includes:

[0012] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, determining that the detection method is a biopsy;

[0013] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, determining that the detection method is no need for detection;

[0014] If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required;

[0015] The digestive tract portion corresponding to the stained image has been stained; the cancer probability value represented by the first classification result is greater than the cancer probability value represented by the second classification result;

[0016] If the digestive tract image is an unstained image, the image classification result includes a third classification result, a fourth classification result, or a fifth classification result, and the method of determining the detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result includes:

[0017] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, determining that the detection method is a biopsy;

[0018] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, determining that the detection method is staining;

[0019] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection;

[0020] If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required;

[0021] Among them, the digestive tract part corresponding to the unstained image is not stained; the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

[0022] According to a method for detecting digestive tract cancer provided by the present invention, if the digestive tract image is a stained image, inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model includes:

[0023] Inputting the digestive tract image into a first image classification model corresponding to the stained image to obtain an image classification result output by the first image classification model;

[0024] If the digestive tract image is an unstained image, inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model includes:

[0025] Inputting the digestive tract image into a second image classification model corresponding to the unstained image to obtain an image classification result output by the second image classification model;

[0026] If the digestive tract image is a stained image, inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes:

[0027] Inputting the digestive tract image into a first semantic segmentation model corresponding to the stained image to obtain a semantic segmentation result output by the first semantic segmentation model;

[0028] If the digestive tract image is an unstained image, inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes:

[0029] Inputting the digestive tract image into a second semantic segmentation model corresponding to the unstained image to obtain a semantic segmentation result output by the second semantic segmentation model;

[0030] The digestive tract portion corresponding to the stained image has been stained, and the digestive tract portion corresponding to the unstained image has not been stained.

[0031] According to a method for detecting digestive tract cancer provided by the present invention, the semantic segmentation model is an integrated semantic segmentation model, and the integrated semantic segmentation model includes a plurality of different segmentation models;

[0032] For any pixel of the digestive tract image, the integrated semantic segmentation model is used to determine multiple category prediction results of the pixel, any of the category prediction results including lesion pixels or non-lesion pixels. If any category prediction result among the multiple category prediction results is a lesion pixel, the final category prediction result of the pixel is determined as a lesion pixel. If all category prediction results among the multiple category prediction results are non-lesion pixels, the final category prediction result of the pixel is determined as a non-lesion pixel.

[0033] According to a method for detecting digestive tract cancer provided by the present invention, before inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model, the method further includes:

[0034] Inputting the digestive tract image into a feature extraction model to obtain image features output by the feature extraction model;

[0035] Accordingly, the step of inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model includes:

[0036] Inputting the image features into an image classification model to obtain an image classification result output by the image classification model;

[0037] Accordingly, the step of inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes:

[0038] The image features are input into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0039] According to a method for detecting digestive tract cancer provided by the present invention, the image classification result includes a target classification result or a plurality of non-target classification results, and the cancer probability values ​​represented by the plurality of non-target classification results are all smaller than the cancer probability value represented by the target classification result;

[0040] Before determining a detection method of a digestive tract portion corresponding to the digestive tract image based on the semantic segmentation result and the image classification result, the method further includes:

[0041] If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the target classification result, adjusting the segmentation threshold of the semantic segmentation model to obtain and output the lesion area in the digestive tract image;

[0042] Among them, for any pixel of the digestive tract image, the semantic segmentation model determines the category prediction result of the pixel based on the segmentation threshold, and the category prediction result of the pixel includes a lesion pixel or a non-lesion pixel.

[0043] According to a method for detecting digestive tract cancer provided by the present invention, the step of adjusting the segmentation threshold of the semantic segmentation model includes:

[0044] The lowest segmentation threshold is determined as 0, and the highest segmentation threshold is determined as the current segmentation threshold of the semantic segmentation model;

[0045] Determine a sum of the minimum segmentation threshold and the maximum segmentation threshold, and adjust the segmentation threshold of the semantic segmentation model to half of the sum;

[0046] Based on the adjusted segmentation threshold, updating the semantic segmentation result;

[0047] If it is determined based on the updated semantic segmentation result that the proportion of the lesion area of ​​the digestive tract image is greater than the first preset proportion, the minimum segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the minimum segmentation threshold and the maximum segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum;

[0048] If it is determined based on the updated semantic segmentation result that the proportion of the lesion area of ​​the digestive tract image is less than the second preset proportion, the highest segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the lowest segmentation threshold and the highest segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum;

[0049] Among them, for any pixel of the digestive tract image, the semantic segmentation model outputs the predicted value of the pixel, and if the predicted value of the pixel is greater than or equal to the segmentation threshold, the category prediction result of the pixel is determined to be a lesion pixel; the lesion area ratio is determined based on the ratio of the lesion area in the digestive tract image to the area of ​​the digestive tract image, and the first preset ratio is greater than the second preset ratio.

[0050] According to a method for detecting digestive tract cancer provided by the present invention, the method further comprises:

[0051] Inputting the digestive tract image into a digestive tract part recognition model to obtain a digestive tract part recognition result output by the digestive tract part recognition model;

[0052] Outputting prompt information corresponding to the digestive tract part identification result to indicate the digestive tract part currently being examined;

[0053] The digestive tract part recognition model is obtained by training based on sample digestive tract images and digestive tract part recognition result labels corresponding to the sample digestive tract images.

[0054] According to a method for detecting digestive tract cancer provided by the present invention, after inputting the digestive tract image into a digestive tract part recognition model and obtaining a digestive tract part recognition result output by the digestive tract part recognition model, the method further comprises:

[0055] When the digestive tract part indicated by the digestive tract part identification result has been inspected, determining the digestive tract parts that have not been inspected from the target digestive tract parts set;

[0056] Output prompt information corresponding to the digestive tract part that has not been examined.

[0057] According to a method for detecting digestive tract cancer provided by the present invention, after inputting the digestive tract image into a digestive tract part recognition model and obtaining a digestive tract part recognition result output by the digestive tract part recognition model, the method further comprises:

[0058] Determining the mirror-out durations of the plurality of digestive tract parts based on historical digestive tract part recognition results of the plurality of digestive tract images that have been collected and the collection times of the plurality of digestive tract images that have been collected;

[0059] Based on the withdrawal times of the multiple digestive tract locations, a total withdrawal time of the inspection scope for acquiring digestive tract images is determined.

[0060] According to a method for detecting digestive tract cancer provided by the present invention, before inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model, the method further includes:

[0061] Inputting the digestive tract image into a digestive tract mucosa cleanliness recognition model to obtain a digestive tract mucosa cleanliness recognition result output by the digestive tract mucosa cleanliness recognition model; the digestive tract mucosa cleanliness recognition result is used to indicate whether the digestive tract image is a cleanliness qualified image;

[0062] If it is determined based on the digestive tract mucosal cleanliness recognition result that the digestive tract image is an image with unqualified cleanliness, outputting flushing prompt information to prompt flushing of the digestive tract portion corresponding to the digestive tract image;

[0063] The digestive tract mucosa cleanliness recognition model is obtained by training based on sample digestive tract images and digestive tract mucosa cleanliness recognition result labels corresponding to the sample digestive tract images.

[0064] The present invention also provides a digestive tract cancer detection device, comprising:

[0065] An image classification module, used to input the digestive tract image into the image classification model to obtain an image classification result output by the image classification model; the image classification result is used to represent the recognition result of cancer recognition for the digestive tract image;

[0066] A semantic segmentation module, used for inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used for indicating whether there is a lesion area in the digestive tract image;

[0067] A method determination module, used to determine a detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result;

[0068] The prompt output module is used to output prompt information corresponding to the detection method when the detection method indicates that the digestive tract part corresponding to the digestive tract image needs to be detected.

[0069] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for detecting digestive tract cancer as described above is implemented.

[0070] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for detecting digestive tract cancer as described in any one of the above is implemented.

[0071] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for detecting digestive tract cancer as described in any one of the above is implemented.

[0072] The digestive tract cancer detection method, device, equipment, storage medium and program product provided by the present invention input the digestive tract image into an image classification model to obtain the image classification result output by the image classification model, and input the digestive tract image into a semantic segmentation model to obtain the semantic segmentation result output by the semantic segmentation model, and the image classification result is used to characterize the recognition result of cancer recognition for the digestive tract image, and the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image and locate the position range of the lesion area, so that the fusion of semantic segmentation and image classification can improve the recognition accuracy of digestive tract cancer, thereby improving the sensitivity of digestive tract cancer recognition, and any one of the image classification result and the semantic segmentation result indicates In the case of the presence of digestive tract cancer, it can be preliminarily determined that digestive tract cancer exists, thereby improving the sensitivity of digestive tract cancer identification, ensuring a certain biopsy positivity rate, and reducing missed detections; and based on the semantic segmentation results and the image classification results, the detection method of the digestive tract part corresponding to the digestive tract image is determined to assist the doctor in making decisions on the detection method, thereby improving the detection efficiency of digestive tract cancer, and improving the sensitivity of digestive tract cancer identification, and reducing missed detections; and in the case where the detection method indicates that the digestive tract part corresponding to the digestive tract image needs to be detected, the prompt information corresponding to the detection method is output to prompt the doctor for further detection methods, thereby improving the detection efficiency of digestive tract cancer, and improving the sensitivity of digestive tract cancer identification, and reducing missed detections. In summary, the present invention can achieve efficient, highly sensitive, and highly accurate digestive tract cancer detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 This is one of the flow charts of the method for detecting digestive tract cancer provided by the present invention.

[0075] Figure 2 This is the second flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0076] Figure 3 This is the third flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0077] Figure 4 This is the fourth flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0078] Figure 5 This is the fifth flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0079] Figure 6 This is the sixth flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0080] Figure 7 This is the seventh flow chart of the method for detecting digestive tract cancer provided by the present invention.

[0081] Figure 8 It is a schematic diagram of the structure of the digestive tract cancer detection device provided by the present invention.

[0082] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] At present, through target detection on digestive tract images, it is determined whether there is a lesion area in the digestive tract image based on the target detection result, that is, the lesion area suspected of cancer is framed with a rectangular frame to assist doctors in making further diagnosis. However, the resolution of the collected digestive tract images is different. Some are collected by close-up lens, and some are collected by telephoto lens, which leads to the reduction of the accuracy of target detection through a target detection model, which may cause missed detection. That is, the sensitivity of digestive tract cancer recognition is not high at present; and the target detection result is to select the lesion area with a rectangular frame, but the lesion area may not be a rectangular area, so the positioning accuracy of the lesion area is not high, resulting in the doctor still needing to determine the lesion area by himself in the rectangular frame, that is, the efficiency of digestive tract cancer recognition is not high at present.

[0085] For example, obtain the real-time gastric image collected during the electronic staining endoscopy; identify and annotate the lesion area based on the pre-built gastric early cancer recognition model; when the identified lesion areas overlap, determine whether they belong to the same lesion area. This method uses target detection technology to achieve image recognition and position positioning at the same time, but the lesion area is selected using a rectangular frame, and the position positioning is not accurate enough. The lesion area may not be a rectangular area, and the doctor needs to make a judgment within the rectangular frame. And using only one target detection model, it is difficult to apply it to both close-up and long-distance lenses. The sensitivity of early cancer recognition is not high enough, and many missed detections will occur, causing economic losses or even life-threatening consequences to patients.

[0086] In addition, by classifying the digestive tract images, whether it is suspected early cancer is determined based on the image classification results, thereby assisting doctors in making further diagnoses. However, the accuracy of digestive tract cancer identification based solely on image classification results is not high, which may result in missed detections, that is, the sensitivity of current digestive tract cancer identification is not high; and image classification cannot determine the lesion area, resulting in the need for doctors to determine the lesion area in the digestive tract image themselves, that is, the efficiency of current digestive tract cancer identification is not high.

[0087] For example, a sample set of early gastric cancer images is obtained, and feature recognition is performed on each lesion image in the sample set of early gastric cancer images to obtain a feature vector of each lesion image; a preset initial model is trained based on the sample set of early gastric cancer images and the feature vector to obtain an early gastric cancer recognition model. However, this method only provides a method for early gastric cancer recognition and does not provide a function for locating the location of early cancer lesions.

[0088] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 7 The method for detecting digestive tract cancer of the present invention is described.

[0089] Figure 1 FIG. 1 is one of the flow charts of the method for detecting digestive tract cancer provided by the present invention, such as Figure 1 As shown, the method for detecting digestive tract cancer includes the following steps 110, 120, 130 and 140.

[0090] Step 110, inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model.

[0091] Here, digestive tract cancer may be early digestive tract cancer, advanced digestive tract cancer, etc. The embodiment of the present invention is described by taking early digestive tract cancer (early digestive tract malignant tumor) as an example.

[0092] Here, the digestive tract image is an image acquired by collecting the digestive tract. The digestive tract image can be acquired by digestive tract endoscopy.

[0093] Exemplarily, the parts of the digestive tract that can be collected by an upper gastrointestinal endoscope (gastroscopy) include: oropharynx, upper esophagus, middle esophagus, lower esophagus, esophagogastric junction, upper gastric body, posterior wall of upper gastric body, anterior wall of upper gastric body, greater curvature of upper gastric body, lesser curvature of upper gastric body, middle gastric body, posterior wall of middle gastric body, anterior wall of middle gastric body, greater curvature of middle gastric body, lesser curvature of middle gastric body, lower gastric body, posterior wall of lower gastric body, anterior wall of lower gastric body, greater curvature of lower gastric body, lesser curvature of lower gastric body, greater curvature of antrum-body junction, gastric angle, posterior wall of gastric angle, anterior wall of gastric angle, gastric antrum, posterior wall of gastric antrum, anterior wall of gastric antrum, greater curvature of gastric antrum, lesser curvature of gastric antrum, pylorus, duodenal bulb, descending duodenum, gastric fundus, posterior wall of gastric fundus, anterior wall of gastric fundus, greater curvature of gastric fundus, lesser curvature of gastric fundus and cardia. The parts of the digestive tract that can be sampled by colonoscopy include: the terminal ileum, appendix orifice, ileocecal valve, ascending colon, hepatic flexure of the colon, transverse colon, splenic flexure, descending colon, sigmoid colon, rectum, and rectal junction.

[0094] Further, the collected digestive tract images are preprocessed. In one embodiment, the collected digestive tract images are uniformly adjusted to digestive tract images of a preset input size, for example, the preset input size is .

[0095] The image classification result is used to characterize the recognition result of cancer recognition of the digestive tract image. That is, the image classification model is used to recognize cancer in the digestive tract image. Specifically, the image classification model is used to classify the digestive tract image into classification results with different cancer probability values, and then determine whether it is suspected digestive tract cancer.

[0096] In one embodiment, the image classification model is trained based on sample digestive tract images and image classification result labels corresponding to the sample digestive tract images.

[0097] In some embodiments, it is first determined whether the digestive tract image is a stained image; if the digestive tract image is a stained image, the digestive tract image is input into a first image classification model corresponding to the stained image to obtain an image classification result output by the first image classification model; if the digestive tract image is an unstained image, the digestive tract image is input into a second image classification model corresponding to the unstained image to obtain an image classification result output by the second image classification model; wherein the digestive tract portion corresponding to the stained image is stained, and the digestive tract portion corresponding to the unstained image is unstained. That is, considering that the stained image and the unstained image are quite different, different image classification models are set to classify the images, thereby improving the accuracy of image classification, and then improving the accuracy of digestive tract cancer recognition, and finally improving the specificity of digestive tract cancer recognition.

[0098] In other embodiments, regardless of whether the digestive tract image is a stained image or not, the digestive tract image is input into an image classification model to obtain an image classification result output by the image classification model.

[0099] Step 120: input the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0100] The semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image and locate the position range of the lesion area. The semantic segmentation result is a pixel-level segmentation result, that is, the semantic segmentation model is used to determine the category prediction result of each pixel in the digestive tract image, that is, to determine whether each pixel in the digestive tract image is a lesion pixel, so that irregular lesion areas can be determined, that is, the actual lesion area can be determined, thereby improving the positioning accuracy of the lesion area, that is, accurately locating the position range of the lesion area, assisting doctors in determining the lesion area, thereby improving the recognition accuracy of digestive tract cancer while improving the recognition efficiency of digestive tract cancer.

[0101] Furthermore, the semantic segmentation model is an integrated semantic segmentation model, which includes multiple different segmentation models. Compared with a single semantic segmentation model, the integrated semantic segmentation model can improve the accuracy of the semantic segmentation results, thereby improving the recognition accuracy of digestive tract cancer. Furthermore, the integrated semantic segmentation model is composed of multiple single semantic segmentation models based on the bagging method.

[0102] In one embodiment, the semantic segmentation model is trained based on sample digestive tract images and semantic segmentation result labels corresponding to the sample digestive tract images.

[0103] In some embodiments, it is first determined whether the digestive tract image is a stained image; if the digestive tract image is a stained image, the digestive tract image is input into the first semantic segmentation model corresponding to the stained image to obtain the semantic segmentation result output by the first semantic segmentation model; if the digestive tract image is an unstained image, the digestive tract image is input into the second semantic segmentation model corresponding to the unstained image to obtain the semantic segmentation result output by the second semantic segmentation model; wherein the digestive tract portion corresponding to the stained image is stained, and the digestive tract portion corresponding to the unstained image is not stained. That is, considering that the stained image and the unstained image are quite different, different semantic segmentation models are set to perform semantic segmentation on them, thereby improving the accuracy of semantic segmentation, and then improving the accuracy of digestive tract cancer recognition, and finally improving the specificity of digestive tract cancer recognition.

[0104] In other embodiments, regardless of whether the digestive tract image is a stained image or not, the digestive tract image is input into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0105] Step 130: Determine a detection method for the digestive tract portion corresponding to the digestive tract image based on the semantic segmentation result and the image classification result.

[0106] Here, the detection method includes biopsy, staining or no detection. This detection method is a further detection method, that is, a further diagnosis result.

[0107] It should be understood that the image classification result is used to characterize the recognition result of cancer identification in the digestive tract image, and the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image and locate the position range of the lesion area, so that the fusion of semantic segmentation and image classification can improve the recognition accuracy of digestive tract cancer, and when any one of the image classification result and the semantic segmentation result indicates the presence of digestive tract cancer, it can be preliminarily judged that there is digestive tract cancer, thereby improving the sensitivity of digestive tract cancer identification, ensuring a certain biopsy positivity rate, and reducing missed detections; and based on the semantic segmentation result and the image classification result, the detection method of the digestive tract part corresponding to the digestive tract image is determined to assist doctors in determining the detection method, thereby improving the detection efficiency of digestive tract cancer.

[0108] In some embodiments, it is first determined whether the digestive tract image is a stained image; if the digestive tract image is a stained image, how to determine the detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result can be referred to the following embodiments, which will not be described one by one here.

[0109] In other embodiments, regardless of whether the digestive tract image is a stained image, the image classification result includes the third classification result, the fourth classification result, or the fifth classification result, and step 130 includes:

[0110] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, determining that the detection method is a biopsy;

[0111] If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required;

[0112] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, and the digestive tract image is an unstained image, determining that the detection method is staining;

[0113] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, and the digestive tract image is a stained image, determining that the detection method is no need for detection;

[0114] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection;

[0115] Among them, the digestive tract part corresponding to the stained image has been stained, the digestive tract part corresponding to the unstained image has not been stained, the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

[0116] It should be noted that if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no need for detection; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image and the image classification result is the fifth classification result, the detection method is determined to be no need for detection; the above two judgment logics are equivalent to determining that if the image classification result is the fifth classification result, the detection method is determined to be no need for detection.

[0117] For example, Figure 2 As shown, the collected digestive tract images are subjected to image preprocessing, and then the features of the digestive tract images after image preprocessing are extracted, and then the image features of the digestive tract images are respectively input into the semantic segmentation model and the image classification model; then, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result. If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no need for detection, and accordingly no prompt is given; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, a subsequent judgment is made based on the image classification result of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion in the digestive tract image area, and the image classification result is the third classification result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, first determine whether the digestive tract image is a stained image. If the digestive tract image is an unstained image, the detection method is determined to be staining, and a staining prompt is given accordingly. If the digestive tract image is a stained image, the detection method is determined to be no need for detection, and no prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, the detection method is determined to be no need for detection, and no prompt is given accordingly.

[0118] The image classification model can output different probabilities. Based on this, if the probability output by the image classification model is greater than the biopsy probability threshold, the image classification result is the third classification result. If the probability output by the image classification model is less than or equal to the biopsy probability threshold and greater than the staining probability threshold, the image classification result is the fourth classification result. If the probability output by the image classification model is less than or equal to the staining probability threshold, the image classification result is the fifth classification result, where the biopsy probability threshold is greater than the staining probability threshold. Therefore, according to the different output probabilities of the image classification model, the judgment logic for determining the detection method is also different.

[0119] For example, Figure 3 As shown in the figure, the collected digestive tract images are subjected to image preprocessing, and then the features of the digestive tract images after image preprocessing are extracted, and then the image features of the digestive tract images are respectively input into the semantic segmentation model and the image classification model; then, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result. If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no need for detection, and accordingly no prompt is given; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, a subsequent judgment is made based on the output probability of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the output probability of the image classification model is If the output probability is greater than the biopsy probability threshold, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the output probability of the image classification model is not greater than the biopsy probability threshold, first determine whether the output probability is greater than the staining probability threshold. If the output probability is not greater than the staining probability threshold, determine that the detection method does not require detection, and no prompt is given accordingly. If the output probability is greater than the staining probability threshold, first determine whether the digestive tract image is a stained image. If the digestive tract image is an unstained image, determine that the detection method is staining, and a staining prompt is given accordingly. If the digestive tract image is a stained image, determine that the detection method is not required, and no prompt is given accordingly.

[0120] In other embodiments, regardless of whether the digestive tract image is a stained image, the image classification result includes the third classification result, the fourth classification result, or the fifth classification result, and step 130 includes:

[0121] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, or the image classification result is the third classification result, determining that the detection method is a biopsy;

[0122] If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the fourth classification result, determining that the detection method is staining;

[0123] If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection;

[0124] Among them, the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

[0125] It should be understood that if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, or the image classification result is the third classification result, and the detection method is determined to be biopsy, that is, if any one of the image classification results and the semantic segmentation results indicates the presence of digestive tract cancer, it can be preliminarily judged that there is digestive tract cancer, and thus the detection method is determined to be biopsy, thereby improving the sensitivity of digestive tract cancer identification, ensuring a certain biopsy positivity rate, and reducing missed detections; and if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the fifth classification result, and the detection method is determined to be no need for detection, that is, if all the results in the image classification results and the semantic segmentation results indicate the absence of digestive tract cancer, then the detection method is determined to be no need for detection, thereby improving the sensitivity of digestive tract cancer identification, ensuring a certain biopsy positivity rate, and reducing missed detections.

[0126] For example, Figure 4 As shown, the collected digestive tract images are subjected to image preprocessing, and then the features of the digestive tract images after image preprocessing are extracted, and then the image features of the digestive tract images are respectively input into the semantic segmentation model and the image classification model; then, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result. If it is determined that there is a lesion area in the digestive tract image based on the semantic segmentation result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly. If it is determined that there is no lesion area in the digestive tract image based on the semantic segmentation result, a subsequent judgment is made based on the image classification result of the image classification model; if the image classification result is the third classification result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined that there is no lesion area in the digestive tract image based on the semantic segmentation result, and the image classification result is the fourth classification result, the detection method is determined to be staining, and a staining prompt is given accordingly; if it is determined that there is no lesion area in the digestive tract image based on the semantic segmentation result, and the image classification result is the fifth classification result, the detection method is determined to be no detection required, and no prompt is given accordingly.

[0127] The image classification model can output different probabilities. Based on this, if the probability output by the image classification model is greater than the biopsy probability threshold, the image classification result is the third classification result. If the probability output by the image classification model is less than or equal to the biopsy probability threshold and greater than the staining probability threshold, the image classification result is the fourth classification result. If the probability output by the image classification model is less than or equal to the staining probability threshold, the image classification result is the fifth classification result, where the biopsy probability threshold is greater than the staining probability threshold. Therefore, according to the different output probabilities of the image classification model, the judgment logic for determining the detection method is also different.

[0128] For example, Figure 5As shown, the collected digestive tract images are subjected to image preprocessing, and then the features of the digestive tract images after image preprocessing are extracted, and then the image features of the digestive tract images are respectively input into the semantic segmentation model and the image classification model; then, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result. If it is determined that there is a lesion area in the digestive tract image based on the semantic segmentation result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly. If it is determined that there is no lesion area in the digestive tract image based on the semantic segmentation result, subsequent judgment is made based on the output probability of the image classification model; if the output probability is greater than the biopsy probability threshold, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if the output probability is not greater than the biopsy probability threshold, it is determined whether the output probability is greater than the staining probability threshold; if the output probability is greater than the staining probability threshold, it is determined that the detection method is staining, and a staining prompt is given accordingly; if the output probability is not greater than the staining probability threshold, it is determined that the detection method is not required for detection, and no prompt is given accordingly.

[0129] Step 140: When the detection method indicates that the digestive tract portion corresponding to the digestive tract image needs to be detected, output prompt information corresponding to the detection method.

[0130] Exemplarily, if the detection method is biopsy, the corresponding prompt information is biopsy prompt information, to prompt further biopsy; if the detection method is staining, the corresponding prompt information is staining prompt information, to prompt further staining; if the detection method is no need for detection, there is no need to output prompt information.

[0131] It should be understood that the prompt information corresponding to the detection method is directly output to prompt the doctor for further detection methods and assist the doctor in making further diagnosis, thereby improving the detection efficiency of gastrointestinal cancer.

[0132] The method for detecting digestive tract cancer provided by the embodiment of the present invention inputs a digestive tract image into an image classification model to obtain an image classification result output by the image classification model, and inputs the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, and the image classification result is used to characterize the recognition result of cancer recognition for the digestive tract image, and the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image, so that the fusion of semantic segmentation and image classification can improve the recognition accuracy of digestive tract cancer, and further improve the sensitivity of digestive tract cancer recognition, and when any one of the image classification result and the semantic segmentation result indicates the presence of digestive tract cancer, It can preliminarily determine the presence of digestive tract cancer, thereby improving the sensitivity of digestive tract cancer identification, ensuring a certain biopsy positivity rate, and reducing missed detections; and based on the semantic segmentation results and the image classification results, determine the detection method of the digestive tract part corresponding to the digestive tract image, assist doctors in determining the detection method, thereby improving the detection efficiency of digestive tract cancer, and improving the sensitivity of digestive tract cancer identification, and reducing missed detections; and when the detection method indicates that the digestive tract part corresponding to the digestive tract image needs to be detected, output prompt information corresponding to the detection method to prompt the doctor for further detection methods, thereby improving the detection efficiency of digestive tract cancer, and improving the sensitivity of digestive tract cancer identification, and reducing missed detections. In summary, the present invention can achieve efficient, highly sensitive, and highly accurate digestive tract cancer detection.

[0133] Based on any of the above embodiments, if the digestive tract image is a stained image, and the image classification result includes a first classification result or a second classification result, the above step 130 includes:

[0134] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, determining that the detection method is a biopsy;

[0135] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, determining that the detection method is no need for detection;

[0136] If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, the detection method is determined as no detection is required.

[0137] The digestive tract part corresponding to the stained image has been stained, and the cancer probability value represented by the first classification result is greater than the cancer probability value represented by the second classification result. For example, the first classification result is suspected cancer, and the second classification result is no cancer.

[0138] Considering that the digestive tract image is a stained image, that is, when the staining detection method has been performed, it is only necessary to further determine whether a biopsy is needed. Based on this, it is only necessary to perform binary classification on the digestive tract image, that is, the image classification model is a binary classification model.

[0139] In some embodiments, the digestive tract image is input into the staining recognition model to obtain a staining recognition result output by the staining recognition model, and the staining recognition result is used to indicate whether the digestive tract image is a stained image. Further, the staining recognition result can also indicate whether the digestive tract image is an electro-stained image or a chemical-stained image, and the staining method corresponding to the electro-stained image is electronic staining, and the staining method corresponding to the chemical-stained image is chemical staining.

[0140] In one embodiment, the staining recognition model is a two-classification model, and the classification result includes a stained image or an unstained image. Further, the staining recognition model is a three-classification model, and the classification result includes an electro-stained image, a chemical-stained image, or a white light image.

[0141] In one embodiment, the staining recognition model is trained based on sample digestive tract images and their corresponding staining recognition result labels.

[0142] Considering that if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, since the digestive tract image is already a stained image, that is, the staining detection method has been performed, the further detection method is determined to be biopsy.

[0143] Considering that if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, since the digestive tract image is already a stained image, that is, the staining detection method has been executed, it is determined that further detection is not required.

[0144] Considering that the semantic segmentation model performs semantic segmentation at the pixel level, that is, the semantic segmentation result is more sensitive, based on this, if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is directly determined as no detection is required.

[0145] If the digestive tract image is an unstained image, the image classification result includes the third classification result, the fourth classification result or the fifth classification result, and the step 130 includes:

[0146] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, determining that the detection method is a biopsy;

[0147] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, determining that the detection method is staining;

[0148] If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection;

[0149] If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, the detection method is determined as no detection is required.

[0150] The digestive tract part corresponding to the unstained image is not stained; the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result. For example, the third classification result is suspected cancer, the fourth classification result is low suspected cancer, and the fifth classification result is no cancer.

[0151] Considering that the digestive tract image is an unstained image (white light image), that is, the detection method without staining is performed, it is necessary to determine whether a biopsy is needed and whether staining is needed. Based on this, the digestive tract image needs to be classified into three categories, that is, the image classification model is a three-category model.

[0152] Considering that if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, since the digestive tract image is an unstained image, that is, the staining detection method is not performed, the further detection method is determined to be staining.

[0153] Considering that the semantic segmentation model performs semantic segmentation at the pixel level, that is, the semantic segmentation result has a higher sensitivity, based on this, if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is directly determined as no detection is required.

[0154] For example, Figure 6As shown, the collected digestive tract images are subjected to image preprocessing, and then feature extraction is performed on the digestive tract images after image preprocessing, and then the image features of the digestive tract images are input into the staining recognition model. If the digestive tract image is determined to be a white light image (unstained image) based on the staining recognition model, the image features are respectively input into the semantic segmentation model and image classification model corresponding to the unstained image, and firstly determine whether there is a lesion area in the digestive tract image based on the semantic segmentation result; if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no detection required, and accordingly no prompt is given; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, subsequent judgment is made based on the image classification result of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, the detection method is determined to be staining, and a staining prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, the detection method is determined to be no detection required, and accordingly no prompt is given. If the digestive tract image is determined to be a stained image based on the staining recognition model, the image features are respectively input into the semantic segmentation model and the image classification model corresponding to the stained image, and firstly, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result; if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no need for detection, and accordingly no prompt is given; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, subsequent judgment is made based on the image classification result of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, the detection method is determined to be no need for detection, and accordingly no prompt is given.

[0155] The image classification model can output different probabilities. Based on this, if the probability output by the image classification model is greater than the biopsy probability threshold, the image classification result is the first classification result or the third classification result. If the probability output by the image classification model is less than or equal to the biopsy probability threshold, the image classification result is the second classification result. If the probability output by the image classification model is less than or equal to the biopsy probability threshold and greater than the staining probability threshold, the image classification result is the fourth classification result. If the probability output by the image classification model is less than or equal to the staining probability threshold, the image classification result is the fifth classification result, where the biopsy probability threshold is greater than the staining probability threshold. Therefore, according to the different output probabilities of the image classification model, the judgment logic for determining the detection method is also different.

[0156] For example, Figure 7 As shown, the collected digestive tract images are subjected to image preprocessing, and then feature extraction is performed on the digestive tract images after image preprocessing, and then the image features of the digestive tract images are input into the staining recognition model. If the digestive tract image is determined to be a white light image (unstained image) based on the staining recognition model, the image features are respectively input into the semantic segmentation model and the image classification model corresponding to the unstained image, and firstly determine whether there is a lesion area in the digestive tract image based on the semantic segmentation result; if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no detection required, and no prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, subsequent judgment is made based on the output probability of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the output probability is greater than the biopsy probability threshold, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the output probability is not greater than the biopsy probability threshold, it is determined whether the output probability is greater than the staining probability threshold; if the output probability is greater than the staining probability threshold, the detection method is determined to be staining, and a staining prompt is given accordingly; if the output probability is not greater than the staining probability threshold, the detection method is determined to be no detection required, and no prompt is given accordingly. If the digestive tract image is determined to be a stained image based on the staining recognition model, the image features are respectively input into the semantic segmentation model and the image classification model corresponding to the stained image, and firstly, it is determined whether there is a lesion area in the digestive tract image based on the semantic segmentation result; if it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, the detection method is determined to be no need for detection, and accordingly no prompt is given; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, subsequent judgment is made based on the output probability of the image classification model; if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the output probability is greater than the biopsy probability threshold, the detection method is determined to be biopsy, and a biopsy prompt is given accordingly; if the output probability is not greater than the biopsy probability threshold, the detection method is determined to be no need for detection, and accordingly no prompt is given.

[0157] The method for detecting digestive tract cancer provided by the embodiment of the present invention first determines whether the digestive tract image is a stained image through the above-mentioned method, and then makes a subsequent judgment on the specific detection method, thereby improving the determination efficiency and accuracy of the detection method, and then improving the detection efficiency of digestive tract cancer, and improving the sensitivity of digestive tract cancer recognition; at the same time, through the determination logic of the above-mentioned detection method, the determination accuracy of the detection method can be improved; and the detection method is determined to be biopsy only when it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result or the third classification result, thereby reducing false positives and improving the specificity of cancer recognition; and even if it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, as long as the image classification result is the second classification result or the fifth classification result, the detection method is determined as no need for detection, thereby reducing false positives and improving the specificity of cancer recognition; and as long as it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, there is no need to determine what the image classification result is, and the detection method is determined as no need for detection, thereby reducing false positives and improving the specificity of cancer recognition.

[0158] Based on any of the above embodiments, taking into account that there is a large difference between stained images and unstained images, based on this, if the digestive tract image is a stained image, the above step 110 includes: inputting the digestive tract image into a first image classification model corresponding to the stained image, and obtaining an image classification result output by the first image classification model; if the digestive tract image is an unstained image, the above step 110 includes: inputting the digestive tract image into a second image classification model corresponding to the unstained image, and obtaining an image classification result output by the second image classification model.

[0159] Taking into account the significant difference between stained images and unstained images, based on this, if the digestive tract image is a stained image, the above step 120 includes: inputting the digestive tract image into a first semantic segmentation model corresponding to the stained image, and obtaining the semantic segmentation result output by the first semantic segmentation model; if the digestive tract image is an unstained image, the above step 120 includes: inputting the digestive tract image into a second semantic segmentation model corresponding to the unstained image, and obtaining the semantic segmentation result output by the second semantic segmentation model.

[0160] The digestive tract portion corresponding to the stained image has been stained, and the digestive tract portion corresponding to the unstained image has not been stained.

[0161] In some embodiments, the digestive tract image is input into a staining recognition model to obtain a staining recognition result output by the staining recognition model, and the staining recognition result is used to indicate whether the digestive tract image is a stained image.

[0162] Furthermore, the staining recognition result can also indicate whether the digestive tract image is an electro-stained image or a chemical-stained image. Considering that there are also differences between electro-stained images and chemical-stained images, based on this, if the digestive tract image is an electro-stained image, the above step 110 includes: inputting the digestive tract image into the first image classification model corresponding to the electro-stained image, and obtaining the image classification result output by the first image classification model; if the digestive tract image is a chemical-stained image, the above step 110 includes: inputting the digestive tract image into the first image classification model corresponding to the chemical-stained image, and obtaining the image classification result output by the first image classification model. If the digestive tract image is an electro-stained image, the above step 120 includes: inputting the digestive tract image into the first semantic segmentation model corresponding to the electro-stained image, and obtaining the semantic segmentation result output by the first semantic segmentation model; if the digestive tract image is a chemical-stained image, the above step 120 includes: inputting the digestive tract image into the first semantic segmentation model corresponding to the chemical-stained image, and obtaining the semantic segmentation result output by the first semantic segmentation model. Therefore, different image classification models are set to perform image classification, thereby improving the accuracy of image classification, further improving the accuracy of digestive tract cancer recognition, and ultimately improving the sensitivity of digestive tract cancer recognition.

[0163] The method for detecting digestive tract cancer provided by the embodiment of the present invention, through the above-mentioned method, respectively sets different image classification models to perform image classification on stained images or unstained images, thereby improving the image classification accuracy, and then improving the recognition accuracy of digestive tract cancer, and ultimately improving the sensitivity of digestive tract cancer recognition; and respectively sets different semantic segmentation models to perform semantic segmentation on stained images or unstained images, thereby improving the semantic segmentation accuracy, and then improving the recognition accuracy of digestive tract cancer, and ultimately improving the sensitivity of digestive tract cancer recognition.

[0164] Based on any of the above embodiments, in the method, the semantic segmentation model is an integrated semantic segmentation model, and the integrated semantic segmentation model includes multiple different segmentation models.

[0165] For any pixel of the digestive tract image, the integrated semantic segmentation model is used to determine multiple category prediction results of the pixel, any of which includes lesion pixels or non-lesion pixels. If any of the multiple category prediction results is a lesion pixel, the final category prediction result of the pixel is determined as a lesion pixel. If all of the multiple category prediction results are non-lesion pixels, the final category prediction result of the pixel is determined as a non-lesion pixel. That is, the embodiment of the present invention uses the union of multiple single semantic segmentation model results to determine the final result.

[0166] The existing bagging ensemble learning method uses voting to take the majority as the final result. The embodiment of the present invention takes into account that the harm of missed detection of digestive tract cancer is greater, so as long as there is a positive result, the final result is positive, that is, if any of the multiple category prediction results is a lesion pixel, the final category prediction result of the pixel is determined as a lesion pixel, thereby improving the sensitivity of digestive tract cancer recognition and reducing missed detection.

[0167] In the method for detecting digestive tract cancer provided by an embodiment of the present invention, the semantic segmentation model is an integrated semantic segmentation model, that is, the integrated semantic segmentation model includes multiple different segmentation models. Compared with a single semantic segmentation model, the integrated semantic segmentation model can improve the accuracy of determining the semantic segmentation results, thereby improving the recognition accuracy of digestive tract cancer; for any pixel of the digestive tract image, the integrated semantic segmentation model is used to determine multiple category prediction results of the pixel, and any category prediction result includes a lesion pixel or a non-lesion pixel. If any category prediction result among the multiple category prediction results is a lesion pixel, the final category prediction result of the pixel is determined as a lesion pixel. If all category prediction results among the multiple category prediction results are non-lesion pixels, the final category prediction result of the pixel is determined as a non-lesion pixel, thereby improving the sensitivity of digestive tract cancer recognition and reducing missed detections.

[0168] Based on any of the above embodiments, before the above step 110, the method further includes: inputting the digestive tract image into a feature extraction model to obtain image features output by the feature extraction model.

[0169] Correspondingly, the above step 110 includes: inputting the image features into an image classification model to obtain an image classification result output by the image classification model.

[0170] Correspondingly, the above step 120 includes: inputting the image features into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0171] In other words, the semantic segmentation model and the image classification model share a feature extraction model. For example, the semantic segmentation model and the image classification model share a backbone network (trunk grid), and then two detection heads are connected on the basis of the backbone network to realize semantic segmentation and image classification respectively.

[0172] Furthermore, the feature extraction model is used to extract high-level abstract features of gastrointestinal images. High-level abstract features have stronger semantic expression capabilities and stronger robustness than low-level features, and can be applied to various complex gastrointestinal image acquisition scenarios, thus having higher accuracy and better generalization capabilities, especially for image classification models, and ultimately improving the accuracy of gastrointestinal cancer identification.

[0173] Furthermore, the feature extraction model can also be shared by models such as a staining recognition model, a digestive tract site recognition model, and a digestive tract mucosal cleanliness recognition model.

[0174] In the method for detecting digestive tract cancer provided by an embodiment of the present invention, the semantic segmentation model and the image classification model share a feature extraction model, thereby effectively reducing the amount of computational effort in model reasoning, thereby improving the detection efficiency of digestive tract cancer and reducing the storage space occupied by the model, so as to be suitable for devices with limited hardware resources.

[0175] Based on any of the above embodiments, the image classification result includes a target classification result or several non-target classification results, and the cancer probability values ​​represented by the several non-target classification results are all smaller than the cancer probability value represented by the target classification result.

[0176] Here, the number of the several non-target classification results can be one or more. For example, when the digestive tract image is a stained image, the target classification result is the first classification result, the several non-target classification results include the second classification result, and the cancer probability value represented by the first classification result is greater than the cancer probability value represented by the second classification result; when the digestive tract image is a non-stained image, the target classification result is the third classification result, the several non-target classification results include the fourth classification result and the fifth classification result, the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

[0177] Accordingly, considering that the image classification result is the target classification result, that is, in the case of a high cancer probability, since the semantic segmentation model does not determine the lesion area, the doctor needs to further locate the lesion area, resulting in a decrease in the detection efficiency of digestive tract cancer; based on this, before the above step 130, the method further includes:

[0178] If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the target classification result, the segmentation threshold of the semantic segmentation model is adjusted to obtain and output the lesion area in the digestive tract image.

[0179] Among them, for any pixel of the digestive tract image, the semantic segmentation model determines the category prediction result of the pixel based on the segmentation threshold, and the category prediction result of the pixel includes a lesion pixel or a non-lesion pixel.

[0180] For example, when the image classification model determines that the image is suspected of cancer but the semantic segmentation model does not output the lesion area, the threshold of the semantic segmentation model is dynamically adjusted. By dynamically adjusting the segmentation threshold of the semantic segmentation model, the lesion area is output for the doctor to locate, observe and analyze the lesion area.

[0181] The method for detecting gastrointestinal cancer provided by an embodiment of the present invention, through the above-mentioned method, when the image classification result is the target classification result, that is, when the probability of cancer is high, even if the semantic segmentation model has not determined the lesion area, the segmentation threshold of the semantic segmentation model can be adjusted to obtain and output the lesion area in the gastrointestinal image, to assist the doctor in further accurately locating the lesion area, thereby improving the detection efficiency of gastrointestinal cancer, and improving the sensitivity of gastrointestinal cancer identification, reducing missed detections.

[0182] Based on any of the above embodiments, in the method, adjusting the segmentation threshold of the semantic segmentation model includes:

[0183] The lowest segmentation threshold is determined as 0, and the highest segmentation threshold is determined as the current segmentation threshold of the semantic segmentation model;

[0184] Determine a sum of the minimum segmentation threshold and the maximum segmentation threshold, and adjust the segmentation threshold of the semantic segmentation model to half of the sum;

[0185] Based on the adjusted segmentation threshold, updating the semantic segmentation result;

[0186] If it is determined based on the updated semantic segmentation result that the proportion of the lesion area of ​​the digestive tract image is greater than the first preset proportion, the minimum segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the minimum segmentation threshold and the maximum segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum;

[0187] If it is determined based on the updated semantic segmentation result that the proportion of the lesion area in the digestive tract image is less than the second preset proportion, the highest segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the lowest segmentation threshold and the highest segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum.

[0188] Among them, for any pixel of the digestive tract image, the semantic segmentation model outputs the predicted value of the pixel, and if the predicted value of the pixel is greater than or equal to the segmentation threshold, the category prediction result of the pixel is determined to be a lesion pixel; the lesion area ratio is determined based on the ratio of the lesion area in the digestive tract image to the area of ​​the digestive tract image, and the first preset ratio is greater than the second preset ratio.

[0189] Here, the highest segmentation threshold is greater than the lowest segmentation threshold. Both the highest segmentation threshold and the lowest segmentation threshold can be dynamically updated.

[0190] Considering that the current segmentation threshold based on the semantic segmentation model does not output the lesion area, and if the segmentation threshold is 0, the lesion area output by the semantic segmentation model is the entire digestive tract image, based on this, the initial minimum segmentation threshold is set to 0, and the initial maximum segmentation threshold is the current segmentation threshold of the semantic segmentation model.

[0191] Here, the first preset proportion and the second preset proportion can be set according to actual conditions. For example, according to data statistical analysis, the lesion area of ​​a positive digestive tract image generally accounts for 5%-80% of the entire positive digestive tract image. Based on this, the first preset proportion is 80% and the second preset proportion is 5%.

[0192] Exemplarily, assuming that the current segmentation threshold of the semantic segmentation model is threshold_init, the first preset proportion is 80%, and the second preset proportion is 5%. Based on this, the segmentation threshold of the semantic segmentation model is adjusted in the open interval (0, threshold_init) until the semantic segmentation model outputs the lesion area. Furthermore, the use of binary search can improve the accuracy and rationality of determining the lesion area; specifically, in the first step, set the lowest segmentation threshold threshold_low=0, the highest segmentation threshold threshold_high=threshold_init, in the second step, set the segmentation threshold threshold of the semantic segmentation model threshold=(threshold_low + threshold_high) / 2, in the third step, based on the adjusted segmentation threshold, update the semantic segmentation result, that is, obtain the lesion area according to the new segmentation threshold, and determine the area ratio P of the lesion area to the entire digestive tract image, in the fourth step, if 0.05≤P≤0.80, determine the current threshold as a reasonable segmentation threshold and output the lesion area corresponding to this segmentation threshold, if P>0.80, then threshold_low=threshold, and return to the second step until the final lesion area satisfies 0.05≤P≤0.80, if P<0.05, then threshold_high=threshold, and return to the second step until the final lesion area satisfies 0.05≤P≤0.80.

[0193] The method for detecting digestive tract cancer provided by an embodiment of the present invention determines the sum of the minimum segmentation threshold and the maximum segmentation threshold in the above manner, and adjusts the segmentation threshold of the semantic segmentation model to half of the sum, so as to accurately and dynamically adjust the segmentation threshold, thereby improving the positioning sensitivity of the lesion area, thereby improving the detection rate of digestive tract cancer; and ensuring that the proportion of the lesion area is between the first preset proportion and the second preset proportion, thereby further improving the positioning accuracy of the lesion area, thereby further improving the detection accuracy of digestive tract cancer.

[0194] Based on any of the above embodiments, the method further includes:

[0195] Inputting the digestive tract image into a digestive tract part recognition model to obtain a digestive tract part recognition result output by the digestive tract part recognition model;

[0196] Output the prompt information corresponding to the digestive tract part identification result to prompt the digestive tract part currently being examined.

[0197] The digestive tract part recognition model is obtained by training based on sample digestive tract images and digestive tract part recognition result labels corresponding to the sample digestive tract images.

[0198] For example, if the digestive tract part identification result is the upper anterior wall of the gastric body, the corresponding prompt information may be "the digestive tract part currently being examined is the upper anterior wall of the gastric body".

[0199] Furthermore, the digestive tract part recognition model can share a feature extraction model with the image classification model and the semantic segmentation model, that is, the image features of the digestive tract image are input into the digestive tract part recognition model to obtain the digestive tract part recognition result output by the digestive tract part recognition model.

[0200] The method for detecting digestive tract cancer provided by the embodiment of the present invention inputs a digestive tract image into a digestive tract part recognition model to obtain a digestive tract part recognition result output by the digestive tract part recognition model; outputs prompt information corresponding to the digestive tract part recognition result to prompt the digestive tract part currently being examined, thereby eliminating the need for the doctor to further judge the digestive tract part currently to be examined, thereby improving the detection efficiency of digestive tract cancer.

[0201] Based on any of the above embodiments, after inputting the digestive tract image into the digestive tract part recognition model to obtain the digestive tract part recognition result output by the digestive tract part recognition model, the method further includes:

[0202] When the digestive tract part indicated by the digestive tract part identification result has been inspected, determining the digestive tract parts that have not been inspected from the target digestive tract parts set;

[0203] Output prompt information corresponding to the digestive tract part that has not been examined.

[0204] In an embodiment of the present invention, the doctor can determine whether the digestive tract part indicated by the digestive tract part identification result has been examined, or can determine that the examination has been completed when the digestive tract endoscope no longer takes pictures of the digestive tract part. Of course, other methods can also be used to determine whether the digestive tract part has been examined, which are not specifically limited here.

[0205] Here, the target digestive tract site set includes a plurality of sites to be examined in the digestive tract.

[0206] In one embodiment, the target digestive tract part set can be set in advance, that is, the digestive tract parts to be inspected are set in advance. In another embodiment, the target digestive tract part set includes all digestive tract parts. In another embodiment, the target digestive tract part set can be updated in real time, such as deleting a digestive tract part after inspecting it.

[0207] Here, the prompt information corresponding to the digestive tract parts that have not been inspected is output to prompt the digestive tract parts that need to be inspected. The number of the digestive tract parts that have not been inspected can be one or more. If there are more than one, all the digestive tract parts that have not been inspected can be prompted, thereby realizing the missed inspection part prompt.

[0208] The method for detecting digestive tract cancer provided by the embodiment of the present invention inputs a digestive tract image into a digestive tract part recognition model to obtain a digestive tract part recognition result output by the digestive tract part recognition model; when the digestive tract part indicated by the digestive tract part recognition result has been examined, the digestive tract parts that have not been examined are collectively determined from the target digestive tract parts; prompt information corresponding to the digestive tract parts that have not been examined is output to prompt the digestive tract parts that have not been examined currently, thereby eliminating the need for doctors to further judge the digestive tract parts that have not been examined, thereby improving the detection efficiency of digestive tract cancer, avoiding missed detection of some digestive tract parts, and improving the comprehensiveness of digestive tract cancer detection.

[0209] Based on any of the above embodiments, after inputting the digestive tract image into the digestive tract part recognition model to obtain the digestive tract part recognition result output by the digestive tract part recognition model, the method further includes:

[0210] Determining the mirror-out durations of the plurality of digestive tract parts based on historical digestive tract part recognition results of the plurality of digestive tract images that have been collected and the collection times of the plurality of digestive tract images that have been collected;

[0211] Based on the withdrawal times of the multiple digestive tract locations, a total withdrawal time of the inspection scope for acquiring digestive tract images is determined.

[0212] It should be noted that, during the gastrointestinal detection process, each gastrointestinal part is detected in turn, that is, image acquisition is performed on each gastrointestinal part in turn. Based on this, based on the historical gastrointestinal part recognition results of multiple acquired gastrointestinal images and the acquisition time of multiple acquired gastrointestinal images, the time of entering a gastrointestinal part and the time of leaving the gastrointestinal part can be determined, thereby determining the withdrawal time of the gastrointestinal part based on the two times.

[0213] Here, the total withdrawal time is greater than or equal to the sum of the withdrawal times of multiple digestive tract locations.

[0214] In one embodiment, if the scope is a gastroscope, the total scope withdrawal time refers to the time it takes for the scope to be withdrawn from the descending part of the duodenum to the outside of the body. If the scope is a colonoscope, the total scope withdrawal time refers to the time it takes for the scope to be withdrawn from the ileocecal part to the outside of the body, and the ileocecal part refers to the terminal ileum, the appendix orifice, and the ileocecal valve area.

[0215] Exemplarily, the withdrawal time of multiple digestive tract sites includes the withdrawal time of the ascending colon segment, the withdrawal time of the transverse colon segment, the withdrawal time of the descending colon segment, and the withdrawal time of the sigmoid colon segment. The total withdrawal time is the sum of the withdrawal time of the ascending colon segment, the withdrawal time of the transverse colon segment, the withdrawal time of the descending colon segment, and the withdrawal time of the sigmoid colon segment.

[0216] Furthermore, the time taken to withdraw the scope at any part of the digestive tract must be deducted from the time taken for biopsy, staining or surgery. Based on this, it is necessary to determine the time taken for biopsy, staining or surgery at that part of the digestive tract.

[0217] After the total scope withdrawal time and the scope withdrawal time of multiple digestive tract sites are obtained, an inspection quality control report can be generated based on at least one of the two for subsequent reading by doctors or patients.

[0218] The method for detecting gastrointestinal cancer provided by an embodiment of the present invention determines the scope withdrawal time of multiple gastrointestinal parts based on historical gastrointestinal part recognition results of multiple gastrointestinal images that have been collected and the collection time of the multiple gastrointestinal images that have been collected; based on the scope withdrawal time of multiple gastrointestinal parts, the total scope withdrawal time of the inspection scope is automatically monitored to improve the detection quality of gastrointestinal cancer.

[0219] Based on any of the above embodiments, before the above step 110, the method further includes:

[0220] Inputting the digestive tract image into a digestive tract mucosa cleanliness recognition model to obtain a digestive tract mucosa cleanliness recognition result output by the digestive tract mucosa cleanliness recognition model; the digestive tract mucosa cleanliness recognition result is used to indicate whether the digestive tract image is a cleanliness qualified image;

[0221] If the digestive tract image is determined to be an image with unqualified cleanliness based on the digestive tract mucosal cleanliness recognition result, flushing prompt information is output to prompt flushing of the digestive tract part corresponding to the digestive tract image.

[0222] The digestive tract mucosa cleanliness recognition model is obtained by training based on sample digestive tract images and digestive tract mucosa cleanliness recognition result labels corresponding to the sample digestive tract images.

[0223] Here, the mucosal cleanliness of the digestive tract corresponding to the qualified cleanliness image is qualified, that is, the digestive tract image meets the input requirements of the semantic segmentation model and the image classification model.

[0224] It can be understood that after the digestive tract portion corresponding to the digestive tract image is flushed, the digestive tract image of the flushed digestive tract portion is recaptured, so as to re-execute the digestive tract cancer detection method based on the digestive tract image.

[0225] Furthermore, the digestive tract mucosal cleanliness recognition model can share a feature extraction model with the image classification model and the semantic segmentation model, that is, the image features of the digestive tract image are input into the digestive tract mucosal cleanliness recognition model to obtain the digestive tract mucosal cleanliness recognition result output by the digestive tract mucosal cleanliness recognition model.

[0226] The method for detecting gastrointestinal cancer provided by the embodiment of the present invention inputs a gastrointestinal image into a gastrointestinal mucosal cleanliness recognition model to obtain a gastrointestinal mucosal cleanliness recognition result output by the gastrointestinal mucosal cleanliness recognition model, and the gastrointestinal mucosal cleanliness recognition result is used to indicate whether the gastrointestinal image is an image with qualified cleanliness. If the gastrointestinal image is determined to be an image with unqualified cleanliness based on the gastrointestinal mucosal cleanliness recognition result, flushing prompt information is output to prompt flushing of the gastrointestinal part corresponding to the gastrointestinal image, thereby ensuring that the gastrointestinal image with qualified cleanliness is input into the image classification model and the semantic segmentation model, ensuring that the gastrointestinal image with qualified cleanliness is used to detect gastrointestinal cancer, thereby improving the detection accuracy of gastrointestinal cancer; assisting doctors in determining whether flushing is needed, and improving the detection efficiency and quality of gastrointestinal cancer.

[0227] Based on any of the above embodiments, the method further includes:

[0228] Inputting the image collected by the inspection mirror into the in-vivo and out-vivo recognition model to obtain an in-vivo and out-vivo recognition result output by the in-vivo and out-vivo recognition model, wherein the in-vivo and out-vivo recognition result is used to indicate whether the input image is an in-vivo image or an out-vivo image;

[0229] Based on the in vivo and in vitro recognition results of multiple images, the total examination time of the digestive tract was determined.

[0230] Among them, the in vivo and in vitro recognition model is trained based on sample images and their corresponding in vivo and in vitro recognition result labels.

[0231] Here, the total inspection time is the time from the moment the inspection scope enters the body from outside the body to the moment the inspection scope leaves the body from inside the body.

[0232] In one embodiment, if the inspection scope is a gastroscope, the total inspection time refers to the time from the time when the inspection scope enters the oral cavity from the outside to the time when the inspection scope leaves the body from the inside to the outside. If the inspection scope is a colonoscope, the total inspection time refers to the time from the time when the inspection scope enters the anus from the outside to the time when the inspection scope leaves the body from the inside to the outside.

[0233] Furthermore, the total examination time must be deducted from the time spent on biopsy, staining or surgery in each part of the digestive tract. Based on this, the biopsy time, staining time or surgery time in each part of the digestive tract needs to be determined.

[0234] After the total inspection time is obtained, an inspection quality control report can be generated based on at least one of the two for subsequent reading by doctors or patients.

[0235] The method for detecting digestive tract cancer provided by the embodiment of the present invention automatically monitors the total inspection time and improves the detection quality of digestive tract cancer through the above-mentioned method.

[0236] Based on any of the above embodiments, considering that the colonoscope also needs to monitor the reaching of the ileocecal part, when the inspection scope is a colonoscope, after inputting the digestive tract image into the digestive tract part recognition model and obtaining the digestive tract part recognition result output by the digestive tract part recognition model, the method further includes:

[0237] Based on the results of digestive tract site identification, determine whether the ileocecal region has been reached.

[0238] It should be understood that if the digestive tract site identification result indicates that the terminal ileum, the appendix or the ileocecal valve has been reached, it means that the ileocecal region has been reached.

[0239] The method for detecting digestive tract cancer provided by the embodiment of the present invention automatically monitors whether the ileocecal part has been reached through the above-mentioned method, thereby improving the detection quality of digestive tract cancer.

[0240] It should be noted that the above embodiments are based on the ideal AI-assisted digestive tract tumor examination and diagnosis technology: (1) follow the guideline process, perform biopsy or other prompts on suspected tumor lesions under ordinary white light endoscopy, stain low-suspected tumor lesions, wait for the physician to perform chemical staining or electronic staining, and then use AI to identify and further discover suspected tumor lesions. This can ensure high specificity and improve the sensitivity of discovering tumor lesions, especially early cancer lesions; (2) with the help of intelligent targeted biopsy that locks the boundary of the lesion area, the positive rate of biopsy of suspected lesions can be improved from the biopsy level.

[0241] Based on the above embodiments, an inspection quality control report (such as an upper gastrointestinal endoscopic tumor screening quality control report) can be generated for subsequent reading by doctors or patients. Exemplarily, in the case of outputting a biopsy prompt, the inspection quality control report includes a digestive tract image of the digestive tract part corresponding to the biopsy prompt and a record of whether the doctor performed a biopsy; in the case of outputting a staining prompt, the inspection quality control report includes a digestive tract image of the digestive tract part corresponding to the staining prompt and a record of whether the doctor performed staining; in the case of outputting a flushing prompt, the inspection quality control report includes a digestive tract image of the digestive tract part corresponding to the flushing prompt and a record of whether the doctor flushed; in the case of outputting a missed inspection prompt, the inspection quality control report includes a record of whether the uninspected digestive tract part was re-inspected; in addition, the inspection quality control report may also include the total inspection time and the mirror withdrawal time.

[0242] Based on the above embodiments, the principle of hyperparameter selection in the training stage of the above models is: to maximize the sensitivity while ensuring that the specificity is not less than 90%. For ease of understanding, assume that the number of positive samples in the sample test set is P, and the number of negative samples is F; the number of true positive samples STP and the number of false positive samples SFP determined by the semantic segmentation results of the semantic segmentation model; the number of true positive samples CTP and the number of false positive samples CFP determined by the image classification results of the image classification model; based on this, the sensitivity is (STP+CTP) / P, and the specificity is (F-SFP-CFP) / F.

[0243] Based on the above embodiments, the present invention can significantly improve the sensitivity of digestive tract cancer identification while ensuring a certain biopsy positivity rate, and at the same time achieve targeted positioning of suspected lesion areas. In addition, according to the different output probabilities of the image classification model, biopsy prompts, staining prompts and no prompts can be used for auxiliary diagnosis suggestions, thereby effectively improving the effect of digestive tract cancer identification and diagnosis.

[0244] The digestive tract cancer detection device provided by the present invention is described below. The digestive tract cancer detection device described below and the digestive tract cancer detection method described above can be referenced to each other.

[0245] Figure 8 Schematic diagram of the structure of the digestive tract cancer detection device provided by the present invention, such as Figure 8 As shown, the detection device for digestive tract cancer includes an image classification module 810, a semantic segmentation module 820, a method determination module 830 and a prompt output module 840.

[0246] The image classification module 810 is used to input the digestive tract image into the image classification model to obtain the image classification result output by the image classification model; the image classification result is used to represent the recognition result of cancer recognition for the digestive tract image.

[0247] The semantic segmentation module 820 is used to input the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image.

[0248] The method determination module 830 is used to determine the detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result.

[0249] The prompt output module 840 is used to output prompt information corresponding to the detection method when the detection method indicates that the digestive tract part corresponding to the digestive tract image needs to be detected.

[0250] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a method for detecting digestive tract cancer, the method comprising: inputting a digestive tract image into an image classification model to obtain an image classification result output by the image classification model; the image classification result is used to characterize a recognition result of cancer recognition for the digestive tract image; inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image; based on the semantic segmentation result and the image classification result, determining a detection method for a digestive tract portion corresponding to the digestive tract image; and outputting prompt information corresponding to the detection method when the detection method indicates that the digestive tract portion corresponding to the digestive tract image needs to be detected.

[0251] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0252] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the gastrointestinal cancer detection method provided by the above-mentioned methods, and the method includes: inputting a gastrointestinal image into an image classification model to obtain an image classification result output by the image classification model; the image classification result is used to characterize the recognition result of cancer identification for the gastrointestinal image; inputting the gastrointestinal image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the gastrointestinal image; based on the semantic segmentation result and the image classification result, determining the detection method of the gastrointestinal part corresponding to the gastrointestinal image; when the detection method indicates that the gastrointestinal part corresponding to the gastrointestinal image needs to be detected, outputting prompt information corresponding to the detection method.

[0253] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the gastrointestinal cancer detection method provided by the above-mentioned methods, the method comprising: inputting a gastrointestinal image into an image classification model to obtain an image classification result output by the image classification model; the image classification result is used to characterize the recognition result of cancer recognition for the gastrointestinal image; inputting the gastrointestinal image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the gastrointestinal image; based on the semantic segmentation result and the image classification result, determining a detection method for the gastrointestinal part corresponding to the gastrointestinal image; and when the detection method indicates that the gastrointestinal part corresponding to the gastrointestinal image needs to be detected, outputting prompt information corresponding to the detection method.

[0254] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0255] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting digestive tract cancer, characterized in that: include: Inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model; The image classification result is used to represent the recognition result of cancer recognition for the digestive tract image; Inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used to indicate whether there is a lesion area in the digestive tract image; Determining a detection method for a digestive tract portion corresponding to the digestive tract image based on the semantic segmentation result and the image classification result; When the detection method indicates that the digestive tract portion corresponding to the digestive tract image needs to be detected, outputting prompt information corresponding to the detection method; If the digestive tract image is a stained image, the image classification result includes a first classification result or a second classification result, and the method of determining the detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result includes: If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, determining that the detection method is a biopsy; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, determining that the detection method is no need for detection; If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required; The digestive tract portion corresponding to the stained image has been stained; the cancer probability value represented by the first classification result is greater than the cancer probability value represented by the second classification result; If the digestive tract image is an unstained image, the image classification result includes a third classification result, a fourth classification result, or a fifth classification result, and the method of determining the detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result includes: If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, determining that the detection method is a biopsy; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, determining that the detection method is staining; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection; If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required; Among them, the digestive tract part corresponding to the unstained image is not stained; the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

2. The method for detecting digestive tract cancer according to claim 1, characterized in that: If the digestive tract image is a stained image, inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model includes: Inputting the digestive tract image into a first image classification model corresponding to the stained image to obtain an image classification result output by the first image classification model; If the digestive tract image is an unstained image, inputting the digestive tract image into an image classification model to obtain an image classification result output by the image classification model includes: Inputting the digestive tract image into a second image classification model corresponding to the unstained image to obtain an image classification result output by the second image classification model; If the digestive tract image is a stained image, inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes: Inputting the digestive tract image into a first semantic segmentation model corresponding to the stained image to obtain a semantic segmentation result output by the first semantic segmentation model; If the digestive tract image is an unstained image, inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes: Inputting the digestive tract image into a second semantic segmentation model corresponding to the unstained image to obtain a semantic segmentation result output by the second semantic segmentation model; The digestive tract portion corresponding to the stained image has been stained, and the digestive tract portion corresponding to the unstained image has not been stained.

3. The method for detecting digestive tract cancer according to claim 1, characterized in that: The semantic segmentation model is an integrated semantic segmentation model, and the integrated semantic segmentation model includes a plurality of different segmentation models; For any pixel of the digestive tract image, the integrated semantic segmentation model is used to determine multiple category prediction results of the pixel, any of the category prediction results including lesion pixels or non-lesion pixels. If any category prediction result among the multiple category prediction results is a lesion pixel, the final category prediction result of the pixel is determined as a lesion pixel. If all category prediction results among the multiple category prediction results are non-lesion pixels, the final category prediction result of the pixel is determined as a non-lesion pixel.

4. The method for detecting digestive tract cancer according to claim 1, characterized in that: Before inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model, the method further includes: Inputting the digestive tract image into a feature extraction model to obtain image features output by the feature extraction model; Accordingly, the step of inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model includes: Inputting the image features into an image classification model to obtain an image classification result output by the image classification model; Accordingly, the step of inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model includes: The image features are input into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

5. The method for detecting digestive tract cancer according to any one of claims 1 to 4, characterized in that: The image classification result includes a target classification result or several non-target classification results, and the cancer probability values ​​represented by the several non-target classification results are all smaller than the cancer probability value represented by the target classification result; Before determining a detection method of a digestive tract portion corresponding to the digestive tract image based on the semantic segmentation result and the image classification result, the method further includes: If it is determined based on the semantic segmentation result that there is no lesion area in the digestive tract image, and the image classification result is the target classification result, adjusting the segmentation threshold of the semantic segmentation model to obtain and output the lesion area in the digestive tract image; Among them, for any pixel of the digestive tract image, the semantic segmentation model determines the category prediction result of the pixel based on the segmentation threshold, and the category prediction result of the pixel includes a lesion pixel or a non-lesion pixel.

6. The method for detecting digestive tract cancer according to claim 5, characterized in that: The adjusting the segmentation threshold of the semantic segmentation model includes: The lowest segmentation threshold is determined as 0, and the highest segmentation threshold is determined as the current segmentation threshold of the semantic segmentation model; Determine a sum of the minimum segmentation threshold and the maximum segmentation threshold, and adjust the segmentation threshold of the semantic segmentation model to half of the sum; Based on the adjusted segmentation threshold, updating the semantic segmentation result; If it is determined based on the updated semantic segmentation result that the proportion of the lesion area of ​​the digestive tract image is greater than the first preset proportion, the minimum segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the minimum segmentation threshold and the maximum segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum; If it is determined based on the updated semantic segmentation result that the proportion of the lesion area of ​​the digestive tract image is less than the second preset proportion, the highest segmentation threshold is updated to the adjusted segmentation threshold, and the step of determining the sum of the lowest segmentation threshold and the highest segmentation threshold is returned to adjust the segmentation threshold of the semantic segmentation model to half of the sum; Among them, for any pixel of the digestive tract image, the semantic segmentation model outputs the predicted value of the pixel, and if the predicted value of the pixel is greater than or equal to the segmentation threshold, the category prediction result of the pixel is determined to be a lesion pixel; the lesion area ratio is determined based on the ratio of the lesion area in the digestive tract image to the area of ​​the digestive tract image, and the first preset ratio is greater than the second preset ratio.

7. The method for detecting digestive tract cancer according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting the digestive tract image into a digestive tract part recognition model to obtain a digestive tract part recognition result output by the digestive tract part recognition model; Outputting prompt information corresponding to the digestive tract part identification result to indicate the digestive tract part currently being examined; The digestive tract part recognition model is obtained by training based on sample digestive tract images and digestive tract part recognition result labels corresponding to the sample digestive tract images.

8. The method for detecting digestive tract cancer according to claim 7, characterized in that: After inputting the digestive tract image into the digestive tract part recognition model to obtain the digestive tract part recognition result output by the digestive tract part recognition model, the method further includes: When the digestive tract part indicated by the digestive tract part identification result has been inspected, determining the digestive tract parts that have not been inspected from the target digestive tract parts set; Output prompt information corresponding to the digestive tract part that has not been inspected.

9. The method for detecting digestive tract cancer according to claim 7, characterized in that: After inputting the digestive tract image into the digestive tract part recognition model to obtain the digestive tract part recognition result output by the digestive tract part recognition model, the method further includes: Determining the mirror-out durations of the plurality of digestive tract parts based on historical digestive tract part recognition results of the plurality of digestive tract images that have been collected and the collection times of the plurality of digestive tract images that have been collected; Based on the withdrawal times of the multiple digestive tract locations, a total withdrawal time of the inspection scope for acquiring digestive tract images is determined.

10. The method for detecting digestive tract cancer according to any one of claims 1 to 4, characterized in that: Before inputting the digestive tract image into the image classification model to obtain the image classification result output by the image classification model, the method further includes: Inputting the digestive tract image into a digestive tract mucosa cleanliness recognition model to obtain a digestive tract mucosa cleanliness recognition result output by the digestive tract mucosa cleanliness recognition model; the digestive tract mucosa cleanliness recognition result is used to indicate whether the digestive tract image is a cleanliness qualified image; If it is determined based on the digestive tract mucosal cleanliness recognition result that the digestive tract image is an image with unqualified cleanliness, outputting flushing prompt information to prompt flushing of the digestive tract portion corresponding to the digestive tract image; The digestive tract mucosa cleanliness recognition model is obtained by training based on sample digestive tract images and digestive tract mucosa cleanliness recognition result labels corresponding to the sample digestive tract images.

11. A digestive tract cancer detection device, characterized in that: include: An image classification module, used to input the digestive tract image into the image classification model to obtain the image classification result output by the image classification model; The image classification result is used to represent the recognition result of cancer recognition for the digestive tract image; A semantic segmentation module, used for inputting the digestive tract image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; the semantic segmentation result is used for indicating whether there is a lesion area in the digestive tract image; A method determination module, used to determine a detection method of the digestive tract part corresponding to the digestive tract image based on the semantic segmentation result and the image classification result; a prompt output module, configured to output prompt information corresponding to the detection method when the detection method indicates that the digestive tract portion corresponding to the digestive tract image needs to be detected; If the digestive tract image is a stained image, the image classification result includes a first classification result or a second classification result, and the mode determination module is used to: If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the first classification result, determining that the detection method is a biopsy; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the second classification result, determining that the detection method is no need for detection; If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required; The digestive tract portion corresponding to the stained image has been stained; the cancer probability value represented by the first classification result is greater than the cancer probability value represented by the second classification result; If the digestive tract image is an unstained image, the image classification result includes the third classification result, the fourth classification result or the fifth classification result, and the mode determination module is used to: If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the third classification result, determining that the detection method is a biopsy; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fourth classification result, determining that the detection method is staining; If it is determined based on the semantic segmentation result that there is a lesion area in the digestive tract image, and the image classification result is the fifth classification result, determining that the detection method is no need for detection; If it is determined based on the semantic segmentation result that no lesion area exists in the digestive tract image, determining that the detection method is that no detection is required; Among them, the digestive tract part corresponding to the unstained image is not stained; the cancer probability value represented by the third classification result is greater than the cancer probability value represented by the fourth classification result, and the cancer probability value represented by the fourth classification result is greater than the cancer probability value represented by the fifth classification result.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for detecting digestive tract cancer according to any one of claims 1 to 10 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting digestive tract cancer according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting digestive tract cancer according to any one of claims 1 to 10 is implemented.

Images