Tumor prediction system, method, and application based on tongue imagery.

Abstract

The present invention relates to the technical field of tumor diagnosis, prediction, and evaluation. Specifically, it relates to a tumor prediction system, method, and its application based on tongue image. The system includes a tongue image acquisition module configured to acquire the tongue image of a test sample, and a data processing module configured to obtain the probability that the test sample belongs to the positive category by the following operations. The data processing module predicts the probability that the test sample belongs to the positive category based on the discriminative features on the tongue image obtained by automatic learning. The discriminative features are derived from a single positive tongue image or negative tongue image. The system further includes an output module. It is intended to apply AI deep learning to perform diagnostic prediction for tumors based on tongue images. The tumor prediction system is simple to operate, low in cost, painless, non-invasive, and through a large number of test cases, it has demonstrated that the prediction system is a forward-looking, economic, non-invasive, and effective screening system for tumors.

Description

[Technical Field]

[0001] This invention relates to the technology of oncology diagnosis, prediction, and evaluation, and more specifically to a tumor prediction system, method, and application based on tongue imagery, thereby achieving economical, non-invasive, and highly accurate tumor prediction by analyzing the relationship between tongue imagery and oncology. [Background technology]

[0002] According to the latest data, gastric cancer (GC) is the third leading cause of cancer-related death worldwide, with 1.09 million new cases and 770,000 deaths in 2020 alone. Of these, 480,000 new cases and 370,000 deaths occurred in China, accounting for approximately half of the global total. Diagnosis and screening of GC still rely on gastroscopy, but its invasiveness, high cost, and the need for specialized endoscopists severely limit its application. Furthermore, the lack of specific symptoms in the early stages of gastric cancer results in relatively poor specificity and sensitivity of clinical disease markers, with over 60% of patients having local or distant metastases by the time of definitive diagnosis. The 5-year survival rate for patients with local early-stage GC exceeds 60%, while it drops significantly to 30% and 5% for patients with local or distant metastases, respectively. Therefore, new GC diagnostic or screening methods are urgently needed to improve the rate of early diagnosis and prognosis for this group.

[0003] Traditional Chinese medicine is a medical science and cultural heritage that has been applied and preserved by the Chinese people for thousands of years, and tongue diagnosis is one of the important bases for diagnosing diseases in traditional Chinese medicine. According to the theory of traditional Chinese medicine, changes in tongue appearance (tongue color, size and shape, tongue coating color, thickness and water content) can reflect the health status of the human body and are closely related in particular to stomach diseases. However, there are no studies that have demonstrated a correspondence between tongue appearance changes and gastrointestinal gland deficiency (GC), or that have demonstrated the value of tongue appearance changes in the diagnosis and screening of GC.

[0004] Artificial intelligence (AI) can be used for the screening, diagnosis, and treatment of various diseases. Scholars such as Cheung CY et al. developed a deep learning system (see references), measured the caliber of retinal blood vessels to evaluate the risk of cardiovascular diseases, and could effectively predict the risk of cardiovascular diseases. Scholars such as Takenaka K et al. developed a deep neural network (see references) for evaluating endoscopic images of patients with ulcerative colitis. The network could recognize patients with endoscopic remission and histological remission with an accuracy of 90.1%, and the positive rate was 92.9%.

[0005] Patent CN110251084A of Fuzhou Data Technology Research Institute Co., Ltd. solves the problems of real-time detection, shooting, storage, and uploading of the tongue body in the tongue image collection process, and provides an artificial intelligence-based tongue image detection and recognition method for recognizing tongue image tongue color, tongue shape, coating quality, and coating color. Its solution mainly relates to the collection and recognition technology of tongue images. Among them, tongue image recognition focuses on extracting characteristics such as the color, texture, coating area, or thickness of the coating of the tongue image. However, these operations do not establish a correspondence between tongue image information and certain special stomach diseases, such as gastric cancer.

[0006] Patent CN111710394A of Shenyang Zhilang Technology Co., Ltd. proposed an artificial intelligence-assisted early gastric cancer screening system. Instead of manual work, it analyzes gastric camera rice images by automation to solve the problem of a large amount of workload for determining gastric cancer positivity. However, such a policy based on gastric camera image analysis first needs to obtain gastric camera images collected by a large number of specialized devices for model learning, and still needs to make decisions based on the gastric camera images of each tester during the test stage. There are still defects such as high time consumption, high material cost, and high tester standards for obtaining gastric camera images, and it is difficult to achieve a comprehensive screening across the country.

[0007] Jiangsu Tianrui Precision Medical Technology Co., Ltd. CN112133427A provides an artificial intelligence-based gastric cancer auxiliary diagnosis system including a diagnosis selection module, a data collection module, a preprocessing module, a diagnosis module and a display output module, and the system can give personalized diagnosis results based on the collected data of the examinee. The data relied on by the diagnosis of the diagnosis system includes the basic information, living diet, infection history, disease history, family history, clinical symptoms and examination items of the examinee, etc. Among them, the data such as clinical symptoms and examination items are relatively difficult to collect, and the information such as basic information, living diet, infection history, disease history, family history alone does not affect the early screening diagnosis effect.

[0008] References: Cheung CY, Xu D, Cheng CY, et al. A deep-learning system for the assessment of cardiovascular disease risk via the measurement of retinal-vessel calibre. Nature biomedical engineering 2021;5(6):498-508. doi:10.1038 / s41551-020-00626-4 [published Online First:2020 / 10 / 14]; Takenaka K, Ohtsuka K, Fujii T, et al. Development and Validation of a Deep Neural Network for Accurate Evaluation of Endoscopic Images From Patients With Ulcerative Colitis. Gastroenterology 2020;158(8):2150-57. doi:10.1053 / j.gastro.2020.02.012[published Online First:2020 / 02 / 16].

[0009] The present invention aims to solve these and other needs to be solved in the art.

Summary of the Invention

[0010] To solve at least one of the technical problems mentioned in the background technology described above, the object of the present invention is intended to provide a tumor prediction system based on tongue imagery, the system is intended to apply AI deep learning to make diagnostic predictions for tumors based on tongue imagery, the tumor prediction system is easy to operate, low-cost, painless, and non-invasive, and has demonstrated through a large number of test cases that the prediction system is a prospective, economical, non-invasive, and effective screening system for tumors. [Means for solving the problem]

[0011] Currently, the application of artificial intelligence to tongue imaging in traditional Chinese medicine primarily focuses on standardizing the extraction of tongue features, eliminating differences caused by human interpretation. This project is the first to apply AI deep learning to construct a GC (Gross Cross) diagnostic model based on tongue images, evaluate its value in GC diagnosis, and provide a scientific basis for traditional Chinese medicine practitioners' tongue image diagnosis theories.

[0012] The present invention relates to a tumor prediction system based on tongue image, wherein the system is: A tongue image acquisition module configured to acquire a tongue image of a test sample, and a data processing module configured to acquire the probability that the test sample belongs to the positive category by the following operation, The system includes a data processing module that predicts the probability that a test sample belongs to the positive category based on discriminative features on tongue images obtained through automated learning.

[0013] In one specific embodiment, the tumor is at least one of the following: gastric cancer, breast cancer, colorectal cancer, esophageal cancer, hepatobiliary and pancreatic cancer, lung cancer, prostate cancer, thyroid cancer, ovarian cancer, neuroblastoma, trophoblastic tumor, or head and neck squamous cell carcinoma.

[0014] In one specific embodiment, the tumor is at least one of the following: gastric cancer, breast cancer, colorectal cancer, esophageal cancer, hepatobiliary and pancreatic cancer, and lung cancer.

[0015] In one specific embodiment, the system further includes an output module configured to output prediction results.

[0016] In one specific embodiment, the output module is configured to output a tongue image and a prediction result.

[0017] In one specific embodiment, the output module outputs in at least one of the following modes: electronic display, audio broadcasting, printing, and network transmission.

[0018] In one specific embodiment, the discriminative features are derived from the relationship between positive and negative categories on the tongue image. The objective is to obtain discriminative features between positive and negative categories by thoroughly comparing, analyzing, and learning the commonalities and differences between and within positive and / or negative tongue images. By deeply discriminating the discriminative features between positive and negative categories on the tongue image of the test sample, it is possible to determine the probability that the test sample belongs to the positive category, thereby enabling tumor prediction of the test sample using tongue images. The discriminative features can be derived from the commonalities and differences between positive and negative tongue images, or from the commonalities and differences between positive and negative categories on a single tongue image. In other words, obtaining discriminative features between positive and negative categories from a tongue image can be used to predict whether a test sample belongs to the positive or negative category.

[0019] The aforementioned discriminative characteristics originate from the positive and negative tongue images of the paired input interactive deep learning model.

[0020] In one specific embodiment, the data processing module is configured to predict the probability that a test sample is positive by the following operations: The interactive deep learning model thoroughly compares positive and negative tongue images input simultaneously, automatically learns the commonalities and differences between positive and negative categories on the tongue images, and predicts the probability that the test sample belongs to the positive category based on the discriminative characteristics between the positive and negative categories. This part of the proposal can acquire discriminative characteristics between positive and negative categories by thoroughly comparing, analyzing, and learning the commonalities and differences between positive and negative tongue images, and predict the probability that the tongue image of the test sample input to the model belongs to the positive category based on these discriminative characteristics. Therefore, any model that can acquire discriminative characteristics between positive and negative categories by comparing, analyzing, and learning the commonalities and differences between positive and negative tongue images can be applied to this part of the proposal and is also included within the scope of protection of this part of the proposal. In particular, this application is not limited to explaining with an example using the APINet model.

[0021] In one specific embodiment, the positive tongue image is collected from a tumor-positive patient.

[0022] In one specific embodiment, the negative tongue image is collected from a tumor-negative patient.

[0023] In one specific embodiment, the interactive deep learning model is the APINet model.

[0024] In one specific embodiment, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: 1) Positive and negative features are extracted and obtained from previously acquired positive and negative tongue images. 2) Train the model with positive and negative features, and output the probability that each feature belongs to each category. 3) Input the tongue image of the test sample into the trained model and output the probability that the test sample belongs to the positive category.

[0025] In a specific embodiment, step 1) of extracting and obtaining the positive and negative features described above is The encoder extracts the feature vector of the image and outputs the positive feature f1 and the negative feature f2, f1, f2 and the combined feature f k , , c ,

[0027] , m , , , , ,

[0028] , , , i , m , , , are simultaneously input into the MLP of the feature selection region, and correspondingly output two control vectors g1 and g2, g1 activates f1 and f2 respectively to form the selected features f1 + and f2 - g2 activates f1 and f2 respectively to form the selected features f1 - and f2 + to form two positive features f1 + and f1 - and two negative features f2 + and f2 - and obtaining them. <s

[0026] In a specific embodiment, the MLP of the feature selection region fully learns the commonality and difference between f1 and f m to output the control vector g1, and similarly learns the commonality and difference between f2 and f m to output the control vector g2.

[0027] ) In a specific embodiment, step 2) of training the model with the positive and negative features described above specifically inputs the positive and negative features into the fully connected layer classifier and outputs the probabilities that these features belong to each category respectively.

[0028] In a specific embodiment, in step 2), when outputting the probabilities that the features belong to each category, according to the categories of the four features, the cross entropy loss function is minimized:

Equation

[0029] f1 + Since it is activated by the control vector g1 corresponding to the positive feature, it contains positive feature information, and f1 - Since it is activated by the control vector g2 corresponding to the negative feature, it contains negative feature information, and f2 + and f2 - Please note that the same applies to the following as well.

[0030] In one specific embodiment, in step 2), if the probability of a feature belonging to each category is output, feature f i + The confidence level of the model in relation to the output is a characteristic of f i - Assuming it should be higher, minimize the sort loss function:

number

[0031] In one specific embodiment, step 3) above, which involves inputting the tongue image of the test sample into the trained model, refers to inputting the tongue image of a single test sample.

[0032] In one specific embodiment, step 3) above, which outputs the probability that a test sample belongs to a category, ultimately means outputting the probability distribution on each category for the corresponding test sample, and designating the category with the highest probability as the predicted category.

[0033] In one specific embodiment, by training and testing using only the circumscribing rectangular portion of the tongue surface region in the tongue image, the influence of the image background on the model can be effectively eliminated.

[0034] In one specific example, during the training process, in order to enrich the sample space of the training set, samples in the training set are randomly flipped with a certain probability, then subgraphs are cut at random positions on the image, and finally, they are linearly interpolated to a fixed-size image, standardized, and input into an interactive deep learning model.

[0035] High-quality sample data is a prerequisite for obtaining a highly generalizable depth model. Therefore, positive and negative tongue image data are obtained in advance from tumor patients and non-tumor groups, respectively. In this part of the plan, only by thoroughly comparing a pair of samples (including positive and negative tongue image data) can their commonalities and differences be discovered. A real-world scene is simulated using the paired images as input. After the encoder extracts the image feature vectors, it outputs positive and negative features, and then combines the combined features to finally output a pair of positive and a pair of negative features. When these are input into a fully connected layer classifier, the probability that these features belong to each category can be output, while simultaneously minimizing the cross-entropy loss function and sort loss function to achieve the objective of the training model. During testing, inputting tongue images of test samples into the system allows for the determination of the probability of tumor positivity. By deeply analyzing the differences between positive and negative tongue images, the system learns the intrinsic relationship between tumors and tongue image information based on deep learning technology. This addresses problems such as low accuracy in early tumor screening and high costs of diagnostic policies by automatically determining the probability of tumor positivity and screening individuals with multiple tumors.

[0036] The aforementioned discriminative characteristics are derived from a single positive or negative tongue image.

[0037] In one specific embodiment, the discriminative feature is derived from cutting the tongue image into n small blocks, forming an input sequence, performing feature extraction, and obtaining deep features advantageous for classification.

[0038] In one specific embodiment, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: The tongue image of the test sample is cut into small blocks, an input vector is formed by linear mapping, a position index is added, a trained deep learning model is introduced to perform feature extraction and feature fusion, deep features favorable for classification after selection are output, and the probability of belonging to each category is obtained.

[0039] In one specific embodiment, the deep learning model completes training in the following steps: a) After cutting the tongue surface image into n small blocks, the cut n small blocks are sequentially constructed to form an input sequence of length n, an input vector is formed by linear mapping, and position indices 0, 1, 2, ..., n-1 are added, b) Feature extraction and feature fusion are performed using an encoder based on the TransFG model, outputting deep features that are advantageous for the selected classification, and finally a softmax classifier outputs the probability distribution to which the deep features belong to each category.

[0040] In one specific embodiment, step a) above, which involves cutting the tongue image into n small blocks, means cutting the tongue image into n non-overlapping square regions.

[0041] In one specific embodiment, in step b), when the encoder performs feature extraction, it includes a total of L+1 Transformer layers, and each layer contains a self-attention mechanism.

[0042] In one specific embodiment, in step b), when the encoder performs feature extraction and feature fusion, in order to remove redundant features, region selection is performed via a feature selection module including a multi-head attention mechanism before inputting deep features to the last layer. The feature selection module returns the index of the front row feature with the largest attention weight, and the selected front row feature is input to the last Transformer layer to perform feature fusion.

[0043] In one specific embodiment, the front row features are the previous K features, where k is one of 1, 2, 3, ..., 20.

[0044] In one specific embodiment, k = 12.

[0045] In one specific embodiment, if step b) the deep features output probability distributions belonging to each category, the cross-entropy loss function is minimized:

number

[0046] In one specific embodiment, in step b), if the deep features output probability distributions belonging to each category, the contrast loss function is minimized:

number

[0047] In this part of the proposal, after cutting the tongue image into small, non-overlapping regions, a sequence is constructed sequentially, and then an input vector is formed by linear mapping. This is input into a TransFG model for feature extraction and feature fusion, generating deep features favorable for classification, and the probability of belonging to each category is output by a softmax classifier. This completes the prediction of the tongue image belonging to a category, and through the automatic learning mode of the deep learning model, the probability of tumor positivity in the test is automatically predicted and screened. Addressing the problems of low accuracy and high cost of conventional early tumor screening, this part of the proposal automatically determines the probability of tumor positivity based on tongue surface images and deep learning technology, and screens for multiple tumors. This part of the proposal is easy to operate, low cost, and high accuracy in the test.

[0048] The aforementioned discriminative characteristics originate from each pixel in the tongue image.

[0049] In one specific embodiment, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: The tongue image of the test sample is input into a trained deep learning model, and the probability that each pixel belongs to positive, negative, or background is output. The category with the highest probability is then used as the predicted category for that pixel. The probability that a test sample is positive is calculated as follows: (Number of pixels predicted to be positive / (Number of pixels predicted to be positive + Number of pixels predicted to be negative)).

[0050] In one specific embodiment, in the process of obtaining the probability that the test sample belongs to the positive category, if the number of pixels predicted to be positive is greater than the number of pixels predicted to be negative, the test sample is ultimately predicted to be positive, and conversely, it is predicted to be negative.

[0051] In one specific embodiment, the deep learning model is trained in the following steps: For positive tongue images or negative tongue images, each pixel is labeled; specifically, pixels in the positive tongue surface area, pixels in the negative tongue surface area, and pixels in the background area are labeled, respectively. The overall algorithmic framework employs an automatic encoding-decoding structure, where an image encoder is used to encode the features of the entire image, and a feature decoder outputs the features as a probability map of the entire image.

[0052] The loss value for each pixel is calculated using the true label of each pixel and the predicted probability in the probability map, and the model parameters are updated until training is complete.

[0053] In one specific embodiment, positive tongue surface region pixels are labeled as 2, negative tongue surface region pixels as 1, and background region pixels as 0.

[0054] In one specific embodiment, the automatic encoding-decoding structure employs a DeeplabV3+ image segmentation network structure and / or a Unet series network structure. Since any automatic encoding-decoding framework capable of generating probability maps can be applied to the present invention, there are many selectable deep network structures. In this invention, the DeeplabV3+ model is preferred, but other frameworks that generate probability maps by automatic encoding-decoding can similarly achieve the objective of the invention, such as the Unet series network structure, which is commonly used in medical image processing.

[0055] In one specific embodiment, a category interpretation module is added after the output layer of the network structure in the automatic encoding-decoding structure, and the final test result is determined based on a probability map of the entire image.

[0056] In one specific embodiment, the category interpretation module probability map employs the interpretation policy shown in the following formula:

number

[0057] In one specific example, t = 0.5 in the interpretation policy formula.

[0058] In one specific example, the deep learning model training process employs a cross-entropy cost function predicted for each pixel:

number

[0059] Based on established clinical diagnoses, labeling collected tongue images of different cases as tumor-positive or tumor-negative, and obtaining sufficient label data, is necessary for optimizing the learning model and improving prediction accuracy. By employing a pixel-by-pixel labeling method for tongue images, and drawing the tongue surface region using existing labeling tools, tags are assigned to each positive region pixel, negative region pixel, and background region pixel. After outputting a probability map through an automatic encoding-decoding structure, the loss difference between the true label of each pixel and the predicted probability in the probability map is calculated, and the model parameters are updated to complete model training. By inputting tongue images of test samples into the model, the probability of tumor positivity can be automatically determined. This allows for screening of tumor-prone groups and overcomes the shortcomings of conventional early tumor diagnosis, such as the high cost of data collection and the difficulty in conducting comprehensive surveys. Internal testing accuracy reached 86.6%, thus demonstrating high clinical application value.

[0060] A method for predicting tumors based on tongue images, wherein the method is: To obtain tongue images of the test sample, This includes inputting a tongue image of a test sample into the system to obtain the tumor-positive probability of the test sample.

[0061] An application of the aforementioned tumor prediction system and / or method based on tongue imagery, wherein the application is This includes performing tumor prediction on a test sample by applying the aforementioned system and / or method.

[0062] Each of the above preferred conditions can be combined with one another, in accordance with common sense in this industry, to obtain specific embodiments. [Effects of the Invention]

[0063] The beneficial effects of the present invention are as follows: This invention provides a multi-tumor prediction system based on tongue images. By directly using tongue images of non-living samples as the target, and analyzing and learning the commonalities and differences between positive and negative features in the tongue images, it can exhibit excellent diagnostic and predictive capabilities for various types of tumors. After analysis and verification of a large number of actual patient samples, the accuracy of the test prediction for gastric cancer can reach approximately 80%, with a sensitivity of 0.741 to 0.826 and an accuracy of 0.785 to 0.806 during internal testing, and a sensitivity of 0.841 to 0.862 and an accuracy of 0.709 to 0.734 during external testing. Both the sensitivity and accuracy of the tests are significantly better than the sensitivity and accuracy of conventional machine learning models that utilize blood tumor markers. Furthermore, the tumor prediction system based on tongue imaging demonstrates excellent diagnostic predictive value for various malignant tumors, including breast cancer, colorectal cancer, esophageal cancer, hepatobiliary and pancreatic cancer, and lung cancer. It is clearly superior to conventional combinations of blood tumor markers, providing a prospective, economical, non-invasive, and effective screening and diagnostic prediction system and method for tumors.

[0064] To achieve the above objectives, the present invention employs the above-mentioned technical solutions, compensating for the shortcomings of the prior art, while also being rationally designed and easy to operate. [Brief explanation of the drawing]

[0065] Some of the accompanying drawings are provided to enable a person skilled in the art to more quickly and clearly understand the above and / or other purposes, features, advantages and examples of the present application. However, it should be noted that the accompanying drawings, schematic embodiments and their descriptions in the specification constituting the present application are for the purpose of providing a further understanding of the present application and do not constitute an unreasonable limitation thereto.

[0066] [Figure 1] This is a schematic diagram of a multicenter clinical study and its patient distribution. [Figure 2] This is a classification framework for APINet models. [Figure 3] This is a visualization of the rationale behind tongue surface image recognition by the APINet model. [Figure 4] These are the ROC and AUC for internal and external validation based on three models (SVM, DT, KNN) for eight hematological malignancies. [Figure 5] These are the ROC and AUC values ​​for internal and external validation of three tongue image-based models (APINet, TransFG, and DeeplabV3+). [Figure 6] These are the ROC and AUC values ​​for the APINet model for GC and other tumors. [Figure 7] This is a classification framework for the TransFG model. [Figure 8] This is a visualization of the results of the domain selection module in the TransFG model. [Figure 9] These are the ROC and AUC values ​​for the TransFG model for GC and other tumors. [Figure 10] This is a schematic diagram of the labeling process for a Deeplab V3+ model training sample. [Figure 11] This is a classification framework for Deeplab V3+ models. [Figure 12] This is a visualization of the prediction results from the Deeplab V3+ model. [Figure 13] This is an output diagram of a Deeplab V3+ model. [Figure 14] These are the ROC and AUC values ​​for the DeepLabV3+ model for GC and other tumors. [Figure 15] These are the external validation probability distributions for the three tongue image models. [Figure 16] These are representative tongue images with different probabilities. [Modes for carrying out the invention]

[0067] Those skilled in the art can implement appropriate substitutions and / or modifications of process parameters by reference to the foregoing specification, but it should be noted in particular that all similar substitutions and / or modifications are obvious to those skilled in the art and are all considered to be included in the present invention. Although the present invention has been illustrated by preferred embodiments, it will be apparent to those involved that the technology of the present invention can be implemented and applied by modifying or appropriately modifying and combining the contents described herein without departing from the spirit and scope of the present invention.

[0068] The following detailed descriptions are illustrative and intended to further illustrate the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art.

[0069] The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the technical solutions of this application. As used herein, the singular form is intended to include the plural form unless otherwise specified by the context, and it should also be understood that when the terms “contains” and / or “includes” are used herein, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0070] APINet model: The APINet model, also known as the attentive pairwise interaction neural network (APINet) model.

[0071] TransFG model: TransFG model, also known as transformer architecture for fine-grained recognition (TransFG) model.

[0072] The present invention will be described in more detail below.

[0073] <Clinical specimen> A nationwide multicenter clinical study was conducted to eliminate the influence of regional, dietary, and center differences on the research, and included 11 centers in 8 cities: Hangzhou, Wenzhou, and Shanghai in the east; Fuzhou in the south; Chengdu in the west; Liaoning and Heilongjiang in the north; and Taiyuan in the central region.

[0074] As shown in Figure 1, from January 2020 to October 2021, 1111 gastric cancer (GC) patients were recruited from 8 centers and 1519 non-gastric cancer (NGC) patients from 3 centers, including 169 healthy controls (HCs), 648 superficial gastritis (SGs), and 702 atrophic gastritis (AGs). 865 cases were randomly selected from the gastric cancer (GC) patients and 1287 cases from the non-gastric cancer (NGC) patients to train and validate the system. These included 448 early-stage GC (TNMI+stage II), 417 late-stage GC (TNMIII+stage IV), 141 healthy controls (HCs), 547 superficial gastritis (SGs), and 599 atrophic gastritis (AGs). Approximately 80% of the cases were used as the training dataset, and approximately 20% were used as the internal validation dataset. Furthermore, 246 cases of gastric cancer (GC) and 232 cases of gastric cancer (NGC) from the three centers were used as an independent external validation dataset, including 162 cases of early-stage GC, 84 cases of late-stage GC, 28 cases of HC, 101 cases of SG, and 103 cases of AG. All of these gastric cancer (GC) patients were newly diagnosed with gastric cancer, had no prior treatment for the disease, and had not undergone surgery, chemotherapy, radiation therapy, targeted therapy, or biotherapy for the disease. All gastric cancer (GC) patients had solitary tumors; that is, patients with two or more types of malignant tumors were excluded. HCs, SGs, and AGs were confirmed by gastroscopy.

[0075] Tongue images and clinical information were collected from all participants, including age, sex, height, weight, family history, smoking, alcohol consumption, TNM staging, and hematological tumor markers. Pathological staging was based on the American Joint Committee on Cancer, 8th edition, 23rd edition. Tongue images for all GC participants were collected on the morning of gastric surgery, and for NGC participants, on the morning of gastroscopy, with a fasting period of more than 8 hours to eliminate the influence of food on tongue images. Table 1 shows general patient information such as age, sex, BMI, smoking, and alcohol consumption for the GC and NGC groups, and these data showed good agreement with training, internal validation, and independent external validation datasets. JPEG0007871484000007.jpg195170

[0076] Additionally, 104 patients with esophageal cancer (EC), 129 with hepatobiliary and pancreatic cancer (HBPC), 116 with colorectal cancer (CRC), 260 with lung cancer (LC), and 154 with breast cancer (BC) were recruited from Zhejiang Provincial Cancer Hospital. Table 2 shows the clinical information of participants with other cancers. With the exception of BC, general information between GC and other cancers, such as age, sex, BMI, smoking, and alcohol consumption, was found to be well-matched. JPEG0007871484000008.jpg111170

[0077] <Statistical analysis> All statistical analyses were performed using SPSS 23.0 software (SPSS Inc., Chicago, IL, USA). Results were expressed as mean ± SD or mean ± SEM. Parametric or non-parametric tests were used depending on whether the data were orthogonal. Count data were analyzed using the chi-squared test. P < 0.05 was considered statistically significant.

[0078] <Ethical Approval> This research was approved by the Central Ethics Committees of 11 participating centers, including the Zhejiang Provincial Cancer Hospital Research Ethics Committee, the First Affiliated Hospital of Wenzhou Medical University, Liaoning Provincial Cancer Hospital, Renji Hospital Affiliated with Shanghai Jiao Tong University, Fujian Provincial Cancer Hospital, Harbin Medical University Affiliated Cancer Hospital, Sichuan Provincial Cancer Hospital, Shanxi Provincial Cancer Hospital, Zhejiang Provincial Tongde Hospital, Zhejiang Provincial Hospital of Traditional Chinese Medicine, and Yuhang Municipal People's Hospital.

[0079] <Clinical Verification> Example 1 The APINet model was validated, and specifically, the system is a tumor diagnostic system based on tongue imagery, and the system is A tongue image acquisition module configured to acquire tongue images of a test sample, A data processing module configured to obtain the probability that a test sample belongs to the positive category through the following operations, The system includes a data processing module that predicts the probability that a test sample belongs to the positive category based on discriminative features on tongue images obtained through automated learning.

[0080] By designing a comparison-based interactive deep learning model and thoroughly comparing a pair of tongue surface images input simultaneously, the model automatically learns the commonalities and differences between positive and negative categories on the tongue surface images, and ultimately predicts the probability that a test sample belongs to the positive category based on discriminative features. As shown in Figure 2, the overall algorithmic framework is divided into three modules: a feature fusion module, a feature selection module, and a classification module.

[0081] Feature fusion module: Simultaneously inputs a pair of tongue images belonging to the positive and negative categories, respectively. First, the encoder extracts the feature vectors from the images and outputs the positive feature f1 and the negative feature f2.

[0082] Feature selection module: f1 and f2 and the combined feature f m The two control vectors g1 and g2(f1+f) are input simultaneously to the MLP of the feature selection region, corresponding to f1 and f2 respectively. m →g1,f2+f m → g2) is output. The control vector g1 activates f1 and f2 respectively, and the selected feature f1 + and f2 - G2 then acts on f1 and f2 to form the selected feature f1 - and f2 + It forms two positive features f1 + and f1 - and two negative characteristics f2 + and f2 - Outputs.

[0083] Classification Module: The selected features are input into the classifier (fully connected layer), and finally, the probability that each of these features belongs to each category is output.

[0084] During the training process, the cross-entropy cost function is minimized based on the category to which the four features belong:

number

[0085] Note that feature f i + The confidence of a highly generalizable model for the output of feature f i - It must be higher than this, so we simultaneously minimize the sort cost function:

number

[0086] A total of 905 relevant patients were tested, of which 427 were internally studied from the same center as the training set, and 478 were used for external studies using data from different centers. The results are shown in Tables 3 and 4 below. JPEG0007871484000011.jpg31170JPEG0007871484000012.jpg31170

[0087] Table 3 shows that the actual number of negative cases was 162 + 52 = 214, and the actual number of positive cases was 37 + 176 = 213. The prediction results showed that 162 cases were accurately predicted as negative, and 52 cases were incorrectly predicted as positive. Therefore, the prediction accuracy in the internal test is (number of cases accurately predicted as negative + number of cases accurately predicted as positive) / total number of test samples = (162 + 176) / (162 + 52 + 37 + 176) = 79%. Similarly, as can be seen from Table 4, the accuracy of the external test can reach 71%. Based on the results of the internal and external tests, this tumor diagnostic system exhibits relatively good predictive accuracy for gastric cancer.

[0088] Figure 3 is a schematic visualization of the model's classification basis. The three test samples in the first row to the left of the dashed line are positive tongue surface images. The second row shows the areas that the model primarily uses as evidence when recognizing based on the tongue surface images. To the right of the dashed line are visualizations of the negative samples and the corresponding tongue surface recognition basis. In the second row, the darker the color, the more the model is focusing on that area. From the results, it was found that the areas on which the model's recognition process is based are mainly concentrated on the tongue surface, and are unaffected by the background, regardless of the black background.

[0089] Clinical symptoms are often masked, diagnostic screening relies on gastrointestinal endoscopy, resulting in low early diagnosis rates for gastrointestinal tumors, poor prognosis, and a heavy social and economic burden. Therefore, there is a pressing need to develop non-invasive, effective screening and diagnostic methods to increase the early diagnosis rate of gastrointestinal tumors. Artificial intelligence plays an increasingly important role in cancer screening and diagnosis, clearly paving the way for an evolving healthcare system filled with higher accuracy and computational power. Our study conducted an observational, prospective, multicenter clinical trial evaluating the value of tongue imaging in the screening and diagnosis of gastrointestinal tumors (GCs) and other tumors.

[0090] To further evaluate the value of tongue imaging as a diagnostic and screening tool for tumors, we compared clinically applied hematological tumor markers with tongue imaging. As a control, we examined tumor prediction using combinations of several classical hematological tumor markers. The selectable hematological tumor markers were alpha-fetoprotein (AFP), carcinoembryonic antigen (CEA), cancer antigen 125 (CA125), cancer antigen 15-3 (CA15-3), cancer antigen 199 (CA199), cancer antigen 72-4 (CA72-4), cancer antigen 242 (CA242), cancer antigen 50 (CA50), non-small cell lung cancer-related antigen (CYFRA21-1), small cell lung cancer-related antigen (neuron-specific enolase, NSE), squamous cell carcinoma antigen (SCC), and total prostate-specific At least one of the following was selected: prostate-specific antigen (TPSA), free prostate-specific antigen (FPSA), alpha-L-fucosidase (AFU), EBV antibody (EBV-VCA), tumor-associated substance (TSGF), ferritin, β2-microglobulin (β2-MG), pancreatic embryo antigen (POA), or gastrin precursor-releasing peptide (PROGRP), and more specifically, at least one of the following was selected: CEA, CA242, CA72-4, CA125, CA199, CA50, AFP, or ferritin, and more specifically, a combination of the above eight blood tumor markers was selected. A prediction method based on the above blood tumor marker indices includes the following steps.

[0091] 1) Data preprocessing: Since there are varying degrees of missing values ​​in serum indicators for all cases, the training data must be complete. Therefore, before model training, it is necessary to impute the data. In this application, the K-neighbor missing value imputation method is used to impute the data, and specifically, the imputed value for a missing serum indicator is the mean of the two neighboring values.

[0092] 2) Model Training: The present invention employs three types of machine learning classification methods, which are support vector machines (SVM), decision trees (DT), and K-neighbor classifiers (KNN). Specifically, eight types of hematological tumor markers for cases (CEA, CA242, CA72-4, CA125, CA199, CA50, AFP, and Ferritin) correspond to the characteristics of the sample, and the negative / positive diagnosis of the case corresponds to the label of the sample. All completed samples are then sent to the three types of classifiers for fitting.

[0093] 3) Model Evaluation: This application evaluates the model using internal and external validation. Internal validation uses data from the same hospital but different cases as the training data, while external validation uses data from different hospitals and cases than the training data. The model is predicted using three indicators, including sensitivity, specificity, and accuracy.

[0094] As shown in Table 5, clinical information on blood tumor markers in related GC patients revealed that the concentrations of blood tumor markers such as CEA, CA424, CA724, CA125, CA199, CA50, AFP, and Ferritin were significantly elevated in GC patients compared to NGC patients. JPEG0007871484000013.jpg131170

[0095] The training, internal validation, and external validation datasets for the model are consistent with the tongue image model (except in cases where blood indicators are missing). The validation results for the sensitivity, specificity, and accuracy of blood tumor markers for GC diagnosis based on three machine learning classification methods are shown in Table 6. For internal and external validation ROC and AUC values, see Figure 4. The range of AUC values ​​for internal validation is 0.682 to 0.715, and the range of AUC values ​​for external validation is 0.694 to 0.760. In the SVM algorithm, the specificity for both internal and external validation reached over 90%, indicating that this algorithm can provide valuable information for gastric cancer diagnosis. However, in DT and KNN, the specificity decreased to some extent, while sensitivity and accuracy improved to varying degrees, allowing for multifaceted information to be provided for gastric cancer diagnosis. JPEG0007871484000014.jpg42170

[0096] It should be made clear that the comparative scheme described above selects eight serum indicators, including CEA, CA242, CA72-4, CA125, CA199, CA50, AFP, and Ferritin, and that increasing, decreasing, or substituting some of these serum indicators can all predict the positive or negative status of tumors, particularly gastric cancer. The comparative scheme employs three types of machine learning classifiers, SVM, DT, and KNN, and other machine learning classifier methods such as logical regression and random forest can also achieve the same objectives.

[0097] The APINet model in this embodiment shows varying degrees of improvement or change in sensitivity, specificity, and accuracy for GC diagnosis compared to the SVM, DT, and KNN models, as shown in Table 7. JPEG0007871484000015.jpg34170

[0098] Table 7 shows the sensitivity, specificity, and accuracy data of the APINet model based on tongue images for GC diagnosis. Internal and external validation showed that the APINet model has significantly higher sensitivity and accuracy for GC diagnosis than the aforementioned SVM, DT, and KNN models based on eight types of hematological tumor markers, providing a prospective, economical, non-invasive, and effective screening and diagnostic prediction method for tumors.

[0099] The APINet model thoroughly compares input data through paired interaction and recognizes comparative cues for classification. Figure 5 shows the ROC (Receiver Operating Characteristic) and AUC (Area Under ROC Curve) of the APINet model for internal and external validation. As can be seen from Figure 5, the APINet model in Figure 5 has an ROC curve that is relatively far from the (0,0)-(1,1) line in both internal and external validation compared to the SVM, DT, and KNN models in Figure 4. Its internal validation AUC value reaches 0.875 and its external validation AUC value reaches 0.792, which are higher than the internal validation AUC values ​​(0.682-0.715) and external validation AUC values ​​(0.694-0.760) of the SVM, DT, and KNN models for eight types of hematological tumor markers. This indicates that the APINet model is a well-representing predictive model. The diagnostic value of AI-based tongue image-based diagnostic models for GC (Gross Cortex) is clearly superior to models that simply apply a combination of eight blood tumor markers.

[0100] We analyzed the correlation between model accuracy and clinical information. See Table 8 for the correlation between the accuracy of specific APINet models and the clinical information of GC patients, and see Table 9 for the correlation between the accuracy of APINet models and the clinical information of NGC patients. We found that in discriminating NGC, the accuracy of the APINet model was related to smoking, alcohol consumption, and blood tumor marker indices, while in discriminating GC, the accuracy of the APINet model was related only to gender. In other words, the function of the APINet model in distinguishing between GC and NGC is less affected by clinical information. JPEG0007871484000016.jpg122170JPEG0007871484000017.jpg71170

[0101] To observe the specificity and effectiveness of the GC diagnostic model APINet based on tongue imagery, we selected 104 EC, 129 HBPC, 116 CRC, 260 LC, and 154 BC patients to evaluate their diagnostic value. As shown in Table 10, the specificity results of the APINet model for GC and other tumors show that the tongue imagery model APINet is most useful for GC diagnosis and plays a certain role in the diagnosis of gastrointestinal tumors such as EC, HBPC, CRC, and LC. As shown in Figure 6, the ROC and AUC of the APINet model for GC and other tumors show that the diagnostic effect of the APINet model is best for GC, while the effects differ for EC, HBPC, CRC, LC, etc., indicating that the APINet model is proactive in predicting the diagnosis of these various tumors. JPEG0007871484000018.jpg25170

[0102] Example 2 The TransFG model was used for verification, and specifically, the system is a tumor diagnostic system based on tongue imagery, and the system is A tongue image acquisition module configured to acquire tongue images of a test sample, A data processing module configured to obtain the probability that a test sample belongs to the positive category through the following operations, The system includes a data processing module that predicts the probability that a test sample belongs to the positive category based on discriminative features on tongue images obtained through automated learning.

[0103] A deep learning model based on Transformers is designed to divide the input tongue surface image into small, non-overlapping blocks, and these small blocks are sequentially constructed and input into a multilayer neural network. Finally, the probability that a test sample belongs to the positive category is predicted based on the extracted high-discrimination features.

[0104] The overall algorithm structure is shown in Figure 7, and the input to the entire model is a tongue surface image. First, the tongue surface image is cut into n small blocks, and then the cut n small blocks are sequentially constructed to form an input sequence of length n. The small image blocks are formed into an input vector by linear mapping, and position indices 0, 1, 2, ..., n-1 are added. This invention performs feature extraction based on the encoder portion of a Transformer model and includes a total of L(L=9)+1 Transformer layers, each layer containing a self-attention mechanism. To remove redundant features, before inputting deep features to the last layer, region selection is first performed via a feature selection module, which includes a multi-head attention mechanism and returns the index of the feature in the previous k(k=12) block with the largest attention weight. The selected k features are input to the last Transformer layer for feature fusion, outputting deep features favorable for classification after selection, and finally outputting the probability distribution of each category by a softmax classifier.

[0105] If the deep features output probability distributions belonging to each category, then minimize the cross-entropy loss function for each:

number

number

[0106] This improves prediction accuracy by further consolidating features within a class and increasing the feature differences between classes.

[0107] A total of 905 relevant patients were tested, of which 427 were used for internal trials from the same center as the training set, and 478 were used for external trials using data from different centers. The trial results are shown in Tables 11 and 12 below. Here, the accuracy rates for internal and external trials reached 81% and 73%, respectively. As can be seen from the results of the internal and external trials, this tumor diagnostic system has relatively good predictive accuracy for gastric cancer. JPEG0007871484000021.jpg31170JPEG0007871484000022.jpg31170

[0108] Figure 8 is a schematic visualization diagram that forms the basis for the model classification. The three test samples in the first row to the left of the dashed line are positive tongue surface images, the small yellow blocks in the second row of images represent the regions in the original image corresponding to the feature indices returned by the region selection module, and the area to the right of the dashed line is the negative sample and region selection result. From the results shown, it was found that the regions that form the basis of the model recognition process are mainly concentrated in the areas with heavy tongue coating in the upper half of the tongue surface, and that there is little correlation with the black background and the lower half of the tongue surface.

[0109] Referring to the above, in order to further evaluate the value of tongue imaging as a diagnostic and screening tool, we compared it with clinically applied hematological malignancy markers. Specifically, we compared the TransFG model based on tongue imaging with the SVM, DT, and KNN models based on hematological malignancy markers. As a result, the TransFG model in this embodiment showed varying degrees of improvement or change in sensitivity, specificity, and accuracy for GC diagnosis, as shown in Table 13. JPEG0007871484000023.jpg28170

[0110] Table 13 shows the sensitivity, specificity, and accuracy data of the TransFG model based on tongue images for GC diagnosis. The TransFG model was found to have significantly higher sensitivity and accuracy for GC diagnosis than the aforementioned SVM, DT, and KNN models based on eight blood tumor markers (CEA, CA242, CA72-4, CA125, CA199, CA50, AFP, and Ferritin) in both internal and external validation. This provides a prospective, economical, non-invasive, and effective screening and diagnostic prediction method for tumors.

[0111] The TransFG model is data-driven and automatically selects regions favorable for classification. Figure 5 shows the ROC and AUC of the TransFG model for internal and external validation. As can be seen from Figure 5, the TransFG model's internal validation AUC = 0.859 and external validation AUC = 0.815 are significantly higher than the internal validation AUC values ​​(0.682~0.715) and external validation AUC values ​​(0.694~0.760) of the SVM, DT, and KNN models based on eight types of hematological tumor markers. This indicates that the TransFG model is a relatively good predictive model. The diagnostic value of the AI ​​diagnostic model based on tongue images for GC is clearly superior to the combination of eight hematological tumor markers.

[0112] The correlation between model accuracy and clinical information was analyzed. For the correlation between the accuracy of specific TransFG models and the clinical information of GC patients, see Table 14. For the correlation between the accuracy of TransFG models and the clinical information of NGC patients, see Table 15. In discriminating NGC, the accuracy of the TransFG model is related to general circumstances such as age, sex, BMI, smoking, and drinking, while in discriminating GC, the TransFG model is related only to sex. Therefore, the function of the TransFG model in distinguishing between GC and NGC is not affected by clinical information. JPEG0007871484000024.jpg137170JPEG0007871484000025.jpg70170

[0113] To observe the specificity and effectiveness of the TransFG GC diagnostic model based on tongue imaging, we selected patients with EC, HBPC, CRC, LC, and BC as described above and aimed to evaluate its diagnostic value. As shown in Table 16, the specificity results of the APINet model for GC and other tumors, the TransFG model, like the APINet model, was found to be most useful for GC diagnosis and to have various effects on the diagnosis of tumors such as EC, HBPC, CRC, and LC. JPEG0007871484000026.jpg34170

[0114] As shown in Figure 9, the ROC and AUC of the TransFG model for GC and other tumors show that the TransFG model has the best diagnostic effect for GC, with an AUC of 0.815. Its AUC for tumor diagnoses such as EC, HBPC, CRC, and LC all exceeded 0.5, indicating a certain level of diagnostic efficacy. Therefore, the TransFG model is found to be proactive in predicting the diagnosis of various tumors, including GC.

[0115] Example 3 The DeeplabV3+ model was used for verification, and specifically, the system is a tumor diagnostic system based on tongue imagery, and the system is A tongue image acquisition module configured to acquire tongue images of a test sample, A data processing module configured to obtain the probability that a test sample belongs to the positive category through the following operations, The system includes a data processing module that predicts the probability that a test sample belongs to the positive category based on discriminative features on tongue images obtained through automated learning.

[0116] Using computer-aided technology, a bottom-up deep learning decision framework is designed, and the determination of whether different test subjects are tumor-positive or mute based on pre-acquired tongue images is performed automatically.

[0117] To learn a better deep learning model for this task, it is first necessary to obtain sufficient labeled data. As shown in Figure 10, a pixel-by-pixel labeling method is adopted for tongue image. Using an existing labeling tool to draw the tongue surface area, tag each pixel. If the sample is a tumor-positive sample, the pixel tag of the tongue surface area is labeled as 2, if the sample is a tumor-negative sample, the pixel of the tongue surface area is labeled as 1, and all background area pixels are set to 0. It should be clear that labeling the aforementioned tongue surface area pixels as 2, 1, and 0 is only exemplary, and all labelings that can distinguish positive sample pixels, negative sample pixels, and background area pixels are acceptable, for example, A, B, C, Jia, Yi, Bing, I, II, III, (1), (2), (3), one, two, three, etc.

[0118] Based on the above labeling method of tongue images, the present invention designs a bottom-up deep learning model for tongue images, automatically learns the tongue surface features of the tongue image, and finally outputs the tumor-positive probability of the corresponding sample based on the tongue image information. The overall algorithm framework adopts an auto-encoding and decoding structure. As shown in Figure 11, the Encoder is an image encoder and is used to encode the features of the entire image. The Decoder is a feature decoder. The decoder outputs as the probability map of the entire image, and each pixel represents the probability of being classified into the specified category. The total number of categories of this task is the number of layers of the model output probability map. Specifically, among all auto-encoding and decoding structures, we adopt the DeeplabV3+ image segmentation network structure. To effectively determine the positive probability of the test sample, we add a category judgment module after the output layer of the DeeplabV3+ network, that is, determine the final test result based on the probability map of the entire image.

[0119] Let the pixel set of the input image be M = {i|i = 1, 2, ……, m}, the category set be C = {c|c = 0, 1, 2}, and 0, 1, 2 in the category set represent background, negative, and positive respectively, P c(i) represents the probability that a given pixel belongs to category c. The category interpretation module employs the interpretation policy shown in the following equation, based on the intermediate result probability map:

number

[0120] During the model training process, a cross-entropy cost function predicted for each pixel is employed, and the corresponding cost function L for the input tongue surface image is as follows:

number

[0121] The detailed steps for predicting the positive probability of a tongue image using the DeeplabV3+ model are as follows:

[0122] 1) During prediction, the model outputs the probability that each pixel belongs to one of three categories (background, negative, positive), corresponding to 0, 1, and 2 in the category interpretation module. By comparing the magnitude of the probability that each pixel belongs to each category, the category with the highest probability is selected as the predicted category. For example, if a pixel has probabilities of belonging to background, negative, and positive of 0.3, 0.5, and 0.2 respectively, that pixel is predicted to be in the negative category.

[0123] 2) The number of pixels predicted to be in the positive and negative categories of the input image is statistically analyzed. If the number of pixels predicted to be positive is greater than the number of pixels predicted to be negative, the input image is ultimately predicted to be positive. That is, if the number of positive pixels / (number of positive pixels + number of negative pixels) is greater than 0.5, the input image is predicted to be positive. The number of positive pixels / (number of positive pixels + number of negative pixels) is the probability that the input image belongs to the positive category.

[0124] To eliminate the influence of the image background on the experiment, we applied only the circumscribing rectangular portion of the tongue surface region in the image for training and testing. During the training process, to enrich the sample space of the training set, we randomly flipped samples in the training set, Gaussian blurred the flipped tongue images by a specified percentage, and randomly selected this percentage within the range of 0 to 0.5 within each training cycle. We collected a total of 678 tongue images, of which 544 were used for training and 134 were used for testing. As shown in Table 17, 10 positive cases were misclassified as negative, and 8 negative cases were misclassified as positive. The model's prediction accuracy was 87%, indicating high prediction accuracy for both negative and positive samples. JPEG0007871484000029.jpg31170

[0125] Figure 12 shows the test results after model visualization. The four test samples were taken from tongue images of negative, positive, and positive / negative cases, respectively. However, our final tongue image category determination results are the same as the directly displayed results. The first column is the input tongue image to be predicted, and the second column is the prediction result for each pixel of the input image. Here, pixels in the green area (area G indicated by the arrow in the figure) are predicted to be negative, pixels corresponding to the yellow area (area Y indicated by the arrow in the figure) are predicted to be positive, and the purple area (area P indicated by the arrow in the figure) is the background area of ​​the model prediction. Therefore, both the negative and positive determination areas are located in the tongue area, and the image category prediction is not affected by the background area. The third column shows the area in the original figure to which the prediction result corresponds. Based on the above formula, the proportion of the yellow area on the entire tongue surface is considered to be the probability that the model predicts the sample to be positive for gastric cancer. If this probability is greater than 0.5, the test sample is considered positive for gastric cancer, and the tongue surface area predicted by the model is considered to be the sum of the positive and negative areas.

[0126] Figure 13 shows the semantic segmentation results of several samples of the DeeplabV3+ model based on tongue images. The three test samples in the first row to the left of the dashed line are positive tongue images, and the entire tongue surface region in the second row of the image corresponds to the region corresponding to the feature index returned in the original image, with pixels marked in yellow being predicted as positive. Similarly, the three test samples in the first row to the right of the dashed line are negative tongue images, and the entire tongue surface region in the second row of the image corresponds to the region corresponding to the feature index returned in the original image, with pixels marked in green being predicted as negative. Therefore, both the positive and negative determination regions are located in the tongue surface region, and the prediction of image categories is not affected by the background region.

[0127] Referring to the above, in order to further evaluate the value of tongue imaging as a means of diagnosis and screening for the tumors in the test samples, we compared tongue imaging with clinically applied hematological tumor markers, specifically comparing the DeeplabV3+ model based on tongue imaging with SVM, DT, and KNN models based on hematological tumor markers. As a result, the DeeplabV3+ in this embodiment showed varying degrees of improvement or change in sensitivity, specificity, and accuracy for GC diagnosis, as shown in Table 18. JPEG0007871484000030.jpg34170

[0128] Table 18 shows that the DeeplabV3+ model based on tongue images exhibits superior sensitivity and accuracy for GC diagnosis, outperforming the aforementioned SVM, DT, and KNN models based on eight blood tumor markers (CEA, CA242, CA72-4, CA125, CA199, CA50, AFP, and Ferritin) in both internal and external validation, resulting in superior sensitivity (0.283-0.566, 0.362-0.539) and accuracy (0.603-0.622, 0.645-0.662). This enriches prospective, economical, non-invasive, and effective tumor screening and diagnostic prediction methods.

[0129] Figure 5 shows the ROC and AUC of the DeeplabV3+ model for internal and external validation. As can be seen from Figure 5, the DeeplabV3+ model's internal validation AUC = 0.836 and external validation AUC = 0.801 are significantly higher than the internal validation AUC values ​​(0.682~0.715) and external validation AUC values ​​(0.694~0.760) of the SVM, DT, and KNN models based on eight types of hematological tumor markers. This indicates that the DeeplabV3+ model is a relatively good predictive model. The diagnostic value of the AI ​​diagnostic model based on tongue images for GC is clearly superior to the combination of eight hematological tumor markers.

[0130] We analyzed the correlation between model accuracy and clinical information. See Table 19 for the specific correlation between DeeplabV3+ model accuracy and clinical information of GC patients, and Table 20 for the correlation between DeeplabV3+ model accuracy and clinical information of NGC patients. We found that in NGC discrimination, DeeplabV3+ model accuracy was related only to sex, while in GC discrimination, DeeplabV3+ model accuracy was related to BMI and tumor location, but not to other clinical information. Therefore, the DeeplabV3+ model's ability to distinguish between GC and NGC is less susceptible to the influence of clinical information. JPEG0007871484000031.jpg135170JPEG0007871484000032.jpg70170

[0131] To observe the specificity and effectiveness of the DeeplabV3+ GC diagnostic model based on tongue images, we selected patients with EC, HBPC, CRC, LC, and BC to evaluate their diagnostic value. As shown in Table 21, the specificity results of the DeeplabV3+ model for GC and other tumors show that, similar to the APINet and TransFG models, the DeeplabV3+ model is most useful for GC diagnosis, but its effectiveness differs for diagnosing tumors such as EC, HBPC, CRC, and LC. JPEG0007871484000033.jpg24170

[0132] As shown in Figure 14, the ROC and AUC of the DeeplabV3+ model for GC and other tumors show that the DeeplabV3+ model is the most effective for diagnosing GC, with an AUC of 0.801. The AUCs for diagnosing tumors such as EC, HBPC, CRC, and LC all exceed 0.5, indicating a certain level of diagnostic effectiveness. Therefore, the DeeplabV3+ model is proactive in predicting the diagnosis of various tumors, including GC.

[0133] A comprehensive analysis of the APINet, TransFG, and DeeplabV3+ models from Examples 1-3 described above revealed that Figure 15 shows the external validation probability distributions (gastric cancer) of the three tongue image models. Most cases are bilaterally distributed, meaning that the three judgments for positive and negative cases are relatively definite, and there are relatively few cases with ambiguous diagnoses between 0.41 and 0.60. This indicates that the diagnostic prediction results of the models for tumors are reliable. Figure 16 shows representative tongue images with different probabilities (gastric cancer, intersection of the three models). Without the intervention of an automated learning model, it is difficult to distinguish positive probabilities simply by intuitively observing the tongue image. Therefore, this invention provides a tumor diagnostic method based on tongue images that exhibits excellent diagnostic prediction value for a variety of tumors, including gastric cancer, and provides a scientific basis for the tongue image diagnosis theory of traditional Chinese medicine.

[0134] Since the prior art described in the above embodiment is known to those skilled in the art, a detailed explanation is omitted here.

[0135] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art can make various modifications or additions to the described specific embodiments, or substitute them in similar ways, without departing from the spirit of the invention or exceeding the scope defined in the appended claims.

[0136] Although the present invention has been described in detail and several specific embodiments have been referenced, it will be apparent to those skilled in the art that various changes or modifications are possible without departing from the spirit and scope of the invention.

[0137] The foregoing describes only preferred embodiments of the present application and is not intended to limit it. To those skilled in the art, various modifications and changes are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the scope of protection.

[0138] Any matters not described in this invention are all known technologies.

Claims

1. A tongue image acquisition module configured to acquire tongue images of a test sample, A data processing module configured to obtain the probability that a test sample belongs to the positive category through the following operations, Includes a data processing module that predicts the probability of a test sample belonging to the positive category based on discriminative features on tongue images obtained through automated learning, The aforementioned discriminative characteristics originate from the positive and negative tongue images of the paired input interactive deep learning model. Specifically, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: 1) Positive and negative features are extracted and obtained from previously acquired positive and negative tongue images. 2) Train the model with positive and negative features, and output the probability that each feature belongs to each category. 3) A tumor prediction system based on tongue images, characterized by inputting tongue images of a test sample into a trained model and outputting the probability that the test sample belongs to the positive category.

2. Step 1), which involves extracting and obtaining positive and negative features as described above, The encoder extracts the feature vectors from the image and identifies the positive features f １ and negative characteristics f ２ To output, f １ and f ２ and the combined feature f ｍ The two control vectors g are simultaneously input to the MLP of the feature selection region, and correspondingly they are input to the MLP. １ and g ２ To output, g １ activates f １ and f ２ respectively, and after activation and selection, the feature f １ ＋ and f ２ － are formed. g ２ activates f １ and f ２ respectively, and after activation and selection, the feature f １ － and f ２ ＋ are formed, obtaining two positive features f １ ＋ and f １ － and two negative features f ２ ＋ and f ２ － The system according to claim 1, characterized in that it includes the above.

3. The system according to claim 1, characterized in that the discriminative feature is derived from a single positive or negative tongue image.

4. Specifically, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: The system according to claim 3, characterized in that it cuts the tongue image of a test sample into small blocks, forms an input vector by linear mapping and adds a position index, introduces a trained deep learning model to perform feature extraction and feature fusion, outputs deep features that are advantageous for classification after selection, and obtains the probability of belonging to each category.

5. The system according to claim 1, characterized in that the discriminative features are derived from each pixel of the tongue image.

6. Specifically, the data processing module is configured to obtain the probability that a test sample belongs to the positive category by the following operation: The tongue image of the test sample is input into a trained deep learning model, and the probability that each pixel belongs to positive, negative, or background is output. The category with the highest probability is then used as the predicted category for that pixel. The system according to claim 5, characterized in that the number of pixels predicted to be positive in the test sample / (number of pixels predicted to be positive + number of pixels predicted to be negative) is the probability that the test sample belongs to the positive category.

7. To obtain tongue images of the test sample, A method for predicting a tumor based on a tongue image, comprising inputting a tongue image of a test sample into the system described in any one of claims 1 to 6 to obtain the tumor-positive probability of the test sample.

Images