Intestinal tract detection method and system based on multi-modal data

Abstract

The invention discloses an intestinal tract detection method and system based on multi-modal data, and relates to the technical field of intestinal tract detection.Biochemical data of an intestinal tract is constructed according to each dominant flora, and a plurality of life characteristics are determined according to recognition of a life event; the multi-modal data is determined according to the multiple living characteristics, the biochemical data of the intestinal tract and the lesion data, and the accuracy of the multi-modal data is improved. Therefore, a plurality of intestinal data combinations are determined based on the detection of the multi-modal data, the environmental characteristics of the intestinal tract are determined according to the recognition of each intestinal data combination, and the intestinal tract distribution diagram is determined according to the plurality of environmental characteristics and the intestinal tract image of the user; according to the method, the corresponding intestinal tract abnormal area is determined based on the intestinal tract distribution diagram, the intestinal tract abnormal event is determined according to the identification of the intestinal tract abnormal area, and the abnormal living habits of the user are determined based on the tracing of the intestinal tract abnormal event, so that the accuracy of the intestinal tract distribution diagram is improved, and the abnormal living habits of the user are conveniently identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intestinal detection, and in particular to an intestinal detection method and system based on multi-modal data. BACKGROUND

[0002] With the development of technology, in the intestinal detection of a user, more is to output a corresponding intestinal image, and a doctor predicts the intestinal condition of the user through the intestinal image. At this time, the intestinal image can be a CT image, or an image taken by an endoscope can be introduced. In the prior art, lesion data is determined based on the recognition of the intestinal image, and the abnormality of the intestine is predicted through the lesion data, and the influence of multiple life characteristics and biochemical data of the intestine is ignored, the environmental characteristics of the intestine cannot be recognized, and the abnormal life habits of the user are ignored. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art, and provides an intestinal detection method and system based on multi-modal data.

[0004] The present application provides an intestinal detection method based on multi-modal data, which comprises the following steps:

[0005] Collecting a lesion area of the intestine, determining a corresponding lesion morphology based on the synchronous detection of each lesion area, and determining lesion data of the intestine according to the area position and lesion morphology of each lesion area;

[0006] Collecting the intestinal flora distribution data, determining each dominant flora based on the detection of the intestinal flora distribution data, and constructing biochemical data of the intestine according to each dominant flora;

[0007] Collecting a life event of the user within two adjacent intestinal detection times, determining multiple life characteristics according to the recognition of the life event, and determining multi-modal data according to the multiple life characteristics, the biochemical data of the intestine, and the lesion data;

[0008] Determining multiple intestinal data combinations based on the detection of the multi-modal data, determining environmental characteristics of the intestine according to the recognition of each intestinal data combination, collecting an intestinal image of the user at the same time, and determining an intestinal distribution map according to the multiple environmental characteristics and the intestinal image of the user;

[0009] Determining a corresponding intestinal abnormal area based on the intestinal distribution map, determining an intestinal abnormal event according to the recognition of the intestinal abnormal area, and determining an abnormal life habit of the user based on the tracing of the intestinal abnormal event.

[0010] The present application provides an intestinal detection system based on multi-modal data, which is applied to the intestinal detection method based on multi-modal data described above, and comprises:

[0011] a lesion data module, configured to collect lesion regions of the intestinal tract, determine corresponding lesion morphologies based on synchronous detection of the lesion regions, and determine lesion data of the intestinal tract according to region positions of the lesion regions and the lesion morphologies;

[0012] a biochemical data module, configured to collect flora distribution data of the intestinal tract, determine each dominant flora based on detection of the flora distribution data, and construct biochemical data of the intestinal tract according to the dominant floras;

[0013] a multi-modal data module, configured to collect life events of the user within adjacent intestinal tract detection time, determine a plurality of life features according to recognition of the life events, and determine multi-modal data according to the life features, the biochemical data and the lesion data of the intestinal tract;

[0014] an intestinal tract distribution map module, configured to determine a plurality of intestinal tract data combinations based on detection of the multi-modal data, determine environmental features of the intestinal tract according to recognition of the intestinal tract data combinations, collect intestinal tract images of the user, and determine an intestinal tract distribution map according to the environmental features and the intestinal tract images of the user;

[0015] an intestinal tract abnormal event module, configured to determine corresponding intestinal tract abnormal regions based on the intestinal tract distribution map, determine an intestinal tract abnormal event according to recognition of the intestinal tract abnormal regions, and determine abnormal life habits of the user based on tracing of the intestinal tract abnormal event.

[0016] Compared with the prior art, the present application has the following advantages:

[0017] In the embodiment of the present application, the lesion regions of the intestinal tract are collected, the corresponding lesion morphologies are determined based on synchronous detection of the lesion regions, and the lesion data of the intestinal tract is determined according to the region positions of the lesion regions and the lesion morphologies; the flora distribution data of the intestinal tract is collected, each dominant flora is determined based on detection of the flora distribution data, and the biochemical data of the intestinal tract is constructed according to the dominant floras; the life events of the user within adjacent intestinal tract detection time are collected, a plurality of life features are determined according to recognition of the life events, and the multi-modal data is determined according to the life features, the biochemical data and the lesion data of the intestinal tract, which is compatible with overall consideration of the life features, the biochemical data and the lesion data of the intestinal tract, and improves the accuracy of the multi-modal data.

[0018] Therefore, based on the detection of the multi-modal data, a plurality of intestinal tract data combinations are determined, the environmental characteristics of the intestinal tract are determined according to the identification of each intestinal tract data combination, meanwhile, the intestinal tract image of the user is collected, the intestinal tract distribution map is determined according to the plurality of environmental characteristics and the intestinal tract image of the user, the corresponding intestinal tract abnormal area is determined based on the intestinal tract distribution map, the intestinal tract abnormal event is determined according to the identification of the intestinal tract abnormal area, and the abnormal living habit of the user is determined based on the tracing of the intestinal tract abnormal event, the environmental characteristics of the intestinal tract are introduced, the overall consideration of the plurality of environmental characteristics and the intestinal tract image of the user is realized, and the accuracy of the intestinal tract distribution map is improved, so as to identify the abnormal living habit of the user. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0020] Figure 2 is a flowchart of step S11 in the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0021] Figure 3 is a flowchart of step S12 in the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0022] Figure 4 is a flowchart of step S13 in the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0023] Figure 5 is a flowchart of step S14 in the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0024] Figure 6 is a flowchart of step S15 in the intestinal tract detection method based on multi-modal data in the embodiment of the application;

[0025] Figure 7 is a structural composition diagram of the intestinal tract detection system based on multi-modal data in the embodiment of the application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application.

[0027] Please refer to Figures 1 to 7 A kind of intestinal tract detection method based on multi-modal data, applied to intestinal tract detection scene;Intestinal tract detection method based on multi-modal data includes:

[0028] Step S11: Collecting the lesion area of the intestinal tract, determining the corresponding lesion morphology based on the synchronous detection of each lesion area, and determining the lesion data of the intestinal tract according to the region position and lesion morphology of each lesion area;

[0029] Step S12: Collecting the intestinal flora distribution data of the intestinal tract, determining each dominant flora based on the detection of the intestinal flora distribution data, and constructing the biochemical data of the intestinal tract according to each dominant flora;

[0030] Step S13: Collecting the life events of the user within the adjacent two intestinal tract detection times, determining the multiple life characteristics according to the recognition of the life events, and determining the multi-modal data according to the multiple life characteristics, the biochemical data and the lesion data of the intestinal tract;

[0031] Step S14: Determining the multiple intestinal tract data combinations based on the detection of the multi-modal data, determining the environmental characteristics of the intestinal tract according to the recognition of each intestinal tract data combination, and collecting the intestinal tract image of the user, and determining the intestinal tract distribution map according to the multiple environmental characteristics and the intestinal tract image of the user;

[0032] Step S15: Determining the corresponding intestinal tract abnormal area based on the intestinal tract distribution map, determining the abnormal life habit of the user according to the recognition of the intestinal tract abnormal area, and determining the abnormal life habit of the user based on the tracing of the intestinal tract abnormal event;

[0033] Reference Figure 2 In step S11, the lesion area of the intestinal tract is collected, the corresponding lesion morphology is determined based on the synchronous detection of each lesion area, and the lesion data of the intestinal tract is determined according to the region position and lesion morphology of each lesion area;

[0034] In the specific implementation process of the present application, the specific steps are as follows:

[0035] S111: When the endoscope is in the intestinal tract, the endoscope collects the internal activity video of the intestinal tract, determines multiple sub-videos according to the detection of the internal activity video, determines the corresponding lesion node based on the recognition of the multiple sub-videos, and determines the lesion area of the intestinal tract along the detection of the lesion node;

[0036] S112: Collecting multiple lesion areas, synchronously detecting each lesion area, and outputting the corresponding region schematic diagram, determining the corresponding lesion morphology according to the recognition of the region schematic diagram, and marking the region position of each lesion area;

[0037] S113: In each lesion area, the data combination of the lesion area is determined based on the region position and lesion morphology of each lesion area, and the lesion data of the intestinal tract is determined according to the data combination of each lesion area.

[0038] In the embodiments of the present application, when the endoscope is in the intestinal tract, the endoscope collects internal activity video of the intestinal tract, determines a plurality of sub-videos according to detection of the internal activity video, determines corresponding lesion nodes based on recognition of the plurality of sub-videos, detects along the lesion nodes, and determines a lesion area of the intestinal tract, which is compatible with overall consideration of recognition of the plurality of sub-videos and guarantees accuracy of the corresponding lesion nodes.

[0039] At this time, when the endoscope is inserted into the intestinal tract (such as the colon or stomach) through the anus or mouth, the doctor will slowly advance the endoscope and observe the real-time picture on the display; the camera of the endoscope will continuously capture images of the mucosa inside the intestinal tract, which are transmitted to a recording device (such as an endoscope host, a computer or a dedicated recording device) in the form of a video stream; the video content usually includes: the overall appearance of the intestinal mucosa (color, luster, fold shape); dynamic changes in the intestinal tract (such as peristalsis of the intestinal wall, movement of the contents); any visible abnormal structure (such as polyps, ulcers, bleeding points, lumps, etc.).

[0040] The internal activity video is usually very long (up to several tens of minutes) and contains a large number of normal mucosa pictures; directly processing the entire long video is inefficient and will introduce a large amount of redundant information; therefore, it is necessary to first extract the segments containing valuable information (especially lesion information), i.e. "sub-videos", from the video; at the same time, significant changes between image frames are detected, which correspond to rapid movement of the endoscope view or capture of dynamic lesions (such as bleeding); computer vision algorithms (such as color histogram analysis, texture feature extraction such as LBP, Haralick feature) are used to identify regions that are significantly different from the surrounding normal mucosa; for example, red regions (indicating inflammation or vascular abnormalities), pale color regions (indicating ischemia or lipomas) or rough, granular textures (indicating precancerous lesions) are detected; a deep learning model (such as a convolutional neural network CNN) is trained to learn the visual features of normal mucosa and common lesions (such as polyps, ulcers); the video frames are input into the model, which will output the probability of each frame containing lesions or directly frame the suspicious regions; according to the output of the model, the continuous frames with high probability or containing suspicious frames are combined into "sub-videos"; multiple "sub-video" segments are output, each of which is considered to contain one or more potential lesions or regions worthy of attention.

[0041] In the "sub-video" that has been screened, a specific "point" or small area suspected of being a lesion is further pinpointed; this "point" or area is the "lesion node", which is the basis for subsequent detailed analysis; a more accurate target detection model (such as YOLO, FasterR-CNN) is run on each frame of the "sub-video" to directly frame the approximate location of the lesion; if the lesion has certain dynamic changes (such as slight movement) in the "sub-video", a tracking algorithm (such as optical flow method, feature-based tracking) can be used to stably locate the lesion between consecutive frames; the most clear and typical frames (key frames) of the lesion are extracted from each "sub-video", labeled by AI model or manually, to determine the center point or bounding box of the lesion; the coordinates of this center point or bounding box are the "lesion node"; one or more "lesion nodes" are output for each sub-video, usually represented as image coordinates such as (x, y) or video timestamp such as 10 minutes 38 seconds.

[0042] The "lesion node" is only a rough location reference; in order to more accurately describe the lesion, more detailed detection needs to be performed around this node to determine the actual range and boundary of the lesion, forming a "lesion area"; focus on the local image area containing the "lesion node" (for example, take a square area with a side length of 100 pixels centered on the node), and perform more detailed analysis; at the same time, use Canny edge detection, region growing algorithm, etc. to try to outline the boundary between the lesion and the normal mucosa; more detailed texture and color statistics are performed within the local area to quantify the characteristics of the lesion; if other data (such as NBI enhanced images, OCT images) are collected synchronously during endoscopy, these data can be corresponded to the "lesion node" at this time, providing more rich information to assist in determining the boundary; for example, NBI makes the blood vessel pattern clearer, which helps to judge the benignity or malignancy of the polyp; OCT can provide sub-surface structure information; an explicit "lesion area" is output, usually represented as a pixel-level mask image, where the lesion part is marked (for example, white), and the non-lesion part is black; or a bounding box, but usually the mask can provide more accurate range information.

[0043] Further, multiple lesion areas are collected, each lesion area is synchronously detected, and a corresponding area diagram is output, the corresponding lesion morphology is determined according to the recognition of the area diagram, and the area position of each lesion area is marked, which is compatible with the overall consideration of the recognition of the area diagram, and ensures the accuracy of the corresponding lesion morphology.

[0044] At this time, multiple lesion areas are collected, and for each collected lesion area, not only data of one modality is used for analysis, but also data of multiple modalities (if available) are combined for comprehensive judgment; through image processing and AI algorithms, multiple modalities of the lesion area are deeply analyzed to extract more abundant features.

[0045] For white light images (WLI), color analysis: calculate average hue, saturation, brightness, and compare with surrounding normal tissue; texture analysis: calculate gray level co-occurrence matrix (GLCM) features, local binary pattern (LBP), etc., to analyze surface roughness and graininess; shape analysis: calculate area, perimeter, circularity, convex hull, and other geometric parameters; for NBI (narrow band imaging) images, blood vessel pattern analysis: detect the density, direction, and morphology (such as grid-like, twisted, irregular) of microvessels; surface structure analysis: enhance the fine structure of the mucosal surface.

[0046] For OCT (optical coherence tomography) data: hierarchical structure analysis: check the thickness, continuity, and presence or absence of abnormal stratification of each layer of the intestinal wall (mucosal layer, submucosal layer, etc.); abnormal structure detection: identify changes in hierarchical structure caused by tumor hyperplasia, ulcers, and inflammatory infiltration; endoscopic ultrasound (EUS) data (if applicable): wall layer analysis: more clearly display the hierarchical structure of the intestinal wall; external structure analysis: assess the condition of surrounding lymph nodes, blood vessels, and adjacent organs; at the same time, use deep learning models (such as CNN) to directly train the fused multi-modal features to identify lesion types or assess malignancy.

[0047] The detection results of the lesion area are presented in an intuitive and visual manner to facilitate the understanding and judgment of doctors; visualization of detection results: mark lesion boundaries: outline the precise contours of the lesion area with different colors or lines; highlight features: use color coding or superimposed graphics to represent specific features such as color abnormal areas, texture abnormal areas, and blood vessel patterns; add annotations: label key measurements (such as size, shape parameters), preliminary judgments (such as "suspected inflammation" "suspected tubular adenoma"); different modalities of detection results can be superimposed on the same schematic diagram, or multiple view switching can be provided; format: can be static images (such as PNG, JPG), or interactive interfaces that allow doctors to zoom, pan, and switch views.

[0048] Based on the comprehensive information of lesion boundaries, colors, textures, vascular patterns, hierarchical structures, etc. presented on the regional sketch map; according to medical knowledge and pre-set classification criteria, the sketch map is analyzed; the lesion morphology is classified into pre-defined categories; for example: polyp morphology: can be divided into uplift type (with peduncle, sub-peduncle, no peduncle), flat type, concave type; according to the surface structure, it can be divided into tubular, villous, tubular villous; inflammation morphology: can be divided into erosion, ulcer (can be divided into benign ulcer, malignant ulcer characteristics according to size, depth, edge morphology), hyperemia and edema; tumor morphology: can be divided into lump type, infiltration type, etc.; one or more morphological description labels are given to each lesion area.

[0049] Mark the location of the region, mark the specific location of the lesion area on the overall intestinal tract map or video timeline, facilitate subsequent navigation and recording; record the pixel coordinates of the lesion center point or boundary box in the original video frame; add the spatial position information of each lesion area.

[0050] Therefore, in each lesion area, the data combination of the lesion area is determined based on the regional position and lesion morphology of each lesion area, and the lesion data of the intestinal tract is determined according to the data combination of each lesion area, which is compatible with the overall consideration of the data combination of each lesion area, and ensures the accuracy of the lesion data of the intestinal tract.

[0051] At this time, two key information determined in S112 are used: the accurate position of each lesion area (such as coordinates, intestinal tract anatomical segment description) and the identified lesion morphology (such as “mucosal hyperemia and erosion” “pedunculated tubular adenoma” and the like); the “data combination” here refers to integrating the position information and morphology information of the lesion, as well as other related features extracted from S111 / S112, into a structured data structure; this structure can be a dictionary, a JSON object, a database record, or a feature vector.

[0052] In addition to position and morphology, this data combination also includes: lesion size: diameter or area estimated by image analysis; lesion color: such as red, yellow, white, etc., quantified as RGB value or chroma value; surface structure: such as flat, uplift, concave, granular, nodular, etc.; boundary features: such as clear, blurred, irregular, etc.; vascular pattern: such as regular, disorderly, no blood vessels, etc. (especially in NBI mode); relationship with surrounding tissue: such as whether there is a peduncle, whether it is infiltrated, etc.; video frame number or timestamp: for tracking; the description of a single lesion area is converted from scattered information (a coordinate, a text description) to a unified, structured data unit, which is convenient for computer processing, storage and subsequent analysis.

[0053] The data combination of all single lesion areas is combined to form a summary data of the lesions found in the intestinal examination; this can be regarded as a list, array or database table, where each item is a "data combination of lesion area" generated in sub-step 1; the "intestinal lesion data" is a structured report or data set containing the following information: total number of lesions: the number of lesions found in this examination; lesion list: contains the data combination of all single lesion areas; lesion distribution profile: for example, where are the lesions mainly distributed (sigmoid colon, descending colon, transverse colon, etc.), whether there is an aggregation of lesions in a specific segment; lesion severity assessment: a preliminary quantitative assessment based on lesion morphology (such as adenoma level, inflammation severity); lesions that need special attention: for example, large adenomas, suspicious lesions, etc.; integrate the information scattered in various lesion areas into a complete lesion report that can provide the doctor with a comprehensive understanding of the patient's intestinal condition; this report is the basis for subsequent steps (such as analysis combined with flora data, lifestyle data).

[0054] Reference Figure 3 In step S12, the flora distribution data of the intestine is collected, the dominant flora is determined based on the detection of the flora distribution data, and the biochemical data of the intestine is constructed according to the dominant flora;

[0055] In the specific implementation of the present application, the specific steps are as follows:

[0056] S121: performing flora detection on the feces excreted by the intestine, determining the flora distribution areas in the feces based on the flora detection of the feces, and predicting the flora distribution data of the intestine according to the area positions of the flora distribution areas and the flora distribution conditions of the flora distribution areas;

[0057] S122: data detection of the flora distribution data, output of the species and relative proportions of each flora, determination of the gradient distribution map of the flora based on the species and relative proportions of each flora, determination of the dominant flora based on the traversal of the gradient distribution map of the flora; collection of the dominant flora, determination of a plurality of dominant flora combinations based on the cross combination of the dominant flora, and construction of the biochemical data of the intestine based on the plurality of dominant flora combinations.

[0058] In the embodiment of the present application, the flora detection is performed on the feces excreted by the intestine, the dominant flora is determined based on the flora detection of the feces, and the biochemical data of the intestine is constructed according to the dominant flora, which is compatible with the overall consideration of the flora detection of the feces and ensures the accuracy of each flora distribution area in the feces.

[0059] At this time, the fecal sample is collected and the microbiota is detected using molecular biology techniques (such as 16S rRNA gene sequencing, metagenomic sequencing, metabolomics analysis, etc.) to identify and quantify the types and abundances of microorganisms present in the sample; for example, 16S rRNA sequencing can identify to the genus or even species level, while metagenomic sequencing can provide more comprehensive gene function information.

[0060] The microbiota information detected in the fecal sample is inferred back to their distribution areas in the intestinal tract; this is usually based on the typical distribution patterns of known microbiota in different regions of the intestinal tract; for example, we know that some butyrate-producing bacterial genera (such as Faecalibacterium) are usually more abundant in the colon, while some opportunistic pathogens (such as Escherichia / Shigella) are more common in inflammatory areas; this is more like a model or knowledge-based inference, rather than a direct spatial positioning.

[0061] In combination with the inferred microbiota distribution areas (such as "colon dominant area" "ileum tendency area") and the specific microbiota composition and abundance (group distribution) in these areas, a predictive description or model of the overall intestinal microbiota distribution is generated; this is a spatialized microbiota distribution map (although it is indirect based on fecal data), or a list containing the main microbiota in each area and their relative abundance.

[0062] Specifically, the patient submits a fecal sample; the laboratory detects the presence of Bacteroidetes, Firmicutes, Proteobacteria, etc. major phyla, and specific genera using 16S rRNA sequencing technology.

[0063] Based on the high abundance of Faecalibacterium and the relatively low abundance of Escherichia / Shigella, the system infers that the microbiota structure in the colon region is relatively healthy, and there is no significant overgrowth of inflammation-related microbiota in the ileum or cecum region; although the feces mix the microbiota of the entire intestinal tract, the regional tendency can be judged by the abundance difference and known patterns.

[0064] The system predicts the patient's intestinal microbiota distribution data as follows: "colon region: dominated by Bacteroidetes and Firmicutes, high abundance of Faecalibacterium, indicating a healthy state; ileum region: slightly higher relative proportion of Proteobacteria, Escherichia / Shigella detectable, indicating a slight inflammation or translocation risk; cecum region: moderate proportion of Bacteroidetes" this is the predicted intestinal microbiota distribution data.

[0065] Further, data detection is performed on the flora distribution data, and the types and relative proportions of each flora are output, the gradient distribution map of the flora is determined according to the types and relative proportions of each flora, and each dominant flora is determined based on traversal of the gradient distribution map of the flora; each dominant flora is collected, a plurality of dominant flora combinations are determined according to cross combinations of each dominant flora, and the biochemical data of the intestinal tract is constructed according to the plurality of dominant flora combinations, which comprehensively considers the cross combinations of each dominant flora and guarantees the accuracy of the plurality of dominant flora combinations.

[0066] At this time, the predictive flora distribution data generated by S121 is standardized and quantified; all identified flora (or selected important flora) are listed, and their percentage or relative abundance in the respective inferred area (or the whole) is calculated.

[0067] Specifically, based on the prediction data of S121, the specific relative proportions are output: “colon region: Bacteroidetes 50%, Firmicutes 45%, Proteobacteria 3%, Faecalibacterium 15%; ileum region: Bacteroidetes 30%, Firmicutes 50%, Proteobacteria 15%, Escherichia / Shigella 10%; cecum region: Bacteroidetes 40%, Firmicutes 55%, Proteobacteria 3%”.

[0068] The relative proportions of flora in different regions (or from the proximal to the distal end of the intestinal tract) are visualized to form a “gradient map”; this can directly show the trend of flora composition changing with the position of the intestinal tract; for example, a column chart or a heat map can be drawn, with the horizontal axis representing the position of the intestinal tract (such as the ileum, cecum, ascending colon, transverse colon, descending colon, sigmoid colon), and the vertical axis representing the relative proportion of the main flora.

[0069] Specifically, a heat map is generated, with the horizontal axis being the ileum, cecum, ascending colon, transverse colon, descending colon, and sigmoid colon, and the vertical axis being Bacteroidetes, Firmicutes, and Proteobacteria; the color depth of the heat map represents the relative proportion; it can be seen that the proportion of Bacteroidetes gradually increases from the ileum to the colon, the proportion of Firmicutes is relatively stable, and the proportion of Proteobacteria is relatively deep in the ileum region.

[0070] By analyzing the gradient distribution map, dominant flora in a specific region or the whole intestinal tract is identified; generally, “dominant flora” refers to flora with a relative abundance exceeding a certain threshold (such as 10%, 20%); global dominant flora (dominant in the whole intestinal tract) and local dominant flora (dominant only in a specific region) can be identified.

[0071] The identified dominant flora are combined in pairs or more to form "dominant flora combinations"; then, based on the known microbe-metabolism association knowledge base, the metabolites produced or biochemical pathways involved in these combinations are inferred, thereby constructing "biochemical data" reflecting the biochemical state of the intestinal tract; for example, certain flora combinations are associated with the production of short-chain fatty acids (such as butyric acid), while other combinations are associated with the production of inflammation-related metabolites (such as indole).

[0072] Specifically, the gradient distribution map is traversed to determine the dominant flora: "global dominant flora: Firmicutes (occupies more than 40% in all regions); local dominant flora: Bacteroidetes (occupies more than 40% in the colon region), and Proteobacteria (occupies 15% in the ileum region, although it does not reach the global dominance, but is significant in this region)". Assuming we only focus on the highest proportion, it is Firmicutes and Bacteroidetes.

[0073] Specifically, the dominant flora are collected: Firmicutes and Bacteroidetes; the dominant flora combinations are determined: combination 1: Firmicutes + Bacteroidetes; combination 2: Firmicutes (ileum) + Proteobacteria (ileum); and the biochemical data of the intestinal tract are constructed:

[0074] For combination 1 (Firmicutes + Bacteroidetes): it is known that these two phyla contain many butyrate-producing bacterial genera; therefore, it can be inferred that the "intestinal butyrate production potential is high" biochemical data item; for combination 2 (Firmicutes (ileum) + Proteobacteria (ileum)): the increase of Proteobacteria in the ileum region is usually associated with inflammation; Firmicutes is also dominant in this region; this suggests that "there are inflammation-related metabolic changes in the ileum region", such as "inflammation-related metabolite (such as indole) production potential increases" or "intestinal barrier function is impaired" biochemical data items; the final intestinal biochemical data includes: butyrate production potential (high), inflammation-related metabolite production potential (ileum region, moderate), intestinal flora diversity (based on the number and evenness of dominant flora, moderate), etc.

[0075] By analyzing the fecal samples, the spatial distribution of intestinal flora is inferred, the flora composition is quantified, the dominant flora and their combinations are identified, and finally these microbial information is converted into biochemical state description more directly related to host health; this step lays the foundation for the subsequent fusion of flora data with morphological data, lifestyle data to form a comprehensive intestinal environment characteristics; for example, combined with the inflammation area found in S11, if S12 shows that the flora combination near this area tends to produce inflammation-related metabolites, it can more strongly support the judgment that there is inflammation in this area.

[0076] Reference Figure 4In step S13, the life events of the user within the adjacent two intestinal detection times are collected, a plurality of life features are determined according to the recognition of the life events, and a plurality of modal data are determined according to the plurality of life features, biochemical data of the intestine and pathological data;

[0077] In the implementation of the present application, the specific steps are:

[0078] S131: Collect the intestinal detection time of the user, determine the adjacent two intestinal detection times according to the current time and the intestinal detection time, and trace the life habits of the adjacent two intestinal detection times to output the life events of the user within the adjacent two intestinal detection times;

[0079] S132: Collect the life events, determine a plurality of sub-life items according to the analysis of the life events, determine corresponding life behaviors according to the detection of the plurality of sub-life items, and determine corresponding life features based on the recognition of the life behaviors;

[0080] S133: Collect a plurality of life features, determine first modal data according to the plurality of life features and biochemical data of the intestine, determine second modal data according to the plurality of life features and pathological data, and determine modal data based on the cross-training of the first modal data and the second modal data.

[0081] In the embodiment of the present application, the intestinal detection time of the user is collected, the adjacent two intestinal detection times are determined according to the current time and the intestinal detection time, and the life habits of the adjacent two intestinal detection times are traced to output the life events of the user within the adjacent two intestinal detection times, which is compatible with the overall consideration of the current time and the intestinal detection time, and ensures the accuracy of the adjacent two intestinal detection times.

[0082] At this time, the system first needs to record the specific time point of each intestinal detection (whether it is an imaging examination, a fecal detection, etc.) of the user; for example, the user performed the first detection on January 1, 2024, and the second detection on June 1, 2024; the system will find all consecutive two detection times according to the time sequence; in the above example, it is this pair of (January 1, 2024, June 1, 2024).

[0083] For each pair of adjacent detection times, the system needs to trace the life events that occurred to the user during this time interval; this usually requires combining multiple sources, such as user-initiated inputs, wearable device data, third-party application data (e.g., diet tracking apps, exercise apps), etc.; the goal of the tracing is to reconstruct as comprehensively as possible the changes in the user's lifestyle during this time; for example, between January 1 and June 1, the user recorded events such as "started regular exercise", "changed diet structure (increased vegetarianism)", "experienced an infection", "increased work stress", etc.

[0084] These life events traced are sorted and output to form a list of life events corresponding to a time period; for example, for the time period (January 1, 2024, June 1, 2024), the output is: {"started regular jogging (3 times a week)", "increased vegetable proportion in diet by 50%", "had a cold for a week in February", "started taking probiotic supplements in March"}.

[0085] Further, the life events are collected, a plurality of sub-life items are determined according to the analysis of the life events, a corresponding life behavior is determined according to the detection of the plurality of sub-life items, and a corresponding life feature is determined based on the recognition of the life behavior, which is compatible with the overall consideration of the recognition of the life behavior, and ensures the accuracy of the corresponding life feature.

[0086] At this time, the life event list output by S131 is used as input; for example, the collected life events are: {"started regular jogging (3 times a week)", "increased vegetable proportion in diet by 50%", "had a cold for a week in February", "started taking probiotic supplements in March"}; each macro life event is decomposed into more specific and quantifiable sub-items; this requires semantic understanding and classification of life events; for example: the event "started regular jogging (3 times a week)" can be parsed into sub-items: {"exercise type: jogging", "exercise frequency: 3 times a week", "exercise start time: after January 1"}; the event "increased vegetable proportion in diet by 50%" can be parsed into sub-items: {"diet category: vegetables", "diet change: increased proportion by 50%", "change start time: after January 1"}; the event "had a cold for a week in February" can be parsed into sub-items: {"health status: infection", "infection type: cold", "duration: one week", "occurrence time: February"}; the event "started taking probiotic supplements in March" can be parsed into sub-items: {"drug / supplement: probiotics", "start taking time: March"}.

[0087] Verify or quantify the parsed sub-life items to confirm that the user has indeed performed the behavior; for example, confirm that the user has indeed jogged 3 times a week through the sports APP data, convert the qualitative description into quantitative data; for example, "increase the proportion of vegetables in diet by 50%" calculates the percentage change of actual vegetable intake through the data of the diet record APP; output life behavior: for example, the confirmed life behavior is: { "jogging frequency per week: 3", "daily vegetable intake: increase about 100g", "recent infection history: yes (cold)", "probiotic taking status: continuous taking"}.

[0088] Extract the detected and confirmed life behaviors into representative life characteristics; these characteristics should be relatively stable and can reflect the user's long-term or stage life habits; for example: the life behavior "jogging frequency per week: 3" can be identified as the life characteristic "regular moderate intensity exercise habit"; the life behavior "daily vegetable intake: increase about 100g" can be identified as the life characteristic "high fiber diet tendency"; the life behavior "recent infection history: yes (cold)" can be identified as the life characteristic "recent immune fluctuation / stress event"; the life behavior "probiotic taking status: continuous taking" can be identified as the life characteristic "active intestinal microecological intervention".

[0089] Therefore, multiple life characteristics are collected, the first modal data is determined according to the multiple life characteristics and the biochemical data of the intestinal tract, the second modal data is determined according to the multiple life characteristics and the lesion data, and the multi-modal data is determined based on the cross-training of the first modal data and the second modal data, which is compatible with the overall consideration of the cross-training of the first modal data and the second modal data, and ensures the accuracy of the multi-modal data. At the same time, it is compatible with the overall consideration of multiple life characteristics, biochemical data of the intestinal tract and lesion data, and improves the accuracy of multi-modal data.

[0090] At this time, the life characteristic set output by S132 is used as input; for example, the collected life characteristics are: { "regular moderate intensity exercise habit", "high fiber diet tendency", "recent immune fluctuation / stress event", "active intestinal microecological intervention"}.

[0091] Determine the first modality data (lifestyle features + biochemical data): input multiple lifestyle features + S12 generated intestinal biochemical data (e.g., dominant flora combination, biochemical function assessment: high potential for butyrate production, low level of inflammation-related metabolites, etc.), and perform correlation analysis between lifestyle features and biochemical data; for example, analyze the relationship between "high fiber diet tendency" and "high potential for butyrate production"; the relationship between "actively intervening in intestinal microecology" and "increased abundance of specific probiotics"; the relationship between "recent immune fluctuations / stress events" and "inflammation-related metabolite levels"; output: first modality data; this can be a structured data set containing lifestyle features and corresponding biochemical indicators, as well as the correlation strength or pattern between them; for example: {lifestyle feature: "high fiber diet tendency", biochemical indicator: "potential for butyrate production", correlation pattern: "positive correlation"}; {lifestyle feature: "actively intervening in intestinal microecology", biochemical indicator: "relative proportion of Bifidobacterium", correlation pattern: "positive correlation"}; {lifestyle feature: "recent immune fluctuations / stress events", biochemical indicator: "potential for inflammation-related metabolite production", correlation pattern: "potential positive correlation (requires time series analysis)"}; the first modality data reflects how lifestyle habits affect the composition and function of intestinal microorganisms.

[0092] Determine the second modality data (lifestyle features + lesion data): input multiple lifestyle features + S11 generated intestinal lesion data (e.g., lesion location: middle sigmoid colon; lesion morphology: mild inflammation; lesion location: beginning of transverse colon; lesion morphology: adenoma; severity score, etc.); perform correlation analysis between lifestyle features and lesion data; for example, analyze the relationship between "high fiber diet tendency" and "degree of inflammation in the sigmoid colon"; the relationship between "regular moderate-intensity exercise habits" and "adenoma detection rate / size"; the relationship between "recent immune fluctuations / stress events" and "inflammatory area range"; output the second modality data; also a structured data set containing lifestyle features and corresponding lesion information, as well as the correlation pattern between them; for example: {lifestyle feature: "high fiber diet tendency", lesion information: "sigmoid colon inflammation", correlation pattern: "potential negative correlation (fiber helps to reduce inflammation)"}; {lifestyle feature: "regular moderate-intensity exercise habits", lesion information: "transverse colon adenoma", correlation pattern: "no significant correlation or potential negative correlation (exercise reduces the risk of certain intestinal carcinogenesis)"}; {lifestyle feature: "recent immune fluctuations / stress events", lesion information: "range of sigmoid colon inflammation", correlation pattern: "potential positive correlation (stress exacerbates inflammation)"}; the second modality data reflects how lifestyle habits affect the occurrence, development, or state of intestinal lesions.

[0093] Determine multi-modal data based on cross-training of first and second modal data, input first modal data + second modal data, fuse data of two modalities to obtain a more comprehensive and in-depth understanding of the intestinal state; this involves: combining relevant features in the first and second modalities to create new, more comprehensive features; for example, combining "high-fiber diet inclination" with "butyrate production potential" and "sigmoid colon inflammation" information to create a "fiber diet-colon flora function-inflammation state" comprehensive feature; use machine learning or statistical models to consider life characteristics, biochemical data and lesion data to predict or explain a certain intestinal state or risk; for example, train a model to predict the probability of recurrence of intestinal inflammation within a year, input features include changes in lifestyle, flora function state and current / historical lesion information; use life characteristics, biochemical data and lesion data as nodes, and the relationship between them as edges to build a knowledge graph to visually show the complex interaction between the three; output multi-modal data; this is the final fused data representation, which is no longer single-source information, but integrates comprehensive information about user lifestyle, intestinal microbial state and intestinal morphological lesions; this multi-modal data can be used for more accurate health assessment, risk prediction, personalized intervention recommendations or as input for downstream artificial intelligence models.

[0094] Reference Figure 5 In step S14, determine a plurality of intestinal data combinations based on the detection of multi-modal data, determine the environmental characteristics of the intestine according to the identification of each intestinal data combination, and collect the intestinal image of the user, and determine the intestinal distribution map according to the plurality of environmental characteristics and the intestinal image of the user;

[0095] In the specific implementation process of the present application, the specific steps are:

[0096] S141: Collect multi-modal data, perform data detection on the multi-modal data, and output a plurality of data categories, divide the multi-modal data based on each data category to determine a plurality of intestinal data combinations, and identify the plurality of intestinal data combinations to determine the environmental characteristics of the intestine;

[0097] S142: Perform CT detection on the user's intestine and output the user's intestinal image, and determine a plurality of image regions based on the division of the user's intestinal image;

[0098] S143: Mark the region position and region morphology of the plurality of image regions; construct an intestinal distribution map according to the environmental characteristics of the intestine, the region position and the region morphology of the plurality of image regions.

[0099] In the embodiments of the present application, multi-modal data is collected, data detection is performed on the multi-modal data, and multiple data categories are output. Based on each data category, the multi-modal data is divided to determine multiple intestinal data combinations. The intestinal data combinations are identified to determine the environmental characteristics of the intestine. The environmental characteristics of the intestine are introduced.

[0100] At this time, multi-modal data (for example, a vector Z that integrates lifestyle, biochemical indicators, and lesion information, or a data set containing multiple feature tables) is collected. Data detection is performed on the multi-modal data to check the quality and integrity of the data. For example, it is checked whether there are missing values, abnormal values (such as a biochemical indicator far exceeding the normal range), whether the data is within a reasonable range, and the like. This helps to ensure the accuracy of subsequent analysis. The multi-modal data is classified according to the source or nature of the data. For example, it can be divided into: output multiple data categories: the multi-modal data is classified according to the source or nature of the data. For example, it can be divided into: lifestyle-related data: such as exercise frequency, diet type, stress level, sleep duration, and the like; biochemical indicator-related data: such as fecal calprotectin (FCP), short-chain fatty acid (SCFA) concentration, intestinal flora diversity index, and the like; lesion-related data: such as the location, size, shape, and number of adenomas detected previously, the severity and range of chronic inflammation, and the like; time-related data: such as the time points of data collection and the time interval between two detections, and the like.

[0101] According to the results of the previous classification, the original multi-modal data is split into different data subsets or views. For example: combination 1: contains all lifestyle data and corresponding biochemical indicator data (for example, exercise frequency, ketogenic diet status vs FCP, butyrate level); combination 2: contains all lifestyle data and corresponding lesion data (for example, stress level, sleep duration vs adenoma number, inflammation score); combination 3: contains all biochemical indicator data and corresponding lesion data (for example, FCP, butyrate level vs adenoma characteristics, inflammation range); combination 4: contains time data combined with any other category of data for analyzing time trends.

[0102] Each data combination is analyzed to extract features that can reflect the overall or specific aspects of the state of the intestine. This involves: statistical analysis: calculating mean, standard deviation, correlation, and the like; machine learning models: using classification, regression, or clustering algorithms to identify patterns; for example, using combination 1 to train a model to predict the "intestinal inflammation risk" score; using combination 2 to train a model to predict the "adenoma occurrence risk" score; feature engineering: creating new, more meaningful features; for example, calculating a "stress-sleep composite index" stress level (8 - average sleep duration)), or a "diet-flora balance index" (butyrate / acetate ratio).

[0103] The final output is a series of "gut environment features", such as: overall gut inflammation risk score: 78 / 100; specific area (such as right colon) adenoma risk score: 45 / 100; gut flora imbalance index: 1.2 (higher than normal value 1.0). Intestinal barrier function score: 65 / 100 (based on FCP and other indicators); stress state on the intestinal effect score: 70 / 100 (based on stress, sleep, inflammation indicators).

[0104] Further, the user's intestine is detected by CT, and the user's intestine image is output, and the plurality of image regions are determined based on the division of the user's intestine image, which is compatible with the overall consideration of the division of the user's intestine image, and ensures the accuracy of the plurality of image regions.

[0105] At this time, the image data of the user's intestine is obtained using computed tomography (CT) technology; this usually requires the user to prepare the intestine (such as enema, taking contrast agent) to obtain clearer images; process the original CT data to generate a series of two-dimensional or three-dimensional intestinal images; these images can show the structure of the intestine, wall thickness, presence or absence of lumps, presence or absence of inflammation signs (such as thickening of the intestinal wall, blurring of the surrounding fat), etc.

[0106] The continuous intestinal images are divided into different meaningful parts; this can be achieved by: dividing by anatomical structure: dividing the intestine into standard areas such as cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectum, etc.; dividing by function or blood flow area: for example, according to the blood supply artery (ileocolic artery, middle colic artery, left colic artery, superior rectal artery); dividing by image features: using image processing algorithms (such as threshold segmentation, edge detection, region growing, machine learning segmentation model) to automatically or semi-automatically divide the image into "suspected lesion area", "normal mucosa area", "intestinal wall area", "perienteric fat area", etc.; dividing by lesion location: if there is a known lesion (such as adenoma in S133), mark these areas and their surrounding range on the image; finally, a series of discrete "image regions" are obtained, each with clear boundary and location information.

[0107] Therefore, the region position and region morphology of the plurality of image regions are marked; the intestinal distribution map is constructed according to the environmental characteristics of the intestine, the region position and region morphology of the plurality of image regions, which is compatible with the overall consideration of the environmental characteristics of the intestine, the region position and region morphology of the plurality of image regions, and ensures the accuracy of the intestinal distribution map.

[0108] At this time, the location of each image region in the intestinal tract is accurately recorded; for example, using anatomical markers (such as "sigmoid colon 15 cm from the anus"), or using a coordinate system such as (x, y, z) coordinates in a three-dimensional reconstructed image), or using distance and angle relative to a certain reference point (such as the appendix opening); describe the shape and appearance characteristics of each image region; for example: shape: round, oval, irregular, linear, sheet; size: diameter, area, volume of the region.

[0109] Boundary: clear, fuzzy, lobulated, spiculated; internal features: uniform, non-uniform, with or without calcification, with or without necrosis; relationship with surrounding tissues: whether it infiltrates the intestinal wall, whether it invades the surrounding fat, whether it has lymph node enlargement; (for normal regions): whether the mucosa is smooth, whether there is hyperemia, edema; these marker information gives each image region a detailed "geographical" and "appearance" label.

[0110] The "intestinal environment characteristics" obtained in S141 are associated with the "image region position, shape information" obtained in S143 to generate a visual or structured "intestinal distribution map"; regions with similar environmental characteristics, or environmental characteristics associated with image morphology, are corresponded in space; for example: if the "right hemicolon adenoma risk score is high", and "boundary clear, irregular shape, soft tissue shadow with mild lobulation" is found in the right hemicolon on the image, then this image region can be marked as "high-risk adenoma suspicious area" and highlighted on the distribution map; if the "intestinal barrier function score is low", and "mild thickening of the intestinal wall, increased density of the surrounding fat" is found in multiple regions on the image, then these regions can be marked with different colors or textures on the distribution map, indicating "potential inflammation or barrier damage area"; if the "dysbiosis index is high", although there is no direct corresponding image morphology, it can be used as a background information to represent the whole intestine in a "unstable state" on the distribution map with the whole color or annotation.

[0111] By first extracting abstract "intestinal environment characteristics" from multi-modal data (S141), then obtaining specific imaging information (S142), and finally combining the two and positioning and visualizing on the intestinal anatomic structure (S143), the "intestinal distribution map" is finally generated; this distribution map goes beyond a single image or biochemical result, providing an integrated multi-information, spatialized, visualized intestinal health assessment tool, which helps to more accurately understand the intestinal state, locate high-risk areas and guide subsequent management and treatment.

[0112] Reference Figure 6In step S15, a corresponding intestinal abnormal region is determined based on the intestinal distribution map, an intestinal abnormal event is determined according to identification of the intestinal abnormal region, and an abnormal living habit of the user is determined based on tracing of the intestinal abnormal event;

[0113] In the implementation of the present application, the specific steps are as follows:

[0114] S151: Collecting an intestinal distribution map, performing abnormal detection on the intestinal distribution map, and outputting corresponding abnormal detection nodes, determining corresponding abnormal data according to detection of each abnormal detection node, and determining a corresponding intestinal abnormal region according to the node position of each abnormal detection node, corresponding abnormal data, and the intestinal distribution map;

[0115] S152: Determining a corresponding abnormal morphology based on identification of the intestinal abnormal region, determining a corresponding intestinal abnormal event according to the abnormal morphology, the intestinal distribution map, and the age of the user, and tracing the intestinal abnormal event to collect corresponding tracing data; determining corresponding living behavior data based on screening of the tracing data, and determining an abnormal living habit of the user according to the living behavior data, behavior records of the user, and the intestinal abnormal event.

[0116] In the embodiment of the present application, the intestinal distribution map is collected, the intestinal distribution map is subjected to abnormal detection, and corresponding abnormal detection nodes are outputted, corresponding abnormal data is determined according to detection of each abnormal detection node, and a corresponding intestinal abnormal region is determined according to the node position of each abnormal detection node, corresponding abnormal data, and the intestinal distribution map, which is compatible with the overall consideration of the node position of each abnormal detection node, corresponding abnormal data, and the intestinal distribution map, and ensures the accuracy of the corresponding intestinal abnormal region.

[0117] At this time, the system needs to obtain the intestinal distribution map generated in the previous step (such as S14); this distribution map is a comprehensive data structure that not only contains intestinal morphology information (such as the thickness of the intestinal wall in different regions, whether there is filling defect, whether there is a mass, etc.) from CT images, but also superimposes intestinal environment characteristics (such as the degree of dysbiosis, the level of inflammation indicators, and the intestinal motility score) obtained from multi-modal data (such as fecal flora detection, metabolite analysis, and living habit records), and these information is usually presented in a spatialized manner, that is, each feature is associated with a specific location on the intestine.

[0118] Abnormality detection on the intestinal map: the system will use pre-set rules or machine learning models to scan the intestinal map; these rules / models will focus on which intestinal features deviate from the "normal" range; for example: based on threshold: intestinal wall thickness exceeds a certain threshold (such as 3mm) is marked as abnormal; based on statistical model: the risk value of dysbiosis in a certain area is much higher than the user's own historical average level or the reference range of healthy people; based on pattern recognition: typical polyp or mass morphology is detected in the CT image; based on spatial distribution: a certain abnormal feature (such as elevated inflammation indicators) presents discontinuous aggregation in space; when an abnormality is detected, the system will create an "abnormality detection node" at the specific location where the abnormality occurs; this node is a data point that marks the approximate location of the abnormality; output a series of "abnormality detection nodes"; each node contains approximate spatial location information (for example, coordinates on the intestinal map, or descriptive location such as "mid-ascending colon").

[0119] Input the list of abnormality detection nodes and the original intestinal map data; input: abnormality detection node list and original intestinal map data; for node A (mid-ascending colon, intestinal wall thickening), the system needs to extract the specific intestinal wall thickness value (4mm) of this area, the degree of deviation from the normal value (+1mm), the intestinal wall thickness situation around this area, etc.; for node B (descending colon, high risk of dysbiosis), the system needs to extract the dysbiosis risk score of this area (such as 8 / 10), the main missing bacterial species (such as bifidobacterium), the main excess bacterial species (such as certain opportunistic pathogens), the risk score trend relative to the user's history, etc.; for node C (end of transverse colon, filling defect), the system needs to extract the diameter of the defect on the CT image (1cm), the shape description (irregular), whether the edge is clear, the density value (CT value), etc.; these extracted specific numerical values and descriptions constitute "abnormal data"; output a set of "abnormal data" corresponding to each abnormality detection node.

[0120] Input: list of detected abnormal nodes, abnormal data of each node, original intestinal profile; extend and fuse the previously detected point-like abnormalities (nodes) into regional abnormalities; the system will consider: location of the node: the specific anatomical location where the abnormality occurs; nature and severity of the abnormal data: for example, a slight thickening of the intestinal wall only constitutes a small abnormal region, while a significant mass requires a larger region to be delineated; a high-risk dysbiosis requires a region covering a larger intestinal segment; continuity on the intestinal profile: the system will look around the node to see if there are similar or related abnormal features; for example, if a thickening is detected in the middle of the ascending colon (node A), the system will look for other signs of intestinal wall thickening or inflammation in the vicinity of the region, and appropriately expand the abnormal region to form a continuous "intestinal wall thickening abnormal region"; if two nodes are very close and the abnormal types are similar (e.g., two small polyps), the system will combine them to define a larger region containing both abnormalities; finally, the system will clearly delineate several "intestinal abnormal regions", each with clear boundaries and main abnormal features; output: one or more clearly defined "intestinal abnormal regions", each containing its spatial range, main abnormal feature description and severity.

[0121] Specifically, assume that Mr. Zhang has just completed a comprehensive intestinal health examination, including CT enterography and fecal microbiota detection; in step S14, the system has generated an intestinal profile based on the CT image and microbiota data; this profile is a data structure containing multiple layers: layer 1: CT image raw data, indicating the anatomical structure of the intestinal tract; layer 2: intestinal wall thickness map, showing the thickness values of each segment of the intestinal wall; layer 3: dysbiosis risk map, based on microbiota detection results, indicating different risk levels in different regions with different colors; layer 4: metabolite abnormality map, indicating certain abnormal regions of metabolites related to intestinal function; the system first needs to load this structured intestinal profile data completely, preparing for the next step of abnormality detection.

[0122] The system begins scanning Mr. Zhang's intestinal distribution map: on the intestinal wall thickness map (layer 2), the system finds that the thickness of a region in the middle of the ascending colon reaches 4mm, exceeding the preset threshold of 3mm; the system creates an anomaly detection node A here and records its location as "middle of the ascending colon"; on the dysbiosis risk map (layer 3), the system finds that the risk value of the descending colon region is consistently in the "high risk" color area, and the value is higher than Mr. Zhang's historical average level; the system creates an anomaly detection node B here and records the location as "descending colon"; on the CT image original data (layer 1), the system (with the help of image recognition algorithms) detects a filling defect of about 1cm in diameter at the end of the transverse colon, with irregular shape; the system creates an anomaly detection node C and records the location as "end of the transverse colon"; the final output: three anomaly detection nodes A (middle of the ascending colon, intestinal wall thickening), B (descending colon, high risk of dysbiosis), C (end of the transverse colon, filling defect).

[0123] The system extracts detailed information for each node: Node A's abnormal data: location "middle of the ascending colon", intestinal wall thickness 4.0mm, exceeding threshold +1.0mm, surrounding area thickness 3.2-3.8mm, risk level "moderate"; Node B's abnormal data: location "descending colon", dysbiosis risk score 8 / 10, main missing flora "Bifidobacterium", main excess flora "Escherichia coli", risk score increased by 2 points from last detection, risk level "high"; Node C's abnormal data: location "end of the transverse colon", filling defect diameter 1.0cm, shape "irregular", edge "blurred", CT value about 50HU, risk level "high" (based on morphological judgment); output: detailed data of node A, detailed data of node B, detailed data of node C.

[0124] System integrates information to delineate regions: for node A and its data, the system looks around the middle ascending colon and finds no other obvious abnormalities, so it delineates a small region: "Abnormal region of thickened intestinal wall in the middle ascending colon", roughly the intestinal segment 2 cm before and after node A, the main feature is diffuse thickening of the intestinal wall (4.0 mm), risk level "moderate"; for node B and its data, the system looks at the descending colon region and finds that the dysbiosis risk map shows that the region is at high risk and extends to the beginning of the sigmoid colon; therefore, it delineates a larger region: "High-risk region of dysbiosis in the descending colon-sigmoid colon", covering from the middle of the descending colon to the beginning of the sigmoid colon, the main feature is significant dysbiosis (score 8 / 10, lack of Bifidobacterium, overabundance of Escherichia coli), risk level "high"; for node C and its data, the system accurately outlines the location of the filling defect based on the CT image and delineates a clear point or small range region: "Abnormal region of filling defect in the end of the transverse colon", accurate location, the main feature is irregular filling defect (1.0 cm, fuzzy edge), risk level "high"; output three clear intestinal abnormal regions and their detailed descriptions; as follows:

[0125] Intestinal abnormal region #1: Abnormal region of thickened intestinal wall in the middle ascending colon

[0126] Location: middle ascending colon (about 25-29 cm from the ileocecal valve); main abnormal data: diffuse thickening of the intestinal wall, maximum thickness 4.0 mm, exceeding the normal threshold (3 mm) + 1.0 mm; surrounding region thickness 3.2-3.8 mm; risk level: moderate;

[0127] Intestinal abnormal region #2: High-risk region of dysbiosis in the descending colon-sigmoid colon

[0128] Location: middle of the descending colon to the beginning of the sigmoid colon; main abnormal data: dysbiosis risk score 8 / 10 (high), main missing bacteria Bifidobacterium, main excess bacteria Escherichia coli; risk score increased by 2 points from the last detection; risk level: high;

[0129] Intestinal abnormal region #3: Abnormal region of filling defect in the end of the transverse colon

[0130] Location: end of the transverse colon (about 15 cm from the hepatic flexure of the colon); main abnormal data: irregular filling defect, diameter 1.0 cm, fuzzy edge, CT value about 50 HU; risk level: high.

[0131] Further, based on the identification of the intestinal abnormal area, the corresponding abnormal morphology is determined, and based on the abnormal morphology, the intestinal distribution map and the age of the user, the corresponding intestinal abnormal event is determined, and the intestinal abnormal event is traced back to collect the corresponding trace data; based on the screening of the trace data, the corresponding life behavior data is determined, and based on the life behavior data, the behavior record of the user and the intestinal abnormal event, the abnormal life habit of the user is determined, which comprehensively considers the life behavior data, the behavior record of the user and the intestinal abnormal event, ensures the accuracy of the abnormal life habit of the user, and at the same time, introduces the environmental characteristics of the intestine, realizes the overall consideration of multiple environmental characteristics and the intestinal image of the user, and improves the accuracy of the intestinal distribution map, so as to identify the abnormal life habit of the user.

[0132] At this time, the intestinal abnormal area information output from S151 is input, including the area position, the main abnormal data (such as the intestinal wall thickness, the flora score, the filling defect size and morphology, etc.); the system needs to further analyze the details of these abnormal data and extract more specific morphological descriptions; for example, for intestinal wall thickening, is it diffuse or localized? Is it uniform thickening or irregular thickening? For filling defects, is it round, oval or irregular? Is the edge smooth, lobulated or rough and blurred? Is the density uniform or non-uniform? These morphological characteristics are crucial for determining the nature of the lesion (such as inflammation, polyps, tumors, etc.); output the detailed morphological description corresponding to each intestinal abnormal area.

[0133] The system combines morphological characteristics, environmental characteristics of the area (such as inflammation level, flora state, intestinal wall blood supply, etc.) and the age of the user, and uses built-in knowledge base or machine learning model to preliminarily judge and classify the nature of the abnormal area; for example, a filling defect with blurred and irregular edge, combined with local inflammation environment and advanced age, is more inclined to be judged as "suspected inflammatory polyps or early tumors"; while diffuse intestinal wall thickening with local flora imbalance is judged as "suspected chronic inflammation or reactive hyperplasia"; this process is similar to the doctor's preliminary diagnosis based on images and clinical information; output the preliminary judgment event corresponding to each intestinal abnormal area based on the current information, such as "suspected inflammatory polyps", "suspected early tumors", "suspected chronic inflammation", "flora imbalance related reactive changes", etc.

[0134] Specifically, S151 has determined three abnormal regions; morphological analysis is performed on these regions: Abnormal Region #1 (Mid-ascending colon intestinal wall thickening): Morphological description: "Diffuse, mild irregular thickening of intestinal wall, no clear boundary, surface still smooth"; Abnormal Region #2 (Descending colon-sigmoid colon flora imbalance): Morphological description: "The regional flora distribution presents a 'patchy' imbalance pattern, with local areas lacking Bifidobacterium signals and significantly enhanced Escherichia coli signals"; Abnormal Region #3 (Terminal transverse colon filling defect): Morphological description: "The filling defect is 'irregular' in shape, about 1.0 cm in diameter, with 'blurred' and 'rough' edges, and 'uneven' internal density."

[0135] In combination with Mr. Zhang's age (60 years old), abnormal morphology and environmental characteristics, the following judgments are made: Abnormal Region #1 (diffuse, irregular thickening, with mild imbalance in surrounding flora): In combination with Mr. Zhang's age of 60 years old, the event is judged to be "suspected chronic inflammation or reactive hyperplasia, with a warning of malignant transformation"; Abnormal Region #2 (patchy flora imbalance): The event is judged to be "flora imbalance related intestinal dysfunction"; Abnormal Region #3 (irregular, blurred, uneven filling defect): In combination with Mr. Zhang's age of 60 years old, the event is judged to be "suspected inflammatory polyp or early tumor, further biopsy is recommended".

[0136] The system triggers a data tracing mechanism based on the type of abnormal event determined; this usually involves querying the user's historical health record database; for example, if it is judged to be "suspected early tumor", the system will look for whether there are relevant symptom records (such as hematochezia, weight loss, abdominal pain) in the past few years, whether there are similar imaging findings, whether there are tumor marker abnormalities, whether there are relevant family history, etc.; The time range of the trace can be dynamically adjusted according to the severity and type of the abnormal event; output the set of historical tracing data related to each intestinal abnormal event, including past examination reports, symptom records, treatment records, family history information, etc.

[0137] The system filters out the part related to life habits from the tracing data; this requires certain rules or models to identify; for example, pay attention to past dietary records (high-fat, high-protein, low-fiber diet frequency), exercise records (changes in exercise amount), sleep records (changes in sleep quality), stress records (major life events, work stress), drug use records (especially antibiotics, painkillers, and psychotropic drugs); The system extracts these behavior data segments related to intestinal health or specific abnormal events; filtered life behavior data segments related to intestinal abnormal events.

[0138] The system compares the screened historical life behavior data with the user's current behavior record, and conducts correlation analysis combined with the nature of the intestinal abnormal event; determine which current habits match the patterns shown in the change points or abnormal events in the historical data, or are related to the pathological mechanism of the abnormal event; for example, if the abnormal event is "suspected early tumor", and the historical data shows that the change in dietary structure (such as increased red meat intake) coincides with the time point of the symptoms, and the user still maintains the habit of high red meat intake, the system will mark "high red meat intake" as "abnormal life habits"; this process requires certain logical reasoning or machine learning model support; output the list of "abnormal life habits" related to the specific intestinal abnormal event that the user currently has, and attach the correlation strength or confidence, and show the user or doctor the abnormal life habits summary report related to the specific intestinal abnormal area / event.

[0139] Specifically, the system traces Mr. Zhang's three abnormal events: for the "suspected chronic inflammation or reactive hyperplasia" event, trace back the colonoscopy report, stool routine + occult blood, symptom record (abdominal pain, diarrhea frequency) in the past 3 years; for the "intestinal dysfunction related to dysbiosis" event, trace back the intestinal symptom diary, diet record, antibiotic use history in the past 6 months; for the "suspected inflammatory polyps or early tumors" event, trace back the colonoscopy report, fecal occult blood test, CEA / CA19-9 tumor markers, and whether there is a family history of digestive tract tumors in the past 5 years.

[0140] Assuming the tracing results are as follows: abnormal area #1: the past 3 years of colonoscopy report shows that this area has mild inflammation, but the current thickening is more obvious; fecal occult blood is occasionally positive; symptom record shows that the frequency of abdominal pain has increased recently; abnormal area #2: the past 6 months of symptom diary shows that the frequency of diarrhea is unstable and related to dietary changes; no antibiotic use history; abnormal area #3: the past 5 years of colonoscopy report shows no abnormalities in this area; fecal occult blood has been negative; no relevant family history.

[0141] The system screens life behavior data from Mr. Zhang's tracing data: for abnormal area #1: screen out the increase in high-fat diet frequency in the past 3 months (4-5 times of takeout per week), and the decrease in exercise amount (from 3 times per week to 1 time); for abnormal area #2: screen out the stress event record (work change, family conflict) in the past 6 months, and the diet record shows that the frequency of eating out is high (food is unclean or the structure changes); for abnormal area #3: screen out no special life behavior change record in the past 1 year, but no obvious related behavior is found in the tracing data.

[0142] The system analyzes Zhang's current behavior record (assuming it is part of the multi-modal data): Current behavior record: Mr. Zhang still maintains a high-fat diet with 4-5 takeout meals per week, low exercise (1 walk per week), recent high work pressure, and irregular diet; Analysis: For abnormal region #1 (chronic inflammation / hyperplasia): The current high-fat diet and low exercise are consistent with historical changes, and these factors are related to intestinal inflammation, so "high-fat diet" and "lack of regular exercise" are determined as related abnormal living habits; For abnormal region #2 (dysbiosis): The current high stress and irregular diet are related to historical stress events and dietary changes, and these factors affect the intestinal flora, so "high stress state" and "irregular diet" are determined as related abnormal living habits; For abnormal region #3 (suspected polyps / tumors): The current behavior record has no clear high-risk factors, and historical tracing also has no strong correlation behavior, so it is temporarily impossible to determine clear abnormal living habits, or marked as "no clear correlation living habits found".

[0143] The system finally outputs the report summary to Mr. Zhang as follows:

[0144] "Based on your intestinal distribution map analysis and historical data tracing, we have identified the following abnormal regions and related living habits:

[0145] Mid-ascending colon (chronic inflammation / hyperplasia):

[0146] Related abnormal living habits: long-term high-fat diet (4-5 takeout meals per week), lack of regular exercise (only 1 walk per week); Suggestions: adjust the dietary structure, reduce high-fat takeout, increase vegetable and fruit intake; increase the frequency and intensity of regular exercise;

[0147] Descending colon-sigmoid colon (dysbiosis-related intestinal function disorder):

[0148] Related abnormal living habits: recent high stress, irregular diet; Suggestions: try stress management techniques (such as meditation, exercise), maintain regular meals, avoid overeating;

[0149] End of transverse colon (suspected inflammatory polyps / early tumors):

[0150] No clear correlation between abnormal living habits is found; Suggestions: closely monitor changes in this area, follow doctor's advice for colonoscopy biopsy to confirm the nature; maintain the general principles of a healthy lifestyle".

[0151] Not only does it identify intestinal abnormalities, but it also tries to link these abnormalities to the user's living habits to provide more personalized health advice and intervention direction, which is what creating new features (such as the stress-intestinal health index) aims to do, that is, to dig deep correlations from multi-dimensional data to improve the accuracy of health management.

[0152] Referring to Figure 7 , Figure 7 is a structural composition diagram of an intestinal tract detection system based on multi-modal data in an embodiment of the present application; the intestinal tract detection system based on multi-modal data comprises:

[0153] a lesion data module 21, configured to collect lesion regions of an intestinal tract, determine corresponding lesion morphologies based on synchronous detection of each lesion region, and determine lesion data of the intestinal tract according to region positions of each lesion region and the lesion morphologies;

[0154] a biochemical data module 22, configured to collect flora distribution data of the intestinal tract, determine each dominant flora based on detection of the flora distribution data, and construct biochemical data of the intestinal tract according to each dominant flora;

[0155] a multi-modal data module 23, configured to collect life events of a user within adjacent two intestinal tract detection times, determine a plurality of life features according to recognition of the life events, and determine multi-modal data according to the plurality of life features, the biochemical data and the lesion data of the intestinal tract;

[0156] an intestinal tract distribution map module 24, configured to determine a plurality of intestinal tract data combinations based on detection of the multi-modal data, determine environmental features of the intestinal tract according to recognition of each intestinal tract data combination, collect an intestinal tract image of the user, and determine an intestinal tract distribution map according to the plurality of environmental features and the intestinal tract image of the user;

[0157] an intestinal tract abnormal event module 25, configured to determine corresponding intestinal tract abnormal regions based on the intestinal tract distribution map, determine intestinal tract abnormal events according to recognition of the intestinal tract abnormal regions, and determine abnormal life habits of the user based on tracing of the intestinal tract abnormal events.

[0158] Any combination of the technical features of the above embodiments is possible, and in order to make the description concise, all combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

Claims

1. A method for intestinal detection based on multimodal data, characterized in that, include: The diseased areas of the intestine are collected, and the corresponding lesion morphology is determined based on the synchronous detection of each lesion area. The lesion data of the intestine are determined according to the regional location and lesion morphology of each lesion area. Collect gut microbiota distribution data, identify dominant microbiota based on the detection of microbiota distribution data, and construct gut biochemical data based on each dominant microbiota. Collect user’s life events within two consecutive intestinal testing periods, identify multiple life characteristics based on the identification of these life events, and determine multimodal data based on multiple life characteristics, intestinal biochemical data, and pathological data. Multiple gut data combinations are determined based on the detection of multimodal data, and the environmental characteristics of the gut are determined based on the identification of each gut data combination. At the same time, gut images of users are collected, and gut distribution maps are determined based on multiple environmental characteristics and gut images of users. Based on the intestinal distribution map, the corresponding abnormal intestinal regions are identified. Based on the identification of abnormal intestinal regions, abnormal intestinal events are determined. Based on the tracing of these abnormal intestinal events, abnormal lifestyle habits of users are determined.

2. The intestinal detection method based on multimodal data according to claim 1, characterized in that, The process involves collecting lesion areas from the intestine, determining the corresponding lesion morphology based on simultaneous detection of each lesion area, and determining intestinal lesion data based on the regional location and lesion morphology of each lesion area, including: When the endoscope is inside the intestine, it acquires videos of the internal activity of the intestine. Based on the detection of the internal activity videos, multiple sub-videos are determined. Based on the identification of multiple sub-videos, the corresponding lesion nodes are determined. The lesion area of ​​the intestine is determined by detecting along the lesion node. Multiple lesion areas are collected, each lesion area is detected simultaneously, and corresponding regional schematic diagrams are output. Based on the identification of the regional schematic diagrams, the corresponding lesion morphology is determined, and the regional location of each lesion area is marked. In each lesion region, the data combination of the lesion region is determined based on the regional location and lesion morphology of each lesion region, and the intestinal lesion data is determined based on the data combination of each lesion region.

3. The intestinal detection method based on multimodal data according to claim 1, characterized in that, The process involves collecting intestinal flora distribution data, identifying dominant bacterial groups based on the detection of this data, and constructing intestinal biochemical data based on these dominant bacterial groups, including: The gut microbiota is analyzed in the feces excreted through the intestine. Based on the fecal microbiota analysis, the distribution areas of each microbiota in the feces are determined. The distribution data of the gut microbiota is predicted based on the regional location and the distribution of each microbiota in the distribution areas. Data detection of microbial community distribution data is performed, and the types and relative proportions of each microbial community are output. A gradient distribution map of the microbial community is determined based on the types and relative proportions of each microbial community, and each dominant microbial community is determined based on the traversal of the gradient distribution map. Each dominant microbial community is collected, and multiple dominant microbial community combinations are determined based on the cross-combination of each dominant microbial community. Biochemical data of the gut are constructed based on multiple dominant microbial community combinations.

4. The intestinal detection method based on multimodal data according to claim 1, characterized in that, The system collects user life events within two consecutive intestinal testing intervals. Based on the identification of these life events, multiple life characteristics are determined. Multimodal data is then determined based on these multiple life characteristics, intestinal biochemical data, and pathological data, including: Collect the user's various intestinal testing times, determine the two adjacent intestinal testing times based on the current time and the individual intestinal testing times, and trace the user's lifestyle habits between the two adjacent intestinal testing times to output the user's life events within the two adjacent intestinal testing times. The process involves collecting data on a life event, analyzing the event to identify multiple sub-life items, detecting these sub-life items to determine corresponding life behaviors, and identifying these behaviors to determine corresponding life characteristics.

5. The intestinal detection method based on multimodal data according to claim 4, characterized in that, The method of collecting user's life events within two consecutive intestinal testing time periods, identifying multiple life characteristics based on the identification of these life events, and determining multimodal data based on these multiple life characteristics, intestinal biochemical data, and pathological data, also includes: Multiple lifestyle characteristics are collected, and the first modality data is determined based on the multiple lifestyle characteristics and intestinal biochemical data. The second modality data is determined based on the multiple lifestyle characteristics and lesion data. Multimodal data is determined based on the cross-training of the first modality data and the second modality data.

6. The intestinal detection method based on multimodal data according to claim 1, characterized in that, The detection based on multimodal data determines multiple combinations of intestinal data, and the identification of each intestinal data combination determines the environmental characteristics of the intestine. Simultaneously, intestinal images of the user are acquired, and an intestinal distribution map is determined based on multiple environmental characteristics and the user's intestinal images, including: Multimodal data is collected, data detection is performed on the multimodal data, and multiple data categories are output. The multimodal data is divided based on each data category to determine multiple gut data combinations. The multiple gut data combinations are then identified to determine the environmental characteristics of the gut.

7. The intestinal detection method based on multimodal data according to claim 6, characterized in that, The method involves determining multiple intestinal data combinations based on multimodal data detection, identifying intestinal environmental features based on the recognition of each intestinal data combination, and simultaneously acquiring the user's intestinal images. The method also includes determining an intestinal distribution map based on multiple environmental features and the user's intestinal images. The system performs CT scans on the user's intestines and outputs images of the user's intestines, defining multiple image regions based on the segmentation of the user's intestine images; Mark the location and morphology of multiple image regions; construct an intestinal distribution map based on the intestinal environment characteristics, the location and morphology of multiple image regions.

8. The intestinal detection method based on multimodal data according to claim 1, characterized in that, The process of identifying corresponding abnormal intestinal regions based on intestinal distribution maps, determining abnormal intestinal events based on the identification of these abnormal regions, and identifying abnormal lifestyle habits of users based on the tracing of these abnormal intestinal events includes: Collect intestinal distribution maps, perform anomaly detection on the intestinal distribution maps, and output the corresponding anomaly detection nodes. Determine the corresponding abnormal data based on the detection of each anomaly detection node, and determine the corresponding abnormal intestinal region based on the node position of each anomaly detection node, the corresponding abnormal data, and the intestinal distribution map.

9. The intestinal detection method based on multimodal data according to claim 8, characterized in that, The process of determining corresponding abnormal intestinal regions based on intestinal distribution maps, identifying abnormal intestinal events based on the identification of abnormal intestinal regions, and determining abnormal lifestyle habits of users based on the tracing of these abnormal intestinal events also includes: Based on the identification of abnormal intestinal regions, the corresponding abnormal morphology is determined, and the corresponding intestinal abnormal events are determined according to the abnormal morphology, intestinal distribution map and user age. The intestinal abnormal events are traced to collect corresponding trace data. Based on the screening of trace data, the corresponding lifestyle data is determined, and the abnormal lifestyle habits of users are determined according to lifestyle data, user behavior records and intestinal abnormal events.

10. An intestinal detection system based on multimodal data, characterized in that, The intestinal detection system based on multimodal data is applied to the intestinal detection method based on multimodal data as described in any one of claims 1-9, wherein the intestinal detection system based on multimodal data comprises: The lesion data module is used to collect lesion areas in the intestine, determine the corresponding lesion morphology based on the synchronous detection of each lesion area, and determine the lesion data of the intestine based on the regional location and lesion morphology of each lesion area. The biochemical data module is used to collect gut microbiota distribution data, identify dominant microbiota based on the detection of microbiota distribution data, and construct gut biochemical data based on each dominant microbiota. The multimodal data module is used to collect the user's life events within two adjacent intestinal testing periods, identify multiple life characteristics based on the identification of these life events, and determine multimodal data based on multiple life characteristics, intestinal biochemical data, and lesion data. The intestinal distribution mapping module is used to determine multiple intestinal data combinations based on the detection of multimodal data, determine the environmental features of the intestine based on the identification of each intestinal data combination, and simultaneously collect the user's intestinal images to determine the intestinal distribution map based on multiple environmental features and the user's intestinal images. The intestinal abnormality event module is used to determine the corresponding intestinal abnormality area based on the intestinal distribution map, determine the intestinal abnormality event based on the identification of the intestinal abnormality area, and determine the user's abnormal lifestyle habits based on the tracing of the intestinal abnormality event.

Images