Colorectal cancer diagnosis auxiliary method and system based on deep learning

Abstract

The invention relates to the technical field of colorectal image analysis, in particular to a colorectal cancer diagnosis auxiliary method and system based on deep learning. The method comprises the following steps: acquiring colorectum endoscope image data, and performing region-of-interest positioning on the colorectum endoscope image data to obtain colorectum region-of-interest data; performing multi-stage image segmentation on the colorectal endoscope image data to obtain colorectal image multi-stage segmentation data; performing depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and performing three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data; and carrying out cell relation feature extraction on the colorectal image three-dimensional network data to obtain colorectal image cell relation feature data so as to carry out colorectal cancer diagnosis auxiliary operation. According to the invention, the early diagnosis rate and diagnosis precision of colorectal cancer are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of colorectal image analysis, and in particular to a colorectal cancer diagnosis auxiliary method and system based on deep learning. Background Art

[0002] Colorectal cancer is one of the most common digestive tract malignancies worldwide, with the highest incidence and mortality rates. Early detection and diagnosis are the key to improving the survival rate and quality of life of patients with colorectal cancer. However, since the symptoms of early colorectal cancer are often not obvious, many patients are already in the middle and late stages when diagnosed. Traditional colorectal cancer diagnosis methods mainly rely on doctors' empirical judgment of endoscopic examination results, which are easily affected by doctors' subjective factors, resulting in unstable diagnostic accuracy and high misdiagnosis and missed diagnosis rates. With the development of artificial intelligence technology, especially the widespread application of deep learning in the field of medical image analysis, new possibilities have been provided for the early diagnosis of colorectal cancer and the development of intelligent auxiliary systems. How to combine artificial intelligence with colorectal cancer diagnosis has become a problem. Summary of the invention

[0003] In order to solve the above-mentioned technical problems, the present invention proposes a colorectal cancer diagnosis auxiliary method and system based on deep learning to solve at least one of the above-mentioned technical problems.

[0004] The present application provides a colorectal cancer diagnosis auxiliary method based on deep learning, comprising the following steps:

[0005] Step S1: acquiring colorectal endoscope image data, and locating a region of interest on the colorectal endoscope image data to obtain colorectal region of interest data;

[0006] Step S2: performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data;

[0007] Step S3: performing depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and performing three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data;

[0008] Step S4: extracting cell relationship features from the colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and performing auxiliary colorectal cancer diagnosis operations based on the colorectal image depth feature data and the colorectal image cell relationship feature data.

[0009] In the present invention, by locating the region of interest (ROI), the focus is focused on the area where the species of interest may exist, reducing the interference of irrelevant areas and improving the diagnostic efficiency. The multi-level segmentation method can refine the image in a hierarchical and precise manner, not only can the macroscopic structural information (such as the outline of the entire region of interest) be extracted, but also the microscopic details (such as the tissue structure inside the region of interest) can be captured, which is helpful for accurate diagnosis. For the region of interest with complex morphology (such as irregular tumors or multiple lesions), the multi-level segmentation can analyze its morphological characteristics step by step, and enhance the robustness of the model when dealing with complex situations. By extracting high-level image features (such as edges, textures, shapes, etc.) through a deep learning network, the essential features of the region of interest can be expressed more accurately, and the recognition ability of the model for the region of interest can be improved. The three-dimensional network can integrate multiple two-dimensional slice images into three-dimensional volume data, capture the spatial structure of the region of interest and the relationship between tissues, and is particularly suitable for detecting the expansion and distribution of the species of interest. Cell relationship features can reveal the interaction patterns (such as density, connectivity, etc.) between cells or regions of interest, and help identify the microenvironmental characteristics of the region of interest and the diffusion path of cancer cells.

[0010] Preferably, the colorectal endoscope image data includes first colorectal endoscope image data and second colorectal endoscope image data, and step S1 is specifically:

[0011] Acquiring a colorectal image using a high-definition camera at a first light source brightness threshold and a first advancement speed threshold to obtain first colorectal endoscope image data;

[0012] Acquiring a colorectal image using an ultra-high-definition camera with a second light source brightness threshold and a second advancement speed threshold to obtain second colorectal endoscopic image data, wherein the first light source brightness threshold is greater than or equal to the second light source brightness threshold, and the first advancement speed threshold is less than or equal to the second advancement speed threshold;

[0013] Extracting texture features from colorectal endoscope image data to obtain colorectal texture feature data;

[0014] Performing clustering calculation on the colorectal texture feature data to obtain colorectal texture clustering feature data;

[0015] Performing similarity calculation based on the colorectal texture clustering feature data and the preset conventional colorectal texture clustering feature data to obtain texture clustering feature similarity data;

[0016] The colorectal endoscopy image data is multi-level labeled according to the texture clustering feature similarity data to obtain the colorectal region of interest data.

[0017] The higher light source brightness and slower advancement speed in the present invention can capture more details in low-resolution images, especially in dark or insufficiently illuminated areas, improving the clarity and detail expression of the image. The ultra-high-definition camera can capture more details under low light source brightness conditions, and the fast advancement speed can improve the acquisition efficiency and reduce the patient's discomfort. The data collected by the high-definition camera and the ultra-high-definition camera complement each other. High-definition images are helpful for macroscopic observation, and ultra-high-definition images are helpful for microscopic detail analysis. By clustering the texture feature data (such as K-means, DBSCAN, etc.), pixels with similar texture features can be clustered together, which helps to identify areas with similar features of interest. Cluster analysis can reveal subtle differences between different regions and improve diagnostic accuracy. By gradually labeling different similarity thresholds, the region can be refined, from initial interest to high interest, and the hierarchical labeling effectively improves the accuracy and reliability of the labeling.

[0018] Preferably, the colorectal region of interest data includes first colorectal region of interest data, second colorectal region of interest data and third colorectal region of interest data, and the multi-level annotation is specifically as follows:

[0019] When it is determined that the texture clustering feature similarity data is less than or equal to a preset first texture clustering feature similarity threshold, a first region of interest labeling is performed on an image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data;

[0020] When it is determined that the texture clustering feature similarity data is greater than a preset first texture clustering feature similarity threshold and less than or equal to a preset second texture clustering feature similarity threshold, high-frequency component feature extraction is performed on the colorectal texture clustering feature data to obtain colorectal texture high-frequency component data, and shape feature extraction is performed on the colorectal endoscope image data to obtain colorectal image shape feature data;

[0021] Calculating the interest index according to the colorectal texture high frequency component data and the colorectal image shape feature data to obtain the colorectal image interest index data;

[0022] When it is determined that the colorectal image interest index data is less than or equal to a preset colorectal image interest index threshold, a first region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data;

[0023] When it is determined that the colorectal image interest index data is greater than a preset colorectal image interest index threshold, a second region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain second colorectal region of interest data;

[0024] When it is determined that the texture clustering feature similarity data is greater than the preset second texture clustering feature similarity threshold, a third region of interest annotation is performed on the image pixel area corresponding to the colorectal endoscopy image data corresponding to the texture clustering feature similarity data to obtain third colorectal region of interest data, wherein the first region of interest annotation, the second region of interest annotation and the third region of interest annotation are region of interest annotation methods containing different region of interest annotation contents.

[0025] In the present invention, the colorectal region of interest data is divided into three different annotation levels, namely the first region of interest, the second region of interest and the third region of interest, which correspond to different rigid interest levels respectively. The multi-level annotation strategy can effectively improve the recognition accuracy and classification effect of the lesion. Through the division of the first, second and third region of interest data, the lesion area is hierarchically managed according to its similarity and importance. By refining layer by layer, the complex features of the region can be better described, and the limitations of a single annotation method can be effectively avoided. Annotation is performed separately under different feature similarity thresholds, which helps to filter out background areas and irrelevant low-similarity areas and reduce noise interference. By setting a multi-level threshold, the misjudgment caused by image blur or uneven illumination can be effectively reduced, and the accuracy and stability of the region of interest annotation can be improved. The annotation judgment is performed in combination with texture features and high-frequency component features. By introducing high-frequency component features, the small and fine structural changes in the lesion area can be more keenly captured. The fusion of multiple features improves the ability to capture local structures and fine textures in the lesion area. For lesion areas with blurred boundaries and complex morphology, the introduction of shape features can effectively distinguish the morphological differences between lesions and normal tissues, and improve the recognition and labeling capabilities of complex lesions.

[0026] Preferably, step S2 specifically includes:

[0027] Performing a first image segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain first colorectal image segmentation data;

[0028] The colorectal endoscopic image data is subjected to a second image segmentation according to the third colorectal region of interest data to obtain second colorectal image segmentation data, wherein the first image segmentation and the second image segmentation are different image segmentation methods.

[0029] In the present invention, different segmentation methods are used for the second colorectal region of interest (medium risk area) and the third colorectal region of interest (high risk area), respectively, which can more accurately adapt to different types of lesion characteristics and morphological characteristics, avoid the limitations of a single segmentation method in dealing with diverse lesions, and effectively improve the accuracy and adaptability of segmentation. The first image segmentation may focus on fast segmentation and overall area extraction, while the second image segmentation pays more attention to details and complex structure processing. The combination of different segmentation methods can improve the ability to recognize detailed lesions while ensuring segmentation efficiency. Through the coarse-to-fine segmentation method, the lesion area can be refined layer by layer, from the whole to the details, and the accuracy and precision of the segmentation can be gradually improved.

[0030] Preferably, the first image segmentation is specifically:

[0031] Performing edge enhancement on colorectal endoscope image data to obtain colorectal image edge enhancement data;

[0032] Performing image segmentation on the edge enhancement data of the colorectal image according to the second colorectal region of interest data to obtain primary first colorectal image segmentation data;

[0033] Performing regional segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain secondary first colorectal image segmentation data;

[0034] The primary first colorectal image segmentation data and the secondary first colorectal image segmentation data are fused to obtain the first colorectal image segmentation data.

[0035] The edge enhancement in the present invention can effectively enhance the edge significance of the lesion area in the colorectal endoscope image and improve the contrast between the lesion area and the surrounding normal tissue. Primary segmentation is performed on the basis of edge enhancement, and the approximate area of ​​the lesion can be quickly located. The image after edge enhancement can better express the morphological characteristics of the lesion, especially for the lesion area with complex boundaries, and can effectively improve the accuracy of the initial segmentation. According to the second region of interest data, the image after edge enhancement is cut, and the approximate outline of the lesion area can be quickly segmented. The main purpose of the primary segmentation process is to roughly identify the lesion area, and the complex image segmentation task is preliminarily simplified, which helps to improve the overall segmentation efficiency. According to the second region of interest data, the original image is segmented again, and the detailed features inside the lesion area can be captured more finely. Secondary segmentation makes up for the shortcomings of primary segmentation in detail recognition, and captures the small and complex structural changes in the lesion area through a more refined segmentation method. The results of the two segmentation methods are merged, and the segmentation accuracy of the internal structure can be improved while ensuring the accuracy of the overall segmentation contour.

[0036] Preferably, the second image segmentation is specifically:

[0037] Gaussian distribution feature extraction is performed based on colorectal texture clustering feature data to obtain texture clustering Gaussian distribution feature data;

[0038] Performing pixel classification on the colorectal endoscope image data according to the texture clustering Gaussian distribution feature data and the third colorectal region of interest data to obtain colorectal image pixel classification data;

[0039] Mapping and segmenting the colorectal image pixel classification data to obtain second colorectal image segmentation data.

[0040] In the present invention, the local and global features of the pixel distribution in the colorectal image can be captured more delicately through Gaussian distribution feature extraction, especially for the lesion area with complex texture and blurred boundary, the correlation and spatial distribution pattern between pixels can be expressed more accurately through Gaussian distribution features. Gaussian distribution features can express and model the multimodal features (such as different textures, colors, brightness, etc.) in the image in a probabilistic manner, so that information of different feature dimensions can be unified, integrated and analyzed, and the feature expression ability and robustness of the segmentation model are improved. Gaussian distribution features can probabilistically distinguish the pixel distribution of the lesion area from the pixel distribution of the normal tissue background, thereby providing an effective basis for distinguishing subsequent pixel classification, reducing the confusion between the region of interest and the background area, and improving the accuracy of segmentation. Combined with the texture clustering Gaussian distribution feature data and the third region of interest data, each pixel in the endoscopic image can be finely classified to ensure that each pixel in the region of interest can be effectively identified. Through the delicate expression of Gaussian distribution features and refined processing of pixel classification, the high-dimensional feature data is organically combined with pixel-level classification information, which can effectively improve the segmentation model's ability to understand and express the region of interest and ensure the accuracy and stability of the segmentation results.

[0041] Preferably, step S3 specifically includes:

[0042] Performing multi-scale feature extraction on the first colorectal image segmentation data to obtain multi-scale feature data of the colorectal image;

[0043] Perform multi-head self-attention calculation on the multi-scale feature data of the colorectal image to obtain the deep feature data of the colorectal image;

[0044] Slice stacking is performed on the second colorectal image segmentation data to obtain slice stacking data;

[0045] Performing three-dimensional transformation on the slice stacking data to obtain slice three-dimensional transformation data;

[0046] The slice 3D conversion data is 3D meshed to obtain the 3D network data of the colorectal image.

[0047] The multi-scale feature extraction in the present invention can capture multi-level information of the region of interest from different scales (such as large-scale overall contours and small-scale detailed textures). Multi-scale feature extraction can adapt to image data of different resolutions, capture the overall contour of the lesion at low resolution, and capture the detailed information of the lesion at high resolution, which helps to enhance the expressiveness and adaptability of the features. The multi-head self-attention mechanism can capture the dependencies between the pixels in the lesion area from a global perspective, identify the correlation between long-distance pixels, and help understand the overall structural characteristics of the region of interest. Slicing and stacking the second colorectal image segmentation data can integrate spatial information at different levels into a three-dimensional volume data to achieve the integration and expression of multi-level information. Slice stacking can make up for the problem of insufficient information in a single slice data and provide a more complete spatial feature representation. By performing a three-dimensional conversion on the slice stacking data, a three-dimensional structural model of the region of interest can be generated to accurately express the position, morphology and structural information of the region of interest in three-dimensional space.

[0048] Preferably, step S4 is specifically:

[0049] Perform voxel region node extraction and voxel region relationship variation extraction on the three-dimensional network data of colorectal images to obtain voxel region node data and voxel region relationship edge data respectively;

[0050] Relationship graph data is constructed according to voxel region node data and voxel region relationship edge data to obtain colorectal relationship graph data;

[0051] Performing local neighborhood feature extraction and global relationship feature extraction on the colorectal relationship graph data to obtain colorectal local neighborhood feature data and colorectal global relationship feature data respectively;

[0052] Performing feature fusion on the local neighborhood feature data of the colorectum and the global relationship feature data of the colorectum to obtain the cell relationship feature data of the colorectal image;

[0053] Colorectal cancer diagnosis auxiliary tasks are performed based on colorectal image depth feature data and colorectal image cell relationship feature data.

[0054] In the present invention, by extracting nodes from voxel regions in the three-dimensional network data of colorectal images, the characteristic nodes of each region of interest can be accurately located in three-dimensional space. Each voxel node carries its spatial position, texture, density and other characteristic information, and can accurately express the distribution and morphological characteristics of the region of interest in three-dimensional space. The extraction of voxel relationship edges can describe the spatial adjacency relationship and association strength between voxel nodes, and reflect the topological structure and distribution pattern of the region of interest in three-dimensional space. The introduction of voxel relationship edges provides richer information for the spatial structure analysis of the region of interest, which helps to understand the interaction and spatial relationship between voxel nodes. By constructing a relationship graph with voxel nodes and relationship edge data, the three-dimensional spatial structure of the region of interest can be converted into structured graph data, where each node represents a voxel in the region of interest, and the edge represents the spatial relationship between nodes. The relationship graph structure can better express the spatial topological structure and global layout of the region of interest. The relationship graph data can simultaneously capture the relationship between voxels within the region of interest (such as the tightness and similarity of the internal structure, etc.) and the relationship between the region of interest and the surrounding tissues (such as the expansion direction, the spatial distribution of adjacent areas, etc.), and comprehensively analyze the structure of the region of interest from multiple levels and multiple angles. Local features can reflect the microscopic relationship and local morphological features within the region of interest, and improve the model's ability to express the detailed features of the region of interest. The introduction of global features can improve the model's ability to understand the overall structure of the region of interest and avoid one-sided analysis caused by local features.

[0055] Preferably, local neighborhood feature extraction is specifically as follows:

[0056] Performing neighborhood extraction on the colorectal relationship graph data to obtain colorectal neighborhood data;

[0057] Extracting local geometric features from colorectal neighborhood data to obtain local geometric feature data;

[0058] The local morphological features are extracted from the local geometric feature data to obtain the local neighborhood feature data of the colorectum.

[0059] Neighborhood extraction in the present invention can identify the neighborhood structure around each node (voxel) from the relationship graph data, and capture its local spatial relationship, including the distance, direction and connection mode between adjacent nodes. Local geometric feature extraction can quantify the geometric properties of the neighborhood structure, including the distance distribution, direction distribution, density distribution, etc. between neighborhood nodes. Local geometric features can keenly capture the tiny geometric changes inside the region of interest, such as tiny protrusions, depressions, boundary curvatures, etc. The extraction of geometric features can better distinguish regions of interest of different morphologies and complexities, help improve the ability to distinguish features, and provide an effective geometric basis for further local morphological feature extraction. Local morphological feature extraction can capture the morphological features of local areas of the region of interest, including surface shape, edge curvature, concavity, etc., and can carefully depict the surface morphological changes of the local area, especially for lesion areas with irregular morphology and complex boundaries. The introduction of morphological features can more accurately express its morphological features.

[0060] Preferably, the present application also provides a colorectal cancer diagnosis assistance system based on deep learning, which is used to execute the colorectal cancer diagnosis assistance method based on deep learning as described above. The colorectal cancer diagnosis assistance system based on deep learning includes:

[0061] A colorectal region of interest positioning module is used to obtain colorectal endoscopic image data and locate the region of interest on the colorectal endoscopic image data to obtain colorectal region of interest data;

[0062] A colorectal image multi-level segmentation module, used for performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data;

[0063] A colorectal image segmentation depth processing module is used to perform depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and to perform three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data;

[0064] The colorectal cancer diagnosis auxiliary operation module is used to extract cell relationship features from colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and perform colorectal cancer diagnosis auxiliary operations based on colorectal image depth feature data and colorectal image cell relationship feature data.

[0065] The beneficial effects of the present invention are: through a multi-level image segmentation strategy, combined with different regions of interest data, the first colorectal image segmentation data and the second colorectal image segmentation data are generated respectively, which can more accurately capture the subtle features of the region of interest, and improve the accuracy and stability of image segmentation. Through the hierarchical management of the first colorectal image segmentation data and the second colorectal image segmentation data, the region of interest can be described in detail from different angles and levels, avoiding the problem of insufficient information or redundancy caused by a single segmentation method. By performing deep feature extraction on the first colorectal image segmentation data, multi-level image feature information can be learned, including texture, shape, structure and other features, and effective combination can better express the key features in the image. By constructing the second colorectal image segmentation data as three-dimensional network data, the spatial features of the region of interest can be extended from a two-dimensional plane to a three-dimensional space, and the distribution, morphology and structural relationship of the region of interest in the three-dimensional space can be accurately expressed. Cell relationship feature extraction can deeply explore the details of local features in three-dimensional network data, capture detailed information of each voxel node and its surrounding neighborhood, and realize a refined description of the local structure inside the region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting implementations made with reference to the following drawings:

[0067] Figure 1 A flowchart of a method for assisting diagnosis of colorectal cancer based on deep learning according to an embodiment is shown;

[0068] Figure 2 A flowchart showing the steps of a method for locating a colorectal region of interest according to an embodiment is shown;

[0069] Figure 3 A flowchart of the steps of a colorectal image multi-level segmentation method according to an embodiment is shown;

[0070] Figure 4 A flowchart of the steps of a colorectal image segmentation depth processing method according to an embodiment is shown;

[0071] Figure 5 A flowchart of the steps of a method for colorectal image cell relationship feature extraction and colorectal cancer diagnosis assistance operation according to an embodiment is shown. DETAILED DESCRIPTION

[0072] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0073] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0074] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0075] A dataset of real colorectal endoscopy images from a medical center, including endoscopic examination images of 500 patients, totaling about 10,000 high-resolution colorectal images. Each image has been annotated by clinical experts, with different lesion areas (such as polyps, adenomas, tumors, etc.) marked. The dataset is divided into a training set (70%), a validation set (15%), and a test set (15%) to evaluate the performance of the model.

[0076] All images were standardized, such as illumination correction, image alignment, and noise removal. The images were resized to 512×512 pixels, and the initial region of interest (ROI) was generated based on the annotation data. The YOLO (You Only Look Once) target detection algorithm based on deep learning was used to quickly detect and locate the lesion area in the image. The preprocessed colorectal endoscopy image data was input. The region of interest data was output, including the bounding box coordinates and confidence scores of the possible lesion area in each image. The endoscopic image was input into the pre-trained YOLO model, and the model output multiple bounding boxes and the confidence of each box. By setting a confidence threshold (such as 0.5), the detection results with low confidence were filtered out. The region of interest (ROI) data was generated for each detected lesion area, and the corresponding image fragment was saved for subsequent processing.

[0077] Use a U-Net-based deep convolutional neural network model for primary segmentation. Input image segments in the region of interest data. Output preliminary segmented lesion contour images (such as the contours of polyps, tumors, etc.). Input each ROI image segment into the U-Net model, and the model outputs a binary segmentation image of the lesion area. Perform morphological operations (such as denoising and filling) on ​​the segmentation results to optimize the lesion boundaries.

[0078] Use the Mask R-CNN model for more refined lesion segmentation. Input the primary segmentation result and the original region of interest image fragment. Output a refined lesion segmentation image, including the detailed features inside the lesion. Use the primary segmentation result as the prior information of Mask R-CNN to improve the model's attention to the lesion boundary and internal structure. The Mask R-CNN model outputs a finely segmented image of the lesion area and annotates the lesion type (such as polyps, adenomas, etc.). Combine the primary and secondary segmentation results to generate multi-level segmentation data.

[0079] Use the pre-trained ResNet-50 model for deep feature extraction. Input the lesion area image obtained by multi-level segmentation. Output the deep feature vector of the lesion area, which represents the high-level feature information of the lesion. Input the lesion segmentation image into the ResNet-50 model to extract the global feature vector of the image (such as texture, shape, edge, etc.). Perform dimensionality reduction and normalization on the feature vector as input data for subsequent diagnostic assistance.

[0080] The lesion slice data obtained by multi-level segmentation are stacked in spatial order to generate three-dimensional volume data. The stacked data are converted into three dimensions using a three-dimensional reconstruction algorithm (such as Marching Cubes) to generate a three-dimensional model. The three-dimensional model is converted into a regular grid structure using a three-dimensional gridding technique (such as Voxelization), and each voxel represents the lesion feature of a spatial unit. The three-dimensional network data is constructed, including the feature vector of each voxel and the spatial relationship between voxels.

[0081] Extract the features of lesion voxels from the three-dimensional grid data as node data, including the spatial position, texture features, density, etc. of each voxel. Analyze the spatial relationship and similarity between adjacent voxels to construct relationship edge data. Integrate the voxel node and relationship edge data into a three-dimensional relationship graph. Use the graph convolutional network (GCN) to process the three-dimensional relationship graph and extract local and global features of the lesion area. Analyze the neighborhood structure of each lesion node and extract local features such as distribution characteristics, connection strength and similarity of the neighborhood nodes. Analyze the global structural features of the entire relationship graph, such as the shortest path between nodes, global clustering coefficient, connectivity, etc. Feature fusion of local neighborhood features and global relationship features to generate lesion cell relationship feature data. Fuse with the deep feature data of the two-dimensional image to form a comprehensive lesion feature representation.

[0082] Use feature data to train classification models such as support vector machines (SVM) or random forests to classify lesions and predict risks. Build decision tree models for graded diagnosis of lesions and generation of treatment recommendations.

[0083] Real-time analysis of image data during endoscopy to identify early lesions and generate diagnostic reports. Real-time provision of 3D models and spatial relationships of lesions during surgery to assist doctors in precise positioning and resection. Regular analysis of postoperative endoscopic data to monitor lesion changes and predict recurrence risk.

[0084] In the test set, the accuracy of the diagnostic model reached 93%, and the recognition rate of early lesions was 90%, which is significantly higher than that of traditional methods. The recognition rate of complex lesions (such as multiple lesions with blurred boundaries) was significantly improved, effectively reducing misdiagnosis and missed diagnosis.

[0085] See also Figures 1 to 5 The present application provides a colorectal cancer diagnosis auxiliary method based on deep learning, comprising the following steps:

[0086] Step S1: acquiring colorectal endoscope image data, and locating a region of interest on the colorectal endoscope image data to obtain colorectal region of interest data;

[0087] In one embodiment, raw image data of the colorectal area is collected by an endoscopic device, and the data includes endoscopic views under different angles and lighting conditions. The acquired images are preprocessed, including denoising, lighting correction, image enhancement, etc., to improve the accuracy of subsequent positioning of the region of interest. The image is segmented using a sliding window technique or a region proposal method to preliminarily determine the possible lesion area. Feature extraction is performed on the cropped image blocks to identify features such as edges, textures, and shapes in order to identify possible lesion locations. A series of regions of interest (ROIs) are generated based on the feature information, which are candidate regions that may contain lesions. Morphological processing or image segmentation technology is used to further screen the candidate regions, remove regions with large errors, and obtain colorectal region of interest data.

[0088] Step S2: performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data;

[0089] In one embodiment, a preliminary image segmentation technique (such as threshold segmentation or edge detection) is used to perform preliminary segmentation of the region of interest to obtain a rough lesion outline. The preliminary segmentation result is subjected to boundary optimization and contour refinement to eliminate noise and artifacts. Segmentation techniques, such as region growing or level set methods, are used to obtain a more accurate segmentation result (first colorectal image segmentation data). Based on the first segmentation, a second segmentation is performed in combination with contextual information or other hierarchical information. Multi-scale processing or different feature fusion strategies are used to obtain different levels of lesion information (second colorectal image segmentation data). Multi-level segmentation data are fused to generate a comprehensive segmentation result containing different information levels, providing a more comprehensive description of the lesion area.

[0090] Step S3: performing depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and performing three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data;

[0091] In one embodiment, the first colorectal image segmentation data is standardized, such as size normalization, grayscale normalization, etc., to ensure the stability of feature extraction. High-level depth features are extracted from the processed image segmentation data. The feature extraction process includes convolution operations, pooling operations, etc. During the feature extraction process, high-level features such as texture, shape, and edges of the lesion area may be captured to obtain a set of high-dimensional depth feature vectors. The extracted depth features are high-dimensional vectors. In order to facilitate subsequent analysis, the feature vectors need to be reduced in dimension or clustered to form depth feature data.

[0092] According to the second colorectal image segmentation data, the 2D image data at different angles are spliced ​​into a three-dimensional structure to form three-dimensional image data. The three-dimensional image data is meshed or voxelized to establish a three-dimensional network topology. In the three-dimensional network, each node represents an image voxel, and the edges between nodes represent the spatial relationship between voxels. Feature information is assigned to each node, such as the depth feature and position of the voxel. Relational features are assigned to the edges between nodes, such as the similarity and distance of adjacent voxels.

[0093] Step S4: extracting cell relationship features from the colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and performing auxiliary colorectal cancer diagnosis operations based on the colorectal image depth feature data and the colorectal image cell relationship feature data.

[0094] In one embodiment, in a three-dimensional network, with each node as the center, the features of its surrounding neighboring nodes are extracted to analyze local spatial relationships, such as local clustering, density, etc. By traversing the entire three-dimensional network, global features such as connectivity, topological structure, etc. are extracted to analyze the spatial distribution and structural features of the entire three-dimensional network. The extracted local and global relationship features are formed into a relationship graph to describe the spatial relationship and structural pattern between lesion cells.

[0095] The colorectal image depth feature data and cell relationship feature data are fused to form a comprehensive feature vector. The fusion method can be simple splicing or weighted averaging. The fused feature vector is input into the auxiliary diagnosis model. The model can be a classification model, a regression model, etc., which determines whether the lesion is cancerous based on the input features. The model outputs the diagnosis result, analyzes the location and severity of the lesion, and provides auxiliary diagnosis suggestions for doctors.

[0096] In the present invention, by locating the region of interest (ROI), the focus is focused on the area where the species of interest may exist, reducing the interference of irrelevant areas and improving the diagnostic efficiency. The multi-level segmentation method can refine the image in a hierarchical and precise manner, not only can the macroscopic structural information (such as the outline of the entire region of interest) be extracted, but also the microscopic details (such as the tissue structure inside the region of interest) can be captured, which is helpful for accurate diagnosis. For the region of interest with complex morphology (such as irregular tumors or multiple lesions), the multi-level segmentation can analyze its morphological characteristics step by step, and enhance the robustness of the model when dealing with complex situations. By extracting high-level image features (such as edges, textures, shapes, etc.) through a deep learning network, the essential features of the region of interest can be expressed more accurately, and the recognition ability of the model for the region of interest can be improved. The three-dimensional network can integrate multiple two-dimensional slice images into three-dimensional volume data, capture the spatial structure of the region of interest and the relationship between tissues, and is particularly suitable for detecting the expansion and distribution of the species of interest. Cell relationship features can reveal the interaction patterns (such as density, connectivity, etc.) between cells or regions of interest, and help identify the microenvironmental characteristics of the region of interest and the diffusion path of cancer cells.

[0097] Preferably, the colorectal endoscope image data includes first colorectal endoscope image data and second colorectal endoscope image data, and step S1 is specifically:

[0098] Step S11: acquiring a colorectal image using a high-definition camera at a first light source brightness threshold and a first advancement speed threshold to obtain first colorectal endoscope image data;

[0099] In one embodiment, the focal length and focus parameters of the high-definition camera are adjusted to ensure that the internal tissue structure of the colorectum can be clearly captured. The light source brightness is set to the first light source brightness threshold to ensure that the brightness during image acquisition is sufficient but not overexposed. The propulsion speed is adjusted to the first propulsion speed threshold to ensure stability during image acquisition and reduce image blur. According to the preset brightness and propulsion speed, endoscopic images are collected from the colon to the rectum in sequence. During the acquisition process, the light source brightness and propulsion speed are adjusted in real time to ensure uniform illumination and image clarity of the entire colorectal area.

[0100] Step S12: acquiring a colorectal image using an ultra-high-definition camera with a second light source brightness threshold and a second advancement speed threshold to obtain second colorectal endoscopic image data, wherein the first light source brightness threshold is greater than or equal to the second light source brightness threshold, and the first advancement speed threshold is less than or equal to the second advancement speed threshold;

[0101] In one embodiment, an ultra-high-definition camera is used, and the focal length and focus parameters are adjusted to obtain images with higher resolution. The light source brightness is set to the second light source brightness threshold to ensure that the ultra-high-definition camera can still capture images with rich details at lower brightness. The propulsion speed is adjusted to the second propulsion speed threshold to allow a slightly faster propulsion speed and improve the collection efficiency.

[0102] The second set of colorectal endoscope image data is collected by preset light source brightness and advancement speed, focusing on the detailed areas that the HD camera cannot fully capture. During the advancement process, the HD camera data is combined for real-time comparison and correction to ensure the consistency of the images collected by the ultra-high-definition camera and the HD image data.

[0103] Step S13: extracting texture features from the colorectal endoscope image data to obtain colorectal texture feature data;

[0104] In one embodiment, edge detection or region segmentation technology is used to segment the first and second colorectal endoscopic image data to separate the colorectal wall tissue from the background. For the segmented image, local texture features are extracted, such as gray-level co-occurrence matrix (GLCM), local binary pattern (LBP), etc., to capture the texture patterns of different regions. Global features of the image are extracted, such as histograms, texture direction distribution, etc., to comprehensively describe the overall texture characteristics of the colorectal surface. All extracted texture features are quantized to generate high-dimensional feature vectors, each of which corresponds to specific texture feature information in an image.

[0105] Step S14: performing clustering calculation on the colorectal texture feature data to obtain colorectal texture clustering feature data;

[0106] In one embodiment, the texture feature vector is standardized to eliminate the dimensional differences between different features and ensure the accuracy of clustering calculation. The high-dimensional feature vector is subjected to dimensionality reduction processing, such as principal component analysis (PCA) and other methods, to reduce the computational complexity and retain the main feature information. The feature data after dimensionality reduction is clustered. By calculating the distance between the feature vectors, image data with similar texture features are classified into the same category. The clustering result assigns a cluster label to each image, indicating the texture category to which it belongs.

[0107] Step S15: performing similarity calculation based on the colorectal texture clustering feature data and the preset conventional colorectal texture clustering feature data to obtain texture clustering feature similarity data;

[0108] In one embodiment, a database containing conventional colorectal texture features is established, including texture features of healthy tissues and different types of interest. Each texture feature obtained by clustering is compared with the data in the preset feature library to calculate its similarity. The similarity calculation can use a metric method such as Euclidean distance and cosine similarity. According to the similarity calculation result, the colorectal texture cluster feature data is matched with the closest feature in the feature library, and its category is marked.

[0109] Step S16: Perform multi-level annotation on the colorectal endoscope image data according to the texture clustering feature similarity data to obtain the colorectal region of interest data.

[0110] In one embodiment, based on the similarity data, each clustered area in the image is labeled as a specific texture type, such as healthy tissue, polyps, cancerous areas, etc. The annotation information is attached to the original image in the form of color or shape markings to form a preliminary annotated image. Combined with the primary annotation results, the contextual relationship of the annotated area is further analyzed to perform multi-level annotation. For example, multiple adjacent lesions are labeled as high-level areas of interest. A confidence evaluation mechanism is added to the multi-level annotation, and the credibility of the annotation results is evaluated to provide a confidence score for each annotated area. The areas with high confidence in the multi-level annotation are extracted to form the area of ​​interest data for the colorectum. These areas may be high-risk lesions that require further diagnosis or treatment.

[0111] Preferably, the colorectal region of interest data includes first colorectal region of interest data, second colorectal region of interest data and third colorectal region of interest data, and the multi-level annotation is specifically as follows:

[0112] When it is determined that the texture clustering feature similarity data is less than or equal to a preset first texture clustering feature similarity threshold, a first region of interest labeling is performed on an image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data;

[0113] In one embodiment, the texture clustering feature similarity of all colorectal endoscopic image data is compared with a preset first similarity threshold to determine which image pixel areas have a similarity less than or equal to the threshold. Pixel areas with a similarity less than or equal to the threshold are extracted, and their specific locations in the endoscopic image are marked. These areas usually correspond to healthy or known low-risk tissue types. These pixel areas are labeled as the first region of interest and marked on the image using a specific color or symbol to form the first colorectal region of interest data. The labeling indicates that these areas in the image meet the preset healthy or low-risk texture features and need to be marked as areas of concern.

[0114] When it is determined that the texture clustering feature similarity data is greater than a preset first texture clustering feature similarity threshold and less than or equal to a preset second texture clustering feature similarity threshold, high-frequency component feature extraction is performed on the colorectal texture clustering feature data to obtain colorectal texture high-frequency component data, and shape feature extraction is performed on the colorectal endoscope image data to obtain colorectal image shape feature data;

[0115] In one embodiment, image pixel regions whose texture clustering feature similarity is greater than a first threshold and less than or equal to a second threshold are screened out. The similarity of these regions is at an intermediate level and contains potential tissues of interest. High-frequency component features are extracted from the screened image regions. Detailed parts of the image, such as edges, texture mutation regions, etc., are extracted through edge detection or filtering technology to obtain colorectal texture high-frequency component data. Shape description methods, such as shape moments or contour features, are used to analyze the shape of the target area in the image to obtain colorectal image shape feature data. Shape features can describe the geometric properties of lesions, such as aspect ratio, area, boundary complexity, etc.

[0116] Calculating the interest index according to the colorectal texture high frequency component data and the colorectal image shape feature data to obtain the colorectal image interest index data;

[0117] In one embodiment, the colorectal texture high frequency component data and the colorectal image shape feature data are fused. A comprehensive feature vector is generated by feature splicing, weighted averaging, etc. The feature vector is used to calculate the interest index. The index can be calculated based on the distance metric, weighted sum, etc. of the feature vector, indicating the probability or risk level of the target area becoming a category of interest. The higher the interest index, the greater the possibility of the target area being a category of interest, and further annotation and attention are required.

[0118] When it is determined that the colorectal image interest index data is less than or equal to a preset colorectal image interest index threshold, a first region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data;

[0119] In one embodiment, the calculated interest index is compared with a preset interest index threshold, and regions with an interest index less than or equal to the threshold are screened out. These regions are marked as low-risk regions using the same or different marking colors or symbols as in step 1. The first colorectal region of interest data is updated, including the original low-similarity region and the current low interest index region.

[0120] When it is determined that the colorectal image interest index data is greater than a preset colorectal image interest index threshold, a second region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain second colorectal region of interest data;

[0121] In one embodiment, regions with an interest index greater than a preset threshold are determined, which have a higher probability of the category of interest or abnormality. These high-risk regions are marked as second regions of interest, highlighted with different colors or symbols, and identified as key regions that require further analysis. Second colorectal region of interest data are generated, with emphasis on marking all high-risk regions.

[0122] When it is determined that the texture clustering feature similarity data is greater than the preset second texture clustering feature similarity threshold, a third region of interest annotation is performed on the image pixel area corresponding to the colorectal endoscopy image data corresponding to the texture clustering feature similarity data to obtain third colorectal region of interest data, wherein the first region of interest annotation, the second region of interest annotation and the third region of interest annotation are region of interest annotation methods containing different region of interest annotation contents.

[0123] In one embodiment, pixel regions whose texture clustering feature similarity is greater than a second threshold are screened out. The similarity of these regions is very high, corresponding to known high-risk categories of interest features. These extremely high-risk categories of interest regions are annotated as third regions of interest. The annotation method uses more eye-catching symbols or colors to indicate warnings or special attention. A third colorectal region of interest data is generated, which includes information on all high-similarity categories of interest regions.

[0124] Preferably, step S2 specifically includes:

[0125] Step S21: performing a first image segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain first colorectal image segmentation data;

[0126] In one embodiment, the second colorectal region of interest data is used to extract all image portions marked as the second region of interest from the endoscopic image. These regions generally correspond to regions of interest of medium risk category. During the extraction process, the original image can be cropped or non-regions of interest can be masked to reduce the interference of non-target regions on the segmentation. The extracted region of interest is preprocessed. This includes operations such as denoising, histogram equalization, and contrast enhancement to highlight the features of the region of interest category. If necessary, the image can be scaled or normalized to ensure the stability of the segmentation algorithm at different image scales;

[0127] An image segmentation method based on edge detection and region growing is applied. This method first identifies obvious boundaries in the image through edge detection, and then uses region growing technology to aggregate similar pixels into a region. During the segmentation process, appropriate seed points and growth thresholds are set to ensure that region growing can accurately outline the lesion boundaries;

[0128] Optimize the boundaries of the region obtained by the preliminary segmentation. Noise and artifacts can be removed by morphological processing (such as dilation, erosion) or contour smoothing algorithms (such as level set method) to make the lesion boundary smoother and more accurate. The optimized segmentation data is the first colorectal image segmentation data. Compare the segmentation results with the annotated data of the second region of interest to verify the accuracy and completeness of the segmented region. For inaccurately segmented areas, manual or semi-automatic adjustments and corrections can be made.

[0129] Step S22: performing a second image segmentation on the colorectal endoscopic image data according to the third colorectal region of interest data to obtain second colorectal image segmentation data, wherein the first image segmentation and the second image segmentation are different image segmentation methods.

[0130] In one embodiment, the third colorectal region of interest data is used to extract all image portions marked as the third region of interest from the endoscopic image. These regions usually correspond to high-risk categories of interest or known lesion regions. These regions are processed separately to remove background information unrelated to the lesion to reduce segmentation noise;

[0131] Perform more sophisticated preprocessing on the extracted region of interest. This includes denoising methods (such as multi-level fusion filtering, for example, filtering low-frequency filtering and high-frequency filtering and fusing the results), multi-scale image enhancement (such as Laplacian pyramid), etc., to retain the detailed information of the lesion to the greatest extent. For high-resolution images, block processing can be used to ensure that the details are not ignored;

[0132] A deep learning-based segmentation method is applied. A pre-trained neural network model is used to classify the image at the pixel level, and each pixel in the image is divided into a lesion area and a non-lesion area. During the segmentation process, the input image undergoes a multi-layer convolution operation to extract high-level features, and combines contextual information to accurately identify complex lesion shapes and boundaries. The same image area is segmented multiple times using different segmentation models to obtain multiple segmentation results, and then the final segmentation result is obtained through a fusion strategy (such as majority voting). The segmentation results output by the deep learning model are post-processed to remove small areas of noise and refine the boundaries of the lesion area. Methods such as conditional random fields (CRF) can be used to take into account the spatial consistency of pixels to improve the accuracy of segmentation.

[0133] Preferably, the first image segmentation is specifically:

[0134] Performing edge enhancement on colorectal endoscope image data to obtain colorectal image edge enhancement data;

[0135] In one embodiment, an edge detection operator, such as a gradient operator or a differential operator (such as Sobel, Prewitt, etc.), is applied to the preprocessed image to calculate the gradient size and direction of each pixel in the image. Based on the calculated gradient information, pixels with large gradient values ​​are enhanced so that the pixel values ​​of the edge portion of the image are higher and the pixel values ​​of the non-edge portion are reduced, thereby forming an edge-enhanced image. In order to avoid noise amplification caused by over-enhancement, a non-maximum suppression method can be used to suppress insignificant edges, and a double threshold process can be used to filter out weak edges.

[0136] Performing image segmentation on the edge enhancement data of the colorectal image according to the second colorectal region of interest data to obtain primary first colorectal image segmentation data;

[0137] In one embodiment, the second colorectal region of interest data is used to generate an image mask, in which all regions of interest are marked, and other non-interested regions are set to black or transparent. The generated mask is used to cut the edge enhancement data of the colorectal image. The region of interest corresponding to the mask is retained, and the non-interested regions are set to zero or masked. The cut image only contains the enhanced edge regions of interest, and the edge information of these regions is clear and obvious. The cut region of interest data is binarized to further separate the edge of the lesion from the background. The binarization results are subjected to morphological processing, such as dilation, corrosion, etc., to remove isolated noise points and fill small holes to make the outline of the lesion area more coherent. The primary first colorectal image segmentation data is obtained, which contains the preliminary segmented lesion boundary information.

[0138] Performing regional segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain secondary first colorectal image segmentation data;

[0139] In one embodiment, in the original colorectal endoscopy image, the region of interest is extracted using the data of the second region of interest to obtain the original image of the region of interest. The extracted region of interest is smoothed using morphological operations (such as opening and closing operations) to remove local small noise. A region-based segmentation method, such as region growing or threshold-based segmentation, is applied to the extracted region of interest. The region growing method clusters similar pixels into the same region starting from the seed point based on pixel similarity. For threshold segmentation, a multi-level threshold method can be used to automatically select different segmentation thresholds for different brightness or texture features in the region of interest to ensure that lesion areas of different properties can be accurately segmented. The segmented regional data is subjected to boundary detection, and the boundary contours of each segmented region are extracted to form the secondary first colorectal image segmentation data. The segmentation results are subjected to connected domain analysis to remove isolated segmented regions with too small an area, and multiple connected domains are merged to form a continuous lesion segmentation region.

[0140] The primary first colorectal image segmentation data and the secondary first colorectal image segmentation data are fused to obtain the first colorectal image segmentation data.

[0141] In one embodiment, the primary and secondary segmentation data are spatially aligned. To ensure that the two are consistent in coordinate system and pixel resolution, an image registration algorithm can be used to accurately align them. If the primary and secondary segmentation data are different in size, interpolation or resampling methods can be used to adjust them to the same size.

[0142] Fuse the primary and secondary segmentation data. A pixel-level fusion strategy can be used to overlay the edge information in the primary data with the region information in the secondary data. For overlapping areas, the boundary information in the primary segmentation data is retained first, and the area in the secondary data is filled into the lesion to ensure the integrity of the segmented area. Alternatively, the weighted averaging method is used to perform weighted fusion of the two segmentation data to obtain a smoother and more accurate segmentation result.

[0143] The fused image segmentation data is subjected to boundary smoothing to eliminate the noise and discontinuous boundaries generated during the fusion process. The fused results are subjected to regional merging and boundary refinement to ensure that the contour of the lesion area is clear and complete. The first colorectal image segmentation data is output, which includes the complete lesion boundary and internal area information after fusion.

[0144] The edge enhancement in the present invention can effectively enhance the edge significance of the lesion area in the colorectal endoscope image and improve the contrast between the lesion area and the surrounding normal tissue. Primary segmentation is performed on the basis of edge enhancement, and the approximate area of ​​the lesion can be quickly located. The image after edge enhancement can better express the morphological characteristics of the lesion, especially for the lesion area with complex boundaries, and can effectively improve the accuracy of the initial segmentation. According to the second region of interest data, the image after edge enhancement is cut, and the approximate outline of the lesion area can be quickly segmented. The main purpose of the primary segmentation process is to roughly identify the lesion area, and the complex image segmentation task is preliminarily simplified, which helps to improve the overall segmentation efficiency. According to the second region of interest data, the original image is segmented again, and the detailed features inside the lesion area can be captured more finely. Secondary segmentation makes up for the shortcomings of primary segmentation in detail recognition, and captures the small and complex structural changes in the lesion area through a more refined segmentation method. The results of the two segmentation methods are merged, and the segmentation accuracy of the internal structure can be improved while ensuring the accuracy of the overall segmentation contour.

[0145] Preferably, the second image segmentation is specifically:

[0146] Gaussian distribution feature extraction is performed based on colorectal texture clustering feature data to obtain texture clustering Gaussian distribution feature data;

[0147] In one embodiment, texture clustering feature data of each region of interest in the colorectal endoscopy image is extracted. These feature data may include information such as grayscale, texture, shape, etc. of different regions in the image. The extracted feature data is standardized to ensure the dimensional consistency between different feature dimensions, providing a good data basis for subsequent Gaussian distribution modeling.

[0148] Model the standardized texture clustering feature data. In general, these data conform to the multidimensional Gaussian distribution. Calculate the mean vector and covariance matrix of each feature data set to obtain the parameters of the multidimensional Gaussian distribution. Use these parameters to generate a Gaussian distribution model, and perform probability density calculation on each feature data set to obtain the probability value of each feature vector under the distribution. The distribution characteristics of the probability density are used to analyze the feature distribution of different types of interest. For different types of lesion areas, multiple Gaussian distribution models are constructed to distinguish different types of interest. For the texture clustering feature data of each region of interest, its probability density value is calculated according to the Gaussian distribution model. The higher the probability density value, the more the characteristics of the region conform to the description of the Gaussian distribution model. The calculated probability density value is used as the texture clustering Gaussian distribution feature data. These data will be used to measure the possibility that each pixel area belongs to different types of interest.

[0149] Performing pixel classification on the colorectal endoscope image data according to the texture clustering Gaussian distribution feature data and the third colorectal region of interest data to obtain colorectal image pixel classification data;

[0150] In one embodiment, the third colorectal region of interest data is used to extract all high-interest regions in the colorectal endoscope image. The pixels in these regions are used as the target objects for classification, and other regions do not participate in pixel classification. The extracted region of interest data is preprocessed, such as denoising, contrast enhancement, etc., to ensure the accuracy of pixel features during classification. For each pixel in the region of interest, its texture features (such as grayscale, texture direction, etc.) and geometric features (such as position, shape, etc.) are extracted. These pixel features are matched with the texture clustering Gaussian distribution feature data generated in step 1, and the probability density value of each pixel under different Gaussian distribution models is calculated. For each pixel, the most likely category of interest is determined according to its probability density value under different Gaussian distribution models. The category corresponding to the model with the highest probability density value is selected as the classification result of the pixel. Using a soft classification strategy, probability weights of multiple categories can be assigned to each pixel, indicating the probability distribution of pixels belonging to different categories of interest. Finally, the classification results of each pixel are summarized to obtain pixel classification data of the colorectal image. The data contains labels or probability information that each pixel belongs to different categories of interest.

[0151] Mapping and segmenting the colorectal image pixel classification data to obtain second colorectal image segmentation data.

[0152] In one embodiment, a mapping strategy is formulated based on the pixel classification data in step 2. Different categories of interest correspond to different mapping rules. Color coding or label numbering can be used to map pixels of different categories to corresponding segmented areas. If a probability classification result is used, a threshold can be set to map pixels with probability values ​​higher than a certain threshold to corresponding categories of interest. For multiple categories, the final category can be determined by majority voting or weighted average.

[0153] A mapping strategy is applied to the pixel classification data to map each pixel to its corresponding category of interest to generate a segmentation mask image. Each mask region represents a different type of interest, such as healthy interest, benign interest, malignant interest, etc. Connected domain analysis is performed on the mask image to merge isolated small areas into the surrounding main areas, remove the influence of noise, and improve the continuity and integrity of the segmentation results. The boundary of the mask image after preliminary segmentation is refined. Morphological processing (such as dilation and erosion) can be used to smooth the segmentation boundary and remove small burrs or irregular edges. The boundary of the mask image is further processed using an image smoothing algorithm (such as Gaussian smoothing) to ensure that the boundary of the segmented area is smooth and natural. The final mapped segmentation mask image is output as the second colorectal image segmentation data. This data contains detailed segmentation information for all high-risk interest category regions and identifies the specific distribution of different interest category categories.

[0154] Preferably, step S3 is specifically:

[0155] Step S31: performing multi-scale feature extraction on the first colorectal image segmentation data to obtain multi-scale feature data of the colorectal image;

[0156] In one embodiment, multi-scale processing is performed on the first colorectal image segmentation data. An image pyramid method is usually used to generate image representations of different resolutions. For example, the original image can be reduced step by step to form a multi-layer pyramid structure. Each layer of the pyramid image represents a different spatial scale, the low-resolution layer captures global features, and the high-resolution layer retains detail information;

[0157] At each scale level, different filters or convolution kernels are used to extract image features. These filters can capture information such as edges, textures, shapes, etc. of the image. For each scale level, features such as local binary patterns (LBP) and histograms of oriented gradients (HOG) are extracted to form feature vectors for each scale layer.

[0158] Features at different scale levels are fused. Features of different resolutions can be combined into a multidimensional feature vector using methods such as feature splicing and weighted averaging to represent the features of each pixel or region at different scales. The fusion strategy eliminates redundant information between features of different scales to form the final multi-scale feature data. The feature vector after multi-scale fusion is used as the feature representation of each pixel or region to obtain multi-scale feature data of colorectal images. This data contains image information at different scales and can more comprehensively describe the structure and texture characteristics of the region of interest.

[0159] Step S32: performing multi-head self-attention calculation on the multi-scale feature data of the colorectal image to obtain the depth feature data of the colorectal image;

[0160] In one embodiment, for each pixel or region feature in the multi-scale feature data, a multi-head self-attention mechanism is used for feature interaction. Each attention head independently calculates the similarity weight between features to capture feature dependencies at different levels and in different local areas;

[0161] Calculate the similarity between each feature vector and all other feature vectors to generate an attention weight matrix. The similarity calculation is usually based on the dot product or cosine similarity of the feature vectors. Use the attention weight matrix to weighted sum all features to generate a self-attention feature representation for each pixel or region; the feature representations of multiple self-attention heads are concatenated or weighted averaged to generate a fused deep feature representation. Each head captures different contextual information, making the final feature representation richer and more comprehensive. The information of each head is integrated into a unified deep feature representation through feature fusion. The fused feature representation is used as the deep feature data of the colorectal image. This data contains rich spatial context information and multi-scale feature interaction information, which can more accurately describe the complex features of the region of interest.

[0162] Step S33: Slice and stack the second colorectal image segmentation data to obtain slice stack data;

[0163] In one embodiment, the second colorectal image segmentation data is divided into multiple continuous two-dimensional slices in sequence. These slices represent the distribution of the rigid interest regions at different levels of the colorectum. Ensure that there is a certain degree of overlap between the slices to ensure the continuity and consistency of subsequent stacking and three-dimensional reconstruction. Each slice is standardized, such as size normalization, grayscale normalization, etc., to ensure the consistency of all slices in space and intensity. Remove noise and artifacts in the slices to ensure smooth transitions between slices. All slices are stacked in sequence to form a three-dimensional data cube. During the stacking process, the spatial consistency of the slices must be maintained to avoid dislocation and mismatch. Interpolation methods can be used to fill the gaps between slices to enhance the continuity of slice stacking. After the stacking is completed, a three-dimensional data cube is generated as slice stacking data. The data contains segmentation information at different levels of the colorectum, laying the foundation for subsequent three-dimensional conversion and analysis.

[0164] Step S34: performing three-dimensional conversion on the slice stacking data to obtain slice three-dimensional conversion data;

[0165] In one embodiment, the pixels of each two-dimensional slice in the stacked data are mapped into a three-dimensional space. Each slice corresponds to a fixed z coordinate value in the stacking order. According to the position and intensity information of the pixel, the two-dimensional pixel is converted into a three-dimensional voxel (volumetric pixel) to construct a segmentation representation of the three-dimensional space. All slices are three-dimensionally reconstructed to generate a three-dimensional distribution map of the region of interest. A three-dimensional interpolation method can be used during the reconstruction process to smooth the transition between slices. The overlapping information between slices is integrated into the three-dimensional structure to avoid data breaks caused by slice intervals. The three-dimensional reconstructed data is standardized to ensure that the intensity value and spatial position of each voxel are consistent with the original data. Isolated voxels or irregular areas in the three-dimensional data are removed to enhance the integrity of the three-dimensional data. The generated slice three-dimensional conversion data contains the three-dimensional spatial distribution information of all slices, providing a basis for subsequent three-dimensional gridding.

[0166] Step S35: Perform three-dimensional meshing on the slice three-dimensional conversion data to obtain three-dimensional network data of the colorectal image.

[0167] In one embodiment, the slice three-dimensional conversion data is gridded to convert the three-dimensional voxel data into a three-dimensional grid representation. The gridding method generally uses triangulation or voxelization technology. In each three-dimensional grid unit, the corresponding voxel information is retained, including the strength, position, neighborhood relationship, etc. of the voxel. The topological structure of the three-dimensional grid is established to define the connection relationship between each grid unit. Octree or grid division technology can be used to divide the three-dimensional space into regular or irregular grid units. Ensure that the connection relationship between all grid units is accurate and continuous, and provide a stable three-dimensional network structure for subsequent analysis. The feature information of interest in the slice three-dimensional conversion data is assigned to the corresponding three-dimensional grid unit. The attributes of each grid unit may include information such as voxel density and category of interest. Spatial relationship attributes such as distance, adjacent relationship, etc. are assigned to the edges between grid units to form complete three-dimensional grid network data. The final colorectal image three-dimensional network data contains the attribute information and topological structure of all three-dimensional grid units. These data can accurately describe the three-dimensional spatial distribution and internal structural relationship of the colorectal region of interest, providing a reliable basis for further analysis and processing.

[0168] Preferably, step S4 is specifically:

[0169] Step S41: performing voxel region node extraction and voxel region relationship edge extraction on the colorectal image three-dimensional network data to obtain voxel region node data and voxel region relationship edge data respectively;

[0170] In one embodiment, each voxel in the three-dimensional network data is regarded as a node. According to the location and attributes (such as density, category of interest, etc.) of the voxel, all voxel node information is extracted. For continuous voxel regions with the same attributes, they can be merged into a node representation. The merging criterion can be based on the spatial connectivity and attribute similarity of the voxels, and adjacent homogeneous voxels can be regarded as the same node. The node information includes the spatial position (x, y, z coordinates) of the voxel, the attribute value (such as category of interest, intensity), and the volume of the node.

[0171] For each pair of adjacent voxel nodes, the corresponding relationship edge is established based on their spatial distance, attribute similarity and other information. If two nodes are adjacent in space and have high similarity in attributes, they are connected as an edge, indicating that there is a certain relationship between them (such as the expansion trend of the region of interest). For non-adjacent but strongly correlated nodes, long-range relationship edges can also be established, which reflect the similarity or linkage of the region of interest at a long distance.

[0172] All extracted node information is aggregated to generate voxel region node data. Each node contains its location and attribute information. All relationship edge information is aggregated to generate voxel region relationship edge data. Each edge contains its associated node and its relationship strength (such as similarity, spatial distance).

[0173] Step S42: constructing relationship graph data according to the voxel region node data and the voxel region relationship edge data to obtain colorectal relationship graph data;

[0174] In one embodiment, an empty graph data structure is created, the node set of the graph is empty, and the edge set is empty. Each node information in the voxel region node data is added to the node set of the graph, and a unique identifier (ID) is assigned to each node. According to the voxel region relationship edge data, the two node IDs corresponding to each edge are added to the edge set of the graph. A weight attribute is added to each edge, and the weight can be calculated based on the similarity, spatial distance, or other relationship strength between the nodes. Each node attribute (such as category, volume, etc.) in the voxel region node data is assigned to the corresponding node in the graph as the attribute information of the node. Each edge attribute (such as similarity, distance, etc.) in the voxel region relationship edge data is assigned to the corresponding edge in the graph as the attribute information of the edge. The generated colorectal relationship graph data contains complete information of all nodes and edges, reflecting the spatial distribution of the colorectal region of interest and its internal relationship structure.

[0175] Step S43: extracting local neighborhood features and global relationship features from the colorectal relationship graph data to obtain colorectal local neighborhood feature data and colorectal global relationship feature data, respectively;

[0176] In one embodiment, local neighborhood feature extraction: for each node, local features are extracted based on its direct neighbor node set (i.e., first-degree neighborhood). Local features may include: the number (degree) of neighbor nodes. Attribute distribution of neighbor nodes (such as category ratio and volume distribution of neighbor nodes). Similarity distribution between neighbor nodes (such as average similarity, maximum and minimum similarity). For each node, its neighborhood feature vector is quantized to form a local neighborhood feature vector;

[0177] Global relationship feature extraction: Analyze the global topological features of the entire relationship graph. Global features include: the overall connectivity of the graph (such as the diameter of the graph and the average shortest path). The clustering coefficient of the graph (indicating the degree of connectivity of the node among its neighbors). The centrality of the graph (such as degree centrality, betweenness centrality, etc., indicating the relative importance of the node in the graph). For each node, calculate its characteristic value in the global graph structure (such as global centrality, global influence, etc.) to form a global relationship feature vector;

[0178] The local neighborhood feature vectors of each node are aggregated to form the local neighborhood feature data of the colorectum. The global relationship feature vectors of each node are aggregated to form the global relationship feature data of the colorectum.

[0179] Step S44: performing feature fusion on the colorectal local neighborhood feature data and the colorectal global relationship feature data to obtain colorectal image cell relationship feature data;

[0180] In one embodiment, the local neighborhood feature data and the global relationship feature data are standardized respectively. The standardization method can use mean-variance normalization or minimum-maximum normalization to ensure the dimensional consistency between different feature dimensions. The local neighborhood feature vector and the global relationship feature vector of each node are spliced ​​to form a fused feature vector. The spliced ​​vector contains the local information and global information of the node. If the feature dimension is too high, a dimensionality reduction method (such as principal component analysis PCA) can be used to reduce the feature dimension to an appropriate range, retain the main information, and remove redundant features. The obtained colorectal image cell relationship feature data contains a comprehensive feature representation of each node, which can reflect the multidimensional information of the colorectal cell area.

[0181] Step S45: performing auxiliary colorectal cancer diagnosis operations based on the colorectal image depth feature data and the colorectal image cell relationship feature data.

[0182] In one embodiment, the colorectal image depth feature data is fused with the colorectal image cell relationship feature data. The fusion method can use feature splicing, weighted averaging, etc. to combine the image features with the graph structure features. The fused feature data is input into the pre-trained diagnostic model. The model is a classification model or a regression model, and its purpose is to predict the lesion type or severity of colorectal cancer based on the input features. The diagnostic model calculates based on the input features and outputs the diagnosis result. The result can be a lesion category (such as benign or malignant) or a score of the degree of the lesion. Based on the output results of the diagnostic model, corresponding auxiliary diagnosis suggestions are generated, such as area annotations that require attention, further examination suggestions, etc. The diagnostic results are visualized with the original image and the three-dimensional relationship diagram to provide doctors with intuitive areas of interest.

[0183] Preferably, local neighborhood feature extraction is specifically as follows:

[0184] Performing neighborhood extraction on the colorectal relationship graph data to obtain colorectal neighborhood data;

[0185] In one embodiment, for each node (voxel node) in the colorectal relationship graph data, its neighborhood is defined. The neighborhood is defined based on different criteria:

[0186] 1. Direct neighborhood (1st-order neighborhood): It consists of all nodes directly connected to the node, that is, the set of all nodes in the graph that are connected to the current node by edges.

[0187] 2. Extended neighborhood (n-th order neighborhood): It consists of multi-order nodes, including all direct neighbor nodes of the current node and the neighbors of these neighbors, up to the n-th order node.

[0188] 3. Distance-based neighborhood: It consists of nodes whose Euclidean distance or other similarity measure to the current node is within a certain range.

[0189] Traverse each node in the colorectal relationship graph and extract its neighborhood node set. For each node, record the IDs of all its neighboring nodes and the connection information between them and the current node (such as edge weight, spatial distance, etc.). Store the extracted neighborhood information as neighborhood data. Each node corresponds to a neighborhood data structure, which contains the ID of the neighboring node, node attributes (such as category, location) and its relationship with the current node (such as edge weight). Generate colorectal neighborhood data, which contains the neighborhood information of all nodes in the graph. This data will serve as the basis for subsequent local feature extraction.

[0190] In one embodiment, more importantly, neighborhood extraction is specifically:

[0191] Perform multi-level neighborhood initialization on the colorectal relationship graph data to obtain colorectal multi-level neighborhood data; perform local density feature extraction based on the multi-level colorectal multi-level neighborhood data to obtain neighborhood local density feature data; perform neighborhood adjustment on the colorectal multi-level neighborhood data based on the neighborhood local density feature data to obtain colorectal multi-level neighborhood adjustment data; perform multi-scale neighborhood merging on the colorectal multi-level neighborhood adjustment data to obtain colorectal neighborhood data;

[0192] For each node (voxel node) in the colorectal relationship graph data, a multi-level neighborhood structure is initialized. It includes direct neighborhood (1st order neighborhood), extended neighborhood (2nd order and above neighborhood) and adaptive neighborhood. The neighborhoods at each level are different in neighborhood size (i.e., neighborhood radius) and neighborhood depth (i.e., neighborhood level). The default neighborhood radius (e.g., 2mm, 5mm, 10mm) and number of neighborhood layers (e.g., 1st order, 2nd order, 3rd order) are set during initialization. By analyzing the local density and lesion information of the current node, the size and number of layers of the neighborhood are adaptively adjusted: if the node density in the neighborhood is high (e.g., the lesion area is densely distributed), the neighborhood radius is reduced to avoid noise interference. For highly suspicious regions of interest (e.g., nodes with high probability of historical regions of interest), the neighborhood radius and number of layers are increased to capture more neighborhood context information. Through this adjustment mechanism, different nodes can have neighborhoods of different sizes and depths to adapt to different lesion distributions. For each node, the neighborhood information of different levels is merged. The hierarchical information and neighborhood radius information of each neighborhood node are retained during the merging process. For the merged multi-scale neighborhood, the weight of the neighborhood node is calculated. The weight can be set based on factors such as the distance between the node and the central node, the lesion category of the node, and the number of neighborhood layers.

[0193] By merging and weighting multi-scale neighborhoods, the spatial structure and complex relationships around the lesion can be more comprehensively described.

[0194] Extracting local geometric features from colorectal neighborhood data to obtain local geometric feature data;

[0195] In one embodiment, the geometric properties of the neighborhood data of each node are calculated. The properties include: the spatial distribution of the neighborhood nodes, such as calculating the spatial distribution of the neighborhood nodes relative to the current node, such as the average position, variance, etc.; the spatial distance distribution of the neighborhood nodes, such as calculating the average distance, maximum and minimum distance, distance variance, etc. between the neighborhood nodes and the current node; the neighborhood density, such as calculating the ratio of the number of neighborhood nodes to the spatial volume, indicating the node density of the local area.

[0196] The neighborhood geometric attributes of each node are represented as feature vectors. Each geometric attribute value (such as average position, density, etc.) is used as a dimension of the feature vector, and all geometric attribute values ​​are combined to form a complete geometric feature vector. The feature vectors are normalized to ensure that the geometric feature vectors of different nodes can be compared at the same scale. The geometric feature vectors of each node are aggregated to generate local geometric feature data. This data contains the local geometric attribute information of all nodes and is used to describe the spatial distribution of nodes in their neighborhoods.

[0197] The local morphological features are extracted from the local geometric feature data to obtain the local neighborhood feature data of the colorectum.

[0198] In one embodiment, for each node, morphological attributes are calculated based on its local geometric feature data. The morphological attributes include:

[0199] 1. Local shape features are used to calculate the shape distribution of neighborhood nodes, such as the convex hull area, aspect ratio, surface area to volume ratio, etc. of the neighborhood;

[0200] 2. Local directional features are used to analyze the directional distribution of neighborhood nodes, such as the main direction and the variance of the directional distribution, to indicate the directionality and symmetry of the neighborhood;

[0201] 3. Neighborhood compactness is to calculate the compactness of neighborhood nodes, such as the average distance between neighborhood nodes, the radius of the minimum enclosing sphere, etc., which indicates the compactness of the neighborhood.

[0202] The morphological attributes of each node are represented as feature vectors. Each morphological attribute value (such as convex hull area, main direction, etc.) is used as a dimension of the feature vector, and all morphological attribute values ​​are combined to form a complete morphological feature vector. The morphological feature vectors are normalized to facilitate the comparison of morphological feature vectors of different nodes at the same scale. The geometric features are concatenated with the morphological features to form a complete local neighborhood feature vector. The local neighborhood feature vector includes the geometric and morphological information of the node in its neighborhood, and can describe the local environmental characteristics of the node. Generate colorectal local neighborhood feature data, which contains the local neighborhood feature vectors of all nodes.

[0203] Neighborhood extraction in the present invention can identify the neighborhood structure around each node (voxel) from the relationship graph data, and capture its local spatial relationship, including the distance, direction and connection mode between adjacent nodes. Local geometric feature extraction can quantify the geometric properties of the neighborhood structure, including the distance distribution, direction distribution, density distribution, etc. between neighborhood nodes. Local geometric features can keenly capture the tiny geometric changes inside the region of interest, such as tiny protrusions, depressions, boundary curvatures, etc. The extraction of geometric features can better distinguish regions of interest of different morphologies and complexities, help improve the ability to distinguish features, and provide an effective geometric basis for further local morphological feature extraction. Local morphological feature extraction can capture the morphological features of local areas of the region of interest, including surface shape, edge curvature, concavity, etc., and can carefully depict the surface morphological changes of the local area, especially for lesion areas with irregular morphology and complex boundaries. The introduction of morphological features can more accurately express its morphological features.

[0204] Preferably, the present application also provides a colorectal cancer diagnosis assistance system based on deep learning, which is used to execute the colorectal cancer diagnosis assistance method based on deep learning as described above. The colorectal cancer diagnosis assistance system based on deep learning includes:

[0205] A colorectal region of interest positioning module is used to obtain colorectal endoscopic image data and locate the region of interest on the colorectal endoscopic image data to obtain colorectal region of interest data;

[0206] A colorectal image multi-level segmentation module, used for performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data;

[0207] A colorectal image segmentation depth processing module is used to perform depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and to perform three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data;

[0208] The colorectal cancer diagnosis auxiliary operation module is used to extract cell relationship features from colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and perform colorectal cancer diagnosis auxiliary operations based on colorectal image depth feature data and colorectal image cell relationship feature data.

[0209] Therefore, from any point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes falling within the meaning and scope of equivalent elements of the application documents are included in the present invention.

[0210] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A colorectal cancer diagnosis auxiliary method based on deep learning, characterized in that: The following steps are involved: Step S1: acquiring colorectal endoscope image data, and locating a region of interest on the colorectal endoscope image data to obtain colorectal region of interest data; Step S2: performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data; Step S3: performing depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and performing three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data; Step S4: extracting cell relationship features from the colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and performing auxiliary colorectal cancer diagnosis operations based on the colorectal image depth feature data and the colorectal image cell relationship feature data.

2. The method according to claim 1, characterized in that The colorectal endoscope image data includes first colorectal endoscope image data and second colorectal endoscope image data, and step S1 is specifically as follows: Acquiring a colorectal image using a high-definition camera at a first light source brightness threshold and a first advancement speed threshold to obtain first colorectal endoscope image data; Acquiring a colorectal image using an ultra-high-definition camera with a second light source brightness threshold and a second advancement speed threshold to obtain second colorectal endoscopic image data, wherein the first light source brightness threshold is greater than or equal to the second light source brightness threshold, and the first advancement speed threshold is less than or equal to the second advancement speed threshold; Extracting texture features from colorectal endoscope image data to obtain colorectal texture feature data; Performing clustering calculation on the colorectal texture feature data to obtain colorectal texture clustering feature data; Performing similarity calculation based on the colorectal texture clustering feature data and the preset conventional colorectal texture clustering feature data to obtain texture clustering feature similarity data; The colorectal endoscopy image data is multi-level labeled according to the texture clustering feature similarity data to obtain the colorectal region of interest data.

3. The method according to claim 2, characterized in that The colorectal region of interest data includes the first colorectal region of interest data, the second colorectal region of interest data and the third colorectal region of interest data. The multi-level annotation is specifically as follows: When it is determined that the texture clustering feature similarity data is less than or equal to a preset first texture clustering feature similarity threshold, a first region of interest labeling is performed on an image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data; When it is determined that the texture clustering feature similarity data is greater than a preset first texture clustering feature similarity threshold and less than or equal to a preset second texture clustering feature similarity threshold, high-frequency component feature extraction is performed on the colorectal texture clustering feature data to obtain colorectal texture high-frequency component data, and shape feature extraction is performed on the colorectal endoscope image data to obtain colorectal image shape feature data; Calculating the interest index according to the colorectal texture high frequency component data and the colorectal image shape feature data to obtain the colorectal image interest index data; When it is determined that the colorectal image interest index data is less than or equal to a preset colorectal image interest index threshold, a first region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain first colorectal region of interest data; When it is determined that the colorectal image interest index data is greater than a preset colorectal image interest index threshold, a second region of interest labeling is performed on the image pixel area corresponding to the colorectal endoscope image data corresponding to the texture clustering feature similarity data to obtain second colorectal region of interest data; When it is determined that the texture clustering feature similarity data is greater than the preset second texture clustering feature similarity threshold, a third region of interest annotation is performed on the image pixel area corresponding to the colorectal endoscopy image data corresponding to the texture clustering feature similarity data to obtain third colorectal region of interest data, wherein the first region of interest annotation, the second region of interest annotation and the third region of interest annotation are region of interest annotation methods containing different region of interest annotation contents.

4. The method according to claim 3, characterized in that Step S2 is specifically as follows: Performing a first image segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain first colorectal image segmentation data; The colorectal endoscopic image data is subjected to a second image segmentation according to the third colorectal region of interest data to obtain second colorectal image segmentation data, wherein the first image segmentation and the second image segmentation are different image segmentation methods.

5. The method according to claim 4, characterized in that The first image segmentation is specifically as follows: Performing edge enhancement on colorectal endoscope image data to obtain colorectal image edge enhancement data; Performing image segmentation on the edge enhancement data of the colorectal image according to the second colorectal region of interest data to obtain primary first colorectal image segmentation data; Performing regional segmentation on the colorectal endoscope image data according to the second colorectal region of interest data to obtain secondary first colorectal image segmentation data; The primary first colorectal image segmentation data and the secondary first colorectal image segmentation data are fused to obtain the first colorectal image segmentation data.

6. The method according to claim 4, characterized in that The second image segmentation is specifically as follows: Gaussian distribution feature extraction is performed based on colorectal texture clustering feature data to obtain texture clustering Gaussian distribution feature data; Performing pixel classification on the colorectal endoscope image data according to the texture clustering Gaussian distribution feature data and the third colorectal region of interest data to obtain colorectal image pixel classification data; Mapping and segmenting the colorectal image pixel classification data to obtain second colorectal image segmentation data.

7. The method according to claim 1, characterized in that Step S3 is specifically as follows: Performing multi-scale feature extraction on the first colorectal image segmentation data to obtain multi-scale feature data of the colorectal image; Perform multi-head self-attention calculation on the multi-scale feature data of the colorectal image to obtain the deep feature data of the colorectal image; Slice stacking is performed on the second colorectal image segmentation data to obtain slice stacking data; Performing three-dimensional transformation on the slice stacking data to obtain slice three-dimensional transformation data; The slice 3D conversion data is 3D meshed to obtain the 3D network data of the colorectal image.

8. The method according to claim 1, characterized in that: Step S4 is specifically as follows: Perform voxel region node extraction and voxel region relationship variation extraction on the three-dimensional network data of colorectal images to obtain voxel region node data and voxel region relationship edge data respectively; Relationship graph data is constructed according to voxel region node data and voxel region relationship edge data to obtain colorectal relationship graph data; Performing local neighborhood feature extraction and global relationship feature extraction on the colorectal relationship graph data to obtain colorectal local neighborhood feature data and colorectal global relationship feature data respectively; Performing feature fusion on the local neighborhood feature data of the colorectum and the global relationship feature data of the colorectum to obtain the cell relationship feature data of the colorectal image; Colorectal cancer diagnosis auxiliary tasks are performed based on colorectal image depth feature data and colorectal image cell relationship feature data.

9. The method according to claim 8, characterized in that The local neighborhood feature extraction is specifically as follows: Performing neighborhood extraction on the colorectal relationship graph data to obtain colorectal neighborhood data; Extracting local geometric features from colorectal neighborhood data to obtain local geometric feature data; The local morphological features are extracted from the local geometric feature data to obtain the local neighborhood feature data of the colorectum.

10. A colorectal cancer diagnosis auxiliary system based on deep learning, characterized in that: Used to execute the colorectal cancer diagnosis auxiliary method based on deep learning as claimed in claim 1, the colorectal cancer diagnosis auxiliary system based on deep learning comprises: A colorectal region of interest positioning module is used to obtain colorectal endoscopic image data and locate the region of interest on the colorectal endoscopic image data to obtain colorectal region of interest data; A colorectal image multi-level segmentation module, used for performing multi-level image segmentation on the colorectal endoscopic image data according to the colorectal region of interest data to obtain colorectal image multi-level segmentation data, wherein the colorectal image multi-level segmentation data includes first colorectal image segmentation data and second colorectal image segmentation data, and the first colorectal image segmentation data and the second colorectal image segmentation data are different colorectal image segmentation data; A colorectal image segmentation depth processing module is used to perform depth feature extraction on the first colorectal image segmentation data to obtain colorectal image depth feature data, and to perform three-dimensional network construction on the second colorectal image segmentation data to obtain colorectal image three-dimensional network data; The colorectal cancer diagnosis auxiliary operation module is used to extract cell relationship features from colorectal image three-dimensional network data to obtain colorectal image cell relationship feature data, and perform colorectal cancer diagnosis auxiliary operations based on colorectal image depth feature data and colorectal image cell relationship feature data.

Images