Esophageal squamous carcinoma detection method and system based on combination of multi-modal information fusion and artificial intelligence

Abstract

The invention provides an esophageal squamous carcinoma detection method and system based on combination of multi-modal information fusion and artificial intelligence, and relates to the technical field of detection. And inputting the fusion feature map corresponding to the multi-modal image information into a pre-trained esophageal squamous carcinoma detection model to perform focus area detection and segmentation on the fusion feature map, and based on a detection result, performing benign and malignant classification and staging prediction on a detected focus by using an integrated learning method in combination with a classification algorithm to obtain the esophageal squamous carcinoma. Outputting a classification result and a visual heat map; converting the extracted focus area information into three-dimensional point cloud data by using an improved point cloud reconstruction algorithm; performing space division and noise reduction processing on the three-dimensional point cloud data by adopting a self-adaptive octree structure; and generating a smooth lesion three-dimensional model by using an improved Poisson surface reconstruction algorithm in combination with a curved surface subdivision technology.

Description

Technical Field

[0001] The present invention relates to detection technology, and in particular to a detection method and system for esophageal squamous cell carcinoma combining multimodal information fusion with artificial intelligence. Background Art

[0002] Esophageal squamous cell carcinoma is a common digestive tract malignancy with high morbidity and mortality. Early diagnosis and precise localization are crucial for the treatment and prognosis of esophageal squamous cell carcinoma. Currently, the detection of esophageal squamous cell carcinoma mainly relies on endoscopy and biopsy, but these methods have some limitations:

[0003] Traditional endoscopic examination mainly relies on the doctor's experience, which is prone to missed diagnosis and misdiagnosis, especially for the identification of early lesions and flat lesions.

[0004] Biopsy is invasive and may cause complications, and the sampling is somewhat random, which may lead to sampling errors. Two-dimensional images cannot fully reflect the three-dimensional structure and invasion depth of the lesion, affecting the clinical staging and treatment plan. Single-modality imaging examinations are difficult to provide morphological and functional information of the lesion at the same time, limiting the comprehensive evaluation of the lesion.

[0005] In recent years, with the development of medical imaging technology, a variety of new imaging modalities have been introduced into the diagnosis of esophageal squamous cell carcinoma, such as narrow band imaging (NBI), confocal laser endoscopy (CLE), optical coherence tomography (OCT), etc. These technologies can provide richer lesion information, but also bring about the problem of a surge in data volume and increased difficulty in interpretation. Summary of the invention

[0006] The embodiments of the present invention provide a method and system for detecting esophageal squamous cell carcinoma by combining multimodal information fusion with artificial intelligence, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a multimodal information fusion combined with artificial intelligence detection method for esophageal squamous cell carcinoma, including:

[0009] Collecting multimodal image information of esophageal tissue, the multimodal image information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; performing image enhancement processing on the multimodal image information, including contrast adjustment, denoising, and image registration; using a deep learning model to perform feature extraction on the multimodal image information after image enhancement processing to obtain a multidimensional feature vector; inputting the multidimensional feature vector into a pre-trained multimodal fusion network, which uses an attention mechanism and a cross-modal transformer structure to achieve effective fusion of different modal information and output a fusion feature map;

[0010] The fused feature map is input into a pre-trained esophageal squamous cell carcinoma detection model, the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, and the esophageal squamous cell carcinoma detection model is used to detect and segment the fusion feature map, and output the location, size and confidence information of the lesion area; based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant and predict the stages, and the classification results and a visual heat map are output;

[0011] Lesion area information is extracted from the original multimodal image information based on classification results and visualization heat maps. The extracted lesion area information is converted into three-dimensional point cloud data using an improved point cloud reconstruction algorithm. The three-dimensional point cloud data is spatially divided and denoised using an adaptive octree structure. A smooth three-dimensional lesion model is generated using an improved Poisson surface reconstruction algorithm combined with surface subdivision technology. The three-dimensional lesion model is registered and fused with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology, and surrounding tissue relationship.

[0012] In an optional embodiment,

[0013] The multi-dimensional feature vector is input into a pre-trained multimodal fusion network, which adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of different modal information. The output fusion feature map includes:

[0014] Using a pre-trained deep learning model to extract features from the multimodal image information to obtain a plurality of feature vectors; performing an adaptive average pooling operation on the plurality of feature vectors to adjust the plurality of feature vectors to feature vectors of the same dimension;

[0015] The feature vectors of the same dimension are input into a multimodal fusion network, wherein the multimodal fusion network includes a multi-head self-attention module and a cross-modal transformer structure; wherein the multi-head self-attention module performs a linear transformation on each feature vector to obtain a query vector, a key vector and a value vector, calculates the attention weight and performs a weighted sum to obtain an attention output; the cross-modal transformer structure includes a plurality of encoding layers and a decoding layer, wherein the plurality of encoding layers perform self-attention calculation and nonlinear transformation on the input feature vectors, and the decoding layer uses the features output by the encoding layer to realize inter-modal information fusion through a cross-attention mechanism;

[0016] The output of each layer of the multimodal fusion network is retained, and the outputs of different layers are fused by weighted summation at the end, wherein the fusion weights are learned by the softmax layer; the fused feature vector is upsampled to the original image resolution through a transposed convolution layer, and the channel attention mechanism is applied to learn the importance weights of different channels. Finally, the number of channels is adjusted through a convolution layer to obtain a fused feature map.

[0017] In an optional embodiment,

[0018] The fused feature map is input into a pre-trained esophageal squamous cell carcinoma detection model, which adopts an improved YOLOv5 network structure. The esophageal squamous cell carcinoma detection model is used to detect and segment the fusion feature map, and the location, size and confidence information of the lesion area are output, including:

[0019] Normalizing the fused feature map to adjust the pixel value distribution to a range of 0 to 1; performing data enhancement on the normalized fused feature map, including random horizontal flipping, random rotation, and random brightness contrast adjustment; adjusting the enhanced fused feature map to a preset size; inputting the preprocessed fused feature map into the backbone network of the improved YOLOv5 network, wherein the backbone network adds a feature pyramid network structure between the C3 layer, the C4 layer, and the C5 layer;

[0020] Apply spatial attention modules and channel attention modules at each feature pyramid network level; input the attention-enhanced fusion feature map into the anchor-free detection head, which outputs the prediction results of each grid point; parse the tensor output by the detection head to extract the center point offset, width and height, confidence, and category probability information; convert relative coordinates to absolute coordinates to obtain the actual position and size of each detection box;

[0021] Non-maximum suppression is performed independently on each category, and the non-maximum suppression results of all categories are merged to obtain the final detection box list; for each retained detection box, the regional feature map of the corresponding area is extracted from the segmentation head of the network; the extracted regional feature map is input into the fully convolutional network to generate a binary mask of the same size as the detection box; the detection box coordinates and segmentation mask are mapped from the regional feature map scale back to the original image scale; for each detection result, the location, size, confidence, category and precise contour information of the lesion area are integrated.

[0022] In an optional embodiment,

[0023] Based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant types and predict their stages. The output classification results and visualization heat maps include:

[0024] Extract deep features, texture features, morphological features and statistical features from the detection results, where deep features are extracted through a pre-trained ResNet50 model, texture features include gray-level co-occurrence matrix and local binary pattern features, morphological features include geometric features and Fourier descriptors, and statistical features include gray-level histogram features and Gabor filter responses; use the random forest algorithm to calculate feature importance scores, select the top K most important features, where K is determined by cross-validation; perform principal component analysis on the selected features to reduce dimensionality, retain 95% of the variance information, and obtain the feature vector after dimensionality reduction;

[0025] Construct multiple base classifiers, including support vector machine, random forest, XGBoost and deep neural network. The support vector machine uses RBF kernel function, the number of trees in random forest is set to 100, XGBoost uses tree model as base learner, and the deep neural network is a three-layer fully connected network. Use 5-fold cross validation to train and evaluate each base classifier, and use early stopping strategy and Adam optimizer for deep neural network.

[0026] The soft voting method and stacking ensemble are used for model fusion. The soft voting method averages the prediction probabilities of each base classifier, and the stacking ensemble uses logistic regression as the meta-classifier.

[0027] The staging prediction is converted into a multi-class classification problem, including early, middle and late stages, and the lesion size and invasion depth features are added to the original features;

[0028] Use the Grad-CAM algorithm to generate class activation heatmaps for deep learning models, calculate the gradient of the feature map of the last convolutional layer and the category score, and perform global average pooling on the gradient to obtain the channel importance weight; calculate the SHAP value for the tree model to reflect the contribution of each feature to the model prediction, and map the SHAP value back to the original image space;

[0029] The Grad-CAM and SHAP heatmaps are weighted fused to obtain a comprehensive heatmap, which is then superimposed on the original image using a flame map to generate a visual heatmap.

[0030] In an optional embodiment,

[0031] The extracted lesion area information is converted into three-dimensional point cloud data using an improved point cloud reconstruction algorithm; the three-dimensional point cloud data is spatially divided and denoised using an adaptive octree structure, including:

[0032] Performing depth estimation on the two-dimensional lesion detection results to obtain a depth map, wherein the depth estimation adopts an improved monocular depth estimation network, which includes a spatial attention module, a channel attention mechanism, and a multi-scale feature fusion module; based on the depth map, using a back-projection algorithm to convert the two-dimensional pixel coordinates into three-dimensional space coordinates to generate an initial point cloud data set;

[0033] Adaptively adjusting the point cloud density of the initial point cloud data set includes:

[0034] Calculating the local curvature of each point, determining an adaptive sampling rate based on the local curvature, resampling using a Poisson disk sampling algorithm, and applying voxelization filtering to ensure global consistency of point cloud density, thereby obtaining an adjusted point cloud data set;

[0035] Constructing an adaptive octree structure to spatially divide the adjusted point cloud data set, including:

[0036] Calculate the number of points, average curvature and spatial size of each node, define adaptive partitioning conditions based on these parameters, recursively partition the nodes according to the adaptive partitioning conditions, and store statistical information in leaf nodes;

[0037] Outlier detection and removal based on the adaptive octree structure includes:

[0038] Calculate the local statistical features of each leaf node, define the adaptive Z-score threshold, use the improved Z-score method to detect potential outliers, perform local plane fitting and correction on potential outliers, and perform global consistency check to obtain denoised point cloud data;

[0039] Adaptively smoothing the denoised point cloud data by bilateral filtering, including:

[0040] The number of neighbors is adaptively determined based on the local curvature, and the adaptive spatial weight and normal vector weight are calculated. The weight parameters are related to the local density and curvature of the point. The position of the point is iteratively updated, and the edge area is maintained to obtain the final three-dimensional point cloud data of the lesion area.

[0041] In an optional embodiment,

[0042] Using the improved Poisson surface reconstruction algorithm combined with the surface subdivision technology, a smooth 3D model of the lesion is generated, including:

[0043] An adaptive octree is constructed based on the point cloud data. In each leaf node of the adaptive octree, a weighted least square method is used to calculate a local normal vector, wherein the weight function considers the spatial distance and the consistency of the normal vector; a Poisson equation is constructed based on the local normal vector, and a divergence operator is applied to the product of a gradient operator and a scalar field to be solved, so that the divergence operator is applied to the vector field;

[0044] The Poisson equation is solved by a multi-grid method, and a local refinement strategy is used in high curvature areas; an improved Marching Cubes algorithm is used to extract isosurfaces, and the improved Marching Cubes algorithm increases sampling points in high curvature areas;

[0045] Apply the Loop subdivision algorithm to the extracted isosurface for surface subdivision, including:

[0046] Perform 4-1 topological subdivision on each triangle; update the vertex position, where the new vertex position is determined by the weighted sum of the four adjacent vertices, and the original vertex position is determined by the weighted sum of its own position and the positions of the adjacent vertices;

[0047] Employ feature preserving strategies in areas of high curvature, including:

[0048] The average curvature of each vertex is calculated, and when the average curvature is greater than a second preset threshold, the smoothing intensity is dynamically adjusted according to the curvature value.

[0049] According to a second aspect of the embodiments of the present invention,

[0050] Provides a multi-modal information fusion combined with artificial intelligence esophageal squamous cell carcinoma detection system, including:

[0051] The first unit is used to collect multimodal image information of esophageal tissue, wherein the multimodal image information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; perform image enhancement processing on the multimodal image information, including contrast adjustment, denoising, and image registration; perform feature extraction on the multimodal image information subjected to image enhancement processing using a deep learning model to obtain a multidimensional feature vector; and input the multidimensional feature vector into a pre-trained multimodal fusion network, wherein the multimodal fusion network adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of information of different modalities and output a fusion feature map;

[0052] The second unit is used to input the fused feature map into a pre-trained esophageal squamous cell carcinoma detection model, wherein the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, uses the esophageal squamous cell carcinoma detection model to detect and segment the fusion feature map, and outputs the location, size and confidence information of the lesion area; based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant and predict the stages, and outputs the classification results and a visual heat map;

[0053] The third unit is used to extract lesion area information from the original multimodal image information based on classification results and visualization heat maps; use the improved point cloud reconstruction algorithm to convert the extracted lesion area information into three-dimensional point cloud data; use an adaptive octree structure to spatially divide and denoise the three-dimensional point cloud data; use an improved Poisson surface reconstruction algorithm combined with surface subdivision technology to generate a smooth three-dimensional lesion model; align and fuse the three-dimensional lesion model with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology and surrounding tissue relationship.

[0054] According to a third aspect of the embodiments of the present invention,

[0055] An electronic device is provided, comprising:

[0056] processor;

[0057] a memory for storing processor-executable instructions;

[0058] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0059] A fourth aspect of the embodiments of the present invention is:

[0060] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0061] Through the multimodal fusion method, the complementary information from different imaging modalities is effectively integrated, which improves the accuracy and reliability of esophageal squamous cell carcinoma detection. The fused feature map not only retains the key information of each modality, but also captures the complex relationship between modalities through attention mechanism and cross-modal interaction, providing rich and effective feature representation for subsequent lesion detection and segmentation tasks. Through the improved YOLOv5 network structure, combined with the multimodal fusion feature map, efficient and accurate esophageal squamous cell carcinoma lesion detection and segmentation are achieved. This method can not only quickly locate the lesion area, but also provide accurate lesion contour information, which provides strong support for clinical diagnosis and treatment planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a method for detecting esophageal squamous cell carcinoma by combining multimodal information fusion with artificial intelligence according to an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the structure of an esophageal squamous cell carcinoma detection system that combines multimodal information fusion with artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of a method for detecting esophageal squamous cell carcinoma by combining multimodal information fusion with artificial intelligence according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S101. Collect multimodal image information of esophageal tissue, wherein the multimodal image information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; perform image enhancement processing on the multimodal image information, including contrast adjustment, denoising, and image registration; perform feature extraction on the multimodal image information subjected to image enhancement processing using a deep learning model to obtain a multidimensional feature vector; input the multidimensional feature vector into a pre-trained multimodal fusion network, wherein the multimodal fusion network adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of information of different modalities, and output a fusion feature map;

[0068] S102. Input the fused feature map into a pre-trained esophageal squamous cell carcinoma detection model, wherein the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, and uses the esophageal squamous cell carcinoma detection model to detect and segment the lesion area on the fused feature map, and output the location, size and confidence information of the lesion area; based on the detection results, use an integrated learning method in combination with a classification algorithm to classify the detected lesions into benign and malignant and predict the stages, and output the classification results and a visual heat map;

[0069] S103. Extract lesion area information from the original multimodal image information based on the classification results and the visualized heat map; use the improved point cloud reconstruction algorithm to convert the extracted lesion area information into three-dimensional point cloud data; use an adaptive octree structure to perform spatial division and noise reduction on the three-dimensional point cloud data; use an improved Poisson surface reconstruction algorithm combined with surface subdivision technology to generate a smooth three-dimensional lesion model; align and fuse the three-dimensional lesion model with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology and surrounding tissue relationship.

[0070] In an optional embodiment,

[0071] The multi-dimensional feature vector is input into a pre-trained multimodal fusion network, which adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of different modal information. The output fusion feature map includes:

[0072] Using a pre-trained deep learning model to extract features from the multimodal image information to obtain a plurality of feature vectors; performing an adaptive average pooling operation on the plurality of feature vectors to adjust the plurality of feature vectors to feature vectors of the same dimension;

[0073] The feature vectors of the same dimension are input into a multimodal fusion network, wherein the multimodal fusion network includes a multi-head self-attention module and a cross-modal transformer structure; wherein the multi-head self-attention module performs a linear transformation on each feature vector to obtain a query vector, a key vector and a value vector, calculates the attention weight and performs a weighted sum to obtain an attention output; the cross-modal transformer structure includes a plurality of encoding layers and a decoding layer, wherein the plurality of encoding layers perform self-attention calculation and nonlinear transformation on the input feature vectors, and the decoding layer uses the features output by the encoding layer to realize inter-modal information fusion through a cross-attention mechanism;

[0074] The output of each layer of the multimodal fusion network is retained, and the outputs of different layers are fused by weighted summation at the end, wherein the fusion weights are learned by the softmax layer; the fused feature vector is upsampled to the original image resolution through a transposed convolution layer, and the channel attention mechanism is applied to learn the importance weights of different channels. Finally, the number of channels is adjusted through a convolution layer to obtain a fused feature map.

[0075] Exemplarily, feature extraction is performed on multimodal image information. Pre-trained deep learning models, such as ResNet50, DenseNet121, etc., are used to process endoscopic images, narrow-band imaging (NBI) images, and optical coherence tomography (OCT) images, respectively. These pre-trained models are trained on large-scale medical imaging datasets and can effectively capture high-level features of images of different modalities. Taking ResNet50 as an example, the feature map size output by the last convolutional layer is 7x7x2048.

[0076] Next, the extracted feature vectors are subjected to adaptive average pooling. The purpose of this step is to adjust the feature vectors of different modalities and possibly different spatial dimensions to the same dimension to facilitate subsequent fusion processing. Specifically, adaptive average pooling is applied to each feature vector to adjust it to a fixed size, such as 1x1x2048. In this way, a feature representation of uniform dimension can be obtained regardless of the original size of the input image.

[0077] Then, the adjusted feature vector is input into the multimodal fusion network. The core components of the network include a multi-head self-attention module and a cross-modal transformer structure. The multi-head self-attention module first performs a linear transformation on each feature vector to generate a query vector, a key vector, and a value vector. Taking 8 attention heads as an example, each head has a dimension of 256. Then, for the 2048-dimensional input feature, 8 sets of 256-dimensional query, key, and value vectors will be generated. Then the attention weight is calculated by performing a dot product of the query vector and the key vector, and normalizing them through the softmax function to obtain the weight. Finally, these weights are used to perform a weighted summation on the value vector to obtain the attention output.

[0078] The cross-modal transformer structure consists of multiple encoding layers and a decoding layer. Taking three encoding layers as an example, each encoding layer contains a self-attention sublayer and a feedforward neural network sublayer. The operation of the self-attention sublayer is similar to the multi-head self-attention module, while the feedforward neural network sublayer contains two fully connected layers to introduce nonlinear transformations. The decoding layer uses the output of the encoding layer to achieve information interaction and fusion between different modalities through a cross-attention mechanism. Specifically, the decoding layer uses the features of one modality as a query and the features of other modalities as keys and values, calculates the attention weights and performs weighted summation.

[0079] In order to make full use of the features at different levels of the network, the output of each layer is retained after each layer of the multimodal fusion network. Assuming there are 4 layers (3 encoding layers and 1 decoding layer), 4 feature outputs at different levels will be obtained. These features are fused by weighted summation, and the fusion weights are learned through a softmax layer. For example, the weight distribution that may be learned is [0.1, 0.2, 0.3, 0.4], indicating that the network is more inclined to use the features of the later layers.

[0080] Finally, the fused feature vector is upsampled to the original image resolution through the transposed convolution layer. Assuming the original image size is 224x224, multiple upsampling operations are required, each time the spatial dimension of the feature map is doubled until it reaches 224x224. In this process, the channel attention mechanism is applied to learn the importance weights of different channels. The specific implementation can use a structure of global average pooling followed by a fully connected layer to output a weight vector with the same number of channels. Finally, the number of channels is adjusted through a 1x1 convolution layer to obtain the final fused feature map. For example, the number of channels can be adjusted from 2048 to 256 to obtain a fused feature map of size 224x224x256.

[0081] Through this multimodal fusion method, the complementary information from different imaging modalities is effectively integrated, improving the accuracy and reliability of esophageal squamous cell carcinoma detection. The fused feature map not only retains the key information of each modality, but also captures the complex relationship between modalities through attention mechanism and cross-modal interaction, providing rich and effective feature representation for subsequent lesion detection and segmentation tasks.

[0082] In an optional embodiment,

[0083] The fused feature map is input into a pre-trained esophageal squamous cell carcinoma detection model, which adopts an improved YOLOv5 network structure. The esophageal squamous cell carcinoma detection model is used to detect and segment the fusion feature map, and the location, size and confidence information of the lesion area are output, including:

[0084] Normalizing the fused feature map to adjust the pixel value distribution to a range of 0 to 1; performing data enhancement on the normalized fused feature map, including random horizontal flipping, random rotation, and random brightness contrast adjustment; adjusting the enhanced fused feature map to a preset size; inputting the preprocessed fused feature map into the backbone network of the improved YOLOv5 network, wherein the backbone network adds a feature pyramid network structure between the C3 layer, the C4 layer, and the C5 layer;

[0085] Apply spatial attention modules and channel attention modules at each feature pyramid network level; input the attention-enhanced fusion feature map into the anchor-free detection head, which outputs the prediction results of each grid point; parse the tensor output by the detection head to extract the center point offset, width and height, confidence, and category probability information; convert relative coordinates to absolute coordinates to obtain the actual position and size of each detection box;

[0086] Non-maximum suppression is performed independently on each category, and the non-maximum suppression results of all categories are merged to obtain the final detection box list; for each retained detection box, the regional feature map of the corresponding area is extracted from the segmentation head of the network; the extracted regional feature map is input into the fully convolutional network to generate a binary mask of the same size as the detection box; the detection box coordinates and segmentation mask are mapped from the regional feature map scale back to the original image scale; for each detection result, the location, size, confidence, category and precise contour information of the lesion area are integrated.

[0087] Exemplarily, first, the fused feature map is preprocessed. The pixel values ​​of the fused feature map are normalized and adjusted to the range of 0 to 1. This step helps to improve the training stability and generalization ability of the model. For example, for an image with an original pixel value range of 0 to 255, each pixel value can be divided by 255 for normalization.

[0088] Next, the normalized fusion feature map is enhanced. Data enhancement techniques include random horizontal flipping, random rotation, and random brightness and contrast adjustment. Specifically, the probability of horizontal flipping is set to 0.5; the angle range of random rotation is set to -15 degrees to 15 degrees; and the adjustment ranges of brightness and contrast are set to 0.8 to 1.2, respectively. These enhancement techniques help improve the robustness of the model to various image changes.

[0089] Then, the enhanced fused feature map is resized to a preset size. Considering the balance between computational efficiency and detection accuracy, the image is resized to 640x640 pixels. This step ensures the consistency of the input and facilitates subsequent network processing.

[0090] After preprocessing, the fused feature map is input into the backbone network of the improved YOLOv5 network. The backbone network adopts the CSPDarknet53 structure, and adds a feature pyramid network (FPN) structure between the C3 layer, C4 layer, and C5 layer. The FPN structure realizes the fusion of features of different scales through top-down paths and lateral connections, which is conducive to the detection of lesions of different sizes. For example, the feature map size of the C3 layer is 80x80x256, the C4 layer is 40x40x512, and the C5 layer is 20x20x1024.

[0091] At each FPN level, a spatial attention module and a channel attention module are applied. The spatial attention module generates a two-dimensional weight matrix through convolution operations to highlight important spatial locations; the channel attention module generates a one-dimensional weight vector through global average pooling and fully connected layers to emphasize important feature channels. The combination of these two attention mechanisms helps the model better focus on key areas and features.

[0092] The feature maps after attention enhancement are input into the anchor-free detection head. The anchor-free detection head adopts a fully convolutional structure to directly predict the existence, position, size and category of the object at each grid point. For 640x640 input, predictions are made on 20x20, 40x40 and 80x80 feature maps respectively to accommodate lesions of different scales.

[0093] The tensor output by the detection head needs to be parsed to extract the center point offset, width and height, confidence and category probability information. For example, for an 80x80 feature map, each grid point outputs a vector of length 7 (2 for center point offset, 2 for width and height, 1 for confidence, and 2 for category probability).

[0094] Convert relative coordinates to absolute coordinates to get the actual position and size of each detection box. Specifically, add the grid index to the predicted center point offset and multiply it by the step size of the feature map (such as 8, 16, 32) to get the coordinates of the center point of the detection box on the original image. The width and height are calculated using an exponential function and a predefined prior value.

[0095] Non-maximum suppression (NMS) is performed independently on each category, and the NMS results of all categories are merged to obtain the final list of detection boxes. The IoU threshold of NMS is set to 0.5 and the confidence threshold is set to 0.25 to balance the recall rate and precision rate.

[0096] For each retained detection box, extract the feature map of the corresponding area from the segmentation head of the network. The segmentation head adopts a fully convolutional network structure, which contains multiple upsampling and convolution layers. For example, for a 20x20 detection box, extract a 40x40 size regional feature map.

[0097] The extracted regional feature map is input into the full convolutional network to generate a binary mask of the same size as the detection box. The full convolutional network contains 3 convolutional layers and 1 upsampling layer. Finally, the probability map is output through the sigmoid function and then binarized using 0.5 as the threshold.

[0098] Map the detection box coordinates and segmentation mask from the feature map scale back to the original image scale. This needs to take into account the network's downsampling factor and the scaling of the input image. For example, if the original image size is 1280x1280 and the network input is 640x640, the coordinate and mask sizes need to be scaled up by 2.

[0099] Finally, for each detection result, the location, size, confidence, category and precise contour information of the lesion area are integrated. The location and size are expressed in pixel coordinates, the confidence and category probability are expressed in floating point numbers, and the precise contour is given in the form of a binary mask.

[0100] Through this improved YOLOv5 network structure, combined with multimodal fusion feature maps, efficient and accurate esophageal squamous cell carcinoma lesion detection and segmentation are achieved. This method can not only quickly locate the lesion area, but also provide accurate lesion contour information, providing strong support for clinical diagnosis and treatment planning.

[0101] In an optional embodiment,

[0102] Based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant types and predict their stages. The output classification results and visualization heat maps include:

[0103] Extract deep features, texture features, morphological features and statistical features from the detection results, where deep features are extracted through a pre-trained ResNet50 model, texture features include gray-level co-occurrence matrix and local binary pattern features, morphological features include geometric features and Fourier descriptors, and statistical features include gray-level histogram features and Gabor filter responses; use the random forest algorithm to calculate feature importance scores, select the top K most important features, where K is determined by cross-validation; perform principal component analysis on the selected features to reduce dimensionality, retain 95% of the variance information, and obtain the feature vector after dimensionality reduction;

[0104] Construct multiple base classifiers, including support vector machine, random forest, XGBoost and deep neural network. The support vector machine uses RBF kernel function, the number of trees in random forest is set to 100, XGBoost uses tree model as base learner, and the deep neural network is a three-layer fully connected network. Use 5-fold cross validation to train and evaluate each base classifier, and use early stopping strategy and Adam optimizer for deep neural network.

[0105] The soft voting method and stacking ensemble are used for model fusion. The soft voting method averages the prediction probabilities of each base classifier, and the stacking ensemble uses logistic regression as the meta-classifier.

[0106] The staging prediction is converted into a multi-class classification problem, including early, middle and late stages, and the lesion size and invasion depth features are added to the original features;

[0107] Use the Grad-CAM algorithm to generate class activation heatmaps for deep learning models, calculate the gradient of the feature map of the last convolutional layer and the category score, and perform global average pooling on the gradient to obtain the channel importance weight; calculate the SHAP value for the tree model to reflect the contribution of each feature to the model prediction, and map the SHAP value back to the original image space;

[0108] The Grad-CAM and SHAP heatmaps are weighted fused to obtain a comprehensive heatmap, which is then superimposed on the original image using a flame map to generate a visual heatmap.

[0109] Exemplarily, a variety of features are extracted from the detection results. The deep features are extracted by the pre-trained ResNet50 model. Specifically, the detected lesion area is adjusted to a size of 224x224 and input into the feature layer before the last fully connected layer of ResNet50 to obtain a 2048-dimensional feature vector. Texture features include grayscale co-occurrence matrix and local binary pattern features. For the grayscale co-occurrence matrix, the matrices of four directions (0°, 45°, 90°, 135°) are calculated, and 14 statistics such as energy, contrast, correlation and entropy are extracted to obtain 56-dimensional features. The local binary pattern feature uses a circular neighborhood with a radius of 3 to obtain a 256-dimensional histogram feature. Morphological features include geometric features and Fourier descriptors. Geometric features include 10 features such as area, perimeter, circularity, aspect ratio, and the Fourier descriptor takes the first 20 coefficients as features. Statistical features include grayscale histogram features and Gabor filter responses. Grayscale histogram features include 10 statistics such as mean, standard deviation, skewness, and kurtosis. Gabor filtering uses 4 scales and 6 directions to calculate the mean and standard deviation of each filter response to obtain 48-dimensional features.

[0110] Next, use the random forest algorithm to calculate the feature importance score and select the top K most important features. Set the number of trees in the random forest to 500 and measure feature importance by the average impurity drop. Through 5-fold cross validation, try different K values ​​(such as 50, 100, 200) and select the K value that performs best on the validation set. Assume that the top 100 most important features are selected.

[0111] Perform principal component analysis on the selected features to reduce the dimension and retain 95% of the variance information. Specifically, first standardize the features, calculate the covariance matrix, solve the eigenvalues ​​and eigenvectors, and select the first N principal components with a cumulative contribution rate of 95%. Assuming N is 20, a 20-dimensional reduced-dimensional feature vector is obtained.

[0112] Construct multiple base classifiers. The support vector machine uses the RBF kernel function and determines the optimal C and gamma parameters through grid search, such as C = 10, gamma = 0.01. The random forest sets the number of trees to 100, the maximum depth to 10, and the minimum number of leaf node samples to 5. XGBoost uses the tree model as the base learner, sets the maximum depth to 6, the learning rate to 0.1, and the subsampling ratio to 0.8. The deep neural network is a three-layer fully connected network with 64, 32, and 16 neurons in each layer, using the ReLU activation function and the sigmoid function in the output layer.

[0113] Each base classifier was trained and evaluated using 5-fold cross validation. For deep neural networks, an early stopping strategy was used, and training was stopped when the validation set loss did not improve for 5 consecutive epochs. The Adam optimizer was used with a learning rate of 0.001 and a batch size of 32. The performance metrics of each model on the validation set, such as accuracy, precision, recall, and F1 score, were recorded.

[0114] Two strategies, soft voting and stacking ensemble, are used for model fusion. Soft voting averages the prediction probabilities of each base classifier to obtain the final prediction probability. Stacking ensemble uses logistic regression as a meta-classifier and the prediction probabilities of the base classifiers as input features to train the logistic regression model. The performance of the two fusion strategies is compared through cross-validation, and the method with better performance is selected as the final classification model.

[0115] The staging prediction was transformed into a multi-class classification problem, including early, middle and late stages. Lesion size and invasion depth features were added on the basis of the original features. The lesion size can be directly obtained from the detection results, and the invasion depth can be estimated by the segmentation results. For the multi-class classification problem, the support vector machine and XGBoost were changed to multi-class versions, and the random forest and deep neural network adjusted the output layer to adapt to the multi-class problem.

[0116] Use the Grad-CAM algorithm to generate class activation heatmaps for deep learning models. Take ResNet50 as an example, and select the last convolutional layer conv5_block3_out as the target layer. Calculate the gradient of the feature map of this layer with the class score, and perform global average pooling on the gradient to get the channel importance weight. Multiply the weight with the feature map, and then perform ReLU activation to get the heatmap. Upsample the heatmap to the original image size, using the bilinear interpolation method.

[0117] Calculate the SHAP value for the tree model to reflect the contribution of each feature to the model prediction. Taking XGBoost as an example, use TreeExplainer to calculate the SHAP value of each sample. Map the SHAP value back to the original image space, and for each pixel, accumulate the SHAP values ​​of all features related to the pixel to obtain a pixel-level importance heat map.

[0118] The Grad-CAM and SHAP heatmaps are weighted fused to obtain a comprehensive heatmap. The fusion weight can be determined by cross-validation, such as a Grad-CAM weight of 0.6 and a SHAP weight of 0.4. The heatmap is superimposed on the original image using the flame map color map, with red representing high importance areas and blue representing low importance areas.

[0119] Through this integrated learning and multi-feature fusion method, combined with interpretable technology, not only accurate benign and malignant classification and staging prediction of esophageal squamous cell carcinoma are achieved, but also intuitive visualization results are provided to help doctors understand the decision basis of the model and improve the credibility and interpretability of the diagnosis. This method has important value in clinical applications and can assist doctors in making more accurate diagnostic decisions and developing personalized treatment plans for patients.

[0120] In an optional embodiment,

[0121] The extracted lesion area information is converted into three-dimensional point cloud data using an improved point cloud reconstruction algorithm; the three-dimensional point cloud data is spatially divided and denoised using an adaptive octree structure, including:

[0122] Performing depth estimation on the two-dimensional lesion detection results to obtain a depth map, wherein the depth estimation adopts an improved monocular depth estimation network, which includes a spatial attention module, a channel attention mechanism, and a multi-scale feature fusion module; based on the depth map, using a back-projection algorithm to convert the two-dimensional pixel coordinates into three-dimensional space coordinates to generate an initial point cloud data set;

[0123] Adaptively adjusting the point cloud density of the initial point cloud data set includes:

[0124] Calculating the local curvature of each point, determining an adaptive sampling rate based on the local curvature, resampling using a Poisson disk sampling algorithm, and applying voxelization filtering to ensure global consistency of point cloud density, thereby obtaining an adjusted point cloud data set;

[0125] Constructing an adaptive octree structure to spatially divide the adjusted point cloud data set, including:

[0126] Calculate the number of points, average curvature and spatial size of each node, define adaptive partitioning conditions based on these parameters, recursively partition the nodes according to the adaptive partitioning conditions, and store statistical information in leaf nodes;

[0127] Outlier detection and removal based on the adaptive octree structure includes:

[0128] Calculate the local statistical features of each leaf node, define the adaptive Z-score threshold, use the improved Z-score method to detect potential outliers, perform local plane fitting and correction on potential outliers, and perform global consistency check to obtain denoised point cloud data;

[0129] Adaptively smoothing the denoised point cloud data by bilateral filtering, including:

[0130] The number of neighbors is adaptively determined based on the local curvature, and the adaptive spatial weight and normal vector weight are calculated. The weight parameters are related to the local density and curvature of the point. The position of the point is iteratively updated, and the edge area is maintained to obtain the final three-dimensional point cloud data of the lesion area.

[0131] For example, using an improved point cloud reconstruction algorithm to convert lesion area information into three-dimensional point cloud data is a key step in achieving accurate three-dimensional reconstruction. The process includes depth estimation, back-projection, point cloud density adjustment, adaptive space division, outlier detection and removal, and adaptive smoothing.

[0132] First, the depth of the two-dimensional lesion detection results is estimated to obtain a depth map. The depth estimation uses an improved monocular depth estimation network, which includes a spatial attention module, a channel attention mechanism, and a multi-scale feature fusion module. Specifically, the spatial attention module uses a 3x3 convolution layer to generate an attention map, the channel attention mechanism uses global average pooling and maximum pooling to generate channel descriptors, and the multi-scale feature fusion module uses 1x1 convolution and upsampling operations to fuse features of different scales. The network input is a 224x224x3 RGB image, and the output is a depth map of the same size as the input. The NYU Depth V2 dataset is used for pre-training during training, and then fine-tuned on the esophageal endoscopy image dataset. The loss function uses the weighted sum of the depth smoothing L1 loss and the gradient consistency loss.

[0133] Based on the obtained depth map, the back-projection algorithm is used to convert the two-dimensional pixel coordinates into three-dimensional space coordinates to generate the initial point cloud data set. The back-projection process requires the camera intrinsic parameter matrix, which can be obtained through camera calibration. For each pixel point (u, v), its depth value is d. The corresponding three-dimensional space coordinates (X, Y, Z) are calculated through the inverse matrix of the camera intrinsic parameter matrix and the depth value. Assuming a 224x224 depth map, about 50,000 initial three-dimensional points may be generated.

[0134] Adaptive point cloud density adjustment is performed on the initial point cloud dataset. First, the local curvature of each point is calculated, and the neighborhood of each point is found using the K nearest neighbor algorithm (K=10), and the local curvature is calculated by principal component analysis. The adaptive sampling rate is determined based on the local curvature, with high sampling rates in areas with high curvature and low sampling rates in areas with low curvature. Resampling is performed using the Poisson disk sampling algorithm, and the sampling radius is adaptively adjusted according to the local curvature, which may range from 0.5mm to 2mm. Then voxelization filtering is applied to ensure the global consistency of the point cloud density, and the voxel size is set to 0.5mm. After this step, the number of point clouds may be reduced to about 20,000.

[0135] Construct an adaptive octree structure to perform spatial partitioning on the adjusted point cloud dataset. Calculate the number of points, average curvature, and spatial size of each node, and define adaptive partitioning conditions based on these parameters. For example, when the number of node points exceeds 100, or the average curvature exceeds 0.05, or the spatial size is greater than 5mm, continue to partition the node. Recursively partition nodes according to the adaptive partitioning conditions until the stop condition is reached. Store statistical information in the leaf nodes, including the number of points, average position, average normal vector, etc. Assume that the final generated octree has a depth of 6 and the number of leaf nodes is about 1000.

[0136] Outlier detection and removal based on adaptive octree structure. Calculate the local statistical features of each leaf node, including the average distance and standard deviation of the point. Define an adaptive Z-score threshold, which is proportional to the average curvature of the node and may range from 2.5 to 4. Use the improved Z-score method to detect potential outliers, and mark the points whose distance mean exceeds the adaptive threshold as potential outliers. Perform local plane fitting and correction on potential outliers, use the RANSAC algorithm to fit the local plane, and project the outliers onto the fitting plane. Perform a global consistency check to ensure that the normal vectors of the corrected points are consistent with those of the neighboring points. After this step, about 5% of the outliers may be removed.

[0137] Adaptive bilateral filtering is performed on the denoised point cloud data. The number of neighbors is adaptively determined based on the local curvature. A smaller number of neighbors (such as 10) is used in areas with high curvature, and a larger number of neighbors (such as 30) is used in areas with low curvature. Adaptive spatial weights and normal vector weights are calculated, where the weight parameters are related to the local density and curvature of the point. The standard deviation of the spatial weight is inversely proportional to the local density of the point, and may range from 0.5mm to 2mm; the standard deviation of the normal vector weight is inversely proportional to the local curvature of the point, and may range from 10 degrees to 30 degrees. Iteratively update the position of the point, and the number of iterations is set to 5. Keep processing in the edge area, identify the edge points by detecting the rate of change of the normal vector of the point, and use a smaller smoothing intensity for the edge points.

[0138] Through the above steps, the initial two-dimensional lesion detection results are converted into high-quality three-dimensional point cloud data. This improved point cloud reconstruction algorithm can effectively process lesion shapes of different complexities, retaining detailed features while removing noise and abnormal points. For example, for a typical esophageal squamous cell carcinoma lesion, the initial two-dimensional detection result may be a 100x100 pixel area, which generates about 10,000 initial three-dimensional points after depth estimation and back projection. After density adjustment, the number of points may be reduced to about 5,000. Adaptive octree partitioning may generate about 200 leaf nodes. Outlier detection may remove about 250 points, and ultimately obtain high-quality three-dimensional point cloud data of about 4,750 points. This precise three-dimensional reconstruction provides a reliable foundation for subsequent lesion analysis and diagnosis.

[0139] In an optional embodiment,

[0140] Using the improved Poisson surface reconstruction algorithm combined with the surface subdivision technology, a smooth 3D model of the lesion is generated, including:

[0141] An adaptive octree is constructed based on the point cloud data. In each leaf node of the adaptive octree, a weighted least square method is used to calculate a local normal vector, wherein the weight function considers the spatial distance and the consistency of the normal vector; a Poisson equation is constructed based on the local normal vector, and a divergence operator is applied to the product of a gradient operator and a scalar field to be solved, so that the divergence operator is applied to the vector field;

[0142] The Poisson equation is solved by a multi-grid method, and a local refinement strategy is used in high curvature areas; an improved Marching Cubes algorithm is used to extract isosurfaces, and the improved Marching Cubes algorithm increases sampling points in high curvature areas;

[0143] Apply the Loop subdivision algorithm to the extracted isosurface for surface subdivision, including:

[0144] Perform 4-1 topological subdivision on each triangle; update the vertex position, where the new vertex position is determined by the weighted sum of the four adjacent vertices, and the original vertex position is determined by the weighted sum of its own position and the positions of the adjacent vertices;

[0145] Employ feature preserving strategies in areas of high curvature, including:

[0146] The average curvature of each vertex is calculated, and when the average curvature is greater than a second preset threshold, the smoothing intensity is dynamically adjusted according to the curvature value.

[0147] Exemplarily, first, an adaptive octree is constructed based on the point cloud data. Considering the complexity of the lesion shape, an adaptive strategy is used for space division. Starting from the cube root node that surrounds the entire point cloud, the space is recursively divided into eight sub-cubes. During the division process, the number of points in the node, the spatial size, and the local curvature are considered. For example, when a node contains more than 100 points, or the node size exceeds 1mm, or the local curvature exceeds 0.1, the division continues. This adaptive strategy ensures more detailed divisions in high curvature areas. For a lesion point cloud containing about 5,000 points, an octree with a depth of 8 may eventually be generated, with about 2,000 leaf nodes.

[0148] In each leaf node of the adaptive octree, the weighted least squares method is used to calculate the local normal vector. The weight function takes into account both spatial distance and normal vector consistency. Specifically, for each point, its K nearest neighbors (K is usually set to 20) are selected to calculate the normal vector. The spatial distance weight uses a Gaussian function, and the standard deviation is set to the average of the neighbor distances. The normal vector consistency weight is calculated based on the angle between the point normal vector and the current estimated normal vector, using a cosine function. Through iterative optimization, the precise normal vector of each point is finally obtained.

[0149] The Poisson equation is constructed based on the local normal vector. The core idea of ​​the Poisson equation is to apply the divergence operator to the product of the gradient operator and the scalar field to be solved, so that it is equal to the divergence operator acting on the vector field. Here, the vector field is defined by the normal vector of the point cloud, and the scalar field to be solved represents the implicit surface. The construction process of the Poisson equation involves finite difference approximation on a discretized grid, which transforms the continuous problem into a discrete linear system of equations.

[0150] The Poisson equation is solved using a multi-grid approach. The multi-grid approach accelerates the convergence process by alternating the solution on grids of different resolutions. Start with a coarse grid and gradually refine to the finest grid. At each grid level, smoothing is performed using the Gauss-Seidel iteration method. In areas of high curvature, such as lesion edges or wrinkles, a local refinement strategy is used to increase the grid resolution by a factor of 2-4. For example, for a lesion area of ​​10mm x10mm x 5mm, the coarsest grid may be 20x20x10 and the finest grid may be up to 160x160x80.

[0151] The isosurface is extracted using the improved Marching Cubes algorithm. The traditional Marching Cubes algorithm samples the scalar field values ​​at the eight vertices of each voxel and generates triangles based on the intersection of the isosurface and the voxel edge. The improved version adds sampling points in high curvature areas to improve reconstruction accuracy. Specifically, when a drastic change in the scalar field within a voxel is detected (for example, the difference in scalar field values ​​between adjacent sampling points exceeds a threshold), additional sampling points are added inside the voxel. This strategy can better capture detailed features on the surface of the lesion, such as tiny protrusions or depressions.

[0152] Apply the Loop subdivision algorithm to the extracted isosurface for surface subdivision. Loop subdivision is a commonly used triangular mesh subdivision method that can generate smooth surfaces. The subdivision process includes two main steps: topological subdivision and geometric smoothing. In topological subdivision, each triangle is subdivided 4-1, that is, a new vertex is inserted at the midpoint of each edge to divide a triangle into four. In geometric smoothing, the vertex position is updated. The position of the new vertex is determined by the weighted sum of the four adjacent vertices. The weight can be set as: 1 / 8 for each diagonal vertex and 3 / 8 for each adjacent side vertex. The new position of the original vertex is determined by the weighted sum of its own position and the positions of adjacent vertices. Usually, its own weight is 5 / 8, and the weight of adjacent vertices is 3 / 8 divided by the number of adjacent vertices.

[0153] A feature preservation strategy is adopted in high curvature areas to avoid the loss of important details due to over-smoothing. First, the average curvature of each vertex is calculated, which can be estimated by fitting a local quadratic surface. When the average curvature is greater than a preset threshold (for example, the curvature is greater than 0.5mm^-1), the smoothing intensity is dynamically adjusted according to the curvature value. Specifically, the weights of adjacent vertices can be reduced when vertices in high curvature areas are updated, and the weight of their own positions can be increased. For example, for areas with particularly high curvature (such as curvature greater than 1mm^-1), the own weight can be increased to 0.8, and the weights of adjacent vertices can be reduced accordingly.

[0154] Through the above steps, the initial point cloud data is converted into a high-quality smooth 3D model. This improved Poisson surface reconstruction algorithm combined with surface subdivision technology can effectively handle lesions of complex shapes, retaining important local features while maintaining overall smoothness. For example, for a typical early esophageal cancer lesion, the initial point cloud may contain about 5,000 points distributed in a space of 10mm x 10mm x 5mm. After Poisson reconstruction, an initial mesh containing about 10,000 triangles may be generated. After applying two Loop subdivisions, the number of triangles increases to about 160,000, generating a fine lesion surface model. In high curvature areas such as the edge of the lesion, the feature preservation strategy ensures that tiny structures of about 0.5mm in size are retained. This accurate 3D reconstruction provides a reliable basis for subsequent lesion analysis, surgical planning, and treatment evaluation.

[0155] Figure 2 FIG. 1 is a schematic diagram of the structure of the esophageal squamous cell carcinoma detection system combining multimodal information fusion and artificial intelligence according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0156] The first unit is used to collect multimodal image information of esophageal tissue, wherein the multimodal image information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; perform image enhancement processing on the multimodal image information, including contrast adjustment, denoising, and image registration; perform feature extraction on the multimodal image information subjected to image enhancement processing using a deep learning model to obtain a multidimensional feature vector; and input the multidimensional feature vector into a pre-trained multimodal fusion network, wherein the multimodal fusion network adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of information of different modalities and output a fusion feature map;

[0157] The second unit is used to input the fused feature map into a pre-trained esophageal squamous cell carcinoma detection model, wherein the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, uses the esophageal squamous cell carcinoma detection model to detect and segment the fusion feature map, and outputs the location, size and confidence information of the lesion area; based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant and predict the stages, and outputs the classification results and a visual heat map;

[0158] The third unit is used to extract lesion area information from the original multimodal image information based on classification results and visualization heat maps; use the improved point cloud reconstruction algorithm to convert the extracted lesion area information into three-dimensional point cloud data; use an adaptive octree structure to spatially divide and denoise the three-dimensional point cloud data; use an improved Poisson surface reconstruction algorithm combined with surface subdivision technology to generate a smooth three-dimensional lesion model; align and fuse the three-dimensional lesion model with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology and surrounding tissue relationship.

[0159] According to a third aspect of the embodiments of the present invention,

[0160] An electronic device is provided, comprising:

[0161] processor;

[0162] a memory for storing processor-executable instructions;

[0163] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0164] A fourth aspect of the embodiments of the present invention is:

[0165] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0166] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting esophageal squamous cell carcinoma by combining multimodal information fusion with artificial intelligence, characterized in that: include: Acquiring multimodal imaging information of esophageal tissue, wherein the multimodal imaging information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; Performing image enhancement processing on the multimodal image information, including contrast adjustment, denoising and image registration; extracting features of the multimodal image information subjected to image enhancement processing using a deep learning model to obtain a multidimensional feature vector; inputting the multidimensional feature vector into a pre-trained multimodal fusion network, which adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of information of different modalities and output a fusion feature map; Inputting the fused feature map into a pre-trained esophageal squamous cell carcinoma detection model, the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, using the esophageal squamous cell carcinoma detection model to perform lesion area detection and segmentation on the fused feature map, and outputting the location, size and confidence information of the lesion area; Based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant types and predict their stages, and the classification results and a visual heat map are output; Lesion area information is extracted from the original multimodal image information based on classification results and visualization heat maps. The extracted lesion area information is converted into three-dimensional point cloud data using an improved point cloud reconstruction algorithm. The three-dimensional point cloud data is spatially divided and denoised using an adaptive octree structure. A smooth three-dimensional lesion model is generated using an improved Poisson surface reconstruction algorithm combined with surface subdivision technology. The three-dimensional lesion model is registered and fused with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology, and surrounding tissue relationship.

2. The method according to claim 1, characterized in that The multi-dimensional feature vector is input into a pre-trained multimodal fusion network, which adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of different modal information. The output fusion feature map includes: Using a pre-trained deep learning model to extract features from the multimodal image information to obtain a plurality of feature vectors; performing an adaptive average pooling operation on the plurality of feature vectors to adjust the plurality of feature vectors to feature vectors of the same dimension; The feature vectors of the same dimension are input into a multimodal fusion network, wherein the multimodal fusion network includes a multi-head self-attention module and a cross-modal transformer structure; wherein the multi-head self-attention module performs a linear transformation on each feature vector to obtain a query vector, a key vector and a value vector, calculates the attention weight and performs a weighted sum to obtain an attention output; the cross-modal transformer structure includes a plurality of encoding layers and a decoding layer, wherein the plurality of encoding layers perform self-attention calculation and nonlinear transformation on the input feature vectors, and the decoding layer uses the features output by the encoding layer to realize inter-modal information fusion through a cross-attention mechanism; The output of each layer of the multimodal fusion network is retained, and the outputs of different layers are fused by weighted summation at the end, wherein the fusion weights are learned by the softmax layer; the fused feature vector is upsampled to the original image resolution through a transposed convolution layer, and the channel attention mechanism is applied to learn the importance weights of different channels. Finally, the number of channels is adjusted through a convolution layer to obtain a fused feature map.

3. The method according to claim 1, characterized in that The fused feature map is input into a pre-trained esophageal squamous cell carcinoma detection model, which adopts an improved YOLOv5 network structure. The esophageal squamous cell carcinoma detection model is used to detect and segment the fusion feature map, and the location, size and confidence information of the lesion area are output, including: Normalizing the fused feature map to adjust the pixel value distribution to a range of 0 to 1; performing data enhancement on the normalized fused feature map, including random horizontal flipping, random rotation, and random brightness contrast adjustment; adjusting the enhanced fused feature map to a preset size; inputting the preprocessed fused feature map into the backbone network of the improved YOLOv5 network, wherein the backbone network adds a feature pyramid network structure between the C3 layer, the C4 layer, and the C5 layer; Apply spatial attention modules and channel attention modules at each feature pyramid network level; input the attention-enhanced fusion feature map into the anchor-free detection head, which outputs the prediction results of each grid point; parse the tensor output by the detection head to extract the center point offset, width and height, confidence, and category probability information; convert relative coordinates to absolute coordinates to obtain the actual position and size of each detection box; Non-maximum suppression is performed independently on each category, and the non-maximum suppression results of all categories are merged to obtain the final detection box list; for each retained detection box, the regional feature map of the corresponding area is extracted from the segmentation head of the network; the extracted regional feature map is input into the fully convolutional network to generate a binary mask of the same size as the detection box; the detection box coordinates and segmentation mask are mapped from the regional feature map scale back to the original image scale; for each detection result, the location, size, confidence, category and precise contour information of the lesion area are integrated.

4. The method according to claim 1, characterized in that: Based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant types and predict their stages. The output classification results and visualization heat maps include: Extract deep features, texture features, morphological features and statistical features from the detection results, where deep features are extracted through a pre-trained ResNet50 model, texture features include gray-level co-occurrence matrix and local binary pattern features, morphological features include geometric features and Fourier descriptors, and statistical features include gray-level histogram features and Gabor filter responses; use the random forest algorithm to calculate feature importance scores, select the top K most important features, where K is determined by cross-validation; perform principal component analysis on the selected features to reduce dimensionality, retain 95% of the variance information, and obtain the feature vector after dimensionality reduction; Construct multiple base classifiers, including support vector machine, random forest, XGBoost and deep neural network. The support vector machine uses RBF kernel function, the number of trees in random forest is set to 100, XGBoost uses tree model as base learner, and the deep neural network is a three-layer fully connected network. Use 5-fold cross validation to train and evaluate each base classifier, and use early stopping strategy and Adam optimizer for deep neural network. The soft voting method and stacking ensemble are used for model fusion. The soft voting method averages the prediction probabilities of each base classifier, and the stacking ensemble uses logistic regression as the meta-classifier. The staging prediction is converted into a multi-class classification problem, including early, middle and late stages, and the lesion size and invasion depth features are added to the original features; Use the Grad-CAM algorithm to generate class activation heatmaps for deep learning models, calculate the gradient of the feature map of the last convolutional layer and the category score, and perform global average pooling on the gradient to obtain the channel importance weight; calculate the SHAP value for the tree model to reflect the contribution of each feature to the model prediction, and map the SHAP value back to the original image space; The Grad-CAM and SHAP heatmaps are weighted fused to obtain a comprehensive heatmap, which is then superimposed on the original image using a flame map to generate a visual heatmap.

5. The method according to claim 1, characterized in that Using the improved point cloud reconstruction algorithm, the extracted lesion area information is converted into three-dimensional point cloud data; The adaptive octree structure is used to perform spatial division and noise reduction on 3D point cloud data, including: Performing depth estimation on the two-dimensional lesion detection results to obtain a depth map, wherein the depth estimation adopts an improved monocular depth estimation network, which includes a spatial attention module, a channel attention mechanism, and a multi-scale feature fusion module; based on the depth map, using a back-projection algorithm to convert the two-dimensional pixel coordinates into three-dimensional space coordinates to generate an initial point cloud data set; Adaptively adjusting the point cloud density of the initial point cloud data set includes: Calculating the local curvature of each point, determining an adaptive sampling rate based on the local curvature, resampling using a Poisson disk sampling algorithm, and applying voxelization filtering to ensure global consistency of point cloud density, thereby obtaining an adjusted point cloud data set; Constructing an adaptive octree structure to spatially divide the adjusted point cloud data set, including: Calculate the number of points, average curvature and spatial size of each node, define adaptive partitioning conditions based on these parameters, recursively partition the nodes according to the adaptive partitioning conditions, and store statistical information in leaf nodes; Outlier detection and removal based on the adaptive octree structure includes: Calculate the local statistical features of each leaf node, define the adaptive Z-score threshold, use the improved Z-score method to detect potential outliers, perform local plane fitting and correction on potential outliers, and perform global consistency check to obtain denoised point cloud data; Adaptively smoothing the denoised point cloud data by bilateral filtering, including: The number of neighbors is adaptively determined based on the local curvature, and the adaptive spatial weight and normal vector weight are calculated. The weight parameters are related to the local density and curvature of the point. The position of the point is iteratively updated, and the edge area is maintained to obtain the final three-dimensional point cloud data of the lesion area.

6. The method according to claim 1, characterized in that Using the improved Poisson surface reconstruction algorithm combined with the surface subdivision technology, a smooth 3D model of the lesion is generated, including: An adaptive octree is constructed based on the point cloud data. In each leaf node of the adaptive octree, a weighted least square method is used to calculate a local normal vector, wherein the weight function considers the spatial distance and the consistency of the normal vector; a Poisson equation is constructed based on the local normal vector, and a divergence operator is applied to the product of a gradient operator and a scalar field to be solved, so that the divergence operator is applied to the vector field; The Poisson equation is solved by a multi-grid method, and a local refinement strategy is used in high curvature areas; an improved Marching Cubes algorithm is used to extract isosurfaces, and the improved Marching Cubes algorithm increases sampling points in high curvature areas; Apply the Loop subdivision algorithm to the extracted isosurface for surface subdivision, including: Perform 4-1 topological subdivision on each triangle; update the vertex position, where the new vertex position is determined by the weighted sum of the four adjacent vertices, and the original vertex position is determined by the weighted sum of its own position and the positions of the adjacent vertices; Employ feature preserving strategies in areas of high curvature, including: The average curvature of each vertex is calculated, and when the average curvature is greater than a second preset threshold, the smoothing intensity is dynamically adjusted according to the curvature value.

7. A multimodal information fusion combined with artificial intelligence esophageal squamous cell carcinoma detection system, used to implement any of the methods of claims 1-6, characterized in that: include: The first unit is used to collect multimodal image information of esophageal tissue, wherein the multimodal image information includes endoscopic images, narrow-band imaging images, computed tomography images, and magnetic resonance imaging images; perform image enhancement processing on the multimodal image information, including contrast adjustment, denoising, and image registration; perform feature extraction on the multimodal image information subjected to image enhancement processing using a deep learning model to obtain a multidimensional feature vector; and input the multidimensional feature vector into a pre-trained multimodal fusion network, wherein the multimodal fusion network adopts an attention mechanism and a cross-modal transformer structure to achieve effective fusion of information of different modalities and output a fusion feature map; The second unit is used to input the fused feature map into a pre-trained esophageal squamous cell carcinoma detection model, wherein the esophageal squamous cell carcinoma detection model adopts an improved YOLOv5 network structure, uses the esophageal squamous cell carcinoma detection model to perform lesion area detection and segmentation on the fused feature map, and outputs the location, size and confidence information of the lesion area; Based on the detection results, an integrated learning method is used in combination with a classification algorithm to classify the detected lesions into benign and malignant types and predict their stages, and the classification results and a visual heat map are output; The third unit is used to extract lesion area information from the original multimodal image information based on classification results and visualization heat maps; use the improved point cloud reconstruction algorithm to convert the extracted lesion area information into three-dimensional point cloud data; use an adaptive octree structure to spatially divide and denoise the three-dimensional point cloud data; use an improved Poisson surface reconstruction algorithm combined with surface subdivision technology to generate a smooth three-dimensional lesion model; align and fuse the three-dimensional lesion model with the original multimodal image information to construct a comprehensive three-dimensional visualization model that includes the lesion location, size, morphology and surrounding tissue relationship.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images