AI analysis method and system based on straight ethyl mirror image

Abstract

The invention provides an AI analysis method and system based on a straight ethyl mirror image, and relates to the technical field of AI.The AI analysis method comprises the steps that multi-scale features of the straight ethyl mirror image are extracted through a pre-trained deep convolutional neural network, and feature fusion is carried out through a self-adaptive feature weight distribution module; and performing lesion area detection on the fused feature map by using a double-branch detection network, and classifying the detected lesion areas. And performing three-dimensional reconstruction based on a continuous orthoscope image sequence, mapping a detection result into a three-dimensional model, realizing accurate positioning and three-dimensional visualization of a lesion area, and recording three-dimensional coordinate position information of the lesion area. The method can improve the accuracy and efficiency of the image analysis of the straight ethyl mirror, and assists a doctor in diagnosis.

Description

Technical Field

[0001] The present invention relates to AI technology, and in particular to an AI analysis method and system based on a unidirectional mirror image. Background Art

[0002] Colorectal endoscopic examination is the gold standard for diagnosing and treating colorectal diseases. However, traditional colorectal endoscopic examination relies on the physician's visual inspection, which may lead to missed and misdiagnosed lesions, especially small and flat lesions in the early stages. To improve the accuracy and efficiency of colorectal endoscopic examination, computer-aided diagnosis systems have been widely studied and developed. These systems usually use image processing and machine learning techniques to automatically detect and classify lesions in colorectal images.

[0003] Insufficient multi-scale feature fusion: The size, shape and texture of lesions in CT images vary greatly. Some methods directly use the last layer of features of deep convolutional neural networks for detection, ignoring the information of shallow and middle layer features, resulting in insufficient detection performance for lesions of different scales. Effective multi-scale feature fusion can improve the model's ability to detect different lesions. Insufficient lesion localization accuracy: The intestinal wall has complex folds and the boundaries of the lesion area are fuzzy, which makes it difficult for some methods to accurately locate the boundaries of the lesions, affecting subsequent classification and analysis. Improving lesion localization accuracy is crucial for accurate diagnosis and treatment. Lack of three-dimensional spatial information: Many existing methods only focus on lesion detection and classification on two-dimensional images, ignoring the three-dimensional spatial information of lesions. Locating lesions in a three-dimensional model of the intestine can provide more comprehensive lesion information, help doctors more accurately assess the scope and location of lesions, and provide a basis for formulating personalized treatment plans. Summary of the invention

[0004] The embodiments of the present invention provide an AI analysis method and system based on a unidirectional mirror image, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides AI analysis methods based on CT images, including:

[0007] Input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle semantic features, and deep abstract features; use an adaptive feature weight allocation module to fuse the multi-scale feature map, the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map, and generate a fused feature map;

[0008] The fused feature map is input into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; the overlapping candidate frames are screened by a non-maximum suppression algorithm to obtain the final lesion region detection result; the detected lesion regions are classified by deep learning to identify the lesion type;

[0009] Based on a sequence of continuous rectoscopic images, a three-dimensional reconstruction algorithm based on depth estimation is used to model the inner wall of the intestine; the detection results of the lesion area are mapped to the three-dimensional reconstruction model, and the lesion area is accurately located using a spatial registration algorithm; a three-dimensional visualization model containing the spatial position information of the lesion area is constructed, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

[0010] The multi-scale feature map is fused by using an adaptive feature weight allocation module, the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture the key area information in the feature map, and the generated fused feature map includes:

[0011] The channel attention unit is used to perform global information compression on the multi-scale feature map through global average pooling to obtain a channel description vector, and the channel description vector is input into the dimension reduction fully connected layer for feature dimension reduction compression, and then the dimension reduction compressed features are restored to the original channel dimension through the dimension increase fully connected layer to generate a channel weight coefficient; the spatial attention unit is used to perform maximum pooling and average pooling operations on the multi-scale feature map to obtain spatial feature information, and the spatial feature information is processed through a convolution layer after splicing in the channel dimension to generate a spatial attention weight map;

[0012] The channel weight coefficient is weightedly fused with the multi-scale feature map to obtain channel weighted features, and the spatial attention weight map is weightedly fused with the channel weighted features to obtain spatial weighted features; the spatial weighted features are subjected to feature fusion processing to obtain fused features, and a residual connection mechanism is introduced to weightedly superimpose the original information of the multi-scale feature map with the fused features to generate a first fused feature map.

[0013] The fused feature map is input into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; the overlapping candidate frames are screened by a non-maximum suppression algorithm, and the final lesion region detection result is obtained, including:

[0014] The region proposal branch sets anchor points on the fused feature map, generates a reference frame with a preset scale and aspect ratio for each anchor point to construct an initial candidate frame set; uses a 3×3 convolution layer to extract local features of the initial candidate frame set, and calculates the foreground-background binary classification score of each candidate frame through a 1×1 convolution layer to generate the candidate lesion region;

[0015] The classification regression branch uses the RoI Align layer to extract features from the candidate lesion regions, uniformly maps the features of the candidate lesion regions of different sizes into a 7×7 fixed-size feature map, and inputs the fixed-size feature map into two fully connected layers to respectively output lesion category prediction results and position regression parameters, wherein the position regression parameters include the center coordinate offset and width and height scaling of the candidate box relative to the real box;

[0016] The candidate frames are sorted in descending order according to the classification scores, the candidate frame with the highest score is selected as the reference frame, the intersection and union ratio of other candidate frames and the reference frame is calculated, and the candidate frames with an intersection and union ratio greater than a set threshold are deleted. The process is repeated until all candidate frames are processed to obtain the final lesion area detection result.

[0017] The detected lesion areas are classified by deep learning, and the lesion types are identified including:

[0018] Preprocessing the lesion area detection result, uniformly scaling the lesion area detection result to 224×224 pixels using a bilinear interpolation algorithm, performing random horizontal flipping, random vertical flipping, random rotation, and random brightness contrast adjustment on the lesion area detection result to obtain an enhanced lesion area detection result;

[0019] The enhanced lesion area detection result is input into a multi-scale feature fusion module, wherein the multi-scale feature fusion module comprises three parallel feature extraction branches, each of which uses convolution layers with three different convolution kernels to extract features, and the convolution layer in each feature extraction branch is followed by a batch normalization layer and a ReLU activation function to obtain feature maps of three different scales;

[0020] Performing channel attention calculation on the feature maps of the three different scales to obtain corresponding attention weights, and performing element-wise summation on the weighted feature maps to obtain a second fused feature map; inputting the fused feature map into a pyramid pooling module, which includes average pooling operations of four different scales, and encodes spatial information of different scales into feature vectors;

[0021] The feature vector is input into a multilayer perceptron comprising two fully connected layers, and the multilayer perceptron is trained using a focal loss function, which focuses on difficult samples by dynamically adjusting sample weights. The model is optimized using an Adam optimizer with weight decay, and the initial learning rate is set to 0.001. The learning rate is decayed to the original 0.1 every 30 training cycles until the model converges to obtain a lesion classification model; the lesion type is determined according to the category probability distribution output by the lesion classification model.

[0022] Based on the continuous rectoscopic image sequence, the intestinal wall is modeled by a three-dimensional reconstruction algorithm based on depth estimation; the lesion area detection result is mapped to the three-dimensional reconstruction model, and the lesion area is accurately located by a spatial registration algorithm, including:

[0023] Preprocessing the continuous rectilinear image sequence, calculating the optical flow field between adjacent frames, eliminating frames with violent motion according to the average amplitude and directional consistency of the optical flow field, and performing histogram equalization enhancement on the continuous rectilinear image sequence to obtain a preprocessed image sequence;

[0024] Input the preprocessed image sequence into a two-stream depth estimation network, wherein the two-stream depth estimation network includes a spatial feature extraction stream and a temporal feature extraction stream, wherein the spatial feature extraction stream uses a ResNet-50 with a hole convolution as an encoder to extract multi-scale features, and restores spatial resolution through a skip connection and a feature pyramid network as a decoder, and the temporal feature extraction stream uses a three-dimensional convolutional network to capture the temporal dependency between adjacent frames;

[0025] A multi-constraint loss function is constructed by using a depth smoothness constraint, a geometric consistency constraint, and a photometric consistency constraint, the dual-stream depth estimation network is trained to obtain a depth estimation model, and the preprocessed image sequence is input into the depth estimation model to obtain a depth map;

[0026] Determine the relative pose between adjacent frames by matching feature points, restore the camera motion trajectory in combination with the depth map, convert the depth map into a three-dimensional point cloud, generate a three-dimensional mesh model using a Poisson surface reconstruction algorithm, and optimize the three-dimensional mesh model based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine;

[0027] The lesion area detection result is mapped to the intestinal 3D reconstruction model through back projection transformation to obtain the initial spatial position of the lesion area, the Gaussian curvature and the mean curvature of the lesion area are extracted to construct a local feature descriptor, the reprojection error of the lesion area under multiple viewing angles is calculated, and the initial spatial position of the lesion area is optimized based on the matching result of the local feature descriptor and the reprojection error to obtain the precise spatial position of the lesion area.

[0028] The relative pose between adjacent frames is determined by feature point matching, the camera motion trajectory is restored in combination with the depth map, the depth map is converted into a three-dimensional point cloud, a three-dimensional mesh model is generated by a Poisson surface reconstruction algorithm, and the three-dimensional mesh model is optimized based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine, including:

[0029] Adaptively preprocessing adjacent frame images in an endoscopic image sequence by using histogram equalization to obtain a preprocessed image, constructing a Hessian matrix of the preprocessed image, using a fast box filter to approximate the second-order derivative of Gaussian to calculate feature point response values, extracting feature points based on the feature point response values ​​and calculating feature descriptors, and generating feature information of the adjacent frame images;

[0030] Based on the feature information, a nearest neighbor matching algorithm is used to obtain an initial matching pair of feature points, and a RANSAC algorithm is used to eliminate mismatched point pairs in the initial matching pair of feature points to obtain an optimized matching pair of feature points, and an initial relative pose between adjacent frame images is calculated according to the optimized matching pair of feature points;

[0031] A reprojection error minimization objective function is constructed based on the initial relative posture, and the initial relative posture is iteratively optimized using the LM algorithm to obtain the accurate relative posture between the adjacent frame images and the corresponding reprojection error;

[0032] According to the precise relative pose, the depth map is converted into a three-dimensional point cloud in the camera coordinate system, the three-dimensional point cloud is transformed into the world coordinate system to obtain an initial point cloud, a bag-of-words model is used to detect loop frames in the initial point cloud and extract loop constraints, the loop constraints are added to the pose graph optimization framework, and the precise relative pose is optimized to obtain a complete motion trajectory of the camera;

[0033] Based on the complete motion trajectory of the camera, each frame depth map is uniformly converted into a complete three-dimensional point cloud in a world coordinate system, the complete three-dimensional point cloud is statistically filtered to remove noise points to obtain a filtered point cloud, the filtered point cloud is downsampled using a voxel grid to obtain a downsampled point cloud, and the normal vector information of each point in the downsampled point cloud is calculated to obtain point cloud data with a normal vector;

[0034] An adaptive octree is used to spatially divide the point cloud data with normal vectors, a Poisson equation is constructed and solved in each grid of the adaptive octree to obtain an indicator function, and a zero isosurface of the indicator function is extracted to generate an initial three-dimensional grid model;

[0035] The normal vector distribution of the vertices of the initial three-dimensional mesh model is calculated, and a normal vector smoothing constraint is constructed based on the consistency of the normal vectors of adjacent vertices. The normal vector smoothing constraint is introduced into the Laplace smoothing operator, and the initial three-dimensional mesh model is iteratively optimized to obtain a final three-dimensional reconstructed model of the intestine that maintains both geometric details and smoothness.

[0036] A three-dimensional visualization model containing the spatial location information of the lesion area is constructed. The three-dimensional visualization model marks different types of lesion areas by color coding and records the three-dimensional coordinate location information of the lesion area relative to the intestinal reference point, including:

[0037] A cylindrical coordinate system with the intestinal centerline as the main axis is established with the starting end of the intestine as the coordinate origin, reference points are set at preset distances along the intestinal centerline in the cylindrical coordinate system, three-dimensional coordinate position information of the reference points in the world coordinate system is collected, and an intestinal skeleton structure containing the three-dimensional coordinate position information is constructed;

[0038] Locating the lesion area based on the intestinal skeleton structure, obtaining the three-dimensional coordinates of the centroid of the lesion area in the world coordinate system, converting the three-dimensional coordinates into the cylindrical coordinate system to obtain the radial distance, circumferential angle and axial distance of the centroid of the lesion area, and generating the spatial position information of the lesion area;

[0039] Calculate the distance vector from the centroid of the lesion area to the nearest reference point according to the spatial position information of the lesion area, and determine the components of the distance vector in the tangential, normal and binormal directions as the relative position information of the lesion area relative to the reference point;

[0040] Building a three-dimensional visualization model of the lesion area based on the spatial position information and the relative position information, wherein the three-dimensional visualization model includes position information of the lesion area in the world coordinate system and the cylindrical coordinate system;

[0041] Designing a color coding scheme for the lesion area in the three-dimensional visualization model, dividing the lesion area into different levels according to the severity of the lesion and marking them with different color systems respectively, setting different tones in each color system according to the specific type of the lesion, using color transparency to represent the depth information of the lesion area, and applying the color coding scheme to the three-dimensional visualization model;

[0042] The image sequences of the lesion area at different time points are collected, the image sequences are registered, the registered image sequences are mapped to the three-dimensional visualization model, and the change data of the three-dimensional coordinate position information of the lesion area relative to the reference point over time is recorded.

[0043] According to a second aspect of the embodiments of the present invention,

[0044] Provides an AI analysis system based on unidirectional microscope images, including:

[0045] The first unit is used to input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle-level semantic features, and deep-level abstract features; an adaptive feature weight allocation module is used to fuse the multi-scale feature map, and the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map to generate a fused feature map;

[0046] The second unit is used to input the fused feature map into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; a non-maximum suppression algorithm is used to screen overlapping candidate frames to obtain the final lesion region detection result; deep learning classification is performed on the detected lesion region to identify the lesion type;

[0047] The third unit is used to model the inner wall of the intestine based on a continuous rectoscopic image sequence and a three-dimensional reconstruction algorithm based on depth estimation; map the lesion area detection result to the three-dimensional reconstruction model, and use a spatial registration algorithm to accurately locate the lesion area; construct a three-dimensional visualization model containing the spatial position information of the lesion area, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

[0048] According to a third aspect of the embodiments of the present invention,

[0049] An electronic device is provided, comprising:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0053] A fourth aspect of the embodiments of the present invention is:

[0054] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0055] The beneficial effects of this application are as follows:

[0056] 1. Improve the accuracy and efficiency of lesion detection: By extracting multi-scale features through a deep convolutional neural network and combining it with an adaptive feature weight allocation module, the characteristic information of the lesion can be captured more comprehensively, improving the sensitivity and accuracy of lesion detection. The application of a dual-branch detection network and a non-maximum suppression algorithm can accurately locate the lesion area, reduce false detections and missed detections, and improve detection efficiency.

[0057] 2. Accurately identify lesion types: Using deep learning classification models to classify detected lesion areas can identify different lesion types, provide more detailed information for clinical diagnosis, and assist doctors in formulating more accurate treatment plans.

[0058] 3. Provide intuitive 3D visualization model: Based on the 3D reconstruction algorithm and spatial registration algorithm based on depth estimation, a 3D visualization model containing the spatial location information of the lesion area is constructed, which can intuitively display the location, size and type of the lesion area, making it easier for doctors to observe and understand the condition, and improve diagnostic efficiency and accuracy. The 3D coordinate information also provides an important reference for subsequent precision treatment and surgical navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the flow of an AI analysis method based on a straight-mirror image according to an embodiment of the present invention;

[0060] Figure 2 The figure is a schematic diagram of the structure of an AI analysis system based on a rectilinear mirror image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0063] Figure 1 FIG. 1 is a flow chart of an AI analysis method based on a rectilinear mirror image according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] S11. Input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle semantic features, and deep abstract features; use an adaptive feature weight allocation module to fuse the multi-scale feature map, the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map, and generate a fused feature map;

[0065] S12. Input the fused feature map into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; a non-maximum suppression algorithm is used to screen overlapping candidate frames to obtain the final lesion region detection result; deep learning classification is performed on the detected lesion regions to identify the lesion type;

[0066] S13. Based on a sequence of continuous rectoscopic images, a three-dimensional reconstruction algorithm based on depth estimation is used to model the inner wall of the intestine; the detection results of the lesion area are mapped to the three-dimensional reconstruction model, and the lesion area is accurately located using a spatial registration algorithm; a three-dimensional visualization model containing the spatial position information of the lesion area is constructed, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

[0067] In an optional implementation, an adaptive feature weight allocation module is used to fuse the multi-scale feature map, and the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map, and generating the fused feature map includes:

[0068] The channel attention unit is used to perform global information compression on the multi-scale feature map through global average pooling to obtain a channel description vector, and the channel description vector is input into the dimension reduction fully connected layer for feature dimension reduction compression, and then the dimension reduction compressed features are restored to the original channel dimension through the dimension increase fully connected layer to generate a channel weight coefficient; the spatial attention unit is used to perform maximum pooling and average pooling operations on the multi-scale feature map to obtain spatial feature information, and the spatial feature information is processed through a convolution layer after splicing in the channel dimension to generate a spatial attention weight map;

[0069] The channel weight coefficient is weightedly fused with the multi-scale feature map to obtain channel weighted features, and the spatial attention weight map is weightedly fused with the channel weighted features to obtain spatial weighted features; the spatial weighted features are subjected to feature fusion processing to obtain fused features, and a residual connection mechanism is introduced to weightedly superimpose the original information of the multi-scale feature map with the fused features to generate a first fused feature map.

[0070] A multi-scale feature map fusion method based on an adaptive feature weight assignment module is used to improve feature representation capabilities, especially for image recognition and other fields. This method learns the importance of different channels through channel attention units and focuses on key areas through spatial attention units, thereby achieving more effective feature fusion.

[0071] First, prepare multi-scale feature maps as input. For example, you can use a list of three scale feature maps, with scales of 256x256x64, 128x128x128, and 64x64x256, where the dimensions of each scale feature map are expressed as height x width x number of channels.

[0072] Next, a channel attention unit is applied to each scale feature map. The channel attention unit first performs a global average pooling operation on the input feature map to compress the features of each channel into a numerical value to obtain a channel description vector. For example, for a 256x256x64 feature map, a 64-dimensional vector is obtained after global average pooling. Then, the vector is input into a dimensionality reduction fully connected layer, for example, reducing the 64-dimensional to 16-dimensional, for feature compression. Then, through a dimensionality increase fully connected layer, the 16-dimensional features are restored to the original 64-dimensional to obtain the channel weight coefficient.

[0073] At the same time, a spatial attention unit is applied to each scale feature map. The spatial attention unit performs max pooling and average pooling operations on the input feature map to obtain spatial feature information. For example, a 2x2 pooling kernel can be used for the operation. The feature maps obtained by max pooling and average pooling are concatenated in the channel dimension. For example, for a 256x256x64 feature map, after 2x2 pooling, the dimension becomes 128x128x64, and the concatenated dimension is 128x128x128. Then, the concatenated feature map is processed through a convolutional layer to generate a spatial attention weight map, whose dimension is the same as the input feature map, for example, 128x128x1.

[0074] The channel weight coefficient is weightedly fused with the corresponding multi-scale feature map to obtain the channel weighted feature. The specific operation is to multiply the feature value of each channel by the corresponding channel weight coefficient. The spatial attention weight map is weightedly fused with the channel weighted feature to obtain the spatial weighted feature. The specific operation is to multiply the feature vector of each position of the feature map by the corresponding spatial attention weight value.

[0075] The spatial weighted features are subjected to feature fusion processing, for example, channel splicing can be used for fusion. Then, a residual connection mechanism is introduced to weightedly superimpose the original information of the multi-scale feature map with the fused features to generate a first fused feature map. For example, the original feature map and the fused feature map can be added in a 1:1 ratio.

[0076] The solution of this application can:

[0077] Improve feature representation capabilities: By adaptively learning channel and spatial weights, this method can more effectively capture key information, thereby improving feature representation capabilities. Enhance model robustness: By fusing multi-scale features, this method can better cope with targets of different scales and complex backgrounds, enhancing the robustness of the model. Reduce computational complexity: Through feature compression and an efficient attention mechanism, this method can reduce computational complexity while improving performance.

[0078] In an optional implementation, the fused feature map is input into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; wherein the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; a non-maximum suppression algorithm is used to screen overlapping candidate frames, and obtaining the final lesion region detection result includes:

[0079] The region proposal branch sets anchor points on the fused feature map, generates a reference frame with a preset scale and aspect ratio for each anchor point to construct an initial candidate frame set; uses a 3×3 convolution layer to extract local features of the initial candidate frame set, and calculates the foreground-background binary classification score of each candidate frame through a 1×1 convolution layer to generate the candidate lesion region;

[0080] The classification regression branch uses the RoI Align layer to extract features from the candidate lesion regions, uniformly maps the features of the candidate lesion regions of different sizes into a 7×7 fixed-size feature map, and inputs the fixed-size feature map into two fully connected layers to respectively output lesion category prediction results and position regression parameters, wherein the position regression parameters include the center coordinate offset and width and height scaling of the candidate box relative to the real box;

[0081] The candidate frames are sorted in descending order according to the classification scores, the candidate frame with the highest score is selected as the reference frame, the intersection and union ratio of other candidate frames and the reference frame is calculated, and the candidate frames with an intersection and union ratio greater than a set threshold are deleted. The process is repeated until all candidate frames are processed to obtain the final lesion area detection result.

[0082] The fused feature map is input into the dual-branch detection network to detect the lesion area. The specific steps are as follows:

[0083] First, the input image is preprocessed, including but not limited to image resizing, normalization, etc. For example, the input image is resized to 512x512 pixels, and the pixel values ​​are normalized to between 0 and 1. The preprocessed image is input into the feature extraction network.

[0084] Then, the multi-scale features of the image are extracted using a feature extraction network. The feature extraction network can use classic network structures such as ResNet and VGG, or other types of convolutional neural networks. For example, using the ResNet50 network to extract image features, feature maps of different levels are obtained, which are 128x128, 64x64, and 32x32 in size.

[0085] Next, the feature maps of different scales are fused to obtain a fused feature map. The fusion method can use the feature pyramid network FPN or other feature fusion methods. For example, the feature maps of different levels extracted by ResNet50 are fused using the FPN network to obtain a fused feature map of size 256x256.

[0086] The fused feature map is input into a two-branch detection network, which includes a region proposal branch and a classification regression branch.

[0087] In the region proposal branch, first set anchor points on the fused feature map. For example, on a 256x256 feature map, set an anchor point every 16 pixels, for a total of 256 anchor points. For each anchor point, generate a reference frame with a preset scale and aspect ratio. For example, set 3 scales (16x16, 32x32, 64x64) and 3 aspect ratios (1:1, 1:2, 2:1), for a total of 9 reference frames. Use these reference frames as the initial candidate frame set.

[0088] Then, a 3x3 convolution layer is used to extract local features of the initial candidate box set. For example, a convolution layer with a 3x3 convolution kernel, a step size of 1, and a padding of 1 is used to extract features. Then, a 1x1 convolution layer is used to calculate the foreground-background binary classification score of each candidate box to generate a candidate lesion area. For example, a 1x1 convolution kernel is used to calculate the score of each candidate box, and the candidate box with a high score is considered to be the foreground, that is, it may contain a lesion area.

[0089] In the classification and regression branch, the RoI Align layer is used to extract features from the candidate lesion regions. For example, the features of candidate lesion regions of different sizes are uniformly mapped into a 7x7 fixed-size feature map. The fixed-size feature map is input into two fully connected layers to output the lesion category prediction results and position regression parameters respectively. The position regression parameters include the center coordinate offset of the candidate box relative to the real box and the width and height scaling.

[0090] Finally, the non-maximum suppression algorithm is used to screen the overlapping candidate boxes to obtain the final lesion area detection results. The candidate boxes are sorted in descending order according to the classification scores, and the candidate box with the highest score is selected as the reference box. The intersection over union (IoU) of other candidate boxes and the reference box is calculated. For example, the IoU threshold is set to 0.7. Candidate boxes with an IoU greater than 0.7 are deleted. Repeat until all candidate boxes are processed to obtain the final lesion area detection results.

[0091] The solution of this application can:

[0092] Improve detection accuracy: By fusing multi-scale features and using the RoI Align layer, the features of the lesion area can be extracted more accurately, thereby improving detection accuracy. Improve detection efficiency: Through dense sampling and non-maximum suppression algorithms, redundant candidate boxes can be effectively reduced and detection efficiency can be improved. Enhance model robustness: This method has strong adaptability to lesion areas of different scales and aspect ratios, which enhances the robustness of the model.

[0093] In an optional implementation, deep learning classification is performed on the detected lesion area to identify the lesion type including:

[0094] Preprocessing the lesion area detection result, uniformly scaling the lesion area detection result to 224×224 pixels using a bilinear interpolation algorithm, performing random horizontal flipping, random vertical flipping, random rotation, and random brightness contrast adjustment on the lesion area detection result to obtain an enhanced lesion area detection result;

[0095] The enhanced lesion area detection result is input into a multi-scale feature fusion module, wherein the multi-scale feature fusion module comprises three parallel feature extraction branches, each of which uses convolution layers with three different convolution kernels to extract features, and the convolution layer in each feature extraction branch is followed by a batch normalization layer and a ReLU activation function to obtain feature maps of three different scales;

[0096] Performing channel attention calculation on the feature maps of the three different scales to obtain corresponding attention weights, and performing element-wise summation on the weighted feature maps to obtain a second fused feature map; inputting the fused feature map into a pyramid pooling module, which includes average pooling operations of four different scales, and encodes spatial information of different scales into feature vectors;

[0097] The feature vector is input into a multilayer perceptron comprising two fully connected layers, and the multilayer perceptron is trained using a focal loss function, which focuses on difficult samples by dynamically adjusting sample weights. The model is optimized using an Adam optimizer with weight decay, and the initial learning rate is set to 0.001. The learning rate is decayed to the original 0.1 every 30 training cycles until the model converges to obtain a lesion classification model; the lesion type is determined according to the category probability distribution output by the lesion classification model.

[0098] A deep learning-based lesion region classification method is used to identify multiple lesion types. This method extracts key features of lesion regions through multi-scale feature fusion and pyramid pooling modules, and uses the focal loss function to train the classification model, thereby improving the accuracy and robustness of lesion classification.

[0099] First, the acquired lesion area detection results are preprocessed. Each lesion area image is uniformly scaled to 224 pixels wide and 224 pixels high using a bilinear interpolation algorithm. For example, a lesion area image with an original size of 300x400 pixels will become 224x224 pixels after bilinear interpolation scaling. Subsequently, a series of data enhancement operations are performed on the scaled image, including random horizontal flipping, that is, mirror flipping with the vertical axis as the center; random vertical flipping, that is, mirror flipping with the horizontal axis as the center; random rotation, rotating the image by a random angle, such as any angle between 0 and 360 degrees; and random brightness contrast adjustment, randomly adjusting the brightness and contrast of the image, such as the brightness adjustment range is 0.8 to 1.2 times, and the contrast adjustment range is 0.9 to 1.1 times. Through these data enhancement operations, the diversity of training data can be increased, the generalization ability of the model can be improved, and overfitting can be prevented.

[0100] Next, the enhanced lesion area image is input into the multi-scale feature fusion module. This module contains three parallel feature extraction branches. Each branch uses convolution kernels of different sizes for feature extraction. For example, the first branch uses a 3x3 convolution kernel, the second branch uses a 5x5 convolution kernel, and the third branch uses a 7x7 convolution kernel. Each convolution layer is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer can speed up the training of the model and improve the stability of the model. The ReLU activation function can introduce nonlinearity and enhance the expressiveness of the model. In this way, each branch will generate a feature map of different scales.

[0101] Then, channel attention is calculated for these three feature maps of different scales. Specifically, global average pooling is first performed on each feature map to obtain a channel descriptor. Then, the channel descriptor is input into a two-layer fully connected network. The number of output channels of the first layer of the fully connected network is 1 / 16 of the original number of channels, and the number of output channels of the second layer of the fully connected network is the same as the original number of channels. Finally, the output value is normalized to between 0 and 1 using a normalized exponential function to obtain the attention weight of each channel. These weights are multiplied by the corresponding feature map to obtain the weighted feature map. Finally, the three weighted feature maps are element-wise added to obtain the fused feature map, which we call the second fused feature map.

[0102] The second fused feature map is input to the pyramid pooling module. This module contains four average pooling operations of different scales, for example, the feature map is divided into 1x1, 2x2, 3x3 and 6x6 grids. Average pooling is performed on each grid to encode spatial information of different scales into feature vectors. These feature vectors are then concatenated to form a fixed-length feature vector.

[0103] Finally, the concatenated feature vectors are input into a multilayer perceptron containing two fully connected layers. The multilayer perceptron is trained using the focal loss function. The focal loss function dynamically adjusts the sample weights and pays more attention to samples that are difficult to classify, thereby improving the classification performance of the model. The model is optimized using the Adam optimizer with weight decay. The initial learning rate is set to 0.001, and the learning rate is decayed to the original 0.1 every 30 training cycles until the model converges to obtain the final lesion classification model. The lesion type is determined based on the class probability distribution output by the model. For example, if the probability distribution output by the model is [0.1, 0.8, 0.1], the lesion area is classified as the second category.

[0104] The solution of this application can:

[0105] Improve classification accuracy: This method can extract more comprehensive and representative features through multi-scale feature fusion and pyramid pooling modules, thereby improving the accuracy of lesion classification. Enhance model robustness: The use of data augmentation operations and focal loss functions can enhance the robustness of the model to different data distributions and noise. Improve model efficiency: The channel attention mechanism can make the model pay more attention to important feature channels, thereby improving the computational efficiency of the model.

[0106] In an optional embodiment, based on a continuous rectoscopic image sequence, a three-dimensional reconstruction algorithm based on depth estimation is used to model the inner wall of the intestine; the lesion area detection result is mapped to the three-dimensional reconstruction model, and the lesion area is accurately located using a spatial registration algorithm, including:

[0107] Preprocessing the continuous rectilinear image sequence, calculating the optical flow field between adjacent frames, eliminating frames with violent motion according to the average amplitude and directional consistency of the optical flow field, and performing histogram equalization enhancement on the continuous rectilinear image sequence to obtain a preprocessed image sequence;

[0108] Input the preprocessed image sequence into a two-stream depth estimation network, wherein the two-stream depth estimation network includes a spatial feature extraction stream and a temporal feature extraction stream, wherein the spatial feature extraction stream uses a ResNet-50 with a hole convolution as an encoder to extract multi-scale features, and restores spatial resolution through a skip connection and a feature pyramid network as a decoder, and the temporal feature extraction stream uses a three-dimensional convolutional network to capture the temporal dependency between adjacent frames;

[0109] A multi-constraint loss function is constructed by using a depth smoothness constraint, a geometric consistency constraint, and a photometric consistency constraint, the dual-stream depth estimation network is trained to obtain a depth estimation model, and the preprocessed image sequence is input into the depth estimation model to obtain a depth map;

[0110] Determine the relative pose between adjacent frames by matching feature points, restore the camera motion trajectory in combination with the depth map, convert the depth map into a three-dimensional point cloud, generate a three-dimensional mesh model using a Poisson surface reconstruction algorithm, and optimize the three-dimensional mesh model based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine;

[0111] The lesion area detection result is mapped to the intestinal 3D reconstruction model through back projection transformation to obtain the initial spatial position of the lesion area, the Gaussian curvature and the mean curvature of the lesion area are extracted to construct a local feature descriptor, the reprojection error of the lesion area under multiple viewing angles is calculated, and the initial spatial position of the lesion area is optimized based on the matching result of the local feature descriptor and the reprojection error to obtain the precise spatial position of the lesion area.

[0112] The method of three-dimensional reconstruction of continuous rectoscopic image sequences and precise positioning of lesion areas constructs a three-dimensional model of the intestinal wall through depth estimation and three-dimensional reconstruction technology, and maps the lesion detection results into the three-dimensional model to achieve precise positioning of the lesion area.

[0113] First, the acquired continuous collimator image sequence is preprocessed. The optical flow field between every two adjacent images is calculated to describe the movement direction and amplitude of the image pixels. If the average amplitude of the optical flow field between adjacent frames is too large, or the consistency of the optical flow direction is poor, it is considered that there is violent motion in the frame image and it is eliminated. For example, the average optical flow amplitude threshold is set to 10 pixels and the direction consistency threshold is set to 0.8. If the average optical flow amplitude between two frames exceeds 10 pixels, or the direction consistency is lower than 0.8, it is considered that there is violent motion in the frame image and it is eliminated. The remaining image sequence is enhanced by histogram equalization to improve the image contrast and brightness for subsequent processing. Assuming that the image sequence has a total of 100 frames, 80 frames remain after optical flow analysis and elimination. These 80 frames are processed by histogram equalization to obtain the preprocessed image sequence.

[0114] Next, the preprocessed image sequence is input into a specially designed two-stream depth estimation network. The network contains two parallel processing flows: spatial feature extraction flow and temporal feature extraction flow. The spatial feature extraction flow uses the ResNet-50 network as the encoder and combines it with dilated convolution to extract multi-scale spatial features. The decoder uses skip connections and feature pyramid networks to restore spatial resolution and obtain more detailed depth information. The temporal feature extraction flow uses a three-dimensional convolutional network to capture the temporal dependencies between adjacent frames. For example, a convolutional layer with a three-dimensional convolution kernel size of 3x3x3 is used to extract temporal features. The outputs of the two feature extraction flows are fused to obtain the final depth estimation result.

[0115] In order to train this two-stream depth estimation network, a loss function containing multiple constraints is constructed. The loss function contains depth smoothness constraints, geometric consistency constraints, and photometric consistency constraints. The depth smoothness constraint encourages the depth values ​​of adjacent pixels to change smoothly; the geometric consistency constraint ensures that the geometric shape of the 3D reconstruction result is reasonable; the photometric consistency constraint requires that the brightness of the same object under different viewing angles remain consistent. The network is trained using the preprocessed image sequence and the corresponding depth truth data. For example, 80% of the data is used for training and 20% of the data is used for verification. By minimizing the loss function and continuously optimizing the network parameters, a depth estimation model is finally obtained. The preprocessed image sequence is input into the trained depth estimation model to obtain the depth map corresponding to each frame of the image. Assuming that an image with a resolution of 512x512 is input, the output depth map is also 512x512, and each pixel value represents the distance from the pixel to the camera.

[0116] Then, three-dimensional reconstruction is performed based on the depth map and image sequence. The relative pose between adjacent frames is determined by feature point matching. For example, the feature points are extracted and matched using the SIFT algorithm. Combining the depth map and camera pose, the movement trajectory of the camera in the intestine can be restored. The depth map is converted into a three-dimensional point cloud. For example, each pixel on the depth map is converted into a point in three-dimensional space based on the intrinsic and extrinsic parameters of the camera. The Poisson surface reconstruction algorithm is used to convert the three-dimensional point cloud into a three-dimensional mesh model. In order to optimize the surface quality of the three-dimensional mesh model, a method based on the consistency of the surface normal vector is used for smoothing, and finally a three-dimensional reconstructed model of the intestine is obtained.

[0117] Finally, the lesion area detection results are mapped to the three-dimensional reconstruction model to achieve accurate positioning of the lesion area. First, the lesion area on the two-dimensional image is mapped to the three-dimensional space through the back-projection transformation to obtain the initial spatial position of the lesion area. Then, the geometric features such as the Gaussian curvature and the mean curvature of the lesion area are extracted to construct a local feature descriptor. The reprojection error of the lesion area under multiple viewing angles is calculated, that is, the difference between the position of the lesion area in the three-dimensional space projected onto the two-dimensional image and the actual detected position of the lesion area. Based on the matching results and reprojection errors of the local feature descriptors, the initial spatial position of the lesion area is optimized, and finally the precise spatial position of the lesion area is obtained. For example, assuming that the reprojection error of the lesion area is 2 pixels at one viewing angle, it can be reduced to 1 pixel through the optimization algorithm.

[0118] The solution of this application can:

[0119] Improve diagnostic accuracy: Through 3D reconstruction technology, the morphology of the intestinal wall and the spatial position relationship of the lesion area can be more intuitively observed, which helps doctors to more accurately determine the nature and scope of the lesion and improve the accuracy of diagnosis. Assisted surgical planning: Based on accurate 3D models, doctors can be assisted in surgical planning, for example, determining the surgical path and resection range, and improving the safety of surgery. Convenient disease tracking: By comparing 3D models at different time points, it is easier to track the progression of the disease and evaluate the treatment effect.

[0120] In an optional implementation, the relative pose between adjacent frames is determined by feature point matching, the camera motion trajectory is restored in combination with the depth map, the depth map is converted into a three-dimensional point cloud, a three-dimensional mesh model is generated by a Poisson surface reconstruction algorithm, and the three-dimensional mesh model is optimized based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine, including:

[0121] Adaptively preprocessing adjacent frame images in an endoscopic image sequence by using histogram equalization to obtain a preprocessed image, constructing a Hessian matrix of the preprocessed image, using a fast box filter to approximate the second-order derivative of Gaussian to calculate feature point response values, extracting feature points based on the feature point response values ​​and calculating feature descriptors, and generating feature information of the adjacent frame images;

[0122] Based on the feature information, a nearest neighbor matching algorithm is used to obtain an initial matching pair of feature points, and a RANSAC algorithm is used to eliminate mismatched point pairs in the initial matching pair of feature points to obtain an optimized matching pair of feature points, and an initial relative pose between adjacent frame images is calculated according to the optimized matching pair of feature points;

[0123] A reprojection error minimization objective function is constructed based on the initial relative posture, and the initial relative posture is iteratively optimized using the LM algorithm to obtain the accurate relative posture between the adjacent frame images and the corresponding reprojection error;

[0124] According to the precise relative pose, the depth map is converted into a three-dimensional point cloud in the camera coordinate system, the three-dimensional point cloud is transformed into the world coordinate system to obtain an initial point cloud, a bag-of-words model is used to detect loop frames in the initial point cloud and extract loop constraints, the loop constraints are added to the pose graph optimization framework, and the precise relative pose is optimized to obtain a complete motion trajectory of the camera;

[0125] Based on the complete motion trajectory of the camera, each frame depth map is uniformly converted into a complete three-dimensional point cloud in a world coordinate system, the complete three-dimensional point cloud is statistically filtered to remove noise points to obtain a filtered point cloud, the filtered point cloud is downsampled using a voxel grid to obtain a downsampled point cloud, and the normal vector information of each point in the downsampled point cloud is calculated to obtain point cloud data with a normal vector;

[0126] An adaptive octree is used to spatially divide the point cloud data with normal vectors, a Poisson equation is constructed and solved in each grid of the adaptive octree to obtain an indicator function, and a zero isosurface of the indicator function is extracted to generate an initial three-dimensional grid model;

[0127] The normal vector distribution of the vertices of the initial three-dimensional mesh model is calculated, and a normal vector smoothing constraint is constructed based on the consistency of the normal vectors of adjacent vertices. The normal vector smoothing constraint is introduced into the Laplace smoothing operator, and the initial three-dimensional mesh model is iteratively optimized to obtain a final three-dimensional reconstructed model of the intestine that maintains both geometric details and smoothness.

[0128] A 3D reconstruction method for endoscopic image sequences is used to construct an accurate 3D model of the intestine. This method uses feature point matching, depth map fusion and Poisson surface reconstruction techniques, combined with normal vector optimization, to achieve high-quality 3D reconstruction.

[0129] First, the image sequence collected by the endoscope is preprocessed. Each frame of the sequence is subjected to adaptive histogram equalization to enhance the image contrast and detail information for subsequent feature extraction. For example, the original pixel value range of a frame of image is between 50 and 200, and after equalization, the pixel value range is expanded to 0 to 255, making the light and dark details of the image clearer.

[0130] Then, perform feature extraction on the preprocessed image. Construct the Hessian matrix of the image, use the fast box filter to approximate the Gaussian second-order derivative, and calculate the feature point response value. Extract feature points based on the response value, and calculate the descriptor of each feature point. For example, 500 feature points are extracted from a frame of image, and each feature point is described by a 128-dimensional vector.

[0131] Next, feature matching and pose estimation are performed. The feature information of two adjacent frames is matched, and the nearest neighbor matching algorithm is used to obtain the initial matching pairs. Then, the RANSAC algorithm is used to eliminate the mismatched point pairs to obtain the optimized feature point matching pairs. Based on these matching pairs, the initial relative pose between the two frames is calculated, such as the rotation matrix and translation vector. Assume that there are 100 initial matching point pairs between the two frames. After being eliminated by the RANSAC algorithm, 80 matching point pairs are left for pose calculation.

[0132] In order to obtain a more accurate posture, the initial posture is optimized. Taking the initial relative posture as the benchmark, the objective function of minimizing the reprojection error is constructed. The LM algorithm is used to iteratively optimize the initial relative posture, and finally the accurate relative posture and corresponding reprojection error between the two frames are obtained. For example, the initial rotation angle is 10 degrees. After optimization, the accurate rotation angle is 9.5 degrees, and the reprojection error is reduced to 0.5 pixels.

[0133] After obtaining the accurate relative pose, the depth map is converted into a 3D point cloud. Each pixel value in the depth map is converted into a 3D coordinate in the camera coordinate system to obtain the initial point cloud. Then, the initial point cloud is transformed into the world coordinate system. For example, a pixel with a depth value of 10 is converted into a 3D coordinate (x, y, 10).

[0134] In order to construct a complete camera motion trajectory, loop detection and pose graph optimization are performed. The bag-of-words model is used to detect loop frames in the initial point cloud and extract loop constraints. The loop constraints are added to the pose graph optimization framework to optimize the precise relative pose calculated previously to obtain the complete motion trajectory of the camera. For example, if it is detected that the 1st frame and the 100th frame are loop frames, a constraint is added to limit the pose difference between the two frames.

[0135] Based on the complete motion trajectory of the camera, the depth map of each frame is uniformly converted into a complete 3D point cloud in the world coordinate system. The complete 3D point cloud is statistically filtered to remove noise points, for example, points whose average distance from the surrounding points exceeds a certain threshold are removed. Then, the filtered point cloud is downsampled using a voxel grid to reduce the amount of data. Finally, the normal vector information of each point in the downsampled point cloud is calculated. For example, a voxel grid contains 100 points, and only the center point is retained after downsampling.

[0136] Next, the three-dimensional mesh model is constructed. The point cloud data with normal vectors is spatially divided using an adaptive octree. In each mesh, the Poisson equation is constructed and solved to obtain the indicator function. The zero isosurface of the indicator function is extracted to generate the initial three-dimensional mesh model.

[0137] Finally, the initial 3D mesh model is optimized. The normal vector distribution of the vertices of the initial 3D mesh model is calculated, and the normal vector smoothing constraint is constructed based on the consistency of the normal vectors of adjacent vertices. The normal vector smoothing constraint is introduced into the Laplace smoothing operator, and the initial 3D mesh model is iteratively optimized to obtain the final 3D reconstruction model of the intestine.

[0138] The solution of this application can:

[0139] Improve reconstruction accuracy: Accurately calculate the camera pose through feature point matching and pose graph optimization, reduce cumulative errors, and thus improve the accuracy of 3D reconstruction. Enhance model integrity: Use loop detection and depth map fusion to build complete point cloud data and generate a more complete 3D model to avoid local loss. Optimize model smoothness: Through normal vector consistency constraints and Laplace smoothing, optimize the 3D mesh model so that it maintains geometric details and has a smooth surface, which is more in line with the real intestinal morphology.

[0140] In an optional embodiment, a three-dimensional visualization model including spatial position information of the lesion area is constructed, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point, including:

[0141] A cylindrical coordinate system with the intestinal centerline as the main axis is established with the starting end of the intestine as the coordinate origin, reference points are set at preset distances along the intestinal centerline in the cylindrical coordinate system, three-dimensional coordinate position information of the reference points in the world coordinate system is collected, and an intestinal skeleton structure containing the three-dimensional coordinate position information is constructed;

[0142] Locating the lesion area based on the intestinal skeleton structure, obtaining the three-dimensional coordinates of the centroid of the lesion area in the world coordinate system, converting the three-dimensional coordinates into the cylindrical coordinate system to obtain the radial distance, circumferential angle and axial distance of the centroid of the lesion area, and generating the spatial position information of the lesion area;

[0143] Calculate the distance vector from the centroid of the lesion area to the nearest reference point according to the spatial position information of the lesion area, and determine the components of the distance vector in the tangential, normal and binormal directions as the relative position information of the lesion area relative to the reference point;

[0144] Building a three-dimensional visualization model of the lesion area based on the spatial position information and the relative position information, wherein the three-dimensional visualization model includes position information of the lesion area in the world coordinate system and the cylindrical coordinate system;

[0145] Designing a color coding scheme for the lesion area in the three-dimensional visualization model, dividing the lesion area into different levels according to the severity of the lesion and marking them with different color systems respectively, setting different tones in each color system according to the specific type of the lesion, using color transparency to represent the depth information of the lesion area, and applying the color coding scheme to the three-dimensional visualization model;

[0146] The image sequences of the lesion area at different time points are collected, the image sequences are registered, the registered image sequences are mapped to the three-dimensional visualization model, and the change data of the three-dimensional coordinate position information of the lesion area relative to the reference point over time is recorded.

[0147] A method for constructing a three-dimensional intestinal lesion visualization model can intuitively display the type, location, severity and changes of lesions in the intestine over time.

[0148] First, intestinal medical imaging data, such as CT or MRI data, is collected and preprocessed, including denoising, contrast enhancement, and other operations, to improve image quality.

[0149] Then, the preprocessed image is segmented to separate the intestine from the surrounding tissue. The intestinal segmentation can be performed using methods based on region growing or level sets. The segmentation result is a three-dimensional intestinal model.

[0150] Next, extract the intestinal centerline. The intestinal centerline can be extracted using a skeleton-based algorithm to obtain a curve that runs through the intestine. A cylindrical coordinate system is established with the starting end of the intestine as the coordinate origin and the intestinal centerline as the main axis.

[0151] A reference point is set every 1 mm along the centerline of the intestine. The three-dimensional coordinate position information (x, y, z) of each reference point in the world coordinate system (such as the DICOM coordinate system) is collected. These reference points are connected to construct an intestinal skeleton structure containing three-dimensional coordinate position information, for example, the world coordinates of the first reference point are (10, 20, 30), the world coordinates of the second reference point are (11, 21, 31), and so on.

[0152] Based on the constructed intestinal skeleton structure, the lesion area marked manually or by the doctor using the automatic algorithm is located. The three-dimensional coordinates of the center of mass of each lesion area in the world coordinate system are obtained. For example, the world coordinates of the center of mass of a lesion area are (12, 23, 33).

[0153] Convert the world coordinates of the center of mass of the lesion area into radial distance, circumferential angle, and axial distance in the cylindrical coordinate system. For example, the radial distance of the center of mass of the lesion area in the cylindrical coordinate system is 2 mm, the circumferential angle is 45 degrees, and the axial distance is 3 mm. The radial distance refers to the distance from the center of mass of the lesion to the center line of the intestine, the circumferential angle refers to the angle of the center of mass of the lesion on the cross section of the intestine, and the axial distance refers to the distance of the center of mass of the lesion along the center line of the intestine. These three parameters together constitute the spatial position information of the lesion area.

[0154] Calculate the distance vector from the centroid of the lesion area to the nearest reference point. Decompose the distance vector into three directions: tangent, normal, and binormal, and obtain three component values. The tangent refers to the direction along the centerline of the intestine, the normal refers to the direction perpendicular to the centerline of the intestine and pointing to the outer wall of the intestine, and the binormal refers to the direction perpendicular to the tangent and normal. These three component values ​​determine the relative position information of the lesion area relative to the reference point. For example, the distance vector from the centroid of the lesion to the nearest reference point has 0.5mm, 1mm, and 0.2mm components in the tangent, normal, and binormal directions, respectively.

[0155] A three-dimensional visualization model of the lesion area is constructed based on the spatial position information and the relative position information. The model includes the position information of the lesion area in the world coordinate system and the cylindrical coordinate system.

[0156] A color coding scheme was designed to divide the lesion area into three levels: mild, moderate, and severe according to the severity of the lesion, and marked with green, yellow, and red, respectively. Within each color system, different tones were set according to the specific type of lesion. For example, light green was used for inflammation and dark green for polyps; light yellow was used for adenoma and dark yellow was used for cancer. The color transparency was used to characterize the depth information of the lesion area. For example, the closer the lesion area was to the observer, the more opaque it was, and the farther it was, the more transparent it was. The color coding scheme was applied to the 3D visualization model.

[0157] Collect image sequences of the lesion area at different time points, for example, once every month. Perform registration processing on the image sequence, for example, using a feature point-based registration method. Map the registered image sequence to a three-dimensional visualization model, and record the change data of the three-dimensional coordinate position information of the lesion area relative to the reference point over time. For example, the axial distance of a lesion area is 3mm in the first month, and the axial distance becomes 4mm in the second month.

[0158] The solution of this application can:

[0159] Improve diagnostic accuracy: Through color coding and 3D visualization, the location, size, depth and severity of different types of lesions can be clearly displayed, which helps doctors to more accurately judge the nature of the lesions and reduce misdiagnosis and missed diagnoses. Facilitate disease tracking: By recording the change data of the lesion area over time, the development trend of the lesion can be intuitively observed, providing a basis for doctors to formulate treatment plans and evaluate the treatment effect. Improve patient understanding: The 3D visualization model can present complex medical information to patients in an intuitive and easy-to-understand way, helping patients better understand their condition and actively cooperate with treatment.

[0160] Figure 2 FIG. 1 is a schematic diagram of the structure of an AI analysis system based on a straight-lens mirror image according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0161] The first unit is used to input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle-level semantic features, and deep-level abstract features; an adaptive feature weight allocation module is used to fuse the multi-scale feature map, and the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map to generate a fused feature map;

[0162] The second unit is used to input the fused feature map into a dual-branch detection network, which includes a region proposal branch and a classification regression branch; the region proposal branch generates candidate lesion regions by dense sampling, and the classification regression branch accurately locates and classifies the candidate lesion regions; a non-maximum suppression algorithm is used to screen overlapping candidate frames to obtain the final lesion region detection result; deep learning classification is performed on the detected lesion region to identify the lesion type;

[0163] The third unit is used to model the inner wall of the intestine based on a continuous rectoscopic image sequence and a three-dimensional reconstruction algorithm based on depth estimation; map the lesion area detection result to the three-dimensional reconstruction model, and use a spatial registration algorithm to accurately locate the lesion area; construct a three-dimensional visualization model containing the spatial position information of the lesion area, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

[0164] According to a third aspect of the embodiments of the present invention,

[0165] An electronic device is provided, comprising:

[0166] processor;

[0167] a memory for storing processor-executable instructions;

[0168] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0169] A fourth aspect of the embodiments of the present invention is:

[0170] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0171] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The AI ​​analysis method based on the straight mirror image is characterized by: include: Input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle semantic features, and deep abstract features; use an adaptive feature weight allocation module to fuse the multi-scale feature map, the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map, and generate a fused feature map; Inputting the fused feature map into a dual-branch detection network, the dual-branch detection network comprising a region proposal branch and a classification regression branch; The region proposal branch generates candidate lesion regions through dense sampling, and the classification and regression branch accurately locates and classifies the candidate lesion regions. The non-maximum suppression algorithm is used to screen overlapping candidate frames to obtain the final lesion region detection results. The detected lesion regions are classified through deep learning to identify the lesion type. Based on a sequence of continuous rectoscopic images, a three-dimensional reconstruction algorithm based on depth estimation is used to model the inner wall of the intestine; the detection results of the lesion area are mapped to the three-dimensional reconstruction model, and the lesion area is accurately located using a spatial registration algorithm; a three-dimensional visualization model containing the spatial position information of the lesion area is constructed, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

2. The method according to claim 1, characterized in that The multi-scale feature map is fused by using an adaptive feature weight allocation module, the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture the key area information in the feature map, and the generated fused feature map includes: The channel attention unit is used to perform global information compression on the multi-scale feature map through global average pooling to obtain a channel description vector, and the channel description vector is input into the dimension reduction fully connected layer for feature dimension reduction compression, and then the dimension reduction compressed features are restored to the original channel dimension through the dimension increase fully connected layer to generate a channel weight coefficient; the spatial attention unit is used to perform maximum pooling and average pooling operations on the multi-scale feature map to obtain spatial feature information, and the spatial feature information is processed through a convolution layer after splicing in the channel dimension to generate a spatial attention weight map; The channel weight coefficient is weightedly fused with the multi-scale feature map to obtain channel weighted features, and the spatial attention weight map is weightedly fused with the channel weighted features to obtain spatial weighted features; the spatial weighted features are subjected to feature fusion processing to obtain fused features, and a residual connection mechanism is introduced to weightedly superimpose the original information of the multi-scale feature map with the fused features to generate a first fused feature map.

3. The method according to claim 1, characterized in that Inputting the fused feature map into a dual-branch detection network, the dual-branch detection network comprising a region proposal branch and a classification regression branch; The region proposal branch generates candidate lesion regions through dense sampling, and the classification and regression branch accurately locates and classifies the candidate lesion regions. The non-maximum suppression algorithm is used to screen the overlapping candidate boxes, and the final lesion area detection results are obtained, including: The region proposal branch sets anchor points on the fused feature map, generates a reference frame with a preset scale and aspect ratio for each anchor point to construct an initial candidate frame set; uses a 3×3 convolution layer to extract local features of the initial candidate frame set, and calculates the foreground-background binary classification score of each candidate frame through a 1×1 convolution layer to generate the candidate lesion region; The classification regression branch uses the RoI Align layer to extract features from the candidate lesion regions, uniformly maps the features of the candidate lesion regions of different sizes into a 7×7 fixed-size feature map, and inputs the fixed-size feature map into two fully connected layers to respectively output lesion category prediction results and position regression parameters, wherein the position regression parameters include the center coordinate offset and width and height scaling of the candidate box relative to the real box; The candidate frames are sorted in descending order according to the classification scores, the candidate frame with the highest score is selected as the reference frame, the intersection and union ratio of other candidate frames and the reference frame is calculated, and the candidate frames with an intersection and union ratio greater than a set threshold are deleted. The process is repeated until all candidate frames are processed to obtain the final lesion area detection result.

4. The method according to claim 1, characterized in that: The detected lesion areas are classified by deep learning, and the lesion types are identified, including: Preprocessing the lesion area detection result, uniformly scaling the lesion area detection result to 224×224 pixels using a bilinear interpolation algorithm, performing random horizontal flipping, random vertical flipping, random rotation, and random brightness contrast adjustment on the lesion area detection result to obtain an enhanced lesion area detection result; The enhanced lesion area detection result is input into a multi-scale feature fusion module, wherein the multi-scale feature fusion module comprises three parallel feature extraction branches, each of which uses convolution layers with three different convolution kernels to extract features, and the convolution layer in each feature extraction branch is followed by a batch normalization layer and a ReLU activation function to obtain feature maps of three different scales; Performing channel attention calculation on the feature maps of the three different scales to obtain corresponding attention weights, and performing element-wise summation on the weighted feature maps to obtain a second fused feature map; inputting the fused feature map into a pyramid pooling module, which includes average pooling operations of four different scales, and encodes spatial information of different scales into feature vectors; The feature vector is input into a multilayer perceptron comprising two fully connected layers, and the multilayer perceptron is trained using a focal loss function, which focuses on difficult samples by dynamically adjusting sample weights. The model is optimized using an Adam optimizer with weight decay, and the initial learning rate is set to 0.

001. The learning rate is decayed to the original 0.1 every 30 training cycles until the model converges to obtain a lesion classification model; the lesion type is determined according to the category probability distribution output by the lesion classification model.

5. The method according to claim 1, characterized in that Based on the continuous rectoscopic image sequence, the intestinal wall is modeled by a three-dimensional reconstruction algorithm based on depth estimation; the lesion area detection result is mapped to the three-dimensional reconstruction model, and the lesion area is accurately located by a spatial registration algorithm, including: Preprocessing the continuous rectilinear image sequence, calculating the optical flow field between adjacent frames, eliminating frames with violent motion according to the average amplitude and directional consistency of the optical flow field, and performing histogram equalization enhancement on the continuous rectilinear image sequence to obtain a preprocessed image sequence; Input the preprocessed image sequence into a two-stream depth estimation network, wherein the two-stream depth estimation network includes a spatial feature extraction stream and a temporal feature extraction stream, wherein the spatial feature extraction stream uses a ResNet-50 with a hole convolution as an encoder to extract multi-scale features, and restores spatial resolution through a skip connection and a feature pyramid network as a decoder, and the temporal feature extraction stream uses a three-dimensional convolutional network to capture the temporal dependency between adjacent frames; A multi-constraint loss function is constructed by using a depth smoothness constraint, a geometric consistency constraint, and a photometric consistency constraint, the dual-stream depth estimation network is trained to obtain a depth estimation model, and the preprocessed image sequence is input into the depth estimation model to obtain a depth map; Determine the relative pose between adjacent frames by matching feature points, restore the camera motion trajectory in combination with the depth map, convert the depth map into a three-dimensional point cloud, generate a three-dimensional mesh model using a Poisson surface reconstruction algorithm, and optimize the three-dimensional mesh model based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine; The lesion area detection result is mapped to the intestinal 3D reconstruction model through back projection transformation to obtain the initial spatial position of the lesion area, the Gaussian curvature and the mean curvature of the lesion area are extracted to construct a local feature descriptor, the reprojection error of the lesion area under multiple viewing angles is calculated, and the initial spatial position of the lesion area is optimized based on the matching result of the local feature descriptor and the reprojection error to obtain the precise spatial position of the lesion area.

6. The method according to claim 5, characterized in that The relative pose between adjacent frames is determined by feature point matching, the camera motion trajectory is restored in combination with the depth map, the depth map is converted into a three-dimensional point cloud, a three-dimensional mesh model is generated by a Poisson surface reconstruction algorithm, and the three-dimensional mesh model is optimized based on the consistency of the surface normal vector to obtain a three-dimensional reconstruction model of the intestine, including: Adaptively preprocessing adjacent frame images in an endoscopic image sequence by using histogram equalization to obtain a preprocessed image, constructing a Hessian matrix of the preprocessed image, using a fast box filter to approximate the second-order derivative of Gaussian to calculate feature point response values, extracting feature points based on the feature point response values ​​and calculating feature descriptors, and generating feature information of the adjacent frame images; Based on the feature information, a nearest neighbor matching algorithm is used to obtain an initial matching pair of feature points, and a RANSAC algorithm is used to eliminate mismatched point pairs in the initial matching pair of feature points to obtain an optimized matching pair of feature points, and an initial relative pose between adjacent frame images is calculated according to the optimized matching pair of feature points; A reprojection error minimization objective function is constructed based on the initial relative posture, and the initial relative posture is iteratively optimized using the LM algorithm to obtain the accurate relative posture between the adjacent frame images and the corresponding reprojection error; According to the precise relative pose, the depth map is converted into a three-dimensional point cloud in the camera coordinate system, the three-dimensional point cloud is transformed into the world coordinate system to obtain an initial point cloud, a bag-of-words model is used to detect loop frames in the initial point cloud and extract loop constraints, the loop constraints are added to the pose graph optimization framework, and the precise relative pose is optimized to obtain a complete motion trajectory of the camera; Based on the complete motion trajectory of the camera, each frame depth map is uniformly converted into a complete three-dimensional point cloud in a world coordinate system, the complete three-dimensional point cloud is statistically filtered to remove noise points to obtain a filtered point cloud, the filtered point cloud is downsampled using a voxel grid to obtain a downsampled point cloud, and the normal vector information of each point in the downsampled point cloud is calculated to obtain point cloud data with a normal vector; An adaptive octree is used to spatially divide the point cloud data with normal vectors, a Poisson equation is constructed and solved in each grid of the adaptive octree to obtain an indicator function, and a zero isosurface of the indicator function is extracted to generate an initial three-dimensional grid model; The normal vector distribution of the vertices of the initial three-dimensional mesh model is calculated, and a normal vector smoothing constraint is constructed based on the consistency of the normal vectors of adjacent vertices. The normal vector smoothing constraint is introduced into the Laplace smoothing operator, and the initial three-dimensional mesh model is iteratively optimized to obtain a final three-dimensional reconstructed model of the intestine that maintains both geometric details and smoothness.

7. The method according to claim 1, characterized in that A three-dimensional visualization model containing the spatial location information of the lesion area is constructed. The three-dimensional visualization model marks different types of lesion areas by color coding and records the three-dimensional coordinate location information of the lesion area relative to the intestinal reference point, including: A cylindrical coordinate system with the intestinal centerline as the main axis is established with the starting end of the intestine as the coordinate origin, reference points are set at preset distances along the intestinal centerline in the cylindrical coordinate system, three-dimensional coordinate position information of the reference points in the world coordinate system is collected, and an intestinal skeleton structure containing the three-dimensional coordinate position information is constructed; Locating the lesion area based on the intestinal skeleton structure, obtaining the three-dimensional coordinates of the centroid of the lesion area in the world coordinate system, converting the three-dimensional coordinates into the cylindrical coordinate system to obtain the radial distance, circumferential angle and axial distance of the centroid of the lesion area, and generating the spatial position information of the lesion area; Calculate the distance vector from the centroid of the lesion area to the nearest reference point according to the spatial position information of the lesion area, and determine the components of the distance vector in the tangential, normal and binormal directions as the relative position information of the lesion area relative to the reference point; Building a three-dimensional visualization model of the lesion area based on the spatial position information and the relative position information, wherein the three-dimensional visualization model includes position information of the lesion area in the world coordinate system and the cylindrical coordinate system; Designing a color coding scheme for the lesion area in the three-dimensional visualization model, dividing the lesion area into different levels according to the severity of the lesion and marking them with different color systems respectively, setting different tones in each color system according to the specific type of the lesion, using color transparency to represent the depth information of the lesion area, and applying the color coding scheme to the three-dimensional visualization model; The image sequences of the lesion area at different time points are collected, the image sequences are registered, the registered image sequences are mapped to the three-dimensional visualization model, and the change data of the three-dimensional coordinate position information of the lesion area relative to the reference point over time is recorded.

8. An AI analysis system based on a rectilinear mirror image, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to input the collected Z-mirror image into a pre-trained deep convolutional neural network, perform feature extraction through multiple different levels of the deep convolutional neural network, and obtain a multi-scale feature map including shallow texture features, middle-level semantic features, and deep-level abstract features; an adaptive feature weight allocation module is used to fuse the multi-scale feature map, and the adaptive feature weight allocation module includes a channel attention unit and a spatial attention unit, wherein the channel attention unit is used to learn the importance weights of different feature channels, and the spatial attention unit is used to capture key area information in the feature map to generate a fused feature map; A second unit is used to input the fused feature map into a dual-branch detection network, where the dual-branch detection network includes a region proposal branch and a classification regression branch; The region proposal branch generates candidate lesion regions through dense sampling, and the classification and regression branch accurately locates and classifies the candidate lesion regions. The non-maximum suppression algorithm is used to screen overlapping candidate frames to obtain the final lesion region detection results. The detected lesion regions are classified through deep learning to identify the lesion type. The third unit is used to model the inner wall of the intestine based on a continuous rectoscopic image sequence and a three-dimensional reconstruction algorithm based on depth estimation; map the lesion area detection result to the three-dimensional reconstruction model, and use a spatial registration algorithm to accurately locate the lesion area; construct a three-dimensional visualization model containing the spatial position information of the lesion area, and the three-dimensional visualization model marks different types of lesion areas by color coding, and records the three-dimensional coordinate position information of the lesion area relative to the intestinal reference point.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images