Micro intelligent terminal intestinal tract image processing method, processing device, storage medium and computer equipment

Abstract

The application discloses a kind of micro intelligent terminal intestinal tract image processing method, processing device, storage medium and computer equipment.The method comprises: the original intestinal tract image set collected to monocular endoscope is preprocessed, and motion calibration scale factor and texture calibration scale factor are calculated according to preprocessed intestinal tract image set, and conversion coefficient from pixel to actual size is adaptively fused;Polypus area key feature points of each image of preprocessed intestinal tract image set are extracted, and Delaunay triangulation is constructed according to each polypus area key feature point, and non-rigid deformation matching is carried out based on cross-map convolution network and optimal transport algorithm, and matching feature point set is obtained;According to matching feature point set, sparse 3D reconstruction is carried out, and polypus point cloud is obtained, and polypus size is calculated according to the Euclidean distance of the two points farthest apart in polypus point cloud and conversion coefficient.The method can overcome monocular image scale ambiguity problem and non-rigid deformation problem, and obtain more accurate polypus size.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method, processing device, storage medium, and computer equipment for processing intestinal images of a miniature intelligent terminal. Background Technology

[0002] A capsule endoscope is a medical device that integrates core functions such as image acquisition and wireless transmission into a capsule that can be swallowed by the human body. During the examination, the capsule endoscope is swallowed, and it acquires images of the digestive tract inside the body and transmits them to the outside of the body simultaneously, so as to carry out medical examinations based on the acquired image data.

[0003] In the endoscopic diagnosis and treatment of colon polyps, accurate size measurement and anatomical localization are crucial for surgical success. However, existing technologies have the following limitations: Strong hardware dependence: Traditional methods often rely on binocular endoscopes, laser rangefinders, or external sensors (such as IMUs) to provide depth information, resulting in high equipment costs and complex operation, making them difficult to adapt to portable devices such as capsule endoscopes. If a monocular endoscope is used, it can lead to scale ambiguity in the monocular images: Single-frame monocular images lack depth information, making it impossible to directly measure the true size. Existing methods often rely on manual labeling or anatomical prior assumptions (such as the average diameter of the intestine), resulting in significant errors. Simultaneously, there is interference from non-rigid deformation: Intestinal peristalsis causes tissue deformation, and traditional feature matching algorithms have low matching accuracy in non-rigid scenarios, affecting the accuracy of multi-frame reconstruction and localization. Summary of the Invention

[0004] The technical problem solved by this invention is: how to overcome the problem of size ambiguity in monocular images and the problem of low matching accuracy caused by non-rigid deformation of intestinal tissue, so as to improve the accuracy of size estimation.

[0005] This application discloses a method for processing intestinal images using a miniature smart terminal, the method comprising:

[0006] The raw intestinal image set acquired by monocular endoscopy is preprocessed to obtain a preprocessed intestinal image set. The preprocessing includes image brightness uniformization processing and image equalization processing.

[0007] Based on the preprocessed intestinal image set, a motion calibration scaling factor and a texture calibration scaling factor are calculated. The motion calibration scaling factor and the texture calibration scaling factor are then incorporated into an adaptive fusion process to obtain the conversion coefficient from pixel to actual size.

[0008] Key feature points of polyp regions in each image of the preprocessed intestinal image set are extracted, and Delaunay triangulation maps are constructed based on the key feature points of each polyp region.

[0009] Based on the Delaunay triangulation diagram, non-rigid deformation matching is performed using a cross-graph convolutional network and the optimal transport algorithm to obtain a set of matching feature points;

[0010] Sparse 3D reconstruction is performed based on the set of matching feature points to obtain a polyp point cloud. The Euclidean distance between the two farthest points in the polyp point cloud is calculated, and the polyp size is calculated based on the Euclidean distance and the transformation coefficient.

[0011] Optionally, the method for calculating motion calibration scaling factors and texture calibration scaling factors based on the preprocessed intestinal image set, and then incorporating the motion calibration scaling factors and texture calibration scaling factors into adaptive fusion to obtain pixel-to-actual-size conversion coefficients includes:

[0012] The displacement of feature points between adjacent frames of the preprocessed intestinal image set is calculated based on the optical flow algorithm, and the motion calibration scaling factor is calculated based on the intestinal peristalsis model and the displacement of each feature point.

[0013] The pixel distance between adjacent folds in each image of the preprocessed intestinal image set is calculated, and the texture calibration factor is calculated based on the pixel distance and the preset average actual spacing of intestinal folds.

[0014] The variances of the motion calibration scaling factor and the texture calibration factor are calculated, and the weights are calculated based on the variances of the motion calibration scaling factor and the texture calibration factor.

[0015] Adaptive fusion is performed based on the motion calibration scaling factor, texture calibration factor, and weights to obtain the conversion coefficient from pixel to actual size.

[0016] Optionally, the step of extracting key feature points of polyp regions from each image in the preprocessed intestinal image set, and constructing a Delaunay triangulation map based on the key feature points of each polyp region, includes:

[0017] Using the SuperPoint network, key feature points of polyp regions are obtained by filtering candidate feature points of polyp regions from each image of the preprocessed intestinal image set according to preset confidence and non-maximum suppression conditions.

[0018] Delaunay triangulation is performed on the key feature points of the polyp region, and an adjacency matrix and a node feature matrix are constructed based on the results of the Delaunay triangulation.

[0019] Optionally, based on the Delaunay triangulation map, non-rigid deformation matching is performed using a cross-graph convolutional network and an optimal transport algorithm to obtain a set of matching feature points, including:

[0020] The adjacency matrix and the node feature matrix are input into a cross-graph convolutional network. The neighborhood features of each node in the graph are aggregated through the intra-graph convolutional layer, and the node features of the Delaunay triangulation map from another frame image are fused through the cross-graph convolutional layer to generate an enhanced node feature representation.

[0021] Based on the enhanced node feature representation, the similarity matrix of all node pairs between the Delaunay triangulation maps of the two images is calculated, and the similarity matrix is ​​normalized and optimized using the optimal transmission algorithm to obtain a soft assignment matrix representing the matching probability between nodes.

[0022] Based on the soft assignment matrix, node pairs with a matching probability greater than a preset threshold are selected to form the matching feature point set.

[0023] Optionally, sparse 3D reconstruction is performed based on the matching feature point set to obtain a polyp point cloud, including:

[0024] Preliminary triangulation is performed based on the pixel coordinates of each matching feature point in the matching feature point set, the camera parameters of the endoscope, and the moving baseline distance to generate 3D point cloud coordinates.

[0025] The reprojection error and parameter coupling error of the 3D point cloud coordinates obtained by the initial triangulation are optimized to obtain the final polyp point cloud.

[0026] Optionally, the processing method further includes:

[0027] Anatomical landmarks are identified from the preprocessed intestinal image set, and the centroid coordinates of the anatomical landmarks are calculated.

[0028] Once the anatomical landmarks are identified, the cumulative motion displacement of the endoscope is calculated, and the cumulative motion displacement is corrected for depth offset based on the centroid coordinates to generate the reference point of the anatomical landmarks.

[0029] The corrected position of the polyp region is calculated based on the cumulative displacement of the polyp region and the reference point of the anatomical landmark region.

[0030] Optionally, when anatomical landmarks are identified, the cumulative motion displacement of the endoscope is calculated, including:

[0031] The inter-frame movement distance of the endoscope is calculated based on the conversion coefficient and the feature point displacement between adjacent frames of the preprocessed intestinal image set.

[0032] The cumulative motion displacement is obtained by summing the movement distances between each frame.

[0033] This application also discloses a processing device for intestinal images of a miniature smart terminal, the processing device comprising:

[0034] An image preprocessing module is used to preprocess the raw intestinal image set acquired by a monocular endoscope to obtain a preprocessed intestinal image set. The preprocessing includes image brightness uniformization processing and image equalization processing.

[0035] A dynamic scaling module is used to calculate a motion scaling factor and a texture scaling factor based on the preprocessed intestinal image set, and to incorporate the motion scaling factor and the texture scaling factor into adaptive fusion to obtain a conversion coefficient from pixel to actual size.

[0036] The feature extraction module is used to extract key feature points of polyp regions from each image of the preprocessed intestinal image set, and to construct a Delaunay triangulation map based on the key feature points of each polyp region.

[0037] A non-rigid deformation matching module is used to perform non-rigid deformation matching based on the Delaunay triangulation map, using a cross-graph convolutional network and the optimal transfer algorithm, to obtain a set of matching feature points.

[0038] The size estimation module is used to perform sparse 3D reconstruction based on the matching feature point set to obtain a polyp point cloud, calculate the Euclidean distance between the two farthest points in the polyp point cloud, and calculate the polyp size based on the Euclidean distance and the transformation coefficient.

[0039] This application also discloses a computer-readable storage medium storing a processing program for intestinal images of a micro-intelligent terminal. When the processing program for intestinal images of a micro-intelligent terminal is executed by a processor, it implements the above-described processing method for intestinal images of a micro-intelligent terminal.

[0040] This application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a micro-intelligent terminal intestinal image processing program stored in the computer-readable storage medium. When the micro-intelligent terminal intestinal image processing program is executed by the processor, it implements the above-described micro-intelligent terminal intestinal image processing method.

[0041] (III) Beneficial Effects

[0042] The present invention discloses a method, processing device, storage medium, and computer equipment for processing intestinal images using a miniature intelligent terminal, which, compared with the prior art, has the following technical advantages:

[0043] This method can solve the problem of scale ambiguity in monocular images, achieve high-precision matching in non-rigid scenes, and finally obtain more accurate polyp region size and improve polyp localization accuracy. Attached Figure Description

[0044] Figure 1 This is a flowchart of a method for processing intestinal images using a miniature intelligent terminal according to Embodiment 1 of the present invention;

[0045] Figure 2 This is a schematic block diagram of the micro-intelligent terminal intestinal image processing device according to Embodiment 2 of the present invention;

[0046] Figure 3 This is a schematic diagram of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] Before describing the various embodiments of this application in detail, the technical concept of this application is first briefly described: Currently, when measuring real size using monocular images, there are problems such as size ambiguity and non-rigid deformation, leading to inaccurate size measurement. To address this, the key improvement of the micro-intelligent terminal intestinal image processing method provided by this application lies in first calculating the motion calibration scale factor and texture calibration scale factor, and then fusing the two to obtain the pixel-to-actual size conversion coefficient, thus solving the problem of scale ambiguity in monocular images. After extracting the key feature points of the polyp region, Delaunay triangulation is performed, followed by non-rigid deformation matching to achieve high-precision matching in non-rigid scenarios, finally obtaining a more accurate polyp region size.

[0049] Specifically, such as Figure 1 As shown, the method for processing intestinal images using a micro-intelligent terminal in this embodiment includes the following steps:

[0050] Step S10: Preprocess the raw intestinal image set acquired by monocular endoscopy to obtain a preprocessed intestinal image set. The preprocessing includes image brightness uniformization processing and image equalization processing.

[0051] Step S20: Calculate the motion calibration scaling factor and texture calibration scaling factor based on the preprocessed intestinal image set, and then perform adaptive fusion on the motion calibration scaling factor and the texture calibration scaling factor to obtain the conversion coefficient from pixel to actual size.

[0052] Step S30: Extract key feature points of polyp regions from each image of the preprocessed intestinal image set, and construct a Delaunay triangulation map based on the key feature points of each polyp region.

[0053] Step S40: Based on the Delaunay triangulation map, perform non-rigid deformation matching using a cross-graph convolutional network and the optimal transport algorithm to obtain a set of matching feature points;

[0054] Step S50: Perform sparse 3D reconstruction based on the matching feature point set to obtain polyp point cloud, calculate the Euclidean distance between the two farthest points in the polyp point cloud, and calculate the polyp size based on the Euclidean distance and the transformation coefficient.

[0055] For example, a capsule endoscope or a conventional colonoscope can be used to acquire time-series images of the colon, and the acquisition time of each frame can be recorded to form a raw intestinal image set.

[0056] Furthermore, the preprocessing of the image brightness uniformity involves converting the original intestinal image to the HSV color space, performing adaptive gamma correction on the luminance (V) channel, and preserving the hue (H) and saturation (S) channels to avoid color distortion and to prevent overexposure or underexposure. The formula for calculating adaptive gamma correction is as follows:

[0057] ,

[0058] In the formula, The average brightness of the image. This is the corrected brightness.

[0059] Furthermore, the specific details of the preprocessing image equalization are as follows: Using an improved CLAHE algorithm, the image after brightness equalization is divided into 8×8 sub-blocks, and the cropping threshold is dynamically adjusted.

[0060] ,

[0061] Histogram equalization is performed on each sub-block, and inter-block boundary effects are eliminated through bilinear interpolation. Always, This is an adjustment factor, ranging from 0.8 to 1.2, which controls the shear strength. When That's the standard CLAHE. This refers to the total number of pixels in the current image sub-block. For example, an 8×8 sub-block has 64 pixels. This is the number of bars in the grayscale histogram, usually set to 256 levels.

[0062] The method for calculating motion calibration scaling factors and texture calibration scaling factors based on the preprocessed intestinal image set, and then incorporating these factors into adaptive fusion to obtain pixel-to-actual-size conversion coefficients includes:

[0063] First, the feature point displacements between adjacent frames of the preprocessed intestinal image set are calculated based on the optical flow algorithm. Then, the motion calibration scaling factor is calculated based on the intestinal peristalsis model and the displacements of each feature point. The specific details are as follows:

[0064] Based on optical flow velocity inversion, the displacement of feature points between adjacent frames is calculated using an optical flow algorithm (based on the Lucas-Kanade optical flow algorithm). :

[0065] ,

[0066] In the formula, Let be the pixel coordinates of the k-th feature point in the t-th frame image, where the pixel coordinates of the feature point can be extracted by the SuperPoint network. For the first The displacement of a feature point between adjacent frames.

[0067] Furthermore, combining the intestinal peristalsis model (default speed) ), calculate the motion calibration scaling factor:

[0068] ,

[0069] In the formula, The time difference between adjacent frames. The number of feature points to be used in the calculation (by default, the first 100 high-confidence feature points are selected).

[0070] Next, the pixel distance between adjacent folds in each image of the preprocessed intestinal image set is calculated. Based on the pixel distance and the preset average actual spacing of intestinal folds, a texture calibration factor is calculated, as follows:

[0071] Statistical analysis of pixel distances between adjacent folds in an image (Default wrinkle spacing 10mm), calculate texture calibration scaling factor:

[0072] ,

[0073] In the formula, This represents the pixel distance between adjacent folds in the image, which can be calculated automatically after extracting the fold midline using Canny edge detection and Hough transform. Based on statistical values ​​from anatomical literature, the average actual spacing between intestinal folds is 10 mm.

[0074] Then, the variances of the motion calibration scaling factor and the texture calibration factor are calculated, and the weights are calculated based on these variances, as detailed below:

[0075] For N consecutive frames of images, calculate a temporary motion scaling factor for each frame. Then, the dispersion of the sequence is statistically analyzed to obtain the variance of the motion calibration results. This reflects the stability of the optical flow displacement:

[0076] ,

[0077] In the formula, It represents the arithmetic mean of the sequence of motion calibration scale factors for N consecutive frames.

[0078] Calculate the temporary scaling factor sequence in the same frame image. Then, the variance is calculated and used as the variance of the texture calibration result, reflecting the consistency of wrinkle spacing detection.

[0079] ,

[0080] In the formula, It represents the arithmetic mean of the texture calibration scaling factor sequence calculated from the N wrinkle spacings.

[0081] Calculate the weights :

[0082] .

[0083] Finally, adaptive fusion is performed based on the motion calibration scaling factor, texture calibration factor, and weights to obtain the pixel-to-actual-size conversion coefficients:

[0084] ,

[0085] In the formula, It represents the conversion coefficient from pixel to actual size, and can also be called the dynamic scaling factor.

[0086] In one or more embodiments, key feature points of polyp regions are extracted from each image of the preprocessed intestinal image set, and a Delaunay triangulation map is constructed based on the key feature points of each polyp region, including:

[0087] First, the SuperPoint network is used to select candidate feature points for polyp regions from each image in the preprocessed intestinal image set. Based on pre-set confidence and non-maximum suppression conditions, key feature points for polyp regions are obtained from these candidate feature points. Specifically, the SuperPoint network is used to extract key points of polyp regions, extract high-confidence feature points, and perform non-maximum suppression (NMS) to optimize the feature point distribution. The formula for calculating the confidence of SuperPoint feature points is as follows:

[0088] ,

[0089] In the formula, The input image is a preprocessed intestinal image with a size of [size missing]. . This represents the convolutional encoder of the SuperPoint network, with an output feature map size of [size missing]. (Classification scores for 65 channels corresponding to an 8×8 grid). This represents the Sigmoid activation function, which maps the output to the interval [0, 1]. Indicates position The confidence score of a feature point indicates that the higher the value, the more likely it is to be a key point.

[0090] The non-maximum suppression (NMS) condition is: if 0.7 and If the feature points are positive, retain them; otherwise, delete them.

[0091] Next, Delaunay triangulation is performed on the key feature points of the polyp region. Based on the results of the Delaunay triangulation, the adjacency matrix and node feature matrix are constructed. Specifically:

[0092] Delaunay triangulation:

[0093] ,

[0094] In the formula, Indicates the first The pixel coordinates of each feature point (from SuperPoint output). Indicates the first A triangle is formed by three characteristic vertices. Indicates the total number of feature points. This represents the number of triangles generated, approximately It satisfies the Delaunay empty circle property.

[0095] Adjacency matrix construction:

[0096] ,

[0097] Node feature matrix initialization:

[0098] ,

[0099] In the formula, Indicates the feature dimension, default value. This includes coordinates and SuperPoint descriptors. This represents the coordinates of the feature point, normalized to the interval [0, 1]. This indicates that the 128-dimensional descriptor extracted by the SuperPoint network has been reduced to 126 dimensions to fit the coordinate information.

[0100] In one or more embodiments, based on the Delaunay triangulation map, non-rigid deformation matching is performed using a cross-graph convolutional network and an optimal transfer algorithm to obtain a matching feature point set. This includes the following steps: inputting the adjacency matrix and node feature matrix into the cross-graph convolutional network; aggregating the neighborhood features of each node in the graph through an intra-graph convolutional layer; and fusing the node features from the Delaunay triangulation map of another frame image through a cross-graph convolutional layer to generate an enhanced node feature representation; based on the enhanced node feature representation, calculating the similarity matrix of all node pairs between the Delaunay triangulation maps of the two images; and normalizing and optimizing the similarity matrix using the optimal transfer algorithm to obtain a soft assignment matrix representing the matching probability between nodes; and selecting node pairs with a matching probability greater than a preset threshold based on the soft assignment matrix to form the matching feature point set.

[0101] First, in-graph convolution:

[0102] ,

[0103] It is an adjacency matrix, representing the node connection relationships (from Delaunay triangulation).

[0104] This is the node feature matrix of the l-th layer (initialized as coordinates + SuperPoint descriptor).

[0105] It is a trainable weight matrix used to aggregate neighborhood features and retain its own features.

[0106] ReLU activation function enhances nonlinear expressive power.

[0107] Next, perform cross-graph convolution:

[0108] ,

[0109] The cross-graph similarity matrix is ​​calculated using the optimal transmission algorithm. For the node feature matrix of another frame image, These are the trainable weights for cross-graph convolution.

[0110] Perform optimal transmission matching:

[0111] Then, calculate the node similarity:

[0112] ,

[0113] The figures are respectively and picture The node feature vectors. The temperature coefficient controls the smoothness of the similarity distribution (experimental optimized value, range 0.01~1.0). Figure G1 shows the graph structure of the reference frame (e.g., the image at time t), for image I... t The key feature points of the polyp region are obtained by triangulation. Figure G2 shows the graph structure of the target frame (such as the image at time t+1), for image I. t +1 The key feature points of the polyp region were obtained by triangulation.

[0114] Finally, Sinkhorn normalization is performed:

[0115] ,

[0116] is the soft assignment matrix, representing the matching probability between nodes. ⊙ represents element-wise multiplication, updating the soft assignment matrix in conjunction with the similarity matrix. Therefore, after the above non-rigid deformation matching steps, the final output is a high-precision set of matching feature points that can resist intestinal peristalsis.

[0117] In one or more embodiments, sparse 3D reconstruction is performed based on the matching feature point set to obtain a polyp point cloud, specifically including the following:

[0118] First, preliminary triangulation is performed based on the pixel coordinates of each matching feature point in the matching feature point set, the camera parameters of the endoscope, and the moving baseline distance to generate 3D point cloud coordinates.

[0119] Triangulated 3D point cloud coordinates:

[0120] ,

[0121] For the first The pixel coordinates of each feature point. The distance of the capsule endoscope's movement baseline is calculated through trajectory accumulation. This refers to the camera's focal length. For feature point disparity, the horizontal displacement between adjacent frames. .

[0122] Next, the reprojection error and parameter coupling error of the 3D point cloud coordinates obtained from the initial triangulation are optimized to obtain the final polyp point cloud.

[0123] Bundle adjustment plays a crucial role in improving accuracy and suppressing errors in size estimation. After initial triangulation to generate a 3D point cloud, two types of errors exist:

[0124] (1) Reprojection error: The position of the 3D point cloud projected back to the image plane Compared with actual observation points Mismatch.

[0125] (2) Parameter coupling error: camera pose With 3D point cloud coordinates The estimates influence each other.

[0126] The optimized solution is obtained by solving the nonlinear least squares problem. )and :

[0127] ,

[0128] In the formula, Let be the camera rotation matrix and translation vector. Given 3D point cloud coordinates, the optimization objective is to minimize the reprojection error.

[0129] 1. The optimization method for reprojection error is as follows:

[0130] Error definition: Reprojection error is the distance between the "actually seen pixel position" and the "pixel position calculated based on the current 3D point and camera pose".

[0131] Optimization process: The cost function directly aims to minimize the sum of squares of all reprojection errors. Optimization algorithms (such as Levenberg-Marquardt) iteratively fine-tune this process. , and If the calculated point is to the right of the actual observation point, the algorithm will try to slightly move the camera to the left or slightly adjust the 3D point in the opposite direction. Through countless such adjustments, the algorithm eventually ensures that the reprojection point of each 3D point in all visible frames coincides as closely as possible with the actual image feature point.

[0132] 2. The optimization method for parameter coupling error is as follows:

[0133] The nature of the error: inaccurate camera pose This can lead to incorrect 3D points. However, incorrect 3D points can lead to inaccurate calculated camera poses.

[0134] Optimization process: The power of BA lies in "joint optimization." Unlike alternating optimization, which fixes one camera position before optimizing another, BA works by adjusting the poses of all cameras together. The coordinates of all 3D points {Xi} are considered as variables to be optimized and solved within the same problem. During optimization, the algorithm can recognize that adjusting the pose of one camera will affect the reprojection error of multiple 3D points, and vice versa. By solving the Jacobian matrix of this large-scale system, the algorithm can find a set of "globally optimal" or "locally optimal" camera poses and 3D point coordinates that minimize the overall reprojection error. This is equivalent to automatically untying the "deadlock" state of interdependence between parameters.

[0135] Finally, the polyp size was calculated based on the Euclidean distance between the two farthest points in the polyp point cloud and the transformation coefficient. :

[0136] ,

[0137] In the formula, This represents the Euclidean distance between two points in a 3D point cloud, expressed in pixels. The table shows the transformation coefficients, also known as dynamic scaling factors. The coordinates of the 3D point cloud obtained from sparse reconstruction. The unit is pixels, so it needs to be multiplied by . Convert physical dimensions to resolve scale ambiguity in monocular vision.

[0138] In one or more embodiments, the processing method further includes step S60: identifying anatomical landmark regions from a preprocessed intestinal image set and calculating the centroid coordinates of the anatomical landmark regions; when the anatomical landmark regions are identified, calculating the cumulative motion displacement of the endoscope, and performing depth offset correction on the cumulative motion displacement based on the centroid coordinates to generate a reference point for the anatomical landmark regions; and calculating the corrected position of the polyp regions based on the cumulative displacement of the polyp regions and the reference point of the anatomical landmark regions.

[0139] For example, key anatomical landmarks of the colon (ileocecal valve, hepatic flexure, splenic flexure, and sigmoid junction) are detected in real time using a deep learning classifier to provide reference points for motion trajectory correction.

[0140] The output of the multi-label classifier is:

[0141] ,

[0142] In the formula, The input image is a preprocessed intestinal image. , which are four-dimensional probability vectors, corresponding to the detection probabilities of the ileocecal valve, hepatic flexure, splenic flexure, and sigmoid colon junction, respectively. This is a residual network pre-trained on ImageNet, fine-tuned to adapt to anatomical landmark classification tasks.

[0143] Furthermore, the criteria for determining the presence of anatomical landmarks: when Anatomical landmarks exist.

[0144] Calculate the centroid coordinates of anatomical landmarks:

[0145] ,

[0146] In the formula, Anatomical landmarks The Each pixel coordinate. This represents the total number of pixels within the area containing the anatomical landmark. The geometric center pixel coordinates of the anatomical landmark are used for spatial anchoring of the anatomical landmark and serve as the input reference for motion trajectory correction, mapping the image coordinates to the cumulative displacement space.

[0147] Once anatomical landmarks are identified, the cumulative motion displacement of the endoscope is calculated, including: calculating the inter-frame movement distance of the endoscope based on the conversion coefficients and the feature point displacements between adjacent frames of the preprocessed intestinal image set; and summing the inter-frame movement distances to obtain the cumulative motion displacement. The specific calculation formula is as follows.

[0148] The inter-frame movement distance is: .

[0149] In the formula, This represents the distance the endoscope travels in real space, i.e., the inter-frame travel distance (unit: mm). For the first Inter-frame optical flow displacement, i.e., the distance a feature point moves in the image plane (unit: pixels). is the conversion factor.

[0150] Cumulative motion displacement:

[0151] ,

[0152] : No. The cumulative actual displacement of the time capsule (unit: mm).

[0153] The actual displacement of the endoscope in the intestine when anatomical landmarks are detected. (Calculated through cumulative optical flow trajectory) Depth offset correction needs to be performed in conjunction with centroid coordinates to generate an accurate reference point. If anatomical landmarks are off-center from the image (usually due to tilted capsule lens view or intestinal curvature), depth compensation must be calculated using centroid coordinates.

[0154] ,

[0155] Anatomical landmarks The cumulative displacement reference point. To detect anatomical landmarks The timestamp. The offset is triggered by the centroid coordinates:

[0156] ,

[0157] These are the coordinates of the image center. Focal length (in pixels) for the capsule camera. The depth-parallax ratio is 0.8 (determined through calibration experiments). is the conversion factor.

[0158] The corrected location for the polyp area is:

[0159] ,

[0160] This represents the cumulative displacement of the polyp region.

[0161] This represents the total displacement accumulated by the capsule from its starting point at the moment when the capsule endoscope lens is directly facing and capturing an image of the polyp. The calculation method is completely consistent with the principle of generating the overall motion trajectory of the capsule described earlier, except that the endpoint of the calculation is different. The calculation follows the following formula chain:

[0162] (1) Calculate the actual inter-frame displacement:

[0163] ,

[0164] The pixel displacement of the feature point from frame k to frame (k+1) is calculated using the optical flow method.

[0165] : Conversion coefficient (mm / pixel) obtained from dynamic scaling calibration.

[0166] (2) Perform displacement accumulation

[0167] = ,

[0168] : The frame number of the frame in which the polyp was detected. This summation process accumulates from the start of the journey (i=1) until the moment the polyp is detected (i = ...). ).

[0169] , indicating the current time With reference point The time difference. This is the correction factor for intestinal peristalsis. is the coefficient of variation of the trajectory velocity, reflecting the stability of the motion. Wherein,

[0170] ,

[0171] In the formula, It is the average trajectory velocity, that is, the arithmetic mean of the movement velocity of the capsule endoscope over a period of time. It is the standard deviation of the trajectory velocity, used to measure the degree of fluctuation in velocity values.

[0172] Furthermore, such as Figure 2 As shown in Embodiment 2, a micro-intelligent terminal intestinal image processing device is disclosed. The processing device includes an image preprocessing module 100, a dynamic scaling module 200, a feature extraction module 300, a non-rigid deformation matching module 400, a size estimation module 500, and a positioning correction module 600. The image preprocessing module 100 preprocesses the raw intestinal image set acquired by a monocular endoscope to obtain a preprocessed intestinal image set. The preprocessing includes image brightness homogenization and image equalization. The dynamic scaling module 200 calculates motion scaling factors and texture scaling factors based on the preprocessed intestinal image set, and then adaptively fuses these factors to obtain pixel-to-actual-size conversion coefficients. The feature extraction module 300 extracts key feature points of polyp regions from each image in the preprocessed intestinal image set and constructs a Delaunay triangulation map based on these key feature points. The non-rigid deformation matching module performs non-rigid deformation matching based on the Delaunay triangulation map and a cross-graph convolutional network with an optimal transport algorithm to obtain a set of matching feature points. The size estimation module 500 performs sparse 3D reconstruction based on the matching feature point set to obtain a polyp point cloud, calculates the Euclidean distance between the two farthest points in the polyp point cloud, and calculates the polyp size based on the Euclidean distance and transformation coefficients. The localization correction module 600 identifies anatomical landmarks from the preprocessed intestinal image set and calculates the centroid coordinates of the anatomical landmarks. When an anatomical landmark is identified, it calculates the cumulative motion displacement of the endoscope, performs depth offset correction on the cumulative motion displacement based on the centroid coordinates, generates a reference point for the anatomical landmark, and calculates the corrected position of the polyp region based on the cumulative displacement of the polyp region and the reference point of the anatomical landmark region.

[0173] For a more detailed description of the working process of each module of the micro-intelligent terminal intestinal image processing device, please refer to the relevant description of the above method embodiments, which will not be repeated here.

[0174] Furthermore, this embodiment three also discloses a computer-readable storage medium storing a processing program for intestinal images of a micro-intelligent terminal. When the processing program for intestinal images of a micro-intelligent terminal is executed by a processor, it implements the processing method for intestinal images of a micro-intelligent terminal of embodiment one.

[0175] This fourth embodiment also discloses a computer device, at the hardware level, such as... Figure 3 As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. The computer-readable storage medium 11 stores a processing program for the intestinal image of a micro-intelligent terminal. When the processor executes the processing program for the intestinal image of the micro-intelligent terminal, it implements the above-described processing method for the intestinal image of the micro-intelligent terminal.

[0176] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0177] The specific embodiments of the present invention have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that modifications and improvements can be made to these embodiments without departing from the principles and spirit of the present invention as defined by the claims and their equivalents, and such modifications and improvements should also be within the protection scope of the present invention.

Claims

1. A method for processing intestinal images using a miniature intelligent terminal, characterized in that, The processing method includes: The raw intestinal image set acquired by monocular endoscopy is preprocessed to obtain a preprocessed intestinal image set. The preprocessing includes image brightness uniformization processing and image equalization processing. Based on the preprocessed intestinal image set, a motion calibration scaling factor and a texture calibration scaling factor are calculated. The motion calibration scaling factor and the texture calibration scaling factor are then incorporated into an adaptive fusion process to obtain the conversion coefficient from pixel to actual size. Key feature points of polyp regions in each image of the preprocessed intestinal image set are extracted, and Delaunay triangulation maps are constructed based on the key feature points of each polyp region. Based on the Delaunay triangulation diagram, non-rigid deformation matching is performed using a cross-graph convolutional network and the optimal transport algorithm to obtain a set of matching feature points; Sparse 3D reconstruction is performed based on the matching feature point set to obtain a polyp point cloud. The Euclidean distance between the two farthest points in the polyp point cloud is calculated, and the polyp size is calculated based on the Euclidean distance and the transformation coefficient. The method for calculating motion calibration scaling factors and texture calibration scaling factors based on the preprocessed intestinal image set, and then incorporating these factors into adaptive fusion to obtain pixel-to-actual-size conversion coefficients includes: The displacement of feature points between adjacent frames of the preprocessed intestinal image set is calculated based on the optical flow algorithm, and the motion calibration scaling factor is calculated based on the intestinal peristalsis model and the displacement of each feature point. The pixel distance between adjacent folds in each image of the preprocessed intestinal image set is calculated, and the texture calibration factor is calculated based on the pixel distance and the preset average actual spacing of intestinal folds. The variances of the motion calibration scaling factor and the texture calibration factor are calculated, and the weights are calculated based on the variances of the motion calibration scaling factor and the texture calibration factor. Adaptive fusion is performed based on the motion calibration scaling factor, texture calibration factor, and weights to obtain the conversion coefficient from pixel to actual size; The key feature points of polyp regions in each image of the preprocessed intestinal image set are extracted, and a Delaunay triangulation map is constructed based on the key feature points of each polyp region, including: Using the SuperPoint network, key feature points of polyp regions are obtained by filtering candidate feature points of polyp regions from each image of the preprocessed intestinal image set according to preset confidence and non-maximum suppression conditions. Delaunay triangulation is performed on the key feature points of the polyp region, and an adjacency matrix and node feature matrix are constructed based on the results of the Delaunay triangulation. Based on the Delaunay triangulation map, non-rigid deformation matching is performed using a cross-graph convolutional network and the optimal transport algorithm to obtain a set of matching feature points, including: The adjacency matrix and the node feature matrix are input into a cross-graph convolutional network. The neighborhood features of each node in the graph are aggregated through the intra-graph convolutional layer, and the node features of the Delaunay triangulation map from another frame image are fused through the cross-graph convolutional layer to generate an enhanced node feature representation. Based on the enhanced node feature representation, the similarity matrix of all node pairs between the Delaunay triangulation maps of the two images is calculated, and the similarity matrix is ​​normalized and optimized using the optimal transmission algorithm to obtain a soft assignment matrix representing the matching probability between nodes. Based on the soft assignment matrix, node pairs with a matching probability greater than a preset threshold are selected to form the matching feature point set.

2. The method for processing intestinal images using a micro-intelligent terminal according to claim 1, characterized in that, Sparse 3D reconstruction is performed based on the matched feature point set to obtain a polyp point cloud, including: Preliminary triangulation is performed based on the pixel coordinates of each matching feature point in the matching feature point set, the camera parameters of the endoscope, and the moving baseline distance to generate 3D point cloud coordinates. The reprojection error and parameter coupling error of the 3D point cloud coordinates obtained by the initial triangulation are optimized to obtain the final polyp point cloud.

3. The method for processing intestinal images using a micro-intelligent terminal according to claim 1, characterized in that, The processing method further includes: Anatomical landmarks are identified from the preprocessed intestinal image set, and the centroid coordinates of the anatomical landmarks are calculated. Once the anatomical landmarks are identified, the cumulative motion displacement of the endoscope is calculated, and the cumulative motion displacement is corrected for depth offset based on the centroid coordinates to generate the reference point of the anatomical landmarks. The corrected position of the polyp region is calculated based on the cumulative displacement of the polyp region and the reference point of the anatomical landmark region.

4. The method for processing intestinal images using a micro-intelligent terminal according to claim 3, characterized in that, Once anatomical landmarks are identified, the cumulative motion displacement of the endoscope is calculated, including: The inter-frame movement distance of the endoscope is calculated based on the conversion coefficient and the feature point displacement between adjacent frames of the preprocessed intestinal image set. The cumulative motion displacement is obtained by summing the movement distances between each frame.

5. A micro-intelligent terminal intestinal image processing device, characterized in that, The processing device includes: An image preprocessing module is used to preprocess the raw intestinal image set acquired by a monocular endoscope to obtain a preprocessed intestinal image set. The preprocessing includes image brightness uniformization processing and image equalization processing. A dynamic scaling module is used to calculate a motion scaling factor and a texture scaling factor based on the preprocessed intestinal image set, and to incorporate the motion scaling factor and the texture scaling factor into adaptive fusion to obtain a conversion coefficient from pixel to actual size. The feature extraction module is used to extract key feature points of polyp regions from each image of the preprocessed intestinal image set, and to construct a Delaunay triangulation map based on the key feature points of each polyp region. A non-rigid deformation matching module is used to perform non-rigid deformation matching based on the Delaunay triangulation map, using a cross-graph convolutional network and the optimal transfer algorithm, to obtain a set of matching feature points. The size estimation module is used to perform sparse 3D reconstruction based on the matching feature point set to obtain a polyp point cloud, calculate the Euclidean distance between the two farthest points in the polyp point cloud, and calculate the polyp size based on the Euclidean distance and the transformation coefficient. The method for calculating motion calibration scaling factors and texture calibration scaling factors based on the preprocessed intestinal image set, and then incorporating these factors into adaptive fusion to obtain pixel-to-actual-size conversion coefficients includes: The displacement of feature points between adjacent frames of the preprocessed intestinal image set is calculated based on the optical flow algorithm, and the motion calibration scaling factor is calculated based on the intestinal peristalsis model and the displacement of each feature point. The pixel distance between adjacent folds in each image of the preprocessed intestinal image set is calculated, and the texture calibration factor is calculated based on the pixel distance and the preset average actual spacing of intestinal folds. The variances of the motion calibration scaling factor and the texture calibration factor are calculated, and the weights are calculated based on the variances of the motion calibration scaling factor and the texture calibration factor. Adaptive fusion is performed based on the motion calibration scaling factor, texture calibration factor, and weights to obtain the conversion coefficient from pixel to actual size; The key feature points of polyp regions in each image of the preprocessed intestinal image set are extracted, and a Delaunay triangulation map is constructed based on the key feature points of each polyp region, including: Using the SuperPoint network, key feature points of polyp regions are obtained by filtering candidate feature points of polyp regions from each image of the preprocessed intestinal image set according to preset confidence and non-maximum suppression conditions. Delaunay triangulation is performed on the key feature points of the polyp region, and an adjacency matrix and node feature matrix are constructed based on the results of the Delaunay triangulation. Based on the Delaunay triangulation map, non-rigid deformation matching is performed using a cross-graph convolutional network and the optimal transport algorithm to obtain a set of matching feature points, including: The adjacency matrix and the node feature matrix are input into a cross-graph convolutional network. The neighborhood features of each node in the graph are aggregated through the intra-graph convolutional layer, and the node features of the Delaunay triangulation map from another frame image are fused through the cross-graph convolutional layer to generate an enhanced node feature representation. Based on the enhanced node feature representation, the similarity matrix of all node pairs between the Delaunay triangulation maps of the two images is calculated, and the similarity matrix is ​​normalized and optimized using the optimal transmission algorithm to obtain a soft assignment matrix representing the matching probability between nodes. Based on the soft assignment matrix, node pairs with a matching probability greater than a preset threshold are selected to form the matching feature point set.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a processing program for intestinal images of a micro-intelligent terminal, which, when executed by a processor, implements the processing method for intestinal images of a micro-intelligent terminal according to any one of claims 1 to 4.

7. A computer device, characterized in that, The computer device includes a computer-readable storage medium, a processor, and a micro-intelligent terminal intestinal image processing program stored in the computer-readable storage medium, wherein the micro-intelligent terminal intestinal image processing program, when executed by the processor, implements the micro-intelligent terminal intestinal image processing method according to any one of claims 1 to 4.

Images