Image processing-based small intestine capsule endoscopy auxiliary method and system

Abstract

The application discloses a kind of based on image processing Small intestine capsule endoscopy auxiliary method and system, to solve the problem that capsule endoscopy reading film is easy to miss, polyp and submucous tumor sensitivity is insufficient and time sequence is unstable, the first prediction and intermediate feature are generated by decoding and preprocessing, sliding window time sequence attention is combined with feature memory, candidate region is extracted using class activation map and is distinguished by local refinement subnetwork, second prediction is formed by confidence weighted fusion, time sequence is smoothed by combining minimum duration and class transition constraint, frame-level labeling and statistical visualization are output, the technical effect that polyp and submucous tumor sensitivity is improved, lesion positioning accuracy is enhanced and reading efficiency is improved under the premise of not significantly sacrificing processing speed is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of endoscopic assisted recognition, and in particular to an image processing-based small intestine capsule endoscope assisted method and system. BACKGROUND

[0002] Small intestine capsule endoscopy has become an important diagnostic tool for small intestine diseases due to its non-invasive, non-narcotic and good patient compliance. However, a large number of images are usually generated in one examination, and manual frame-by-frame reading is time-consuming and prone to fatigue, with a high risk of missed detection.

[0003] In recent years, artificial intelligence technology has been rapidly applied to capsule endoscopy image analysis. In the early stage, convolutional neural networks were mainly used, and then the Transformer architecture showed advantages in multi-class lesion classification and cross-device generalization. A multi-task reading assistance system was developed, which combined lesion recognition with cumulative time statistics and progress bar marking, improving reading efficiency and visual presentation.

[0004] As the closest prior art, there is a multi-task small intestine capsule endoscopy assistance system based on Transformer, which achieves good overall performance in multi-class frame-level classification and provides cumulative time bar chart and progress bar marking. The inference speed is higher than manual reading. This system also provides Grad-CAM and SHAP explainability analysis and semantic feature visualization. However, there are still the following shortcomings, which are the direction for further improvement:

[0005] 1. Mainly using frame-by-frame classification, lacking explicit temporal feature modeling and consistency constraints, prone to prediction jitter and short-term misjudgment under capsule fast motion, light changes and bubble interference, affecting the stable detection of continuous lesions;

[0006] 2. Lack of candidate region extraction and local refinement mechanism for suspected lesions, the model mainly classifies the whole image, and the sensitivity to small volume, unclear boundary or overlapping with normal mucosa features is low, especially for polyps and submucosal tumors;

[0007] 3. Frame-level labeling can be visualized as cumulative time and progress bar, but the post-processing lacks temporal smoothing strategies such as minimum duration and class transition constraints, resulting in frequent and unstable class switching on the time axis, which is not conducive to quick positioning and review.

[0008] Therefore, there is an urgent need for an assistance method that introduces temporal modeling, candidate region refinement and post-temporal smoothing without significantly sacrificing processing speed, to improve the sensitivity of polyps and submucosal tumors and enhance sequence stability and clinical usability. SUMMARY

[0009] An object of the present application is to provide an image processing-based small intestinal capsule endoscopy auxiliary method, which addresses the problems of lack of temporal modeling in existing frame-by-frame classification, lack of suspected lesion candidate region refinement, and unstable time axis prediction, thereby leading to insufficient sensitivity of polyps and submucosal tumors, and proposes a technical solution comprising video decoding and preprocessing, sliding window temporal attention combined with feature memory to generate first prediction and intermediate features, selection of candidate regions according to class activation maps and discrimination by local refinement subnetwork, confidence weighted fusion to form second prediction, post-temporal smoothing based on minimum duration and class transition constraints, and output of frame-level labeling and statistical visualization. The present application has the technical effects of improving the sensitivity of polyps and submucosal tumors, enhancing lesion positioning accuracy, and improving temporal consistency and reading efficiency without significantly sacrificing processing speed.

[0010] According to an embodiment of the present application, an image processing-based small intestinal capsule endoscopy auxiliary method is provided, characterized by comprising the following steps:

[0011] S1, acquiring a capsule endoscopy video and decoding it into consecutive image frames to obtain a video frame sequence;

[0012] S2, performing image preprocessing on the video frame sequence to output a preprocessed frame sequence;

[0013] S3, inputting the preprocessed frame sequence into a visual transformer neural network to perform temporal modeling and feature extraction, obtaining a first prediction result and intermediate features for each frame with respect to a set of lesion classes, and outputting the first prediction result and intermediate features;

[0014] S4, jointly calculating the intermediate features and the first prediction result to generate a class activation map, selecting suspected lesion regions in the class activation map according to a preset threshold and a quantity threshold, and obtaining a candidate region set;

[0015] S5, aligning the candidate region set with the preprocessed frame sequence and extracting corresponding local patches, inputting the local patches into a local refinement subnetwork to obtain refinement results, and confidence weighted fusing the refinement results with the first prediction result to obtain a second prediction result for each frame;

[0016] S6, inputting the second prediction result into a temporal smoothing module, performing post-temporal smoothing based on a preset temporal consistency constraint, generating smoothed class probabilities, and forming a frame-level labeling sequence accordingly;

[0017] S7, aligning and superimposing the frame-level labeling sequence with the preprocessed frame sequence to generate a labeled frame, generating a statistical result based on the frame-level labeling sequence, and synthesizing the labeled frame and the statistical result to form an output video with labeling.

[0018] Optionally, step S1 is specifically:

[0019] The capsule endoscopy video is obtained through a collection interface of a capsule endoscopy device or a video file generated by the device, and container parsing and decoding are performed on the capsule endoscopy video to obtain continuous image frames arranged in time sequence and represented in a pixel matrix form, time information of each continuous image frame is reserved and a serial number is marked to form a video frame sequence, in the case of existing lost frames, damaged frames or empty frames, a fault-tolerant strategy of discarding, adjacent frame interpolation or repeating previous and subsequent frames is adopted to ensure the continuity of the video frame sequence, and the video frame sequence is output.

[0020] Optionally, step S2 is specifically:

[0021] Each continuous image frame in the video frame sequence is resized according to a preset size, and the preset size is consistent with an input size of the visual transformer neural network; the resizing is performed by scaling and padding or cropping to complete alignment without significantly changing the proportion of an image subject;

[0022] After the resizing is completed, color normalization is performed on each frame based on a preset statistical parameter; the color normalization maps pixel values to a specified range by a channel-level linear transformation and eliminates color distribution differences caused by different device collection conditions;

[0023] Noise suppression is performed on each frame to reduce random noise and common artifacts; the noise suppression is performed by linear filtering or nonlinear filtering to enhance the visibility of a lesion area;

[0024] After the resizing, color normalization and noise suppression are completed, the processed frames are arranged according to an input format of the visual transformer neural network to obtain a preprocessed frame sequence and output the preprocessed frame sequence.

[0025] Optionally, step S3 is specifically:

[0026] The preprocessed frame sequence is divided into sliding windows in time sequence, and the sliding window contains a target frame and adjacent frames thereof;

[0027] Each sliding window is input into a visual transformer neural network containing a time sequence attention module and a feature memory module; the time sequence attention module performs cross-frame weighted aggregation on the features of the frames in the sliding window to obtain time sequence context features of the target frame; and the feature memory module caches and updates the features of the processed frames for use by subsequent windows;

[0028] The time sequence context features and spatial features of the target frame are fused in the network to generate a first prediction result and intermediate features for each frame in the preprocessed frame sequence; the first prediction result includes class probabilities and corresponding confidence degrees for a lesion class set, and the lesion class set at least contains polyps and submucosal tumors; and the intermediate features are feature representations used to generate a class activation map in a subsequent step;

[0029] output the first prediction result and the intermediate feature.

[0030] Optionally, step S4 is specifically:

[0031] The intermediate feature and the first prediction result are jointly calculated, for each category in the lesion category set, the channels of the intermediate feature are weighted and aggregated according to the category probability of the first prediction result, and the aggregation result is normalized and scaled to generate a category activation map of the corresponding frame;

[0032] Based on the category activation map, the response peak position is determined, the response regions meeting the condition are filtered according to the pre-set confidence threshold, and the suspected lesion regions are sorted from high to low according to the response intensity, and the position information and size information of the suspected lesion regions are extracted, forming a candidate region set;

[0033] Each candidate region in the candidate region set is associated with its category and confidence, and the index and time information consistent with the pre-processed frame sequence are kept, and the candidate region set is output.

[0034] Optionally, step S5 is specifically:

[0035] The candidate region set and the pre-processed frame sequence are position-aligned, and according to the position information and size information of each candidate region in the candidate region set, the corresponding local patch is extracted from the pre-processed frame sequence and resampled according to the input size of the local refinement subnetwork, and the local patch sequence is obtained by arranging in time sequence and frame index;

[0036] The local patch sequence is input into the local refinement subnetwork for discrimination to obtain a refinement result, the local refinement subnetwork is a neural network classifier for candidate region classification, and the refinement result includes lesion category probability and confidence for each candidate region;

[0037] For multiple candidate regions of the same frame, the category probabilities of the refinement result are weighted and aggregated according to the confidence of the candidate region, and the candidate region aggregation result of the frame is obtained;

[0038] The candidate region aggregation result and the first prediction result are weighted and fused according to the confidence as the weight, and the second prediction result of each frame is obtained and output.

[0039] Optionally, step S6 is specifically:

[0040] The second prediction result is input into the time sequence smoothing module in time sequence, the time sequence smoothing module performs posterior time sequence smoothing on the category probability of each frame in the second prediction result, generates smoothed category probability according to the minimum duration constraint and category transition constraint, and forms a frame-level label sequence accordingly;

[0041] wherein the minimum duration constraint is used to adjust the segment category to the category with the highest adjacent frame proportion when the duration of a certain category of consecutive frames is less than a preset duration threshold, and the category transition constraint is used to suppress unreasonable adjacent category switching according to a preset category transition matrix and correct the category of the switching boundary to reduce the frequency of low-confidence switching;

[0042] After the posterior timing smoothing is completed, the predicted category and the confidence are determined according to the smoothed category probability of each frame, a frame-level labeling sequence containing the predicted category and the confidence of each frame is obtained, and the frame-level labeling sequence is output.

[0043] Optionally, step S7 is specifically:

[0044] The frame-level labeling sequence is aligned with the preprocessed frame sequence frame by frame, and the predicted category and the confidence of each frame are superimposed on the corresponding preprocessed frame to generate a labeled frame, wherein the superimposed display presents the predicted category and the confidence in a text or graphical manner and keeps consistent with the frame index and time information;

[0045] Based on the frame-level labeling sequence, the continuous segments of each category are accumulated according to the frame rate or the time stamp, the accumulated duration of each category on the time axis is counted, and an accumulated time bar chart is generated;

[0046] The frame-level labeling sequence is mapped to a color segment marker sequence with a color set according to the category to form a progress bar marker, and the color segment marker sequence is corresponded to the frame index to reflect the distribution of each category on the time axis;

[0047] The labeled frame, the accumulated time bar chart and the color segment marker sequence are synthesized to obtain an output video with labeling and statistical results, and the output video with labeling and the statistical results are output.

[0048] A small intestine capsule endoscope auxiliary system based on image processing, comprising:

[0049] A video acquisition module is configured to acquire a capsule endoscope video and decode the video into continuous image frames;

[0050] An image processing module is configured to perform image preprocessing on the video frame sequence;

[0051] A feature extraction module is configured to receive the preprocessed frame sequence and perform timing modeling and feature extraction to obtain a first prediction result and intermediate features for each frame with respect to a set of lesion categories;

[0052] A generation module is configured to generate a category activation map based on the intermediate features and the first prediction result, and select suspected lesion regions according to a preset threshold and a quantity threshold to obtain a candidate region set;

[0053] The refining and fusing module is used for position alignment with the preprocessed frame sequence and extraction of corresponding local patches, input of the local patches into a local refining subnetwork to obtain a refined result, and confidence weighted fusion with the first prediction result to obtain a second prediction result of each frame.

[0054] The timing smoothing module is used for posterior timing smoothing of the second prediction result based on a preset timing consistency constraint to generate a frame-level labeling sequence.

[0055] The visual output module is used for alignment and superimposed display of the frame-level labeling sequence and the preprocessed frame sequence to generate a labeled frame, generation of a statistical result based on the frame-level labeling sequence, and synthesis of the labeled frame and the statistical result into an output video with labeling.

[0056] The present application has the following advantages:

[0057] 1. The cascade discrimination of "category activation map candidate region screening + local refining subnetwork + confidence weighted fusion" improves the detection capability of lesions with small volume, unclear boundary or overlapping with normal mucosa features, and can improve the sensitivity of polyps and submucosal tumors without significantly sacrificing processing speed.

[0058] 2. The forward modeling of "sliding window timing attention and feature memory" combined with the posterior timing smoothing of "minimum duration constraint and category transition constraint" significantly reduces frame-level prediction jitter and short-term misjudgment, suppresses unreasonable category switching, and improves the timing consistency and video-level stability of continuous lesions.

[0059] 3. The alignment and superposition of the frame-level labeling sequence and the preprocessed frame, the synthesis output of the cumulative time column chart and the progress bar mark realize fast positioning and review of lesions, shorten the reading time and reduce the risk of missed detection, and at the same time, the size alignment, color normalization and noise suppression improve the robustness under cross-device and complex imaging conditions. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0061] Figure 1 A flowchart of an image processing-based small intestine capsule endoscopy auxiliary method and system is provided.

[0062] Figure 2This diagram illustrates the categories of lesions, showing representative images of common small bowel capsule endoscopy categories that can be identified by this invention. It is used to illustrate the composition of the lesion category set, which includes at least normal mucosa, vascular malformations, hemorrhage, erosion, erythema, foreign bodies, lymphoid follicles, lymphangiectasia, polyps, submucosal tumors, stenosis, and venous aneurysms. This diagram is used to define the category range of the objects to be treated. The images shown are for illustration only and do not constitute a limitation on the appearance, number, or distribution of each lesion.

[0063] Figure 3 This diagram illustrates the process of generating a category activation map and candidate region extraction based on intermediate features and the first prediction, and then filtering suspected lesion candidate regions. A is the original frame, B is the category activation heatmap, and C is a visualization of the heatmap and the original frame superimposed, with the category and confidence level labeled. This diagram corresponds to step S4 of the present invention and is used to illustrate the generation method of the "category activation map-candidate region set".

[0064] Figure 4 The diagram illustrates the visualization output interface for step S7 of this invention. It includes: a labeled frame with frame-level predicted category and confidence level superimposed on the upper left; a cumulative time bar chart of each category obtained by accumulating frame-level labels on the upper right; and a progress bar marker that maps frame-level labels to color segments at the bottom, which is used to reflect the distribution of each category on the time axis and supports quick location and review. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0066] refer to Figure 1 A method for assisting small bowel capsule endoscopy based on image processing, characterized by comprising the following steps:

[0067] S1. Acquire capsule endoscopy video and decode it into continuous image frames to obtain a video frame sequence;

[0068] S2. Perform image preprocessing on the video frame sequence and output the preprocessed frame sequence;

[0069] S3. Input the preprocessed frame sequence into the visual transformer neural network, perform temporal modeling and feature extraction, obtain the first prediction result and intermediate features for each frame for the lesion category set, and output the first prediction result and intermediate features.

[0070] S4. Combine the intermediate features with the first prediction result to generate a category activation map. Select suspected lesion areas in the category activation map according to the preset threshold and the number threshold to obtain a candidate region set.

[0071] S5. Align the candidate region set with the preprocessed frame sequence and extract the corresponding local patches. Input the local patches into the local thinning sub-network to obtain the thinning result, and then fuse them with the first prediction result according to the confidence level to obtain the second prediction result for each frame.

[0072] S6. Input the second prediction result into the temporal smoothing module, perform posterior temporal smoothing based on the preset temporal consistency constraint, generate smoothed class probabilities and form a frame-level labeled sequence accordingly.

[0073] S7. Align and overlay the frame-level annotation sequence with the preprocessed frame sequence to generate annotated frames. Generate statistical results based on the frame-level annotation sequence. Combine the annotated frames and statistical results to create an annotated output video.

[0074] In this specific embodiment, S1 specifically refers to:

[0075] Raw video data is accessed in real-time through the acquisition interface of the capsule endoscopy device or read from the storage file generated by the device. The video container is parsed and the compressed bitstream is decoded to obtain a continuous sequence of image frames arranged in chronological order. For ease of description, the video container or raw bitstream is referred to as... The container resolution function is The encoding and decoding functions are The output frame sequence is Then we have:

[0076] ;

[0077] in Represents the set of decoded frame sequences. Indicates the first The pixel matrix of a frame. Indicates the first Frame timestamp Represents the frame sequence number and its value range is... , Indicates the total number of frames in the sequence;

[0078] To ensure consistent follow-up processing, the following agreement is made. ,in Indicates image height, Indicates image width, Indicates the number of channels. Represents the set of real numbers;

[0079] To ensure consistency of the timeline, the device's nominal frame rate is used. Calculate the target frame interval ,in The numerical value representing the nominal frame rate (in Hertz). This represents the ideal interval (in seconds) between adjacent valid frames on the time axis, which the system will decode. according to Sort in ascending order and build and The system establishes a one-to-one correspondence and performs decoding integrity and content validity checks on each frame to identify anomalies such as dropped frames, corrupted frames, or empty frames.

[0080] To address abnormal situations, a fault-tolerant strategy is employed to maintain the continuity and availability of the frame sequence. This involves two main approaches: first, discarding severely corrupted frames that cannot be recovered; and second, reconstructing missing frames using time-weighted linear interpolation of neighboring frames when adjacent valid frames exist. The nearest adjacent valid frame indices are denoted as follows: and Their timestamps are respectively and Interpolation weights are , reconstructed frames Then we have:

[0081] and ;

[0082] in express The most recent valid frame number on the left, express The most recent valid frame number on the right, Represents the linear interpolation coefficients obtained proportionally to time. Indicates the first Frame reconstruction pixel matrix, and Representing the pixel matrices of the preceding and following valid frames, respectively. and These represent the timestamps of the preceding and following valid frames, respectively.

[0083] Thirdly, when there is no available or A repetition strategy is used to maintain temporal continuity, i.e. or ,in Indicates the number after repeated substitution Frame pixel matrix;

[0084] After completing the above fault tolerance, the temporal continuity of the sequence is checked and the jitter tolerance is limited. As a criterion for continuity, among which This represents the allowed time jitter threshold, and the final output is a video frame sequence that meets the requirements of temporal order and continuity. The time information and sequence number of each frame are retained for subsequent preprocessing and temporal modeling.

[0085] In this specific embodiment, S2 specifically refers to:

[0086] Frame-by-frame size alignment, color normalization, and noise suppression are performed on the video frame sequence, and the results are then used as input to the visual transformer neural network. First, size alignment is performed to ensure the input matches the network requirements without significantly altering the proportions of the main image elements. Let the th frame be... The original image of the frame is Its height and width are respectively and The number of channels is The network expects the input height and width to be respectively... and Use proportional scaling with center-aligned padding or light cropping. The scaling factor is:

[0087] ;

[0088] in Indicates the proportional scaling factor, and Indicates the height and width of the network's default input dimensions. and Indicates the first The original height and width of the frame, A function representing the minimum value of the two. Indicates frame index and Indicates the first The pixel matrix of a frame, after scaling and alignment, yields a size-aligned image, denoted as . ;

[0089] Color normalization is then performed to reduce color distribution differences caused by device and imaging conditions. A linear transformation is then performed on each channel, setting:

[0090] ;

[0091] in Represents the normalized pixel value, Represents the pixel value before normalization. Represents the pixel coordinates with the top-left corner as the origin. Indicates color channel index, This represents the statistical mean of the channel. This represents the linear scaling factor of the channel. This represents the linear bias of the channel, and if normalization is required, it can be set as follows: And by This represents the standard deviation of the channel;

[0092] Then noise suppression is performed to reduce random noise and common artifacts while preserving details. Linear filtering is achieved by two-dimensional discrete convolution, which is defined as:

[0093]

[0094] wherein denotes the filtered pixel value, denotes the kernel weight centered at , denotes the kernel radius, denotes the relative displacement index, denotes the discrete summation operation within the kernel window, and the kernel weight and are set to 1 to maintain the brightness unchanged. The mirror or copy strategy is adopted for the boundary to avoid out-of-bound access. When stronger edge preservation is needed, the can be replaced by a bilateral weight to form a nonlinear smoothing.

[0095] Finally, the filtered frame is organized into the input representation required by the visual transformer, which is divided into blocks with a side length of to obtain blocks and generate the token matrix by linear projection and position encoding, which is defined as:

[0096]

[0097] wherein denotes the token sequence matrix of the frame, denotes the operator that performs block division and linear projection on the input image and adds the position index, denotes the number of pixels of the side length of each block, denotes the number of blocks and is determined by the input size and together, denotes the frame index and corresponds to one-to-one. Thus, the preprocessed frame sequence that meets the size consistency, color distribution consistency, and noise suppression is obtained for input into the visual transformer neural network containing the temporal attention and feature memory.

[0098] In the specific embodiment, the S3 is specifically:

[0099] The preprocessed frame token sequence is input into the visual transformer neural network containing the temporal attention and feature memory, and the sliding window is used to model the cross-frame context to generate the first prediction and intermediate feature. Assuming that the token sequence matrix of the frame from step S2 is , denotes the frame index, and ​​​​Let T denote the total number of frames, and construct a window index set containing the target frame and its neighboring frames as:

[0100]

[0101] wherein denotes the frame index set within the window, denotes the half window size, denotes the legal index interval;

[0102] Calculate the cross-frame attention weight and obtain the timing context feature within the window by using the query vector of the target frame and the key vector of the neighboring frames, and let:

[0103]

[0104] wherein denotes the target frame the attention weight of the index frame, denotes the Softmax function normalized in the index dimension, denotes the inner product of the query and the key, denotes the scaling by the square root of the feature dimension , denotes the vector space dimension, denotes the spatial feature vector of the frame, denotes the timing context feature obtained by weighted aggregation, denotes the discrete summation of the indices within the set , denotes any frame index within the window, and the query and the key respectively obtained by linear mapping of the frame features;

[0105] In order to enhance long-term dependence and reduce short-term jitter, the processed frames are cached by using an updatable feature memory and written at the current time, and let:

[0106]

[0107] wherein denotes the updated memory state, denotes the memory state at the last time, denotes the memory decay coefficient, denotes the spatial feature vector of the frame;

[0108] Then, the discriminative representation is obtained by fusing the spatial feature of the target frame, the timing context and the memory reading, and let: ​​​​

[0109] ;

[0110] in Represents the fused feature vector, Represents a learnable transformation used for feature fusion. This indicates a connection operation that splices elements by channel or dimension.

[0111] In the classification header Perform linear mapping and normalization to obtain the probability distribution of the lesion category set, and calculate the frame-level confidence score, let:

[0112] and ;

[0113] in Indicates the first Frame in Probability vectors for each category Represents the classification weight matrix, Represents the classification bias vector, The Softmax function represents the normalization of the category dimension. Indicates the first Maximum class confidence of the frame Indicates category index, Indicates the size of the set of lesion categories, and that the set contains at least "polyps" and "submucosal tumors";

[0114] Simultaneously, it outputs intermediate features used to generate the class activation map, denoted as:

[0115] ;

[0116] in Indicates intermediate feature representation, The feature extraction operator of the visual transformer encoder, Indicates the first Token sequence matrix of frames This represents the set of window indices, ultimately yielding the first prediction result for each frame. With intermediate features This is for use in subsequent candidate region generation and discrimination.

[0117] In this specific embodiment, S4 specifically refers to:

[0118] Using intermediate features and the first prediction as input, a category activation map is generated and candidate regions for suspected lesions are screened. Let the first feature be the first feature. The intermediate feature tensor of the frame is The class probability vector of the first prediction is ,in Indicates frame index, and They represent Spatial height and width, Indicates the number of channels, Indicates the size of the lesion category set and includes at least "polyps" and "submucosal tumors". Represents spatial coordinates on the feature map, Indicates channel index, Indicates a category index;

[0119] To obtain class-specific responses, channel weights are first modulated with class probabilities and channel aggregation is performed, let:

[0120] and ;

[0121] in Indicates the first Frame number Class of the first Channel weights, Indicates the first The class's channel baseline weight constant or learnable coefficient, Indicates coordinates The first Class weighted response, Indicates coordinates The first Channel eigenvalues, Indicates the channel index Discrete summation;

[0122] To facilitate threshold filtering and alignment with the original frame, the response map is normalized and upsampled, let:

[0123] and ;

[0124] in Represents the normalized i-th Class activation value, and They represent Minimum and maximum values ​​on the whole graph This represents the smallest positive number used to prevent the denominator from being zero. The activation map represents the scale aligned to the preprocessed frame size. The upsampling operator for bilinear interpolation, and These represent the height and width of the preprocessed frame, respectively.

[0125] exist The above is based on a preset confidence threshold. Perform connected component decomposition to obtain the first Class response region collection For each response, its bounding box and intensity are calculated, and the bounding box is denoted as:

[0126] and ;

[0127] in Indicates the first One satisfies connected pixel set, Indicates the axis-aligned circumscribed rectangle. and Indicates the coordinates of the top left corner of the bounding box. and Representing the width and height of the outer frame respectively, Indicates the candidate's response strength or confidence level. This represents the preset confidence threshold.

[0128] Press within the same frame Sort by high to low and select no more than [number] items. Among the candidates, This represents a threshold number, ultimately forming a candidate set consistent with the frame index and time information. And retain the category to which each candidate belongs. Frame Index With timestamp This is for subsequent local refinement and integration.

[0129] In this specific embodiment, S5 specifically includes:

[0130] Taking the candidate region set and preprocessed frames as input, the system first performs position alignment and local patch extraction. Then, it uses a local refinement sub-network for discrimination and performs weighted aggregation of candidates in the same frame according to confidence. Finally, it adaptively fuses the first prediction with the second prediction according to confidence to form the second prediction result, where the first prediction is denoted as . The candidate set of frames is Indicates frame index, Represents the candidate set of this frame. Indicates the first Class 1 One candidate axis-aligned outer frame, Indicates category index, Indicates the candidate's sequence number within this frame, This represents the candidate confidence obtained from the class activation map. The candidate is aligned with the preprocessed frame, cropped, and resampled to obtain a local patch. Let:

[0131] ;

[0132] where denotes the resampled local patch, denotes the operator of cropping on the image with the bounding box, denotes the preprocessed image of the frame, denotes the resampling operator that scales the input to height and width , and denotes the input size of the local refinement subnetwork;

[0133] obtains the refinement output by inputting the local patch into the local refinement subnetwork, let

[0134] ;

[0135] where denotes the local refinement subnetwork for the candidate classification, denotes the refinement output, denotes the probability vector on classes, denotes the refinement confidence of the candidate, denotes the total number of lesion classes;

[0136] For the aggregation of multiple candidates of the same class in the same frame, the candidate index set of the class in the frame is defined as and the weighted average is performed with the candidate confidence as the weight, let

[0137] and ;

[0138] where denotes the candidate aggregation probability of the class in the frame, denotes the candidate aggregation confidence of the class in the frame, denotes the component of the vector on the class, denotes the discrete summation on the index set, denotes the candidate index set belonging to the class;

[0139] When there is no candidate of a class in the frame, the aggregation is skipped and the first prediction takes over in the subsequent fusion. Then the candidate aggregation result and the first prediction are adaptively fused according to the confidence, let

[0140] and ;

[0141] in The second prediction after fusion is in the first... Probability of class Indicates fusion weights, Indicates the first prediction in the 1st century. Probability of class Indicates frame-level confidence, This indicates an extremely small positive number that prevents the denominator from being zero;

[0142] Finally, the second prediction vector for each frame is obtained. It also maintains consistency with the frame index and time information for subsequent timing smoothing processing.

[0143] In this specific embodiment, S6 specifically refers to:

[0144] The second prediction result is input into the temporal smoothing module in chronological order. Under the combined effect of class transition constraints and minimum duration constraints, posterior temporal smoothing is performed, and a frame-level labeled sequence is generated. Let the first prediction result be... The second prediction probability vector of the frame is and Represents the total number of frames, The total number of categories must include at least polyps and submucosal tumors. To suppress unreasonable switching between adjacent categories, dynamic programming is first performed based on the category transition matrix. The class transition matrix is ​​given by Indicates from category Transfer to Category Prior probability, with Indicates the first Frame category The cumulative cost, with Represents the natural logarithm, with If we denote minimization over the predecessor category, then:

[0145] and ;

[0146] in Indicates the first Frame in The second prediction probability on the class, Indicates the candidate category index of the previous frame;

[0147] The initial path, smoothed only by transition constraints, is obtained by backtracking based on the minimum cumulative cost. and with Indicates the first The initial category label of the frame, and then the minimum duration constraint is applied to correct for short misjudgments, assuming that segments of the same type are... and The length of the fragment is Duration threshold is Fragment category is The categories of adjacent segments on both sides are and ,when Then, relabel the entire segment to the category with the highest adjacent percentage, let:

[0148] and For all ;

[0149] in This represents the category that maximizes the objective function. This represents the discrete summation of the frame index within the segment. Indicates the path label after applying the duration constraint;

[0150] To obtain a smoothed probability distribution, a gentle fusion is performed between the hard labels and the original probabilities, let:

[0151] ;

[0152] in Indicates the first Frame in Smoothed probability of class Indicates the fusion coefficient, Indicates when The Kronecker function takes the value 1 if it is true and 0 otherwise.

[0153] Final basis To determine frame-level annotations and confidence levels, let:

[0154] and ;

[0155] in Indicates the first Frame prediction category, Indicates the first The prediction confidence of each frame is used to form a frame-level annotation sequence containing the prediction category and confidence of each frame for subsequent visualization output.

[0156] In this specific embodiment, S7 specifically refers to:

[0157] The frame-level annotation sequence and the preprocessed frame sequence are aligned frame by frame and then overlaid for display, time statistics, and visualization. First, an annotation frame is generated for each frame using the overlay operator, and then... ,in Indicates the first Frame annotation frames, overlay operator that overlays the predicted class and confidence onto the image in text or graphics, denotes the pre-processed image of the frame, denotes the predicted class label of the frame, denotes the predicted confidence of the frame, denotes the frame index and denotes the total number of frames and keeps consistent with the timestamp to ensure no loss of time information;

[0158] The cumulative time statistics is then performed based on the frame rate to generate the total duration of each class, let where denotes the cumulative duration of class , denotes the Kronecker delta function that takes 1 when and 0 otherwise, denotes the frame rate of the video, denotes the class index, denotes the set of lesion classes, denotes the total number of classes and contains at least "polyp" and "submucosal tumor", and the adjacent timestamp difference is used to replace the text description when the cumulative time is needed ;

[0159] The frame-level annotation sequence is then mapped to the progress bar marker of color segments, let where denotes the color segment image of the progress bar, denotes the rendering operator that arranges the input color sequence into a one-dimensional strip according to the length of the time axis , denotes the color mapping function that maps the class to a preset color.

[0160] The cumulative duration vector is used to draw the cumulative time histogram, let where denotes the histogram image, denotes the drawing operator that draws the cumulative duration of each class into a histogram.

[0161] Finally, the layout operator is used to layout the annotated frames, the histogram and the progress bar into a layout and encode it into the annotated output video, let:

[0162] ;

[0163] where denotes the annotated output video, Compose A synthesis operator for spatial arrangement and time sequence coding of multiple visual elements, thus obtaining an annotated output video and statistical results and ensuring consistency with frame index and timestamp for quick positioning and review.

[0164] The above merely describes a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical scope disclosed by the present application, should be covered within the protection scope of the present application.

[0165] The present application forms a synergistic effect on the technical problems of missed detection, insufficient sensitivity of polyps and submucosal tumors and unstable time sequence in capsule endoscopy reading through the algorithm combination of "forward time sequence modeling + candidate region cascade refinement + posterior time sequence smoothing": sliding window time sequence attention combined with feature memory explicitly aggregates cross-frame context at the feature level, weakening short-term jitter caused by rapid capsule movement, illumination fluctuation and bubble artifacts, and improving the stability of frame-level prediction from the source; on the basis of the first prediction of the whole image, suspected lesion candidates are generated with class activation maps, only local patches are implemented for refinement discrimination and are fused with global prediction according to confidence, so that the detection rate and positioning accuracy of lesions with small volume, unclear boundary or overlapping with normal mucosa are higher; then, posterior smoothing is performed based on the minimum duration and class transition constraint to suppress unreasonable adjacent class switching and correct short-term misjudgment segments, finally outputting frame-level labeling and cumulative time, progress bar visualization, which simultaneously improves the sensitivity, sequence consistency and reading efficiency of polyps and submucosal tumors without significantly sacrificing processing speed.

[0166] On the algorithm structure, the present application makes targeted improvements around the technical problems and forms an engineering path that can be implemented: first, the forward structure of "sliding window time sequence attention + updatable feature memory" is adopted, which significantly enhances the time sequence robustness compared with single-frame classification and takes into account streaming inference; second, a two-level discrimination closed loop of "global classification-CAM candidate-local refinement-confidence fusion" is constructed, which concentrates calculation on high-suspected areas under the constraints of candidate number and threshold, balances recall and speed, and reduces false positives and false negatives through consistency fusion of candidate-level and global-level information; third, the posterior optimization of "class transition matrix + minimum duration" is introduced, which makes the time axis label transition from the probability level to the global consistency of the path level, stabilizing the boundary and class of continuous lesion segments. The above structural improvements cooperate with each other to improve the detection ability and time sequence stability of key lesions from three levels of model representation, candidate screening and post-processing, further strengthening the clinical usability and cross-device robustness.

Claims

1. A method for assisting small bowel capsule endoscopy based on image processing, characterized in that, Includes the following steps: S1. Acquire capsule endoscopy video and decode it into continuous image frames to obtain a video frame sequence; S2. Perform image preprocessing on the video frame sequence and output the preprocessed frame sequence; S3. Input the preprocessed frame sequence into the visual transformer neural network, perform temporal modeling and feature extraction, obtain the first prediction result and intermediate features for each frame for the lesion category set, and output the first prediction result and intermediate features. S4. Combine the intermediate features with the first prediction result to generate a category activation map. Select suspected lesion areas in the category activation map according to the preset threshold and the number threshold to obtain a candidate region set. The step of jointly calculating the intermediate features and the first prediction result to generate the category activation map specifically involves: The intermediate features are jointly calculated with the first prediction result. For each category in the lesion category set, the channels of the intermediate features are weighted and aggregated according to the category probability of the first prediction result. The aggregation result is then normalized and scale-aligned to generate the category activation map of the corresponding frame. S5. Align the candidate region set with the preprocessed frame sequence and extract the corresponding local patches. Input the local patches into the local thinning sub-network to obtain the thinning result, and then fuse them with the first prediction result according to the confidence level to obtain the second prediction result for each frame. S6. Input the second prediction result into the temporal smoothing module, perform posterior temporal smoothing based on the preset temporal consistency constraint, generate smoothed class probabilities and form a frame-level labeled sequence accordingly. S7. Align and overlay the frame-level annotation sequence with the preprocessed frame sequence to generate annotated frames. Generate statistical results based on the frame-level annotation sequence. Combine the annotated frames and statistical results to create an annotated output video.

2. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S1 specifically refers to: Capsule endoscopy video is acquired through the acquisition interface of the capsule endoscopy device or the video file generated by the device. The capsule endoscopy video is parsed and encoded / decoded to obtain a series of consecutive image frames arranged in chronological order and represented in the form of a pixel matrix. Time information is retained for each consecutive image frame and a sequence number is marked to form a video frame sequence. In the case of dropped frames, damaged frames, or empty frames, a fault tolerance strategy of discarding, interpolating adjacent frames, or repeating preceding and following frames is adopted to ensure the continuity of the video frame sequence, and the video frame sequence is output.

3. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S2 specifically refers to: Each consecutive image frame in the video frame sequence is resized according to a preset size, which is consistent with the input size of the visual transformer neural network. The resizing is achieved by scaling and padding or cropping to complete the alignment without significantly changing the proportion of the main image. After the size adjustment is completed, each frame is color normalized based on preset statistical parameters. Color normalization uses channel-level linear transformation to map pixel values ​​to a specified range and eliminate color distribution differences caused by different device acquisition conditions. Noise suppression is performed on each frame to reduce random noise and common artifacts. Noise suppression uses linear or nonlinear filtering to enhance the visibility of lesion areas. After completing the resizing, color normalization, and noise suppression, the processed frames are arranged according to the input format of the visual transformer neural network to obtain a preprocessed frame sequence and output the preprocessed frame sequence.

4. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S3 specifically refers to: The preprocessed frame sequence is divided into sliding windows in chronological order, and each sliding window contains the target frame and its adjacent frames. Each sliding window is input into a visual transformer neural network containing a temporal attention module and a feature memory module. The temporal attention module performs cross-frame weighted aggregation of the features of each frame within the sliding window to obtain the temporal context features of the target frame. The feature memory module caches and updates the features of the processed frames for use by subsequent windows. In the network, temporal context features are fused with spatial features of the target frame to generate a first prediction result and intermediate features for each frame in the preprocessed frame sequence. The first prediction result includes the class probability and corresponding confidence of the lesion class set, which includes at least polyps and submucosal tumors. The intermediate features are feature representations used to generate class activation maps in subsequent steps. Output the first prediction result and intermediate features.

5. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S4 specifically refers to: The intermediate features are jointly calculated with the first prediction result. For each category in the lesion category set, the channels of the intermediate features are weighted and aggregated according to the category probability of the first prediction result. The aggregation result is then normalized and scale-aligned to generate the category activation map of the corresponding frame. The peak position of the response is determined based on the category activation map. Response regions that meet the conditions are filtered according to a preset confidence threshold and sorted from high to low response intensity. After sorting, suspected lesion regions with a number not exceeding the number threshold are selected. The location and size information of the suspected lesion regions are extracted to form a candidate region set. Associate each candidate region in the candidate region set with its category and confidence level, and keep it consistent with the index and time information of the preprocessed frame sequence, and output the candidate region set.

6. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S5 specifically refers to: The candidate region set is aligned with the preprocessed frame sequence. Based on the position and size information of each candidate region in the candidate region set, the corresponding local patches are extracted from the preprocessed frame sequence and resampled according to the input size of the local thinning sub-network. The local patch sequence is then arranged in chronological order and frame index. The local patch sequence is input into the local thinning sub-network for discrimination to obtain the thinning result. The local thinning sub-network is a neural network classifier for candidate region classification. The thinning result includes the lesion category probability and confidence for each candidate region. For multiple candidate regions in the same frame, the category probabilities of the refined results are weighted and aggregated according to the confidence of the candidate regions to obtain the aggregated candidate regions of the frame. The candidate region aggregation result and the first prediction result are weighted and fused according to the confidence level to obtain the second prediction result for each frame and output the second prediction result.

7. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S6 specifically refers to: The second prediction result is input into the temporal smoothing module in chronological order. The temporal smoothing module performs posterior temporal smoothing on the class probability of each frame in the second prediction result, generates smoothed class probabilities based on minimum duration constraints and class transition constraints, and forms a frame-level labeling sequence accordingly. Among them, the minimum duration constraint is used to adjust the category of a segment to the category with the highest proportion of adjacent frames when the duration of a certain category in consecutive frames is less than the preset duration threshold. The category transfer constraint is used to suppress unreasonable adjacent category switching based on the preset category transfer matrix and correct the category of the switching boundary to reduce the frequency of low-reliability switching. After post-abstract temporal smoothing, the predicted class is determined for each frame according to the smoothed class probability and the confidence is calculated to obtain a frame-level labeling sequence containing the predicted class and confidence of each frame, and the frame-level labeling sequence is output.

8. The method for small bowel capsule endoscopy based on image processing according to claim 1, characterized in that, S7 specifically refers to: The frame-level annotation sequence is aligned frame by frame with the preprocessed frame sequence. The predicted category and confidence level of each frame are overlaid on the corresponding preprocessed frame to generate an annotation frame. The overlay display presents the predicted category and confidence level in text or graphic form and maintains consistency with the frame index and time information. Based on the frame-level labeled sequence, the continuous segments of each category are accumulated according to the frame rate or timestamp, and the cumulative occurrence duration of each category on the time axis is counted to generate a cumulative time bar chart. The frame-level label sequence is mapped to a color segment label sequence with colors set according to category to form progress bar labels, so that the color segment label sequence corresponds to the frame index to reflect the distribution of each category on the time axis; The labeled frames, cumulative time bar chart, and color segment marker sequence are combined to obtain the labeled output video and statistical results, and the labeled output video and statistical results are output.

9. An image processing-based small bowel capsule endoscopy-assisted system for performing any one of the image processing-based small bowel capsule endoscopy-assisted methods of claims 1 to 8, comprising: The video acquisition module is used to acquire capsule endoscopy videos and decode them into continuous image frames; The image processing module is used to preprocess the video frame sequence. The feature extraction module is used to receive the preprocessed frame sequence and perform temporal modeling and feature extraction to obtain the first prediction result and intermediate features for each frame for the lesion category set. The generation module is used to generate a category activation map based on intermediate features and the first prediction result, and select suspected lesion areas according to preset thresholds and quantity thresholds to obtain a candidate region set; the generation of the category activation map based on intermediate features and the first prediction result specifically involves: jointly calculating the intermediate features and the first prediction result, and for each category in the lesion category set, weighting and aggregating the channels of the intermediate features according to the category probability of the first prediction result, and normalizing and scaling the aggregation result to generate the category activation map of the corresponding frame; The thinning and fusion module is used to align the position with the preprocessed frame sequence and extract the corresponding local patches. The local patches are input into the local thinning sub-network to obtain the thinning result, and then fused with the first prediction result according to the confidence level to obtain the second prediction result for each frame. The temporal smoothing module is used to perform posterior temporal smoothing on the second prediction result based on a preset temporal consistency constraint, and generate a frame-level labeled sequence. The visualization output module is used to align and overlay the frame-level annotation sequence with the preprocessed frame sequence to generate annotated frames, generate statistical results based on the frame-level annotation sequence, and synthesize the annotated output video with the annotated frames and statistical results.

Images