Colon capsule endoscopy video lesion detection method based on time sequence attention fusion

Abstract

The present application relates to a kind of colon capsule endoscopy video lesion detection methods based on timing attention fusion, belong to medical instrument image processing technical field, comprising the following steps: S1: to colon capsule endoscopy continuous video stream is carried out timing windowing and pre-processing;S2: 3D-CNN is used to extract windowing space-time joint feature, while extracting long timing dependence feature by improving timing Transformer;S3: utilize timing attention weight matrix to space-time feature and long timing feature do dynamic weighted fusion, obtain the fusion feature of accurate representation lesion;S4: to fusion feature decoding output lesion candidate frame, through cosine similarity calculation cross-frame lesion correlation, determine whether it is same lesion;S5: set timing duration threshold, only lesion continuous detection frame number is up to standard when determining as real lesion, combine frame effective rate correction final confidence.

Description

Technical Field

[0001] This invention belongs to the field of medical device image processing technology, and relates to a method for detecting lesions in colon capsule endoscopy video based on temporal attention fusion. Background Technology

[0002] As a core non-invasive screening method for colonic diseases, capsule endoscopy generates tens of thousands of continuous video streams in a single examination. Manual image interpretation is time-consuming and has a high rate of missed diagnoses. Therefore, relying on artificial intelligence to achieve automated lesion-assisted detection has become an industry necessity. At present, most of the technical solutions related to lesion detection in capsule endoscopy are based on single-frame image analysis. The mainstream solutions mostly use single-frame convolutional models to complete the initial lesion screening. They only perform temporal optimization through superficial post-processing methods such as inter-frame difference comparison, fixed window voting, and simple counting filtering. The technical methods in related patents and journal literature have not carried out in-depth modeling of the temporal correlation features of the video stream. They can only achieve basic lesion detection and preliminary artifact removal, which is difficult to meet the accuracy and efficiency requirements of clinical screening. Specifically, there are many shortcomings, such as small lesions being easily missed due to blurred intestinal peristalsis or temporary occlusion, repeated detection of the same lesion across frames, high false positive rate, inability to effectively distinguish between real lesions and intestinal peristalsis artifacts and bubble interference, and low inference efficiency due to repeated calculation of inter-frame features. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method for detecting lesions in colon capsule endoscopy videos based on temporal attention fusion. By deeply modeling the temporal correlation features of the video, it can achieve accurate detection of small lesions, deduplication of lesions across frames, and automatic filtering of artifacts. At the same time, it can improve the inference efficiency of video stream and reduce the missed diagnosis rate and false positive rate in clinical image interpretation.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion includes the following steps: S1: Perform temporal windowing and preprocessing on the continuous video stream of colon capsule endoscopy; S2: 3D-CNN is used to extract spatiotemporal joint features within the window, while long temporal dependency features are extracted by improving the temporal Transformer; S3: Utilize the temporal attention weight matrix to dynamically weight and fuse spatiotemporal features and long-term temporal features to obtain fused features that accurately represent lesions; S4: Decode the fused features to output lesion candidate boxes, calculate the cross-frame lesion correlation degree through cosine similarity, and determine whether they are the same lesion; S5: Set a time-series continuous threshold. Only when the number of consecutive detection frames for a lesion meets the standard is it determined to be a real lesion. The final confidence level is then adjusted based on the frame effectiveness rate.

[0005] Furthermore, in step S1, the continuous video stream from the colon capsule endoscopy is divided into fixed lengths. Divided into time-series sliding windows ,in Indicates the first Frame images; each frame image undergoes normalization and noise reduction preprocessing to obtain standardized frames. .

[0006] Furthermore, in step S2, 3D-CNN is used to extract spatiotemporal joint features within the window. Meanwhile, long-term temporal dependency features are extracted by improving the temporal Transformer. The specific formula is as follows:

[0007] In the formula: For convolution operations, For 3D convolution kernel weights, For bias terms, It is the ReLU activation function. The total number of convolution kernels, For the first One convolutional kernel.

[0008] Furthermore, the formula for calculating the fusion feature in step S3 is as follows:

[0009]

[0010] In the formula For the first Frame-time attention weights It is a multilayer perceptron. For feature dot product operation, This is a feature channel splicing operation. Indicates the first in the sliding window The spatiotemporal joint features corresponding to the frame; Indicates the first in the sliding window Long-term temporal dependency features corresponding to frames; Indicates the first in the sliding window The spatiotemporal joint features corresponding to the frame; Indicates the first in the sliding window Long-term temporal dependency features corresponding to frames; It is the index variable for summation and traversal, representing any frame in the sliding window, and its value range is... ,Cooperate Summing the feature similarities of all frames within the window and performing Softmax normalization ultimately ensures the attention weights of all frames within the window. The sum is 1; This is a feature of fusion.

[0011] Furthermore, in step S4, the fused features are decoded to output candidate lesion boxes. ,in Indicates the first The first to m lesion candidate detection boxes are output after the frame image is decoded by fusion features. The correlation degree of lesions across frames is calculated by cosine similarity to determine whether they are the same lesion. The formula is as follows:

[0012] In the formula, For the first Frame number The depth feature vectors of each candidate bounding box For the first The depth feature vector of the j-th candidate box in frame, with a similarity threshold set. ,when If the same lesion is identified, only the detection frame with the highest confidence level is retained.

[0013] Furthermore, in step S5, a time-series duration threshold is set. A lesion is considered a true lesion only when the number of consecutive detection frames meets the standard. The final confidence score is then adjusted based on the frame effectiveness rate. The calculation formula is as follows:

[0014] In the formula The final confidence level for the lesion. The original detection confidence level for a single frame. The number of effective detection frames for lesions within the window. Set the total number of frames for the sliding window; set the judgment threshold. ,when At that time, the final lesion result is output.

[0015] The beneficial effects of this invention are as follows: This invention utilizes a temporal sliding window + 3D-CNN and Transformer dual-branch feature extraction to break through the limitations of single-frame detection, simultaneously capturing local spatiotemporal details and long-term temporal dependencies of lesions, thus fundamentally solving the problem of missed detection of small polyps; it strengthens the temporal correlation of lesions and weakens background interference through dynamic weight allocation, significantly improving the accuracy of lesion feature representation; it achieves accurate cross-frame matching of the same lesion based on cosine similarity, completely eliminating the problem of repeated detection; it combines the number of consecutive detection frames for confidence correction, effectively distinguishing real lesions from intestinal peristalsis artifacts and bubble interference; and it shares feature calculation results within the sliding window, avoiding repeated calculations and significantly improving the efficiency of video stream processing.

[0016] This invention relies on temporal feature deep modeling to effectively capture transiently blurred and locally occluded micropolyp features during intestinal peristalsis. Compared with traditional single-frame detection schemes, it significantly improves the detection rate of microlesions. At the same time, through cross-frame deduplication and temporal filtering dual mechanisms, it can eliminate most peristalsis artifacts and bubble interference, significantly reducing the false positive rate and duplicate detection rate. It is not only suitable for video stream scenarios with different intestinal preparation quality and capsule movement speed, but also significantly enhances its anti-interference ability against complex interferences such as light fluctuations and motion blur. In addition, the inter-frame feature reuse design effectively improves the overall processing efficiency of the video stream, which can meet the needs of clinical real-time screening. Moreover, the detection logic is consistent with the doctor's reading habits, which can greatly reduce the workload of manual review and reduce the risk of missed diagnosis and misdiagnosis. It is perfectly adapted to the entire process of non-invasive screening using colon capsule endoscopy.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of a video lesion detection method for colon capsule endoscopy based on temporal attention fusion. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0022] Example 1: This invention provides a method for lesion detection in colon capsule endoscopy videos based on temporal attention fusion. It achieves end-to-end lesion detection in the colon capsule endoscopy video stream through four core modules: sliding window spatiotemporal feature extraction, adaptive temporal attention fusion, cross-frame lesion feature matching, and temporal persistence constraint filtering. First, continuous video frames are processed by windowing to extract spatial lesion features and temporal motion features. Then, the dual features are fused through an attention mechanism to complete cross-frame lesion association matching. Finally, artifacts are removed through temporal persistence determination, outputting accurate lesion results. The algorithm for lesion detection in colon capsule endoscopy videos based on temporal attention fusion is as follows: Figure 1 As shown, the algorithm steps are as follows.

[0023] Step 1: Video windowing and preprocessing The continuous video stream from the colon capsule endoscopy was cut into fixed lengths. Divided into time-series sliding windows ,in Indicates the first Frame images; each frame image undergoes normalization and noise reduction preprocessing to obtain standardized frames. .

[0024] Step 2: Multi-scale spatiotemporal feature extraction Extracting within the window using 3D-CNN Spatiotemporal joint features, and extraction through an improved temporal Transformer Long-term time-dependent features, the specific formula is as follows:

[0025] In the formula: For convolution operations, For 3D convolution kernel weights, For bias terms, It is the ReLU activation function. This represents the total number of convolution kernels.

[0026] Step 3: Adaptive Temporal Attention Fusion Construct a temporal attention weight matrix to dynamically weight and fuse spatiotemporal features and long-term temporal features, thereby obtaining an accurate representation of lesions. The core formula for feature fusion is as follows:

[0027]

[0028] In the formula: For the first Frame-time attention weights It is a multilayer perceptron. For feature dot product operation, This is a feature channel splicing operation; Indicates the first in the sliding window The spatiotemporal joint features corresponding to the frame are extracted by a 3D-CNN branch, and also include the first... The spatial texture and contour information of frame lesions, as well as the temporal features such as motion and deformation between neighboring frames, represent local spatiotemporal details. The frame number represents the first frame within the window corresponding to this feature. frame( The range of values ​​is ); Indicates the first in the sliding window The long-term temporal dependency features corresponding to the frames are output after the temporal Transformer branch models the entire window sequence, capturing the long-range evolution patterns, motion trends, and inter-frame correlations of lesions in consecutive multiple frames, focusing on the representation of global temporal information. Similarly, the first one in the corresponding window frame. and Completely consistent with the meaning above, that is It is the index variable for summation and traversal, representing any frame in the sliding window, and its value range is also [value range missing]. .Cooperate Summing the feature similarities of all frames within the window and performing Softmax normalization ultimately ensures the attention weights of all frames within the window. The sum is 1. This is the final fusion feature.

[0029] Step 4: Cross-frame lesion feature matching and deduplication Decode the fused features to output lesion candidate boxes ,in Indicates the first The first to m lesion candidate detection boxes are output after the frame image is decoded by fusion features. The correlation degree of lesions across frames is calculated by cosine similarity to determine whether they are the same lesion. The formula is as follows:

[0030] In the formula: For the first Frame number The depth feature vectors of candidate boxes, when When the similarity threshold is reached, the same lesion is identified, and only the detection box with the highest confidence is retained.

[0031] Step 5: Temporal Duration Constraint Filtering Set the duration threshold A lesion is considered a true lesion only when the number of consecutive detection frames meets the standard. The final confidence score is then adjusted based on the frame effectiveness rate. The calculation formula is as follows:

[0032] In the formula: The final confidence level for the lesion. The original detection confidence level for a single frame. The number of effective detection frames for lesions within the window. The total number of frames in the sliding window; when When the (judgment threshold) is reached, the final lesion result is output.

[0033] Example 2: An electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method described in Embodiment 1 when executing the computer program.

[0034] Example 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0035] Example 4: A computer program product includes a computer program that, when executed by a processor, implements the method described in Example 1.

[0036] In the above embodiments, the reference to "this embodiment" in the specification indicates that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple appearances of "this embodiment" do not necessarily refer to the same embodiment.

[0037] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0038] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0039] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0040] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0041] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0042] This invention can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0043] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion, characterized in that: Includes the following steps: S1: Perform temporal windowing and preprocessing on the continuous video stream of colon capsule endoscopy; S2: 3D-CNN is used to extract spatiotemporal joint features within the window, while long temporal dependency features are extracted by improving the temporal Transformer; S3: Utilize the temporal attention weight matrix to dynamically weight and fuse spatiotemporal features and long-term temporal features to obtain fused features that accurately represent lesions; S4: Decode the fused features to output lesion candidate boxes, calculate the cross-frame lesion correlation degree through cosine similarity, and determine whether they are the same lesion; S5: Set a time-series continuous threshold. Only when the number of consecutive detection frames for a lesion meets the standard is it determined to be a real lesion. The final confidence level is then adjusted based on the frame effectiveness rate.

2. The method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion according to claim 1, characterized in that: In step S1, the continuous video stream from the colon capsule endoscopy is divided into fixed lengths. Divided into time-series sliding windows ,in Indicates the first Frame images; each frame image undergoes normalization and noise reduction preprocessing to obtain standardized frames. .

3. The method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion according to claim 1, characterized in that: In step S2, 3D-CNN is used to extract spatiotemporal joint features within the window. Meanwhile, long-term temporal dependency features are extracted by improving the temporal Transformer. The specific formula is as follows: In the formula: For convolution operations, For 3D convolution kernel weights, For bias terms, It is the ReLU activation function. The total number of convolution kernels, For the first One convolutional kernel.

4. The method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion according to claim 1, characterized in that: The formula for calculating the fusion feature in step S3 is as follows: In the formula For the first Frame-time attention weights It is a multilayer perceptron. For feature dot product operation, This is a feature channel splicing operation; Indicates the first in the sliding window The spatiotemporal joint features corresponding to the frame; Indicates the first in the sliding window Long-term temporal dependency features corresponding to frames; Indicates the first in the sliding window The spatiotemporal joint features corresponding to the frame; Indicates the first in the sliding window Long-term temporal dependency features corresponding to frames; It is the index variable for summation and traversal, representing any frame in the sliding window, and its value range is... ,Cooperate Summing the feature similarities of all frames within the window and performing Softmax normalization ultimately ensures the attention weights of all frames within the window. The sum is 1; This is a feature of fusion.

5. The method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion according to claim 1, characterized in that: In step S4, the fused features are decoded to output candidate lesion boxes. ,in Indicates the first The first to m lesion candidate detection boxes are output after the frame image is decoded by fusion features. The correlation degree of lesions across frames is calculated by cosine similarity to determine whether they are the same lesion. The formula is as follows: In the formula, For the first Frame number The depth feature vectors of each candidate bounding box For the first The depth feature vector of the j-th candidate box in frame, with a similarity threshold set. ,when If the same lesion is identified, only the detection frame with the highest confidence level is retained.

6. The method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion according to claim 1, characterized in that: In step S5, the timing duration threshold is set. A lesion is considered a true lesion only when the number of consecutive detection frames meets the standard. The final confidence score is then adjusted based on the frame effectiveness rate. The calculation formula is as follows: In the formula The final confidence level for the lesion. The original detection confidence level for a single frame. The number of effective detection frames for lesions within the window. Set the total number of frames for the sliding window; set the judgment threshold. ,when At that time, the final lesion result is output.

7. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the colon capsule endoscopy video lesion detection method based on temporal attention fusion as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion as described in any one of claims 1-6.

9. A computer program product, characterized in that: The method includes a computer program that, when executed by a processor, implements the method for detecting lesions in colonic capsule endoscopy video based on temporal attention fusion as described in any one of claims 1-6.

Images