Confocal laser endoscope video redundancy removing and filtering method, device and system

Abstract

The invention relates to a confocal laser endoscope video redundancy removing and filtering method, device and system, and relates to the technical field of computers, and the method comprises the following steps: receiving an input confocal laser endoscope CLE video clip; a video clip is processed through a parallel multi-path feature coding architecture, wherein the architecture comprises a microscopic feature coding path, a macroscopic space-time coding path and a time sequence stability coding path; dynamically integrating the feature vectors output by the three paths to generate a fused feature vector; on the basis of the fusion feature vector, a weighted ordinal regression classifier is used for generating classification output, the classifier considers the sequence relation between diagnostic value levels and allocates weights for different categories to deal with the problem of data imbalance, and the classification result accurately corresponds to a predefined operator cognitive intention stage. According to the method, the technical problem that key diagnosis fragments in confocal laser endoscope videos cannot be accurately and efficiently screened due to the design of'literal blindness' and'single integrality 'in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to the fields of artificial intelligence and computer vision. More specifically, this invention relates to a method, apparatus, and system for automated analysis of medical video data, particularly suitable for processing and evaluating videos generated by confocal laser endoscopy (CLE). Background Technology

[0002] Confocal laser endoscopy (CLE), a revolutionary medical imaging technology, provides real-time, cellular-level in vivo tissue images during routine endoscopic examinations, earning it the reputation of "optical biopsy." It has shown great potential in monitoring precancerous lesions such as Barrett's esophagus and inflammatory bowel disease, effectively guiding physicians to perform targeted biopsies, reducing unnecessary random biopsies, and providing unprecedented opportunities for the early and accurate diagnosis of diseases.

[0003] However, the clinical application potential of CLE technology is facing a severe challenge: data review overload and diagnostic fatigue caused by massive amounts of video data. A typical CLE examination can generate approximately 4,500 frames of images in just 15 minutes. This explosive growth in data is impacting clinicians' workflows in an unprecedented way and reveals a common dilemma in the field of modern medical imaging. In other fields such as radiology, the exponential increase in the number and complexity of imaging examinations has become a core driver of physician burnout and diagnostic fatigue. Manually reviewing CLE videos, which are filled with a large amount of non-diagnostic information, is particularly cognitively demanding. Physicians need to perform detailed interpretations at the cellular level, which is inherently a very demanding task. The dynamic nature of the alternating static and dynamic images in the video makes it more likely to cause fatigue than reviewing static images, adding unnecessary external load. Furthermore, the massive sequence images (typically reaching thousands) generated by CLE systems in clinical examinations currently present two main problems: first, a large number of blurred images are caused by factors such as gastrointestinal peristalsis, respiratory movements, and unstable contact between the lens and the mucosa, resulting in a loss of detail and rendering them unusable for clinical diagnosis; second, the multiple frames continuously captured during video stream acquisition of the same microscopic region contain highly redundant content. Therefore, developing an intelligent system capable of automatically filtering out key diagnostic fragments has become an urgent technological need concerning physician well-being, patient safety, and the sustainability of the healthcare system.

[0004] Preliminary explorations have been made in existing technologies to address this problem, but they suffer from the following fundamental drawbacks: Limitation 1: The "Context-Blindness" of Existing Technologies. Early CLE image analysis methods primarily focused on Single-Frame Image Quality Assessment (SF-IQA). These methods detect motion artifacts by analyzing the similarity of adjacent row vectors in an image, or estimate noise intensity by modeling spatially correlated noise, and use machine learning models such as Support Vector Regression (SVR) to score the overall quality of a single frame. While these SF-IQA methods contribute to quantifying whether a single frame is "sharp," their fundamental limitation lies in their "context-blindness." The reliability of clinical diagnosis depends not only on the quality of a single frame but also on the continuity and stability of a video segment over time. For example, in a typical "movement and search" phase, a brief, isolated, sharp image may appear, but if the frames before and after it are blurry and unstable, the entire segment cannot be used for reliable diagnosis. Conversely, a video segment with high diagnostic value in the "focus and diagnosis" phase has all frames that are continuous, stable, and sharp. The SF-IQA method, lacking the ability to model temporal dynamics, cannot distinguish between these two scenarios and therefore fundamentally fails to solve the screening challenges in clinical workflows. This limitation stems from a mischaracterization of the technical problem: existing technologies treat the task as an "image quality assessment" problem, while the true clinical need concerns "workflow efficiency and diagnostic relevance." A clinician needs not fragmented "clear images," but "diagnostically valuable video clips," the latter inherently including temporal stability. Therefore, any technique operating only at the single-frame level, no matter how complex its quality assessment model, cannot meet actual clinical needs due to its inherent "context-blind" nature.

[0005] Defect 2: Suboptimal Nature of General Video Classification Models. In recent years, deep learning has achieved great success in general video classification tasks, such as 3D convolutional neural networks (3D-CNNs) like I3D, and Transformer-based video models like TimeSformer and Swin 3D. While these general video models are powerful, they typically employ a "monolithic" design, treating the video as a single, homogenized input. This "one-size-fits-all" approach is not optimal for processing CLE videos with their unique internal structures. CLE videos contain heterogeneous dimensions of information with drastically different characteristics and scales: for example, microscopic texture information such as cell nuclear morphology and glandular structure; macroscopic content evolution information such as extensive movement of the mucosal surface; and temporal stability information determined by the operator's probe movement patterns. Mixing these heterogeneous information makes it difficult for the model to learn the more discriminative and subtle features needed to differentiate between different clinical intentions. A deeper problem lies in the "signal dilution" effect. In a typical CLE examination video, most of the time is spent in the "entry and navigation" or "movement and search" phase, characterized by vigorous and wide-ranging motion. However, the clinically crucial "focusing and diagnosis" phase, characterized by probe stability and clear microscopic texture, is actually a "minority signal" in the overall video signal. When a single holistic model processes this signal, the strong motion signal becomes dominant, "diluting" or "averaging out" the relatively weak but vital stability signal. This leads the model to learn a mixed, non-discriminatory feature representation, ultimately classifying based on incorrect (i.e., motion-related) cues. Therefore, a specialized architecture capable of explicitly separating and independently analyzing these heterogeneous signals is essential to prevent signal dilution and accurately capture the operator's diagnostic intent.

[0006] In summary, existing technologies either fail to understand clinical needs due to "context blindness" or cannot effectively handle the heterogeneous signals in CLE videos due to their "single-system" design. Consequently, they cannot provide a technical solution that can automatically, accurately, and efficiently analyze CLE videos and filter out truly diagnostically valuable segments. Therefore, there is an urgent need in the field for a novel technical solution that overcomes these shortcomings by understanding the inherent clinical logic of the video and the operator's intent to solve the data overload problem. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a medical video analysis method, device and system based on the operator's cognitive intent. It aims to solve the technical problem that the prior art cannot accurately and efficiently screen key diagnostic segments in confocal laser endoscopy (CLE) videos due to "context blindness" and "single holistic" design.

[0008] To achieve the above objectives, the present invention provides a method for deduplication of confocal laser endoscopy video, comprising the following steps: Receive input confocal laser endoscopy (CLE) video clips; Video segments are processed through a parallel multi-path feature coding architecture, which includes three paths: micro-feature coding path, macro-spatiotemporal coding path, and temporal stability coding path. The feature vectors output from the three paths are dynamically integrated to generate a fused feature vector; Based on the fused feature vector, a weighted ordinal regression classifier is used to generate classification output. The classifier considers the order relationship between diagnostic value levels and assigns weights to different categories to deal with the data imbalance problem. Its classification results accurately correspond to the predefined operator cognitive intention stage. Among them, the micro-feature encoding path uses a convolutional neural network (CNN) to independently process each image frame to extract frame-level feature vectors, and employs a multi-instance learning (MIL) pooling strategy based on an attention mechanism to weight and aggregate them to generate fine-grained spatial texture features for recognizing histological patterns. The macro-spatiotemporal coding path adopts a hierarchical Transformer architecture. First, the spatial features of each frame are extracted independently by a spatial encoder with shared weights. Then, the spatial features of all frames are processed by the temporal fusion module to model the temporal dependencies across frames, thereby capturing the global scene dynamics and long-range temporal dependencies across frames in the video. The temporal stability coding path calculates the inter-frame similarity matrix based on the deep semantic features extracted from each image frame, and then extracts dynamic patterns or global statistics from them through at least one parallel processing path to quantify the feature-level similarity between consecutive frames and generate stability features that represent the operator's cognitive intent.

[0009] Preferably, in the micro-feature encoding path, a ResNet18 network is used as the backbone network, and the processing flow includes: Frame-level feature extraction: Extract a high-dimensional frame-level feature vector from each frame through the ResNet18 backbone network; Fragment-level feature aggregation: A multi-instance learning pooling strategy based on attention mechanism is adopted to calculate the attention weight and gating signal of each frame-level feature, and obtain the fragment-level micro feature vector by weighted summation of all gating-adjusted frame-level features.

[0010] Preferably, in the macroscopic spatiotemporal encoding path, a layered Transformer architecture is adopted, and the processing flow includes: Frame-by-frame spatial coding: Spatial features are extracted from each frame using a shared-weight spatial encoder to generate a spatial feature vector; Cross-frame temporal fusion: Combine the spatial feature vectors of all frames into a sequence and input it into the temporal fusion module to model the temporal dynamics across frames and output feature vectors that encode the macroscopic dynamics of the entire segment.

[0011] Preferably, in the temporal stability coding path, features are extracted from the inter-frame similarity matrix, and the processing flow includes: Inter-frame similarity matrix calculation: Calculate the cosine similarity matrix between frames using high-dimensional semantic features; Multi-path feature extraction: Features are extracted through three paths: dynamic pattern encoding, global statistical encoding, and contextual feature enhancement. Feature fusion and output: Features from the three paths are concatenated and fused through a multilayer perceptron to generate a feature vector that ultimately encodes the stability of the entire segment.

[0012] Preferably, the cross-attention fusion module adopts a three-way cross-attention fusion mechanism, including: Feature space alignment: Projecting the feature vectors of the three branches into the same 256-dimensional space; Multi-directional cross-enhancement: Each feature stream vector is enhanced by using the other two feature streams as context through a multi-head attention mechanism; Final deep fusion: The original projected features and the cross-enhanced features are concatenated and then deep fused and dimensionality reduced through a multilayer perceptron to obtain the final fused feature vector.

[0013] The technical solution of the present invention also provides a confocal laser endoscopy video redundancy filtering device, comprising: The input module is used to receive video segments consisting of a series of image frames; The micro-feature encoding module is used to extract fine-grained spatial texture features from a single image frame for identifying histological patterns. The macro-spatiotemporal coding module is used to model the global scene dynamics and cross-frame temporal dependencies of video clips; The temporal stability coding module is used to generate stability features that represent the operator's cognitive intent by quantifying the feature-level similarity between a series of image frames. The feature fusion module is used to fuse features from the three encoding modules to generate a fused feature vector; The classification module is used to generate classification outputs based on the fused feature vectors.

[0014] Preferably, the micro-feature encoding module includes: Convolutional Neural Networks (CNNs) are used to process each image frame independently to extract frame-level feature vectors. The multi-instance learning MIL pooling layer is used to perform weighted aggregation of frame-level feature vectors to generate spatial texture features.

[0015] Preferably, the macroscopic spatiotemporal coding module includes: The hierarchical Transformer architecture, including a spatial encoder and a temporal fusion module, is used to model the global dynamics and long-range temporal dependencies of video segments.

[0016] The technical solution of the present invention also provides a confocal laser endoscope video redundancy filtering system, comprising: Confocal laser endoscopy video redundancy filtering device; The processing unit is used to automate the workflow of the intelligent classification stage and the intelligent deduplication stage. In the intelligent classification stage, the processing unit calls the device to process the original video stream and classifies and stores the video frames according to the classification results; in the intelligent deduplication stage, the processing unit applies a deduplication algorithm to the video frames with high diagnostic value to generate a set of key frames with high information density and no redundancy.

[0017] The preferred intelligent deduplication stage processing flow includes: Use a deep learning feature extractor to extract deep semantic feature vectors for each frame; Within a user-defined time window, the cosine similarity between feature vectors is calculated, and consecutive frames with highly similar content are grouped together. Within each group of similar frames, the sharpness of each frame is quantified by calculating the variance of the Laplacian operator, and the sharpest frame is retained as the representative.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention significantly improves the accuracy of confocal laser endoscopy (CLE) video analysis through an innovative "divide and conquer" parallel architecture and direct modeling of operator intent. On real clinical datasets, the core evaluation metric, the macro-average F1 score, reached 84.26%, an improvement of over 7.3 percentage points compared to current advanced general-purpose video classification models (such as I3D), with inference speed more than 9 times faster. Prototype system validation shows that this method achieves a data reduction rate of over 95%, effectively reducing the review burden on physicians and revolutionizing clinical workflows. This significant performance improvement enables the invention to more accurately identify video segments with clinical diagnostic value, thereby effectively improving the screening capability and diagnostic accuracy of early precancerous lesions.

[0019] The parallelized and specialized architecture of this invention excels in optimizing computational resource allocation, achieving superior computational efficiency. Its inference speed reaches approximately 5601.7 FPS (frames per second), more than 9 times faster than traditional 3D-CNN models (such as I3D) and about 2.5 times faster than advanced Transformer models (such as TimeSformer). This dual advantage in accuracy and efficiency makes this invention ideally suited for deployment in clinical scenarios requiring rapid processing of large amounts of video data. It can output selected, diagnostically significant image sequences in real-time or near real-time, significantly reducing video review time and accelerating clinical decision-making.

[0020] The system and method proposed in this invention directly solve the problems of data overload and diagnostic fatigue caused by massive amounts of video data. In an application prototype simulating a real clinical workflow, this invention achieves an average data simplification rate of up to 95.89%, condensing videos that originally required a lot of time to review into a set of key images with a controllable number and high information density. This not only reduces the number of images that doctors need to review manually by more than 90%, fundamentally solving the problem of review fatigue caused by data overload, but also frees doctors from high-intensity, repetitive visual screening work, allowing them to concentrate their valuable cognitive resources on the most diagnostically valuable cell morphology interpretation, effectively reducing the risk of burnout and improving diagnostic efficiency and the level of intelligence in clinical workflows.

[0021] This invention also has significant social and economic benefits. It directly addresses the core bottlenecks in the clinical promotion of CLE technology, clearing obstacles to its large-scale, routine application. By deeply integrating artificial intelligence technology with cutting-edge medical imaging equipment, this invention not only enhances the value of a single examination but also strengthens the overall screening capability for early precancerous lesions, potentially leading to earlier diagnosis, more timely treatment, and better patient prognosis, thus generating substantial social benefits. Simultaneously, this system has the potential to be integrated into existing medical information systems, serving as a highly efficient software tool with broad market prospects and economic benefits, and is of vital importance to promoting the development of precision medicine. Attached Figure Description

[0022] Figure 1 This is a comparative schematic diagram of CLE video segments with different diagnostic values ​​used to clarify technical problems in a confocal laser endoscopy video redundancy filtering method, device and system of the present invention; Figure 2 This is a schematic block diagram of the overall MSTA-Net architecture in a confocal laser endoscopy video redundancy filtering method, device, and system according to the present invention. Figure 3This is a detailed structural block diagram of the macroscopic spatiotemporal coding module in a confocal laser endoscopy video redundancy filtering method, device, and system according to the present invention. Figure 4 This is a detailed structural block diagram of the timing stability encoding module in a confocal laser endoscopy video redundancy filtering method, device, and system according to the present invention. Figure 5 This is a flowchart illustrating the two-stage clinical application workflow of a confocal laser endoscopy video redundancy filtering method, device, and system according to the present invention. Figure 6 This illustration shows the ablation study results contributed by each architectural component in the confocal laser endoscopy video redundancy filtering method, device and system of the present invention. Figure 7 This is a schematic diagram of the test results in the method, apparatus and system for deduplication of confocal laser endoscopy video. Figure 8 This is a logical diagram illustrating the two-stage automated workflow of "filtering-summarizing" in a confocal laser endoscopy video redundancy filtering method, device, and system according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention discloses a method, device, and system for deduplication of confocal laser endoscopy (CLE) video, involving a multimodal spatiotemporal sensing network (MSTA-Net). Its core design philosophy is "divide and conquer," aiming to solve the problem of automated screening of key diagnostic segments in CLE video by modeling the operator's cognitive intent.

[0025] The overall architecture of MSTA-Net includes three parallel encoder branches, a three-way cross-attention fusion module, and an ordinal regression classifier. This design, at the computational level, simulates the multi-dimensional, hierarchical cognitive process used by clinical experts when reviewing videos. First, evidence is evaluated from different perspectives, then all information is synthesized for intelligent arbitration, ultimately leading to a conclusion.

[0026] Example 1: This invention discloses a method for deduplication of confocal laser endoscopy video, which includes the following steps: It receives input confocal laser endoscopy (CLE) video clips, which consist of a series of image frames.

[0027] The processing is achieved through a parallel multi-path feature encoding architecture, which includes three specialized paths: a micro-feature encoding path, a macro-spatiotemporal encoding path, and a temporal stability encoding path.

[0028] The micro-feature encoding path uses a convolutional neural network (CNN) to process each image frame independently to extract frame-level feature vectors, and employs a multi-instance learning (MIL) pooling strategy based on an attention mechanism to weight and aggregate them, generating fine-grained spatial texture features for identifying histological patterns, thereby identifying diagnostically significant histological patterns.

[0029] The macro-spatiotemporal coding path adopts a hierarchical Transformer architecture. First, the spatial features of each frame are extracted independently by a spatial encoder with shared weights. Then, the spatial features of all frames are processed by the temporal fusion module to model the temporal dependencies across frames, thereby capturing the global scene dynamics and long-range temporal dependencies across frames and understanding the overall motion context.

[0030] The temporal stability coding path calculates the inter-frame similarity matrix based on the deep semantic features extracted from each image frame, and then extracts dynamic patterns or global statistics from them through at least one parallel processing path to quantify the feature-level similarity between consecutive frames, generating stability features that represent the operator's cognitive intent, which serve as a technical proxy for the operator's cognitive intent.

[0031] The cross-attention fusion module dynamically integrates the feature vectors output from the three paths. The features of any path are used as context by the features of other paths. After dynamic enhancement and weighting by the multi-head attention mechanism, the original features are combined with the enhanced features to generate a fused feature vector.

[0032] Based on the fused feature vectors, a weighted ordinal regression classifier is used to generate classification outputs. This classifier considers the order relationship between diagnostic value levels and assigns weights to different categories to address the data imbalance problem. Its classification results accurately correspond to the predefined operator cognitive intent stages, and the predefined diagnostic value levels include a "high diagnostic value" level corresponding to the "focus and diagnosis" cognitive intent stage, a "mixed value" level corresponding to the "movement and search" cognitive intent stage, and a "low diagnostic value" level corresponding to the "entry and navigation" cognitive intent stage.

[0033] Specifically, the input is a video clip consisting of T frames of grayscale images, represented as a tensor. In a preferred embodiment, T=4 is a fixed number of frames, and H and W are the height and width of the image (e.g., 224x224 pixels), respectively. The t-th frame in the video is represented as... .

[0034] The video clip was fed in parallel into three specialized encoding modules: a micro-feature encoding module. Macro-spatial coding module and timing stability coding module These three encoding modules extract features from three dimensions: micro-texture, macro-dynamics, and temporal stability, respectively. Subsequently, the three feature vectors are fed into a cross-attention fusion module for deep interaction and fusion. Finally, the fused feature vectors are fed into a weighted ordinal regression classifier, which outputs a diagnostic value rating for the video segment, for example, categorized into three classes: "low diagnostic value," "mixed value," and "high diagnostic value" (i.e., number of classes K=3).

[0035] Example 2: Example 2 further discloses a confocal laser endoscopy video redundancy filtering device based on Example 1. The core of this device lies in its multiple functional modules, which work together to implement the confocal laser endoscopy video redundancy filtering method (automated screening method) described in Example 1.

[0036] The core modules of the device include an input module, three parallel coding modules (microscopic feature coding module, macroscopic spatiotemporal coding module, and temporal stability coding module), a feature fusion module, and a classification module.

[0037] The input module is used to receive video segments consisting of a series of image frames.

[0038] The three encoding modules work in parallel to extract deep features from three dimensions: spatial texture, global dynamics, and temporal stability.

[0039] The micro-feature encoding module is used to extract fine-grained spatial texture features for identifying histological patterns from a single image frame. It uses a convolutional neural network (CNN) to process each image frame independently to extract frame-level feature vectors, and uses a multi-instance learning (MIL) pooling strategy based on an attention mechanism to weight and aggregate the frame-level feature vectors to generate spatial texture features.

[0040] The macro-spatiotemporal coding module is used to model the global scene dynamics and cross-frame temporal dependencies of video clips. It is based on a hierarchical Transformer architecture. First, the spatial features of each frame are extracted independently by a spatial encoder with shared weights. Then, the spatial features of all frames are processed by the temporal fusion module to model the cross-frame temporal dependencies.

[0041] The temporal stability coding module is used to generate stability features representing the operator's cognitive intent by quantifying the feature-level similarity between a series of image frames. It first calculates the inter-frame similarity matrix based on the deep semantic features extracted from each image frame, and then extracts dynamic patterns or global statistics from the similarity matrix through at least one parallel processing path to generate stability features as a technical proxy for the operator's cognitive intent.

[0042] The feature fusion module is used to fuse features from the three encoding modules to generate a fused feature vector. It adopts a cross-attention fusion mechanism, so that features from any encoding path are used as context by features from at least one other encoding path. Dynamic enhancement and weighted fusion are performed through a multi-head attention mechanism. Then, the original features are combined with the enhanced features to generate the fused feature vector.

[0043] The modules collaborate with each other. After the input module receives the video segment, the three encoding modules work in parallel to extract microscopic, macroscopic, and temporal features, respectively. The feature fusion module combines these features, and finally, the classification module generates a diagnostic value rating. This design not only improves the accuracy of video analysis but also effectively addresses the problem of data imbalance, enabling automated screening of key diagnostic segments in CLE videos.

[0044] The classification module generates a classification output corresponding to a predefined diagnostic value level for a video segment based on the fused feature vector. It uses a weighted ordinal regression classifier that considers the order relationship between diagnostic value levels and assigns weights to different categories to address the data imbalance problem. The predefined diagnostic value levels include a "high diagnostic value" level corresponding to the "focus and diagnosis" cognitive intent stage, a "mixed value" level corresponding to the "mobile and search" cognitive intent stage, and a "low diagnostic value" level corresponding to the "entry and navigation" cognitive intent stage.

[0045] Specifically, the microscopic feature encoding module aims to answer the question, "Does the image contain high-quality histological details?" Its core objective is to identify the microscopic histological patterns required for the "focusing and diagnosis" stage, such as cell nuclear morphology and glandular structures. In this embodiment, the module selects a convolutional neural network (CNN) as its backbone network due to its strong inductive bias, which gives it a natural advantage in capturing local spatial patterns. Specifically, a ResNet18 network pre-trained on the ImageNet dataset can be used as the backbone. The processing flow is as follows: 1. Frame-level feature extraction: Each frame in a video clip (in First, a high-dimensional frame-level feature vector is extracted independently through the ResNet18 backbone network. .

[0046] 2. Fragment-level Feature Aggregation: To aggregate an information-rich fragment-level feature vector from the features of all frames, this embodiment employs an attention-based multi-instance pooling (MILPooling) strategy. This strategy uses an attention pooling layer with a gating mechanism to aggregate information from each frame-level feature vector. Calculate attention weights and a gating signal The final fragment-level micro-feature vector The weighted summation of all gated frame-level features is obtained as shown in formula (1): (1) Here, ⊙ represents element-wise multiplication. In this way, the module not only identifies "what's in the image," but also uses attention weights. The importance of each frame was quantified and controlled by gating signals. Its relevance was assessed, providing dual-screened micro-evidence for the final decision.

[0047] Specifically, the macro-spatiotemporal coding module aims to capture the overall scene of a video clip and its changes over time, answering the question, "What is the global dynamics of the scene?" It is used to capture large-scale scene changes during the "movement and search" phase. To effectively model the global dynamics and long-range dependencies of video clips, this embodiment employs a layered Transformer architecture to finely decouple the spatial and temporal dimensions. The processing flow of this module is as follows: 1. Frame-by-frame spatial coding: First, a weighted, patch-based spatial encoder independently extracts spatial features from each frame of the video.

[0048] Specifically, each frame First, the image is divided into N fixed-size (e.g., 16x16) blocks. This sequence is then concatenated with a learnable classification token. Subsequently, the sequence is linearly embedded and positionally encoded to obtain the initial tokens. The sequence then proceeds through... Layers (e.g.) =2) The spatial Transformer encoder is used for processing, as shown in formulas (2) and (3): (2) (3) Here, MSA, FFN, and LN represent multi-head self-attention, feedforward network, and layer normalization, respectively. Ultimately, a compact spatial feature vector is generated for each frame. .

[0049] 2. Cross-frame temporal fusion: Spatial feature vectors of all frames They are combined into a sequence and fed into a database containing... One (e.g.) The Transformer module's Temporal Fusion module models the temporal dynamics across frames. Ultimately, this module outputs a feature vector encoding the macroscopic dynamics of the entire segment. .

[0050] Specifically, the temporal stability coding module is designed to directly answer the question, "Is the operator's behavior stable or mobile?" It transforms the abstract concept of "operator's intention" into a computable quantifiable feature, providing the most direct signal to distinguish between "entry and navigation" (high instability) and "focusing and diagnosis" (high stability). This module is the key technical implementation for realizing the core idea of ​​this invention—namely, modeling the operator's cognitive intention. The processing flow of this module is as follows: Inter-frame similarity matrix calculation: First, the high-dimensional semantic frame-level features extracted by the micro-feature encoding module in Example 1 are used. To calculate the cosine similarity matrix between frames Each element The calculation method is shown in formula (4): (4) These high-dimensional semantic features were chosen as the computational basis because they are more robust to low-level changes such as illumination variations and minute displacements, and can more accurately reflect the stability at the content level. Multi-path feature extraction: Subsequently, features are extracted from the similarity matrix through three parallel paths to comprehensively capture stability information. (a) Temporal Pattern Encoding: This path uses independent bidirectional Long Short-Term Memory (LSTM) networks to process the similarity sequences of adjacent frames separately. Similarity sequence with interval frames To capture dynamic patterns in time series.

[0051] (b) Global Statistics Encoding: This path directly calculates global statistics such as the mean and standard deviation of the similarity matrix M to obtain a macroscopic description of the overall stability.

[0052] (c) Feature Enhancement: This path enhances the original frame feature sequence. Input a multi-head self-attention layer to capture broader contextual relationships and enhance features.

[0053] Feature fusion and output: Features from the three paths mentioned above are concatenated and fused using a multilayer perceptron (MLP) to generate a final feature vector that encodes the stability of the entire segment. As shown in formula (5): (5) Specifically, simple feature splicing cannot achieve dynamic interaction between different information dimensions. To address this, the present invention designs a three-way cross-attention fusion module to simulate a dynamic "evidence arbitration" process and intelligently integrate multi-source evidence from three parallel encoders.

[0054] The processing flow of this module is as follows: 1. Feature Space Alignment: First, the feature vectors of the three branches are aligned using independent linear layers. , , Projecting onto the same 256-dimensional space, we get , , ′.

[0055] 2. Multi-directional cross-enhancement: Next, a comprehensive multi-directional enhancement process is implemented. Each feature stream is enhanced using the other two feature streams as context through a multi-head attention (MHA) mechanism. For example, the process of using macroscopic features to enhance microscopic features is shown in Equation (6): (6) Here, Q, K, and V represent the query, key, and value, respectively. This process generates a total of 6 enhanced feature vectors.

[0056] 3. Final Deep Fusion: Finally, the original three projected features and six cross-enhanced features are concatenated, and then deep fusion and dimensionality reduction are performed through an MLP to obtain the final fused feature vector. .

[0057] This architecture allows the model to respond to, for example, “stable” intent signals (from... ), dynamically enhancing the micro-texture (from ) and macro context (from The perception of ) . For example, when the stability signal is strong, the model will place more trust in the detailed information provided by the micro-feature encoder. Such as the results of ablation studies ( Figure 6 As shown in the figure, using the cross-attention fusion module, compared with simple feature concat-fusion, the macro average F1 score is improved from 83.10% to 84.26%, which proves the effectiveness of this dynamic interaction mechanism and is a non-obvious inventive step to achieve the high performance of the present invention.

[0058] Specifically, in order to fuse the feature vectors To convert the data into a final classification result and effectively handle the order relationship between categories (low value < mixed value < high value) and the problem of data imbalance, this embodiment constructs a weighted ordinal regression classifier. The implementation and training method of this classifier are as follows: 1. Task Transformation: Traditional cross-entropy loss ignores the order relationship between categories. To address this issue, this embodiment transforms the K-class classification task into K-1 binary classification subtasks. The fused feature vector... After passing through a classifier (an MLP), it outputs K-1=2 logit values. .

[0059] 2. Weighted Ordinal Loss: To address class imbalance (e.g., high-value segments only account for 17%), weighted ordinal loss is applied to each true class. Pre-calculate a balance weight that is inversely proportional to its frequency. For a true level of The cumulative target label of the sample is , where if and only if hour Weighted ordinal loss The definition is shown in formula (7): (7) in, It represents the model's cumulative predicted probability. The design of this loss function not only imposes a greater penalty on predictions that deviate further from the true level, but also through class weights. It effectively mitigated the impact of data imbalance.

[0060] 3. Reasoning Stage: In the reasoning stage, the level of prediction... It is the number of thresholds with a cumulative probability greater than 0.5, as shown in formula (8): (8) in It is an indicator function.

[0061] Example 3: Example 3, based on Example 2, further discloses a confocal laser endoscopy video redundancy filtering system. This system includes the confocal laser endoscopy video redundancy filtering device (automated screening device) from Example 2, and a processing unit. This processing unit implements an efficient two-stage automated workflow, specifically including an intelligent classification stage and an intelligent deduplication stage.

[0062] During the intelligent classification phase, the processing unit invokes an automated screening device to process the raw video stream, categorizing and storing video frames based on the classification results (e.g., high, mixed, or low diagnostic value). This allows video frames to be organized in an orderly manner according to their diagnostic value level, facilitating subsequent targeted analysis and processing, and providing a preliminary structured data foundation for clinical review. The system workflow is as follows: the user specifies the folder of raw CLE videos to be processed through the data input interface.

[0063] During the intelligent deduplication stage, the processing unit applies a deduplication algorithm specifically to video frames classified as having high diagnostic value, ultimately generating a set of representative key frames with high information density and no redundancy. This greatly improves the efficiency and accuracy of clinical review, helping doctors to obtain key information more quickly and accurately, thereby making more reliable diagnostic decisions.

[0064] This system, through a two-stage automated workflow, fully leverages the functional advantages of automated screening devices, realizing a complete process from video frame classification to key frame extraction, and providing strong support for the efficient analysis and application of medical videos.

[0065] This invention integrates the aforementioned MSTA-Net into a practical tool, transforming its technical value into actionable clinical value through a novel two-stage automated workflow of "filter-then-summarize". The system includes a data input interface, a processing unit that executes the MSTA-Net model, and a data output interface. Its workflow is as follows: Phase 1: Intelligent Classification Based on MSTA-Net (Intent-Based Filtering) 1. Users specify the original CLE video folder to be processed through the data input interface.

[0066] 2. After the system starts, the processing unit uses a sliding window strategy (e.g., a window size of 4 frames and a step size of 1 frame) to extract video segments from the original video.

[0067] 3. Each video segment is input into the MSTA-Net device for inference to obtain its diagnostic value category (high, mixed, low).

[0068] 4. Based on the predicted category, the system automatically categorizes all frames in the video clip and saves them to the corresponding subfolder specified by the data output interface. This stage is the core filter, which utilizes the most important signal in the information hierarchy—the operator's intent—to quickly separate potentially diagnostically valuable (high-value) parts of the massive amount of data from a large amount of irrelevant information.

[0069] Phase Two: Intelligent Deduplication of High-Value Frames (Summary by Content) After the first stage of screening, the content in the "high-value" folder, while highly relevant, may still contain a significant amount of redundancy (e.g., due to the probe's long-term, stable observation of the same area). The second-stage algorithm then uses secondary but equally important signals in the information hierarchy—visual content and image quality—for intelligent simplification.

[0070] 1. For all frames in the “high-value” folder, the processing unit uses a deep learning feature extractor (e.g., reusing the ResNet backbone network in Example 1) to extract a deep semantic feature vector for each frame.

[0071] 2. Within a user-defined time window (e.g., 20 consecutive frames), consecutive frames with highly similar content (similarity higher than a preset threshold) are grouped together by calculating the cosine similarity between feature vectors.

[0072] 3. In each group of similar frames, the sharpness of each frame is quantified by calculating the variance of the Laplacian operator, and only the sharpest frame is retained as the sole representative of the group, while the rest of the redundant frames are discarded.

[0073] This strategy, which combines semantic similarity, temporal locality, and image sharpness, ensures that the final result presented to doctors is not only highly diagnostic but also a summary with the highest information density and no redundancy.

[0074] To quantify the system's actual effectiveness in addressing the core clinical problem of "data overload," this invention tested four unfiltered real clinical CLE examination videos. For example... Figure 7 As shown, the system's automated processing flow achieves a significant data simplification effect.

[0075] The quantitative results demonstrate the practical value of the system of this invention. By condensing videos that originally required tens of minutes to review into a controllable number of key images with high information density, this invention successfully transforms abstract algorithm performance indicators (such as F1 scores) into measurable clinical benefits, namely, an average data filtering capability of over 95%. This directly addresses the urgent clinical need to reduce physician diagnostic fatigue and improve diagnostic efficiency, providing a concrete and efficient example for the successful application of advanced AI technology in the field of endoscopic image analysis.

[0076] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of co-focal laser endomicroscopy video de- cluttering, characterized in that, The method comprises the following steps: receiving an input confocal laser endomicroscopy (CLE) video clip; processing the video clip through a parallel multi-path feature encoding architecture, the architecture comprising three paths: a microscopic feature encoding path, a macroscopic spatio-temporal encoding path, and a temporal stability encoding path; dynamically integrating feature vectors output by the three paths to generate a fused feature vector; based on the fused feature vector, using a weighted ordinal regression classifier to generate a classification output, the classifier considering the order relationship between the diagnostic value grades, assigning weights to different categories to address the data imbalance problem, and the classification result accurately corresponding to the pre-defined operator cognitive intention stage; wherein the microscopic feature encoding path uses a convolutional neural network (CNN) to independently process each image frame to extract a frame-level feature vector, and adopts a multiple-instance learning (MIL) pooling strategy based on an attention mechanism to perform weighted aggregation, generating fine-grained spatial texture features for identifying histological patterns; the macroscopic spatio-temporal encoding path adopts a hierarchical Transformer architecture, first independently extracting the spatial features of each frame through a shared weight spatial encoder, then processing the spatial features of all frames through a temporal fusion module to model the temporal dependence across frames, thereby capturing the global scene dynamics and long-range temporal dependence across frames of the video; the temporal stability encoding path calculates an inter-frame similarity matrix based on deep semantic features extracted from each image frame, and then extracts dynamic patterns or global statistics from the matrix through at least one parallel processing path to quantify the feature-level similarity between consecutive frames, generating stability features representing the operator's cognitive intention.

2. The method of claim 1, wherein the method further comprises: In the microscopic feature encoding path, a ResNet18 network is used as the backbone network, and the processing procedure includes: frame-level feature extraction: each frame is passed through the ResNet18 backbone network to extract a high-dimensional frame-level feature vector; segment-level feature aggregation: a multiple-instance learning (MIL) pooling strategy based on an attention mechanism is used to calculate the attention weights and gating signals of each frame-level feature, and a segment-level microscopic feature vector is obtained by weighted summation of all frame-level features after gating adjustment.

3. The method of claim 2, wherein the method further comprises: In the macroscopic spatio-temporal encoding path, a hierarchical Transformer architecture is adopted, and the processing procedure includes: frame-by-frame spatial encoding: a shared weight spatial encoder is used to extract spatial features from each frame, generating a spatial feature vector; cross-frame temporal fusion: the spatial feature vectors of all frames are combined into a sequence and input into a temporal fusion module to model the temporal dynamics across frames, outputting a feature vector encoding the macroscopic dynamics of the entire segment.

4. The method of claim 3, wherein the method further comprises: In the temporal stability encoding path, features are extracted by calculating an inter-frame similarity matrix, and the processing procedure includes: inter-frame similarity matrix calculation: a high-dimensional semantic feature is used to calculate a cosine similarity matrix between frames; multi-path feature extraction: features are extracted through three paths: dynamic pattern encoding, global statistics encoding, and context feature enhancement; feature fusion and output: the features from the three paths are concatenated and fused through a multi-layer perceptron to generate a final feature vector encoding the stability of the entire segment.

5. The method of claim 4, wherein the method further comprises: The cross-attention fusion module adopts a three-way cross-attention fusion mechanism, including: Feature space alignment: project the feature vectors of the three branches into the same 256-dimensional space; Multi-directional cross-enhancement: each feature stream vector is used as context by the other two feature streams, enhanced through multi-head attention mechanism; Final deep fusion: concatenate the original projected features and the cross-enhanced features, and perform deep fusion and dimension reduction through multi-layer perceptron to obtain the final fused feature vector.

6. An apparatus for performing the method of video de-noising by confocal laser endoscopy according to any one of claims 1 to 5, characterized in that Comprise: an input module for receiving a video clip composed of a series of image frames; a microscopic feature encoding module for extracting fine-grained spatial texture features for identifying histological patterns from individual image frames; a macroscopic spatio-temporal encoding module for modeling the global scene dynamics and cross-frame temporal dependencies of the video clip; a temporal stability encoding module for generating stability features representing the operator's cognitive intention by quantifying the feature-level similarity between a series of image frames; a feature fusion module for fusing features from the three encoding modules to generate a fused feature vector; a classification module for generating a classification output based on the fused feature vector.

7. A video de-aliasing apparatus for a confocal endoscope according to claim 6, wherein The microscopic feature encoding module is provided with: a convolutional neural network (CNN) for independently processing each image frame to extract frame-level feature vectors; a multiple-instance learning (MIL) pooling layer for weighted aggregation of frame-level feature vectors to generate spatial texture features.

8. The method, device and system of claim 7, wherein, The macroscopic spatio-temporal encoding module is provided with: a hierarchical Transformer architecture including a spatial encoder and a temporal fusion module for modeling the global dynamics and long-range temporal dependencies of the video clip.

9. A confocal laser endomicroscopy video de-bulk filtering system, comprising: Comprise: the confocal laser endoscopy video de-redundancy filtering device of any one of claims 6-8; a processing unit for implementing an automatic workflow of the intelligent classification stage and the intelligent de-redundancy stage; wherein, in the intelligent classification stage, the processing unit calls the device to process the original video stream, and stores the video frames according to the classification results; in the intelligent de-redundancy stage, the processing unit applies a de-redundancy algorithm to the video frames with high diagnostic value to generate a set of key frames with high information density and no redundancy.

10. The video de-noising system of claim 9, wherein the system further comprises a confocal laser endoscope. 10 The processing flow of the intelligent de-redundancy stage includes: using a deep learning feature extractor to extract deep semantic feature vectors for each frame; within a user-defined time window, calculate the cosine similarity between the feature vectors, and group the consecutive frames with highly similar content into a group; in each similar frame group, quantify the clarity of each frame by calculating the variance of the Laplacian operator, and keep the clearest frame as the representative.

Images