A confocal image mapping panoramic stitching method, device, medium and equipment
By performing time-series analysis and white-light video mapping on confocal image sequences, the problems of identifying and stitching mucus and red blood cell artifacts were solved, improving the diagnostic accuracy and image quality of gastrointestinal mucosal lesions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIZHOU ENZE MEDICAL CENT GROUP
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-26
AI Technical Summary
In existing confocal laser microendoscopy techniques, black dot artifacts produced by mucus and red blood cells are easily confused with goblet cells, leading to a high misdiagnosis rate; traditional field of view is limited, making macroscopic anatomical localization difficult; there is a lack of reliable image acquisition methods, resulting in large errors in lesion boundary localization; and the low signal-to-noise ratio of single-frame images affects the identification of small lesions.
By performing time-series analysis on confocal image sequences, artifacts are identified and repaired, goblet cells are distinguished, and coordinate mapping relationships are established by combining white light video mapping. Image stitching and edge fusion are then performed to generate confocal panoramic images.
It improves the diagnostic accuracy of gastrointestinal mucosal lesions, reduces the misdiagnosis rate, enhances the precision of lesion boundary localization, and improves the image signal-to-noise ratio.
Smart Images
Figure CN122288976A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of confocal laser microscopy (pCLE) imaging technology, and more specifically, to a panoramic stitching method, apparatus, medium, and device for confocal image fixing. Background Technology
[0002] Existing CLE technology has the following drawbacks: black dot artifacts produced by mucus and red blood cells are easily confused with goblet cells (misdiagnosis rate of 18-22%); traditional CLE has a limited field of view (approximately 240 μm in diameter), making it difficult to establish macroscopic anatomical localization; the lack of reliable image acquisition methods leads to lesion boundary localization errors >2 mm; and the signal-to-noise ratio (SNR) of a single frame image is usually below 15 dB, affecting the identification of small lesions. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention provides a panoramic stitching method, apparatus, medium, and device for confocal image positioning.
[0004] According to one aspect of the present invention, a panoramic stitching method for confocal image positioning is provided, comprising: Temporal analysis was performed on the confocal image sequence to identify and repair artifacts caused by mucus and red blood cells, and goblet cells were distinguished to obtain the original confocal image frame sequence without repair and the corresponding repaired confocal image frame sequence. Based on the unrepaired original confocal image frame sequence, the optimal observation frame is determined, and the white light fixed image is extracted from the white light video at the corresponding time. The image registration model is used to scale down and map each confocal field of view in the unrepaired original confocal image frame sequence to the corresponding region of the white light fixation image, thus establishing a coordinate mapping relationship between the white light fixation image and the confocal field of view. Based on the coordinate mapping relationship, the white light fixed image is used as the macroscopic reference coordinate system. The overlapping region deduplication process is performed on the unrepaired original confocal image mapped to the macroscopic reference coordinate system to obtain the overlapping region selection strategy. The overlapping region selection strategy is applied to repair confocal image frame sequences, and image stitching and edge fusion are performed in a macroscopic reference coordinate system to generate confocal panoramic images.
[0005] According to another aspect of the present invention, a panoramic stitching device for confocal image fixing is provided, comprising: The identification and repair module is used to perform time-series analysis on the confocal image sequence, identify and repair artifacts caused by mucus and red blood cells, distinguish goblet cells, and obtain the original confocal image frame sequence without repair and the corresponding repaired confocal image frame sequence. The extraction module is used to determine the optimal observation frame based on the unrepaired original confocal image frame sequence, and extract the white light fixed image from the white light video at the corresponding time. A module is established to scale down and map each confocal field of view in the unrestored original confocal image frame sequence to the corresponding region of the white light fixed image using an image registration model, and to establish a coordinate mapping relationship between the white light fixed image and the confocal field of view. The deduplication module is used to perform overlapping region deduplication on the unrepaired original confocal image mapped to the macroscopic reference coordinate system based on the coordinate mapping relationship, taking the white light fixed image as the macroscopic reference coordinate system, and obtaining the overlapping region selection strategy. The fusion module is used to apply the overlapping region selection strategy to repair the confocal image frame sequence, perform image stitching and edge fusion in the macroscopic reference coordinate system, and generate a confocal panoramic image.
[0006] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.
[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.
[0008] Therefore, this invention performs temporal analysis on confocal image sequences, identifies and repairs artifacts caused by mucus and red blood cells, and distinguishes goblet cells, obtaining an unrepaired original confocal image frame sequence and its corresponding repaired confocal image frame sequence. Based on the unrepaired original confocal image frame sequence, the optimal observation frame is determined, and a white light fixed image is extracted from the white light video at the corresponding time. Using an image registration model, each confocal field of view in the unrepaired original confocal image frame sequence is scaled down and mapped to the corresponding region of the white light fixed image, establishing a coordinate mapping relationship between the white light fixed image and the confocal field of view. Based on this coordinate mapping relationship, using the white light fixed image as a macroscopic reference coordinate system, overlapping region deduplication processing is performed on the unrepaired original confocal images mapped to the macroscopic reference coordinate system to obtain an overlapping region selection strategy. This overlapping region selection strategy is applied to the repaired confocal image frame sequence, and image stitching and edge fusion are performed in the macroscopic reference coordinate system to generate a confocal panoramic image. This improves the diagnostic accuracy of gastrointestinal mucosal lesions. Attached Figure Description
[0009] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures: Figure 1This is a schematic flowchart of a panoramic stitching method for confocal image positioning provided by an exemplary embodiment of the present invention; Figure 2 This is a functional schematic diagram of an artifact recognition unit provided in an exemplary embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a panoramic stitching device for confocal image positioning provided in an exemplary embodiment of the present invention; Figure 4 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation
[0010] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.
[0011] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention.
[0012] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0013] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.
[0014] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.
[0015] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.
[0016] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0017] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0018] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.
[0019] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.
[0020] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0021] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0022] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0023] Exemplary methods Figure 1 This is a schematic flowchart of a panoramic stitching method for confocal image positioning provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, the panoramic stitching method 100 for confocal image positioning includes the following steps: Step 101: Perform time-series analysis on the confocal image sequence, identify and repair artifacts caused by mucus and red blood cells, and distinguish goblet cells to obtain the unrepaired original confocal image frame sequence and the corresponding repaired confocal image frame sequence. Step 102: Based on the unrepaired original confocal image frame sequence, determine the optimal observation frame and extract the white light fixation image from the white light video at the corresponding time. Step 103: Using the image registration model, each confocal field of view in the unrepaired original confocal image frame sequence is scaled down and mapped to the corresponding area of the white light fixed image, thus establishing a coordinate mapping relationship between the white light fixed image and the confocal field of view. Step 104: Based on the coordinate mapping relationship, the white light fixed image is used as the macroscopic reference coordinate system. The overlapping region deduplication process is performed on the unrepaired original confocal image mapped to the macroscopic reference coordinate system to obtain the overlapping region selection strategy. Step 105: Apply the overlapping region selection strategy to repair the confocal image frame sequence, perform image stitching and edge fusion in the macroscopic reference coordinate system, and generate a confocal panoramic image.
[0024] Specifically, this invention combines white light-guided confocal image artifact removal, image acquisition, and panoramic stitching methods to improve the diagnostic accuracy of gastrointestinal mucosal lesions. In CLE (confocal laser microscopy) images, the dynamic changes of mucus artifacts, erythrocyte artifacts, and goblet cells during the respiratory cycle show significant differences and can be differentiated through temporal analysis and morphodynamic characteristics. The specific implementation steps are as follows: Step S1: Artifact Recognition Unit, functional modules as follows Figure 2 As shown, it includes four modules: respiratory segment recognition module, segmentation module, feature extraction module, and identification module.
[0025] Step S1.1, Respiratory Segment Recognition Module 1) Image sharpness is quantified based on the gradient energy method, which calculates the sum of squares of image gradients (such as the Sobel operator). A higher value indicates sharper edges. .
[0026] 2) The image clarity of the inspiratory phase is compared with that of the expiratory phase as shown in Table 1 below.
[0027] Table 1
[0028] Respiratory phase division: ① Store the changes in resolution values over time.
[0029] ② Calculate the slope of the change in sharpness. When the slope continues to decrease and the slope is small, the corresponding time period is the inhalation phase.
[0030] ③The slope is relatively large, and the clarity increases → briefly peaks → decreases. This process is the exhalation phase.
[0031] ④ During the exhalation phase, the frame preceding the maximum clarity value is the optimal frame for observation.
[0032] Step S1.2, Segmentation Module Artifacts such as black dots produced by mucus and red blood cells, goblet cells, and easily confused targets (those that are difficult for the segmentation model to distinguish).
[0033] Step S1.3, Feature Extraction Module For easily confused targets, the following features are extracted: 1) Motion amplitude: Calculate the centroid of each easily confused target. Based on the respiratory phase time interval obtained in step S1.1, calculate the maximum centroid offset of each easily confused target within a complete respiratory cycle. ; 2) Motion frequency: Based on the centroid of each easily confused target in 1), calculate the time-varying function of the centroid of each easily confused target. and to Fast Fourier Transform yields the principal frequency components. .
[0034] Step S1.4, Identification Module Based on machine learning models (such as RF, DT, etc.) and Feature fitting was performed to train a 3-class classification model, identifying black dot artifacts produced by mucus and red blood cells, as well as goblet cells.
[0035] Step S1.5, Fill the module The black dotted artifact areas caused by mucus and red blood cells identified in steps S1.2 and S1.4 are filled based on the surrounding pixels. The specific steps are as follows: 1) Poisson Reconstruction Method ① Draw the minimum bounding rectangle in the mucus region and the black dot artifact region produced by red blood cells, and then expand the rectangle outward. times, of which It can be 1.5, 2.0, etc., without specific restrictions.
[0036] ② Calculate the gradient field (Sobel operator in the x / y direction) of the non-mucus region and the black dot artifact region produced by non-red blood cells in the outer expansion region.
[0037] ③ Construct a sparse linear system of equations: ,in , a sparse coefficient matrix (where N is the total number of pixels to be repaired). The repaired pixel values to be solved; Constraint vector; ④ Establish a discrete Poisson equation for each pixel p (position (i,j)) to be repaired: ,in Let be the Laplacian value of the source image at that point; and keep the original pixel value as the boundary condition.
[0038] ⑤ Solve iteratively using the conjugate gradient method.
[0039] 2) Mucus filling ①Poisson reconstruction + Gaussian smoothing ,in, Gaussian blur result (σ=1.5, smoothed noise); Smoothing constraints; Smoothing regularization term (based on the Laplace operator).
[0040] 3) Filling the black dot artifact areas produced by red blood cells, the main technical challenges are: to completely preserve microvessels with a diameter of <10μm when eliminating artifacts, to avoid damaging the sharp structure of the edge of the mucosal gland duct, and to compensate for artifact displacement caused by blood flow.
[0041] ① Vascular sensing segmentation: Extracting blood vessel orientation based on the Frangi filter; Divide the blood vessel into sections along its main direction (length to width ratio 1:2 to 1:3). ② Restricted neighborhood filling: Dynamically adjust the sampling window (3×3 to 7×7 pixels); Sampling lines must not cross the contours of blood vessels; ③ Edge sharpening compensation: Unsharpened mask (USM) enhances the blood vessel wall; Sharpening intensity 0.3-0.8 (adaptive to local contrast).
[0042] Thus, step S1 yields two types of image frame sequences: 1) the original CLE image frame sequence without repair. ;2) Repaired CLE image frame sequence ; and One-to-one correspondence.
[0043] Step S2, registration unit, all the following steps are based on the original CLE image frame sequence without repair. Registration with white light images Step S2.1, confocal image thumbnail module 1) Based on the time corresponding to the optimal frame obtained in step S1.1-1)-④, extract the corresponding white light image frame from the white light video. This white light image frame is the white light fixed image. 2) Construct the CLE image thumbnail model as follows: Supervised registration network: Dual-stream architecture processes two images simultaneously → Feature extraction → Correlation calculation → Dense displacement field prediction → Differentiable resampling.
[0044] The dual-stream architecture processes two types of images simultaneously: two independent CNN streams (weights are not shared), each stream containing four downsampling stages (similar to the UNet encoder). Correlation calculation layer: flattens the two-stream feature maps into [b,c,h*w], calculates the dot product of features at all locations using matrix multiplication, and outputs the 5D correlation volume [b,h,w,Δh,Δw]. Displacement field prediction network: Input: 5D correlation volume → expanded to [b,1,h,w,Δh,Δw], 3D convolution processing space - displacement dimension, final output dense displacement field [b,2,h,w] (x / y direction offset); Differentiable resampling: based on bilinear sampling of the displacement field, I_aligned(x,y) = I_src(x+Δx(x,y), y+Δy(x,y)), where Δx / Δy is calculated differentially through bilinear interpolation.
[0045] A CLE thumbnail image is obtained, wherein the size of the thumbnailed CLE image is 1:1 with the size of the corresponding white light region.
[0046] Step S2.2, image registration: find the corresponding field of view under white light for each CLE field of view. 1) Multi-candidate fusion mechanism ① On the white light image, according to the size of the scaled CLE image, the white light image is divided into N candidate regions with a certain step size, and a feature descriptor is constructed for each candidate region: Low-level features: HSV histogram, LBP texture features Intermediate features: SIFT / SURF keypoint distribution density Advanced Feature: Cosine Similarity of Deep Activation Vectors in CNN ② Adaptive weighted voting is adopted: different weights are assigned to different feature dimensions, and the comprehensive score is calculated: S = 0.4 × low-level feature + 0.3 × mid-level feature + 0.3 × high-level feature. The maximum value of S in the N candidate regions is the field of view region under white light corresponding to the current CLE.
[0047] Step S3, panoramic stitching unit Step S3.1, Deduplication Module ① Perform SIFT feature point detection on each CLE thumbnail; ② Based on the calculated field of view area under white light corresponding to CLE, the projection transformation matrix (Homography) of the white light image is calculated using the RANSAC algorithm to transform all CLE thumbnails to the unified coordinate system of the white light image; ③ Calculate the pixel-level intersection region for each pair of CLE images; ④ For each overlapping region (set of pixel coordinates P): a. Calculate the sharpness score of each CLE image in the P region; b. Retain the image data with the highest score; c. Other images are set to the alpha channel in this area.
[0048] Step S3.2, panoramic stitching module 1) Based on steps S3.1-④ The overlapping region processing method for each image is applied to the repaired CLE image frame sequence. ; 2) Perform the following steps at the boundary of the overlapping region after processing step 1): A graph cut-based energy minimization method is used to find the optimal seam path. b. Apply Poisson blending to smooth the transition edges; c. Perform local histogram matching at the stitching area to obtain a CLE panoramic stitched image with well-processed edges; 3) Enlarge the stitched CLE panoramic image inversely proportionally according to the scaling ratio to obtain the real CLE panoramic stitched image, i.e., the confocal panoramic image.
[0049] Therefore, this invention performs temporal analysis on confocal image sequences, identifies and repairs artifacts caused by mucus and red blood cells, and distinguishes goblet cells, obtaining an unrepaired original confocal image frame sequence and its corresponding repaired confocal image frame sequence. Based on the unrepaired original confocal image frame sequence, the optimal observation frame is determined, and a white light fixed image is extracted from the white light video at the corresponding time. Using an image registration model, each confocal field of view in the unrepaired original confocal image frame sequence is scaled down and mapped to the corresponding region of the white light fixed image, establishing a coordinate mapping relationship between the white light fixed image and the confocal field of view. Based on this coordinate mapping relationship, using the white light fixed image as a macroscopic reference coordinate system, overlapping region deduplication processing is performed on the unrepaired original confocal images mapped to the macroscopic reference coordinate system to obtain an overlapping region selection strategy. This overlapping region selection strategy is applied to the repaired confocal image frame sequence, and image stitching and edge fusion are performed in the macroscopic reference coordinate system to generate a confocal panoramic image. This improves the diagnostic accuracy of gastrointestinal mucosal lesions.
[0050] Exemplary device Figure 3 This is a schematic diagram of the structure of a panoramic stitching device for confocal image positioning provided in an exemplary embodiment of the present invention. Figure 3 As shown, the device 300 includes: The identification and repair module 310 is used to perform time-series analysis on the confocal image sequence, identify and repair artifacts caused by mucus and red blood cells, distinguish goblet cells, and obtain the unrepaired original confocal image frame sequence and the corresponding repaired confocal image frame sequence. Extraction module 320 is used to determine the optimal observation frame based on the unrepaired original confocal image frame sequence, and extract the white light fixed image from the white light video at the corresponding time. Module 330 is established to scale down and map each confocal field of view in the unrepaired original confocal image frame sequence to the corresponding region of the white light fixed image through an image registration model, and to establish a coordinate mapping relationship between the white light fixed image and the confocal field of view. The deduplication module 340 is used to perform overlapping region deduplication on the unrepaired original confocal image mapped to the macroscopic reference coordinate system based on the coordinate mapping relationship, taking the white light fixed image as the macroscopic reference coordinate system, to obtain the overlapping region selection strategy. The fusion module 350 is used to apply the overlapping region selection strategy to repair the confocal image frame sequence, perform image stitching and edge fusion in the macroscopic reference coordinate system, and generate a confocal panoramic image.
[0051] Exemplary electronic devices Figure 4 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 4 As shown, the electronic device 40 includes one or more processors 41 and memory 42.
[0052] The processor 41 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0053] The memory 42 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 41 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above, and / or other desired functions. In one example, the electronic device may also include an input device 43 and an output device 44, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0054] In addition, the input device 43 may also include, for example, a keyboard, a mouse, etc.
[0055] The output device 44 can output various information to the outside. The output device 44 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0056] Of course, for the sake of simplicity, Figure 4 Only some of the components of this electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0057] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.
[0058] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present invention. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0059] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.
[0060] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0061] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.
[0062] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0063] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0064] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.
[0065] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0066] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A panoramic stitching method for confocal image positioning, characterized in that, include: Temporal analysis was performed on the confocal image sequence to identify and repair artifacts caused by mucus and red blood cells, and goblet cells were distinguished to obtain the original confocal image frame sequence without repair and the corresponding repaired confocal image frame sequence. Based on the unrepaired original confocal image frame sequence, the optimal observation frame is determined, and the white light fixed image is extracted from the white light video at the corresponding time. The image registration model is used to scale down and map each confocal field of view in the unrepaired original confocal image frame sequence to the corresponding region of the white light fixation image, thereby establishing a coordinate mapping relationship between the white light fixation image and the confocal field of view. Based on the coordinate mapping relationship, the white light fixed image is used as a macroscopic reference coordinate system. The unrepaired original confocal image mapped to the macroscopic reference coordinate system is subjected to overlapping region deduplication processing to obtain the overlapping region selection strategy. The overlapping region selection strategy is applied to the repaired confocal image frame sequence, and image stitching and edge fusion are performed in the macroscopic reference coordinate system to generate a confocal panoramic image.
2. The method according to claim 1, characterized in that, Temporal analysis was performed on the confocal image sequence to identify and repair artifacts caused by mucus and red blood cells, and goblet cells were distinguished. This yielded the original confocal image frame sequence (unrepaired) and its corresponding repaired confocal image frame sequence, including: Based on the change in sharpness value of each confocal image over time, the inhalation phase and the exhalation phase are divided; within the exhalation phase, the frame before the sharpness reaches its maximum value is determined as the optimal observation frame; Segmenting mucus artifacts, red blood cell artifacts, goblet cells, and easily confused target regions in confocal images; For each of the aforementioned easily confused target regions, the amplitude and frequency of movement are calculated within a complete respiratory cycle; Based on the motion amplitude and motion frequency characteristics, the easily confused target region is identified as mucus artifact, red blood cell artifact, or goblet cell using a classification model. The identified mucus artifact region and red blood cell artifact region are filled with pixels using a Poisson reconstruction method based on the gradient field of the surrounding non-artifact region to obtain the repaired confocal image frame sequence.
3. The method according to claim 2, characterized in that, Repairing the red blood cell artifact region includes: The blood vessel orientation is extracted based on the blood vessel structure filter, and the repair area is divided into blocks along the blood vessel orientation. During filling, the pixel sampling window is dynamically adjusted and the sampling line is prohibited from crossing the blood vessel contour. After filling, the edge of the blood vessel wall is sharpened and enhanced.
4. The method according to claim 1, characterized in that, The image registration model is used to scale down and map each confocal field of view in the unrestored original confocal image frame sequence to the corresponding region of the white light fixation image, establishing a coordinate mapping relationship between the white light fixation image and the confocal field of view, including: Each frame of the unrepaired original confocal image frame sequence is input into a two-stream supervised registration network; Features of the confocal image and the white light image are extracted through two independent branches of the dual-stream supervised registration network, and the dense image displacement field is predicted by calculating the correlation of feature points. Based on the image displacement field, the confocal image is resampled, aligned, and scaled in a differentiable manner to obtain a confocal thumbnail image whose size is consistent with the corresponding area of the white light fixed image; On the white light fixed image, multiple candidate regions are generated by sliding according to the size of the confocal thumbnail image and a preset step size; For each candidate region, a fusion feature descriptor containing low-level, mid-level, and high-level features is calculated, and a comprehensive matching score for each candidate region is calculated through adaptive weighting. The candidate region with the highest score is determined as the field of view region corresponding to the confocal field of view on the white light fixation map. Based on the field of view, the coordinate mapping relationship between the white light fixed image and the confocal field of view is established.
5. The method according to claim 1, characterized in that, Based on the coordinate mapping relationship, using the white light fixation image as a macroscopic reference coordinate system, the unrepaired original confocal image mapped to the macroscopic reference coordinate system undergoes overlapping region deduplication processing to obtain an overlapping region selection strategy, including: Based on the coordinate mapping relationship, all confocal thumbnail images are projected onto the macroscopic reference coordinate system of the white light fixed image; Based on the projection results, for the overlapping area of any two confocal thumbnail images, the sharpness score of each image in the overlapping area is calculated, and only the image data with the highest sharpness score is retained to obtain the overlapping area selection strategy.
6. The method according to claim 1, characterized in that, The overlapping region selection strategy is applied to the repaired confocal image frame sequence, and image stitching and edge fusion are performed in the macroscopic reference coordinate system to generate a confocal panoramic image, including: The overlapping region selection strategy is applied to the repaired confocal image frame sequence mapped to the same macroscopic reference coordinate system; At the boundary of the overlapping area between the mapped images, the optimal stitching path is determined by the seam finding method based on energy minimization, and the stitching edge is smoothly transitioned by the Poisson fusion method, finally generating a seamless confocal panoramic image. The image is then inversely magnified according to the scaling ratio to generate the confocal panoramic image.
7. A panoramic stitching device for confocal image fixing, characterized in that, include: The identification and repair module is used to perform time-series analysis on the confocal image sequence, identify and repair artifacts caused by mucus and red blood cells, distinguish goblet cells, and obtain the original confocal image frame sequence without repair and the corresponding repaired confocal image frame sequence. The extraction module is used to determine the optimal observation frame based on the unrepaired original confocal image frame sequence, and extract the white light fixed image from the white light video at the corresponding time. A module is established to scale down and map each confocal field of view in the unrepaired original confocal image frame sequence to the corresponding region of the white light fixed image using an image registration model, thereby establishing a coordinate mapping relationship between the white light fixed image and the confocal field of view. The deduplication module is used to perform overlapping region deduplication processing on the unrepaired original confocal image mapped to the macroscopic reference coordinate system based on the coordinate mapping relationship, using the white light fixed image as the macroscopic reference coordinate system, to obtain an overlapping region selection strategy. The fusion module is used to apply the overlapping region selection strategy to the repaired confocal image frame sequence, and to perform image stitching and edge fusion in the macroscopic reference coordinate system to generate a confocal panoramic image.
8. The apparatus according to claim 7, characterized in that, The identification and repair module includes: Based on the change in sharpness value of each confocal image over time, the inhalation phase and the exhalation phase are divided; within the exhalation phase, the frame before the sharpness reaches its maximum value is determined as the optimal observation frame; Segmenting mucus artifacts, red blood cell artifacts, goblet cells, and easily confused target regions in confocal images; For each of the aforementioned easily confused target regions, the amplitude and frequency of movement are calculated within a complete respiratory cycle; Based on the motion amplitude and motion frequency characteristics, the easily confused target region is identified as mucus artifact, red blood cell artifact, or goblet cell using a classification model. The identified mucus artifact region and red blood cell artifact region are filled with pixels using a Poisson reconstruction method based on the gradient field of the surrounding non-artifact region to obtain the repaired confocal image frame sequence.
9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-6.
10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-6.