3D ultra-high-definition fluorescence medical endoscope imaging method and device

Abstract

The present invention discloses a 3D ultra-high-definition fluorescent medical endoscope imaging method and device, including: acquiring and analyzing the real-time video stream of the endoscope, performing denoising and multi-dimensional feature extraction; using a multi-scale shape reconstruction method to perform initial depth estimation, obtaining a high-precision depth map through stereo disparity optimization and conditional random field methods, and reconstructing the initial three-dimensional point cloud; using a variational registration method based on optimal transmission theory for precise registration, and dynamically resampling and fusing white light and fluorescent image sequences; combining a tissue physical model, performing surface reconstruction, optimization and refinement of the initial three-dimensional point cloud, using a fused image sequence for dynamic texture mapping, and finally rendering and outputting. This method achieves high-quality 3D ultra-high-definition fluorescent medical endoscope imaging.

Description

Technical Field

[0001] The present invention relates to video image processing technology, in particular to a 3D ultra-high definition fluorescence medical endoscope imaging method and device. Background Art

[0002] Endoscopic technology plays a vital role in minimally invasive surgery and early disease diagnosis. The development of 3D ultra-high-definition fluorescence medical endoscopy imaging technology provides unprecedented visual experience and diagnostic capabilities. It not only provides high-resolution three-dimensional images, but also highlights specific tissue structures or lesion areas through fluorescence imaging technology. It improves the accuracy and safety of surgery, and also provides new possibilities for early cancer screening and diagnosis. Especially in complex anatomical structures, such as the digestive tract, respiratory tract or neurosurgery, 3D ultra-high-definition fluorescence imaging technology can help doctors better understand the spatial relationship and functional status of tissues, so as to make more accurate diagnosis and treatment decisions.

[0003] At present, 3D ultra-high-definition fluorescence medical endoscopy imaging technology has made significant progress. Researchers have developed a system that can capture white light and fluorescence images simultaneously and integrate the information of the two modalities through image fusion technology.

[0004] Although the existing technology has made significant progress, there are still some specific technical problems that need to be solved. In terms of image denoising, existing methods often find it difficult to strike a good balance between retaining detail information and removing noise, especially in fluorescence images acquired under low light conditions. Accurate registration of white light images and fluorescence images remains a challenge, especially in the presence of tissue deformation, where traditional rigid registration methods are difficult to achieve ideal results. In addition, existing depth estimation methods often have errors when dealing with reflective or translucent surfaces, which is a common problem in endoscopic imaging. In the process of three-dimensional reconstruction, how to accurately simulate and consider the physical properties of tissues (such as elasticity and deformation) remains an unresolved problem. Therefore, research innovation is needed. Summary of the invention

[0005] The purpose of the invention is to provide a 3D ultra-high-definition fluorescence medical endoscope imaging method and device, in order to solve at least some of the problems existing in the prior art.

[0006] Technical solution, according to one aspect of the present application, a 3D ultra-high-definition fluorescent medical endoscope imaging method comprises the following steps:

[0007] Step S1, acquiring and analyzing the real-time video stream of the endoscope to obtain a white light image sequence and a fluorescence image sequence; performing denoising and multi-dimensional feature extraction on the white light image sequence and the fluorescence image sequence respectively to obtain a multi-dimensional feature map;

[0008] Step S2, obtaining a denoised white light image sequence, a multi-dimensional feature map and a pre-calibrated camera parameter matrix, using a multi-scale shape reconstruction method based on light and dark changes to perform initial depth estimation to obtain an initial depth map; using a stereo disparity optimization technique to calculate and optimize the disparity map to obtain an optimized disparity map; combining the initial depth map and the optimized disparity map, using a conditional random field method to perform depth fusion and refinement to obtain a high-precision depth map; based on the high-precision depth map and the camera parameter matrix, reconstructing an initial three-dimensional point cloud by a triangulation method;

[0009] Step S3, obtaining the denoised white light image sequence and the fluorescence image sequence, and calling the pre-stored nonlinear deformation field, using the variational registration method based on the optimal transmission theory to perform accurate registration, and obtaining the optimal transformation field; based on the optimal transformation field, dynamically resampling and interpolating the white light image sequence and the fluorescence image sequence to obtain the resampled image sequence; using the multi-scale fusion method based on wavelet transform, fusing the resampled white light image sequence and the fluorescence image sequence to obtain a fused image sequence, which is recorded as a fused image sequence;

[0010] Step S4, obtaining a fused image sequence, a high-precision depth map, an initial three-dimensional point cloud and a pre-calibrated camera parameter matrix, and combining the tissue physical model to perform surface reconstruction, optimization and refinement on the initial three-dimensional point cloud to obtain an accurate 3D surface model that takes into account tissue characteristics; using the fused image sequence, dynamically texture mapping the accurate 3D surface model to obtain a 3D model with dynamic texture and render and output it.

[0011] According to another aspect of the present application, a 3D ultra-high definition fluorescent medical endoscope imaging device is also provided, comprising:

[0012] at least one processor; and,

[0013] a memory communicatively connected to at least one of the processors; wherein,

[0014] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the 3D ultra-high-definition fluorescence medical endoscope imaging method described in any of the above technical solutions.

[0015] Beneficial effects: The present invention significantly improves the quality and accuracy of endoscopic imaging. The use of adaptive non-local mean filtering and multi-scale analysis effectively solves the problem of image denoising under low light conditions. The variational registration method based on optimal transmission theory successfully handles the image registration problem caused by tissue deformation. Multi-scale shape reconstruction combined with the conditional random field method greatly improves the accuracy of depth estimation on reflective and translucent surfaces. The introduction of nonlinear hyperelastic models and modified Navier-Stokes equations enables accurate simulation of tissue physical properties. The combination of spectral element method and multi-grid method significantly improves the efficiency and accuracy of dynamic texture mapping. The accuracy and safety of endoscopic examinations and minimally invasive surgery are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 It is a flow chart of step S1 of the present invention.

[0018] Figure 3 It is a flow chart of step S2 of the present invention.

[0019] Figure 4 It is a flow chart of step S3 of the present invention.

[0020] Figure 5 It is a flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0021] According to one aspect of the present application, a 3D ultra-high-definition fluorescent medical endoscope imaging method comprises the following steps:

[0022] Step S1, acquiring and analyzing the real-time video stream of the endoscope to obtain a white light image sequence and a fluorescence image sequence; performing denoising and multi-dimensional feature extraction on the white light image sequence and the fluorescence image sequence respectively to obtain a multi-dimensional feature map;

[0023] Step S2, obtaining a denoised white light image sequence, a multi-dimensional feature map and a pre-calibrated camera parameter matrix, using a multi-scale shape reconstruction method based on light and dark changes to perform initial depth estimation to obtain an initial depth map; using a stereo disparity optimization technique to calculate and optimize the disparity map to obtain an optimized disparity map; combining the initial depth map and the optimized disparity map, using a conditional random field method to perform depth fusion and refinement to obtain a high-precision depth map; based on the high-precision depth map and the camera parameter matrix, reconstructing an initial three-dimensional point cloud by a triangulation method;

[0024] Step S3, obtaining the denoised white light image sequence and the fluorescence image sequence, and calling the pre-stored nonlinear deformation field, using the variational registration method based on the optimal transmission theory to perform accurate registration, and obtaining the optimal transformation field; based on the optimal transformation field, dynamically resampling and interpolating the white light image sequence and the fluorescence image sequence to obtain the resampled image sequence; using the multi-scale fusion method based on wavelet transform, fusing the resampled white light image sequence and the fluorescence image sequence to obtain a fused image sequence, which is recorded as a fused image sequence;

[0025] It should be noted that, in some cases, the input may also include a high-precision depth map Z(x, y) and an initial three-dimensional point cloud P. The relevant scheme is a preferred scheme and will be described in detail later.

[0026] Step S4, obtaining a fused image sequence, a high-precision depth map, an initial three-dimensional point cloud and a pre-calibrated camera parameter matrix, and combining the tissue physical model to perform surface reconstruction, optimization and refinement on the initial three-dimensional point cloud to obtain an accurate 3D surface model that takes into account tissue characteristics; using the fused image sequence, dynamically texture mapping the accurate 3D surface model to obtain a 3D model with dynamic texture and render and output it.

[0027] In order to highlight the innovation points, some processes that can be implemented using existing technologies are briefly described.

[0028] According to one aspect of the present application, step S1 specifically comprises:

[0029] Step S11, using a synchronous acquisition device to simultaneously capture image sequences in a white light imaging mode and a fluorescence imaging mode;

[0030] In the white light imaging mode, a white light image sequence I_w(x, y, t) is obtained; in the white light imaging mode, the natural color and morphological information of the tissue is obtained by visible light illumination.

[0031] In the fluorescence imaging mode, a fluorescence image sequence I_f(x, y, t) is obtained; wherein (x, y) represents the image plane coordinates and t represents the time series; in the fluorescence imaging mode, the fluorescent substance in the tissue is excited by excitation light of a specific wavelength to capture the fluorescence signal emitted by it.

[0032] Step S12, performing denoising processing on the original white light image sequence I_w and the fluorescence image sequence I_f, specifically as follows: obtaining the original white light image sequence I_w and the fluorescence image sequence I_f;

[0033] For each pixel p to be processed, search for similar pixel blocks within the search window;

[0034] Calculate the weight w(p,q) between pixel blocks, w(p,q) = exp(-||P(p) - P(q)|| 2 / h 2 ), where q is the pixel in the search window; P(p) and P(q) represent the image blocks centered at p and q, respectively, and h is the adaptive smoothing parameter;

[0035] Based on the calculated weights, the pixel values ​​are weighted averaged to obtain the denoised pixel value I'(p) = Σw(p,q) * I(q) / Σw(p,q);

[0036] After processing all pixels, the denoised white light image sequence I'_w and the fluorescence image sequence I'_f are obtained;

[0037] It can effectively preserve image details and edge information while significantly reducing noise levels.

[0038] Step S13, obtaining the denoised white light image sequence I'_w and the fluorescence image sequence I'_f;

[0039] The local binary pattern algorithm is used to extract texture features T(x, y);

[0040] Calculate the local gradient direction histogram to obtain edge information E(x, y);

[0041] Extract the fluorescence intensity distribution F(x, y) from the fluorescence image sequence;

[0042] Wavelet scattering network is used to extract multi-scale features and construct the feature S_j[p]x = |x * ψ_j,p| * φ_J; where ψ_j,p is the wavelet function and φ_J is the smoothing function;

[0043] The above extracted features are combined to form a multidimensional feature map C(x, y) = [T(x, y), E(x, y), F(x,y), S_j[p]x]. This provides a rich information basis for subsequent tissue modeling and image analysis.

[0044] In this embodiment, the image sequence of white light and fluorescence imaging modes is captured simultaneously by the synchronous acquisition device, so as to realize efficient dual-modal image acquisition. The problem of time inconsistency between white light and fluorescence images in traditional technology is solved, which lays the foundation for subsequent image fusion and 3D reconstruction. In the denoising process, an adaptive non-local mean filtering algorithm is adopted, which effectively retains the image details while reducing the noise level by finding similar pixel blocks in the search window and calculating the weights. It is particularly suitable for the low signal-to-noise ratio environment commonly used in endoscopic imaging, and improves the accuracy of subsequent processing. In the feature extraction stage, a variety of advanced technologies such as local binary patterns, gradient directional histograms and wavelet scattering networks are combined to construct a multidimensional feature map. It not only captures the texture, edge and multi-scale information of the image, but also contains the fluorescence intensity distribution, which provides a rich source of information for subsequent depth estimation and image fusion. In short, this step improves the image quality and information richness, and lays a solid foundation for the subsequent processing of 3D ultra-high-definition fluorescence medical endoscope imaging devices.

[0045] According to one aspect of the present application, step S2 is specifically:

[0046] Step S21, obtaining a denoised white light image sequence I'_w;

[0047] Construct a Gaussian pyramid {G_0, G_1, ..., G_n}, where G_0 is the denoised white light image sequence I'_w;

[0048] Starting from the top layer of the pyramid, estimate the depth layer by layer; for each layer, construct the energy minimization equation E(D_i) = E_data(D_i) + λE_smooth(D_i); E(D_i) is the total energy function; D_i is the depth estimate in the i-th layer of the pyramid; E_data(D_i) is the data term or reconstruction error term; E_smooth(D_i) is the smoothing term or regularization term; λ is the weight coefficient;

[0049] The energy equation is solved by an iterative optimization algorithm to obtain the depth estimation of the current layer;

[0050] Use this depth estimate as the initial value for the next layer and repeat the optimization process until the bottom layer is processed;

[0051] Get the initial depth map D_init(x, y);

[0052] Step S22, obtaining left and right views of the denoised white light image sequence I'_w;

[0053] Calculate the pixel matching cost C(p, d) between the left and right views;

[0054] Perform cost aggregation along multiple directions and calculate the aggregate matching cost S(p, d);

[0055] For each pixel, select the disparity value that minimizes the aggregate matching cost S(p, d) as the optimal disparity of the pixel;

[0056] Solve the optimization problem through dynamic programming algorithm to obtain a globally consistent disparity map;

[0057] Apply left-right consistency check and sub-pixel refinement processing to obtain the optimized disparity map D_stereo;

[0058] Step S23, obtaining an initial depth map D_init(x, y) and an optimized disparity map D_stereo;

[0059] Convert the initial depth map D_init(x, y) and the optimized disparity map D_stereo to the same depth representation;

[0060] Construct the energy function E(D) = Σ ψ_u(D_a) + Σ ψ_p(D_a, D_b); where ψ_u(D_a) is the single pixel potential energy term; ψ_p(D_a, D_b) is the paired pixel potential energy term; D_a is the depth value of the a-th pixel in the depth map; D_b is the depth value of the b-th pixel in the depth map;

[0061] Use the mean field approximation algorithm to iteratively solve the conditional random field CRF;

[0062] Each iteration updates the depth expectation and variance of each pixel. After the iteration converges, a fused and refined high-precision depth map Z(x, y) is obtained.

[0063] Step S24, obtaining a high-precision depth map Z(x, y) and a pre-calibrated camera parameter matrix K;

[0064] For each pixel (u, v), calculate its normalized coordinates;

[0065] Combined with the depth information Z(u, v), calculate the 3D coordinates corresponding to each pixel;

[0066] Generate an initial 3D point cloud P.

[0067] In this embodiment, the initial depth estimation is performed by a multi-scale shape reconstruction method based on light and dark changes, which effectively solves the common problems of uneven illumination and surface reflection in endoscopic imaging. By constructing a Gaussian pyramid and optimizing layer by layer starting from the top layer, it is possible to capture local details while ensuring global consistency, thereby improving the accuracy and robustness of depth estimation. The introduction of stereo disparity optimization technology further refines the depth information, especially when dealing with complex surfaces commonly seen in the endoscopic field of view. The conditional random field method is used for deep fusion and refinement, which not only integrates multi-source depth information, but also considers the spatial relationship between pixels, effectively eliminating noise and discontinuity in the depth map. It is particularly suitable for processing reflective surfaces and translucent tissues commonly seen in endoscopic images, and improves the accuracy of the reconstruction results. The initial three-dimensional point cloud is reconstructed by the triangulation method, providing a reliable foundation for the subsequent construction of an accurate 3D surface model. This solution enables the system to extract high-precision three-dimensional information from endoscopic images, providing doctors with more intuitive and accurate spatial information, and improving the accuracy of endoscopic examinations and surgeries.

[0068] According to one aspect of the present application, step S3 is specifically:

[0069] Step S31, acquiring a denoised white light image sequence I'_w and a fluorescence image sequence I'_f, and a pre-stored nonlinear deformation field Φ(x, y, z, t);

[0070] The white light image sequence and the fluorescence image sequence are represented as probability distributions μ and ν; the Wasserstein distance W(μ,ν) of the white light image and the fluorescence image is constructed; and the energy functional E[T] = W(T#μ,ν) + R(T); where T is the transformation to be determined, T#μ represents the distribution of μ after transformation by T, and R(T) is the regularization term designed based on Φ;

[0071] The energy functional is transformed into the Euler-Lagrange equation by the variational method. The alternating direction multiplier method (ADMM) is used to solve the equation. A multi-resolution strategy is introduced to gradually optimize the transformation field from coarse to fine. The optimization is iterated until convergence to obtain the optimal transformation field T*(x, y, t).

[0072] Step S32, obtaining the denoised white light image sequence I'_w and the fluorescence image sequence I'_f, and the optimal transformation field T*(x, y, t);

[0073] Based on T*(x, y, t), a time-varying adaptive Delaunay triangulated mesh G(t) is constructed. Cubic B-spline interpolation is applied on G(t) to calculate the resampled image value I_r(x, y, t) = ∑∑∑ c_ijk B_i(x) B_j(y) B_k(t). Where c_ijk is the control point coefficient, and B_i, B_j and B_k are the B-spline basis functions in the x, y and t directions respectively.

[0074] Adaptive tension spline method is used to handle image boundaries and discontinuous areas; motion-compensated time interpolation strategy is used to ensure time continuity;

[0075] Resample the denoised white light image sequence I'_w and the fluorescence image sequence I'_f respectively to obtain the resampled white light image sequence I_r_w(x, y, t) and the fluorescence image sequence I_r_f(x, y, t);

[0076] Step S33, obtaining the resampled white light image sequence I_r_w (x, y, t) and the fluorescence image sequence I_r_f (x, y, t);

[0077] Multi-scale wavelet decomposition is performed on the two image sequences respectively, using biorthogonal wavelet basis;

[0078] For each scale j and direction d, the corresponding wavelet coefficients W_j,dw(x, y, t) and W_j,df(x, y,t) are obtained;

[0079] Design an adaptive fusion rule to calculate the fused wavelet coefficients W_j,d fused(x, y, t) = ω_w(x,y, t) · W_j,dw(x, y, t) + ω_f(x, y, t) · W_j,df(x, y, t); where the weights ω_w and ω_f are dynamically calculated based on local contrast, directional consistency, and temporal stability;

[0080] Perform inverse wavelet transform on the fused wavelet coefficients; apply cycle consistency constraints to ensure that the fusion result is consistent with the original input in each modality; and obtain the fused image sequence I_fused(x, y, t).

[0081] First, the variational registration method based on optimal transmission theory is used to accurately align the white light image and the fluorescence image to obtain the optimal transformation field T*(x, y, t). Then, based on the optimal transformation field T*(x, y, t), the white light image sequence I'_w and the fluorescence image sequence I'_f are dynamically resampled and interpolated to obtain the resampled image sequences I_r_w(x, y,t) and I_r_f(x, y, t). Finally, the resampled white light image sequence and the fluorescence image sequence are fused using a multi-scale fusion method based on wavelet transform to obtain the fused image sequence I_fused(x, y, t). This series of steps achieves accurate registration and seamless fusion of white light images and fluorescence images, providing high-quality fused data for subsequent three-dimensional reconstruction and visualization.

[0082] In this embodiment, by introducing a variational registration method based on optimal transmission theory, accurate registration of white light image sequences and fluorescence image sequences is achieved. Not only does it take into account the global distribution characteristics of the image, but it can also handle local nonlinear deformations, which is particularly suitable for common tissue deformation and respiratory motion in endoscopic imaging. Through the pre-stored nonlinear deformation field, it can quickly adapt to the tissue characteristics of different patients and different parts, improving the accuracy and efficiency of registration. In the resampling process, the dynamic resampling and interpolation technology based on the optimal transformation field is adopted to effectively retain the detail information of the original image and eliminate the image distortion caused by registration. It is particularly suitable for processing common motion blur and deformation problems in endoscopic images. In the image fusion stage, a multi-scale fusion method based on wavelet transform is adopted, which not only realizes the seamless integration of white light and fluorescence information, but also retains image features of different scales through multi-scale analysis. It is particularly suitable for endoscopic images, and can simultaneously highlight the anatomical structure (white light image) and functional information (fluorescence image) of the tissue, providing doctors with comprehensive visual information. In summary, the quality of image registration and fusion is improved, high-quality fused image sequences are provided for subsequent 3D reconstruction and visualization, and the diagnostic and surgical navigation capabilities of the endoscopic imaging system are enhanced.

[0083] According to one aspect of the present application, step S4 is specifically:

[0084] Step S41, obtaining a fused image sequence, a high-precision depth map, an initial three-dimensional point cloud and a pre-calibrated camera parameter matrix;

[0085] Calling the pre-stored tissue physical model Φ;

[0086] Using the improved Poisson surface reconstruction method, the initial implicit function representation is constructed;

[0087] Solve the variational problem Δχ = ▽·V, where V is the normal vector field of the point cloud, and obtain the characteristic function χ;

[0088] Introducing material properties and geometric constraints in the tissue physics model Φ to construct a physics-based regularization term;

[0089] Construct the reconstruction energy function: E(S) = E_data(S) + λ_1 E_smooth(S) + λ_2 E_physics(S); where E_data(S), E_smooth(S), and E_physics(S) are the data term, smoothing term, and physical constraint term, respectively, and λ_1 and λ_2 are weight coefficients;

[0090] The energy function is solved by an alternating optimization strategy, and the surface mesh and implicit function representation are iteratively updated. An adaptive octree is used to accelerate the calculation process, and local refinement is performed in high curvature areas. The level set method is introduced for dynamic topology adjustment, and the mesh quality is improved through the surface subdivision algorithm to obtain the optimized 3D surface model S.

[0091] Step S42, obtaining the optimized 3D surface model S and the fused image sequence;

[0092] A time-varying Riemannian metric g(t) is constructed on the 3D surface model S. Based on the time-varying Riemannian metric, the harmonic mapping equation Δ_g u = 0 is solved, where u is a parameterized mapping. The spectral element method is used to discretize the equation, and the multigrid method is used to accelerate the solution process.

[0093] Decompose the 3D surface model S into multiple topological disks and establish consistent transition functions between the disks;

[0094] Construct a time-varying texture coordinate system u(x, y, z, t);

[0095] The fused image sequence is projected onto the 3D surface model S using the time-varying texture coordinate system u to obtain the initial texture mapping;

[0096] Apply anisotropic filtering algorithm to adaptively adjust the filter kernel according to surface curvature and viewing direction; use multi-resolution texture synthesis technology to supplement the insufficient texture area; apply optical flow constraints between consecutive frames to smooth the temporal changes of texture; use parallax occlusion mapping technology to enhance detail performance; output 3D model M(x, y, z, t) with dynamic texture;

[0097] Step S43, obtaining a 3D model M (x, y, z, t) with dynamic texture; setting virtual camera parameters and lighting conditions;

[0098] Apply physically based rendering techniques to calculate surface reflectance characteristics; implement real-time shadow rendering, such as using shadow mapping or ambient occlusion techniques; perform viewpoint-dependent level of detail adjustments; perform anti-aliasing processing to obtain the final rendering result and output it.

[0099] In this embodiment, by introducing a tissue physics model, the actual physical properties of the tissue, such as elasticity and deformation, are considered in the 3D surface reconstruction process. The physically based reconstruction method improves the authenticity and accuracy of the 3D model, especially when dealing with soft tissue deformation. The improved Poisson surface reconstruction method and variational optimization strategy can not only process the incomplete and noisy data commonly seen in the endoscopic field of view, but also generate a smooth and continuous surface model. By introducing an adaptive octree to accelerate the calculation process, efficient large-scale point cloud processing is achieved, meeting the real-time requirements. In the dynamic texture mapping stage, by constructing a time-varying texture coordinate system, texture mapping that is dynamically updated with tissue deformation is achieved. It is particularly suitable for common tissue movement and deformation scenes in endoscopic imaging, and can provide continuous and natural visual effects. Anisotropic filtering and multi-resolution texture synthesis technology are used to effectively solve the common problems of insufficient texture and change of viewing angle in endoscopic images. By applying physically based rendering technology, high-quality real-time rendering is achieved, including effects such as real-time shadows and ambient light occlusion. These rendering technologies enhance the visual realism of the 3D model and provide doctors with more intuitive and accurate surgical navigation information. In summary, the visualization quality and clinical practicality of the 3D ultra-high-definition fluorescence medical endoscope imaging device have been improved, providing strong technical support for precise diagnosis and minimally invasive surgery.

[0100] According to one aspect of the present application, in step S12, the original white light image sequence I_w and the fluorescence image sequence I_f are subjected to denoising, which may also be:

[0101] Step S121, obtaining the original white light image sequence I_w and the fluorescence image sequence I_f; constructing an image pyramid to decompose the image from coarse to fine;

[0102] Step S122: For each scale, perform the following operations:

[0103] Calculate the local structure tensor, estimate the edge direction and strength, construct an adaptive diffusion coefficient, suppress diffusion at the edge, enhance diffusion in smooth areas, and implement anisotropic diffusion filtering;

[0104] For each pixel p to be processed, similar pixel blocks are searched in the search window; weights between pixel blocks are calculated, taking into account structural similarity and scale information; pixel values ​​are weighted averaged based on the calculated weights;

[0105] Perform wavelet decomposition to obtain coefficients of different scales and directions, apply adaptive soft thresholding, and dynamically adjust the threshold based on local variance estimation to remove residual noise;

[0106] Step S123, reconstructing step by step from coarse scale to fine scale, using cross-scale consistency constraints to ensure the consistency of the results;

[0107] Step S124: After all pixels are processed, a denoised white light image sequence I'_w and a fluorescence image sequence I'_f are obtained.

[0108] In this embodiment, multi-scale denoising is achieved by constructing an image pyramid and decomposing the image from coarse to fine. It can effectively process noise of different scales and is particularly suitable for complex noise patterns commonly found in endoscopic images. At each scale, an adaptive anisotropic diffusion filter is used to dynamically adjust the diffusion coefficient according to the local image structure, effectively retaining edge and detail information while smoothing noise. It is particularly suitable for processing tissue boundaries and fine structures commonly found in endoscopic images. By finding similar pixel blocks in the search window and calculating weights, non-local mean denoising is achieved, effectively retaining the texture information of the image. It performs well in processing repeated textures and complex structures in endoscopic images. Wavelet decomposition and adaptive soft threshold processing are introduced to further remove residual noise while retaining image features of different scales and directions. It is particularly suitable for processing multi-scale structures and directional features in endoscopic images. Through cross-scale consistency constraints, the consistency of denoising results at different scales is ensured, effectively avoiding discontinuity between scales. In short, it not only improves the image quality, but also provides high-quality input for subsequent feature extraction and depth estimation, thereby improving the overall performance of the 3D ultra-high-definition fluorescence medical endoscope imaging device.

[0109] According to one aspect of the present application, in step S23, the process of iteratively solving the conditional random field CRF using the mean field approximation algorithm is specifically as follows:

[0110] Step S231: Based on the initial depth map and image data, a random variable including a pixel-level depth label and a potential function considering image features and spatial relationships are established to obtain a conditional random field model;

[0111] Step S232, assigning an initial probability distribution to each possible depth label of each pixel, and setting a maximum number of iterations and a convergence threshold;

[0112] Step S233: using the constructed compatibility function and comprehensively considering the spatial and intensity relationships between pixels, updating the depth label probability distribution of each pixel, and obtaining the message and edge probability distribution of each pixel;

[0113] Step S234: Calculate the distribution difference between adjacent iterations, stop when the difference is less than a threshold or reaches the maximum number of iterations, select the depth label with the highest probability for each pixel, and obtain the optimized conditional random field solution, that is, the optimized depth map.

[0114] In this embodiment, by constructing a conditional random field model, the depth estimation problem is transformed into a probabilistic reasoning problem, which effectively integrates image features and spatial relationship information. It is particularly suitable for processing common noise and uncertainty in endoscopic images, and can generate a more consistent and smooth depth map. By assigning an initial probability distribution to each pixel, it can effectively process a variety of possible depth values, which is particularly suitable for processing complex surfaces and occlusions in the endoscopic field of view. In the iterative optimization process, by considering the spatial and intensity relationship between pixels, a balance between local consistency and global optimality is achieved. This optimization strategy is particularly suitable for processing continuous surfaces and mutation boundaries in endoscopic images. By setting the convergence threshold and the maximum number of iterations, while ensuring the quality of the results, the real-time requirements are also met. Finally, the depth label with the highest probability for each pixel is selected, and the optimized depth map obtained not only retains the detail information, but also has good spatial consistency. In short, the accuracy and robustness of depth estimation are improved, high-quality depth information is provided for subsequent 3D reconstruction, and the spatial perception capability of the 3D ultra-high-definition fluorescent medical endoscope imaging device is enhanced.

[0115] According to one aspect of the present application, in step S31, the calculation process of the nonlinear deformation field is specifically as follows:

[0116] Step S311, obtaining a pre-stored multi-dimensional feature map C(x, y) and a high-precision depth map Z(x, y);

[0117] Generate a triangular mesh M based on the multi-dimensional feature map C(x, y) and the high-precision depth map Z(x, y);

[0118] Using the texture and fluorescence information in the multidimensional feature map C(x, y), material properties E and ν are assigned to the grid cells;

[0119] Establish a nonlinear hyperelastic model σ= f(E, ν, ε); where σ is the stress tensor, ε is the strain tensor, and f is a nonlinear function; use the improved Mooney-Rivlin model or Ogden model as the function f; solve the nonlinear equations to obtain the mechanical response of the tissue under different loads; and obtain the tissue physical model Φ(M, E, ν, f);

[0120] Step S312, obtaining a denoised white light image sequence I'_w and a tissue physical model Φ;

[0121] Apply the improved optical flow algorithm to the continuous white light image frames to calculate the dense optical flow field V(x, y, t);

[0122] Regularize the dense optical flow field V using the tissue physics model Φ to obtain a physically consistent motion field V';

[0123] Perform time-frequency analysis on the physically consistent motion field V' to extract the main frequency and amplitude information of breathing;

[0124] The motion field is decomposed by wavelet transform method to identify the scale corresponding to the respiratory frequency; the respiratory period T is estimated; the amplitude distribution of the motion field is analyzed to obtain the spatial distribution and amplitude A of the respiratory motion; the respiratory motion model R(V', T,A) is obtained and stored;

[0125] Step S313, obtaining a tissue physical model Φ and a respiratory motion model R;

[0126] Based on the geometric information of the tissue physical model Φ, the Riemannian metric g_ij is defined on the tissue surface;

[0127] Taking the respiratory motion model R as the boundary condition, the modified Navier-Stokes equation is constructed: dv / dt + ▽_gv · v = -▽_g p + μΔ_g v + f; where v is the velocity field, p is the pressure, μ is the viscosity coefficient, and f is the external force term; d is the partial differential symbol; ▽_g and Δ_g are the gradient and Laplace operator based on the Riemann metric, respectively; the finite element method is used to discretize the partial differential equation; the semi-implicit time integration scheme is used to solve the equation to obtain the velocity field and stress distribution inside the tissue; the velocity field is integrated to obtain the displacement field, and the nonlinear deformation field Φ(x, y, z, t) is obtained and output.

[0128] In this embodiment, a triangular mesh is constructed by combining a multidimensional feature map and a high-precision depth map, which provides rich geometric and material information for tissue modeling. It is particularly suitable for processing complex tissue structures in the endoscopic field of view. By assigning material properties to the grid cells and establishing a nonlinear hyperelastic model, an accurate simulation of the physical properties of the tissue is achieved. The authenticity of the model is improved, and it is particularly suitable for simulating the deformation behavior of soft tissue. By applying an improved optical flow algorithm to calculate the dense optical flow field and regularizing it using a tissue physical model, a physically consistent motion field is obtained. The tissue movement in the endoscopic field of view, especially the periodic movement caused by breathing, is effectively captured. The main frequency and amplitude information of breathing are extracted through time-frequency analysis, and a respiratory motion model is constructed, which provides an important reference for subsequent dynamic reconstruction. Finally, by constructing a modified Navier-Stokes equation and solving it using the finite element method, the velocity field and stress distribution inside the tissue are obtained. This fluid dynamics-based method can accurately simulate the complex deformation of tissues, and is particularly suitable for processing common tissue flows and deformations in endoscopic examinations. In summary, accurate modeling of the physical properties and dynamic behavior of tissues was achieved, which provided strong theoretical support for subsequent image registration and 3D reconstruction and improved the accuracy and reliability of 3D ultra-high-definition fluorescence medical endoscope imaging devices.

[0129] According to one aspect of the present application, in step S42, the equation is discretized using the spectral element method, and the multigrid method is used to accelerate the solution process, specifically:

[0130] Step S421, discretize the 3D surface model into hexahedral or tetrahedral elements; use high-order Legendre or Chebyshev polynomial basis functions in each element; construct a global stiffness matrix and a mass matrix; adopt appropriate boundary conditions to form a final set of linear equations;

[0131] Step S422, constructing a grid hierarchy, coarsening from the finest grid layer by layer; applying several Gauss-Seidel smoothing iterations on the finest grid; calculating the residual and interpolating it to the coarse grid; solving the error equation on the coarse grid; interpolating the error solution of the coarse grid back to the fine grid and correcting it; applying smoothing iterations again on the fine grid; until convergence or reaching the maximum number of iterations;

[0132] Step S423, converting the solution obtained by the multi-grid method into node displacements; using the node displacements to update the geometric shape of the 3D surface model; and using the updated geometric information in a subsequent dynamic texture mapping process.

[0133] In this embodiment, high-precision parameterized mapping is achieved by discretizing the harmonic mapping equation using the spectral element method. Using high-order polynomial basis functions, complex geometric shapes can be accurately represented, which is particularly suitable for processing curved surfaces and irregular structures in the endoscopic field of view. By constructing a global stiffness matrix and a mass matrix, the geometric characteristics of the entire 3D surface model can be effectively captured, ensuring the global consistency of texture mapping. The introduction of the multi-grid method accelerates the solution process, and the computational efficiency is significantly improved by constructing a grid hierarchy and iterating at different scales. This multi-scale solution strategy is particularly suitable for processing large-scale and high-resolution endoscopic image data. By applying smooth iterations on fine grids, local discontinuities can be effectively eliminated, generating smooth and accurate parameterized mapping. Finally, the solution results are converted into node displacements and the geometry of the 3D surface model is updated to achieve dynamic texture mapping. This physics-based parameterization method not only improves the quality of texture mapping, but also accurately reflects the deformation of tissues, and is particularly suitable for processing common tissue movements and deformations in endoscopic examinations. In conclusion, the accuracy and efficiency of dynamic texture mapping are improved, providing high-quality, real-time updated visualization for 3D ultra-high-definition fluorescence medical endoscope imaging devices, and enhancing the clinical practicality and diagnostic value of the system.

[0134] According to another aspect of the present application, a 3D ultra-high definition fluorescent medical endoscope imaging device is also provided, comprising:

[0135] at least one processor; and,

[0136] a memory communicatively connected to at least one of the processors; wherein,

[0137] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the 3D ultra-high-definition fluorescence medical endoscope imaging method described in any of the above technical solutions.

[0138] In a further embodiment, the adaptive denoising and enhancement may also be:

[0139] A 3D model M(x, y, z, t) with dynamic texture is obtained, and a multi-stage adaptive processing method is used for denoising and enhancement. First, a surface-based anisotropic diffusion filter is applied for denoising. The diffusion equation ∂I / ∂t = div(c(||▽_SI||)▽_SI) is defined, where ▽_S is the surface gradient operator, c(x) is the adaptive diffusion coefficient, defined as c(x) =exp(-(x / κ)^2), and κ is the local adaptive edge threshold. The equation is discretized and solved on the surface mesh by the finite element method to achieve noise suppression while maintaining edges and details. Then, surface adaptive histogram equalization (Surface-CLAHE) is applied to enhance local contrast. The surface of the 3D model is divided into overlapping local regions, histogram equalization is performed on each region, and the results of adjacent regions are merged by bilinear interpolation. To avoid over-enhancement, a curvature-based adaptive limit is introduced. Finally, a surface-based guided filter is used for edge-aware detail enhancement. Define the filter output as q = a·I + b, where coefficients a and b are obtained by minimizing the energy function E = (q - I)^2 + ε·a^2 on the surface. This process enhances texture details while maintaining edge clarity. The entire process remains consistent in the time dimension to ensure the temporal coherence of the enhancement effect. Output the enhanced 3D model M_enhanced(x, y, z, t), which has clearer texture, higher contrast, and richer details.

[0140] In another embodiment of the present application,

[0141] Step S12 may also use a denoising method based on block matching and 3D filtering (BM3D), and the specific steps are as follows:

[0142] For each reference block, similar blocks are searched within the search window. The search window size can be adjusted dynamically according to the image resolution and is usually set to 2-3 times the reference block size.

[0143] Use a patch similarity metric such as normalized cross-correlation to evaluate the similarity between patches.

[0144] Stack similar blocks into a 3D array.

[0145] Perform a 3D transform on a 3D array (such as a 3D discrete cosine transform).

[0146] Perform hard thresholding or Wiener filtering in the transform domain.

[0147] Perform the inverse transform and aggregate the results into the denoised image.

[0148] A second similar process is performed, but using the preliminary denoised image as a guide for the similarity search.

[0149] This method can effectively preserve image details while significantly reducing noise levels.

[0150] In another embodiment of the present application, step S13 may also use a wavelet scattering network, and the specific structure and training method are as follows:

[0151] The network structure is:

[0152] Input layer: receives the original image.

[0153] Multiple scattering layers: Each layer contains wavelet transform, modulo operation and low-pass filtering.

[0154] Output layer: extracted feature vector.

[0155] The training method is as follows:

[0156] Unsupervised pre-training using medical image datasets.

[0157] A contrastive learning strategy is adopted to maximize the feature similarity of the same image from different perspectives, while minimizing the feature similarity between different images.

[0158] The Adam optimizer was used, the learning rate was initially set to 0.001, and the cosine annealing strategy was used.

[0159] Batch normalization and dropout techniques are used during training to improve generalization ability.

[0160] In another embodiment of the present application, step S21, the Gaussian pyramid construction process, is specifically as follows:

[0161] Initialization: Set the denoised white light image sequence I'_w as the bottom layer G_0 of the pyramid.

[0162] Iteration process: For each layer G_i (i > 0),

[0163] Apply convolution smoothing with a 5x5 Gaussian kernel to G_{i-1}.

[0164] The smoothed image is downsampled to 2x2 using bicubic interpolation.

[0165] The obtained image is taken as G_i,

[0166] Until the predetermined number of pyramid layers is reached (usually 4-6 layers).

[0167] In another embodiment of the present application, in S3, a high-precision depth map Z(x, y) and an initial three-dimensional point cloud P may also be used to optimize subsequent image registration and fusion processes, specifically:

[0168] The obtained high-precision depth map Z(x, y), the initial three-dimensional point cloud P, and the denoised white light image sequence I'_w and fluorescence image sequence I'_f in step S1 are obtained. At the same time, the pre-stored nonlinear deformation field Φ(x, y, z, t) is called.

[0169] A variational registration method based on optimal transfer theory is used for accurate registration. The specific process is as shown in S31 and will not be described in detail here. The same applies below.

[0170] Based on the optimal transformation field T(x, y, t), the white light image sequence I'_w and the fluorescence image sequence I'_f are dynamically resampled and interpolated. The specific process is as shown in step S32.

[0171] The resampled white light image sequence I_r_w(x, y, t) and the fluorescence image sequence I_r_f(x, y, t) are fused using a multi-scale fusion method based on wavelet transform. The specific process is as shown in step S33.

[0172] The obtained fused image sequence I_fused(x, y, t), the high-precision depth map Z(x, y) and the initial 3D point cloud P, as well as the pre-calibrated camera parameter matrix K are obtained. Rendering processing is performed, as specifically described in step S4.

[0173] In step S31, the specific steps of using ADMM (alternating direction multiplier method) to solve the optimal transformation field are as follows:

[0174] Construct the objective function: minimize E[T] = W(T#μ, ν) + R(T); where W is the Wasserstein distance and R(T) is the regularization term based on the pre-stored nonlinear deformation field.

[0175] Iteration steps:

[0176] Introduce auxiliary variables Z and Lagrange multipliers U;

[0177] Alternately optimize T, Z, and U: a) T-subproblem: fix Z and U, optimize T; b) Z-subproblem: fix T and U, optimize Z; c) U update: U = U + ρ(T - Z), ρ is the step size parameter.

[0178] Construct an image pyramid, start solving from the coarsest layer, and use the results of the coarse layer as the initial value of the fine layer.

[0179] The change between the current and the next two iterations is less than the preset threshold, or the maximum number of iterations is reached.

[0180] In step S4, the conversion from the 3D surface model to the dynamic texture model is specifically as follows:

[0181] Generate an initial 3D surface model S; construct a time-varying parameterized mapping u(x, y, z, t); project the fused image sequence onto the 3D surface model; apply anisotropic filtering and multi-resolution texture synthesis; implement optical flow constraints and parallax occlusion; generate the final dynamic textured 3D model M(x, y, z, t);

[0182] In step S41, the adaptive octree accelerates the calculation, specifically:

[0183] Using a pointer octree, each node stores; spatial boundary information is the child node pointer (8) and the surface information of the current node (if it is a leaf node).

[0184] Start from the root node, which represents the entire 3D space; recursively divide the space into 8 subspaces; for each subspace, check whether further subdivision is needed: if it contains enough point cloud data, continue subdividing; otherwise, stop subdividing and store the surface information.

[0185] Starting from the root node, according to the coordinates of the query point, the corresponding child node is selected and traversed recursively downward until a leaf node is reached, and the surface information stored in the leaf node is returned.

[0186] During the reconstruction process, the tree structure is dynamically adjusted according to the surface complexity, increasing the subdivision level in high curvature areas and merging nodes in flat areas to reduce memory usage.

[0187] In step S42, the spectral element method and the multi-grid method may also be implemented as follows:

[0188] Discretize 3D surface models into hexahedral or tetrahedral elements; use high-order polynomial basis functions within each element; construct global stiffness and mass matrices; apply appropriate boundary conditions; solve algebraic equations to obtain nodal displacements;

[0189] Build a hierarchy of meshes, from fine to coarse; apply several smoothing iterations (such as Gauss-Seidel) on the finest mesh.

[0190] Limit the residual to a coarse grid; solve the error equation on the coarse grid; interpolate the coarse grid solution back to the fine grid.

[0191] Smoothing iterations are applied again on the fine mesh; until convergence or the maximum number of iterations is reached.

[0192] In summary, in the present invention, several key problems in 3D ultra-high-definition fluorescent medical endoscope imaging are effectively solved. First, in terms of image denoising, an adaptive non-local mean filtering algorithm is used, combined with multi-scale analysis and anisotropic diffusion filtering, to successfully achieve a good balance between retaining detail information and removing noise. It is particularly suitable for fluorescent images acquired under low light conditions. By finding similar pixel blocks in the search window and calculating weights, the image details are effectively retained while the noise level is significantly reduced. By introducing wavelet decomposition and adaptive soft threshold processing, the residual noise is further removed, providing high-quality input for subsequent processing.

[0193] In terms of accurate registration of white light images and fluorescence images, this method introduces a variational registration method based on optimal transfer theory, combined with a pre-stored nonlinear deformation field, which effectively solves the limitations of traditional rigid registration methods in dealing with tissue deformation. This method not only considers the global distribution characteristics of the image, but also can handle local nonlinear deformation, which is suitable for common tissue deformation and respiratory movement in endoscopic imaging. By constructing a time-varying texture coordinate system, texture mapping that is dynamically updated with tissue deformation is realized, further improving the accuracy and real-time performance of registration.

[0194] In order to solve the common error problem of depth estimation when dealing with reflective or semi-transparent surfaces, this method adopts a multi-scale shape reconstruction method based on light and shade changes, combined with stereo disparity optimization technology and conditional random field method for depth fusion and refinement. It improves the accuracy and robustness of depth estimation, especially when dealing with complex curved surfaces, reflective surfaces and semi-transparent tissues commonly seen in the endoscope field of view. By constructing a conditional random field model, the depth estimation problem is transformed into a probabilistic reasoning problem, which effectively integrates image features and spatial relationship information, and further improves the consistency and smoothness of the depth map.

[0195] In order to accurately simulate and consider the physical properties of tissues (such as elasticity and deformation) during the three-dimensional reconstruction process, this method introduces a physics-based reconstruction technique. By combining a multidimensional feature map and a high-precision depth map to construct a triangular mesh, assigning material properties to the mesh units, and establishing a nonlinear hyperelastic model, an accurate simulation of the physical properties of the tissue is achieved. The improved Poisson surface reconstruction method and variational optimization strategy can not only process the incomplete and noisy data commonly seen in the endoscopic field of view, but also generate a smooth and continuous surface model. By constructing a modified Navier-Stokes equation and solving it using the finite element method, the velocity field and stress distribution inside the tissue are obtained, which can accurately simulate the complex deformation of the tissue.

[0196] In addition, this method combines a variety of advanced technologies such as local binary patterns, gradient directional histograms, and wavelet scattering networks in the feature extraction stage to construct a multidimensional feature map, which provides a rich source of information for subsequent processing. In the dynamic texture mapping stage, the spectral element method is used to discretize the harmonic mapping equation, and the multi-grid method is combined to accelerate the solution process, achieving high-precision and high-efficiency parameterized mapping. Finally, by applying physically based rendering technology, high-quality real-time rendering is achieved, including real-time shadows and ambient occlusion effects, which greatly improves the visual realism of the 3D model.

[0197] In summary, this method not only effectively solves key problems such as image denoising, image registration, depth estimation and 3D reconstruction, but also significantly improves the overall performance of the 3D ultra-high-definition fluorescent medical endoscope imaging device. By considering the physical properties and dynamic behavior of tissues, this method achieves more accurate and realistic 3D reconstruction and visualization effects, providing doctors with more intuitive and accurate diagnostic and surgical navigation information, and enhancing the accuracy and safety of endoscopic examinations and minimally invasive surgeries.

[0198] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A 3D ultra-high definition fluorescence medical endoscope imaging method, characterized in that: The steps include: Step S1, acquiring and analyzing the real-time video stream of the endoscope to obtain a white light image sequence and a fluorescence image sequence; performing denoising and multi-dimensional feature extraction on the white light image sequence and the fluorescence image sequence respectively to obtain a multi-dimensional feature map; Step S2, obtaining a denoised white light image sequence, using a multi-scale shape reconstruction method based on light and dark changes to perform initial depth estimation, and obtaining an initial depth map; obtaining left and right views of the denoised white light image sequence, using a stereo disparity optimization technique to calculate and optimize the disparity map, and obtaining an optimized disparity map; combining the initial depth map and the optimized disparity map, using a conditional random field method to perform depth fusion and refinement, and obtaining a high-precision depth map; based on the high-precision depth map and a pre-calibrated camera parameter matrix, reconstructing an initial three-dimensional point cloud by a triangulation method; Step S3, obtaining the denoised white light image sequence and the fluorescence image sequence, and calling the pre-stored nonlinear deformation field, using the variational registration method based on the optimal transmission theory to perform accurate registration, and obtaining the optimal transformation field; Based on the optimal transformation field, the denoised white light image sequence and the fluorescence image sequence are dynamically resampled and interpolated to obtain a resampled image sequence; the resampled white light image sequence and the fluorescence image sequence are fused using a multi-scale fusion method based on wavelet transform to obtain a fused image sequence, which is recorded as a fused image sequence; Step S4, obtaining a fused image sequence, a high-precision depth map, an initial three-dimensional point cloud and a pre-calibrated camera parameter matrix, and combining the tissue physical model to perform surface reconstruction, optimization and refinement on the initial three-dimensional point cloud to obtain an accurate 3D surface model that takes into account tissue characteristics; using the fused image sequence, dynamically texture mapping the accurate 3D surface model to obtain a 3D model with dynamic texture and render and output it; The step S4 is specifically as follows: Step S41, obtaining a fused image sequence, a high-precision depth map, an initial three-dimensional point cloud and a pre-calibrated camera parameter matrix; Calling the pre-stored tissue physical model Φ; Using the improved Poisson surface reconstruction method, the initial implicit function representation is constructed; Solve the variational problem Δχ = ▽·V, where V is the normal vector field of the point cloud, and obtain the characteristic function χ; Introducing material properties and geometric constraints in the tissue physics model Φ to construct a physics-based regularization term; Construct the reconstruction energy function: E(S) = E_data(S) + λ_1E_smooth(S) + λ_2E_physics(S); where E_data(S), E_smooth(S), and E_physics(S) are the data term, smoothing term, and physical constraint term, respectively, and λ_1 and λ_2 are weight coefficients; The energy function is solved by an alternating optimization strategy, and the surface mesh and implicit function representation are iteratively updated. An adaptive octree is used to accelerate the calculation process, and local refinement is performed in high curvature areas. The level set method is introduced for dynamic topology adjustment, and the mesh quality is improved through the surface subdivision algorithm to obtain the optimized 3D surface model S. Step S42, obtaining the optimized 3D surface model S and the fused image sequence; A time-varying Riemannian metric g(t) is constructed on the 3D surface model S; based on the time-varying Riemannian metric, the harmonic mapping equation Δ_gu'=0 is solved, where Δ_g is the Laplace operator based on the Riemannian metric and u' is the parameterized mapping; the spectral element method is used to discretize the equation, and the multigrid method is used to accelerate the solution process; Decompose the 3D surface model S into multiple topological disks and establish consistent transition functions between the disks; Construct a time-varying texture coordinate system u"(x, y, z, t); where (x, y, z) is a three-dimensional coordinate and t represents a time series; The fused image sequence is projected onto the 3D surface model S using the time-varying texture coordinate system u″ to obtain the initial texture mapping; Apply anisotropic filtering algorithm to adaptively adjust the filter kernel according to surface curvature and viewing direction; use multi-resolution texture synthesis technology to supplement the insufficient texture area; apply optical flow constraints between consecutive frames to smooth the temporal changes of texture; use parallax occlusion mapping technology to enhance detail performance; output 3D model M(x, y, z, t) with dynamic texture; Step S43, obtaining a 3D model M (x, y, z, t) with dynamic texture; setting virtual camera parameters and lighting conditions; Apply physically based rendering techniques to calculate surface reflectance characteristics; use shadow mapping or ambient occlusion techniques to achieve real-time shadow rendering; make viewpoint-dependent level of detail adjustments; perform anti-aliasing processing to obtain the final rendering result and output it.

2. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 1, characterized in that: The step S1 is specifically as follows: Step S11, using a synchronous acquisition device to simultaneously capture image sequences in a white light imaging mode and a fluorescence imaging mode; In white light imaging mode, obtain the white light image sequence I_w(x, y, t); In the fluorescence imaging mode, a fluorescence image sequence I_f(x, y, t) is obtained; where (x, y) represents the image plane coordinates and t represents the time series; Step S12, performing denoising processing on the original white light image sequence I_w and the fluorescence image sequence I_f, specifically as follows: obtaining the original white light image sequence I_w and the fluorescence image sequence I_f; For each pixel p to be processed, search for similar pixel blocks within the search window; Calculate the weight w(p,q) between pixel blocks, w(p,q) = exp(-||P(p) - P(q)|| 2 / h 2 ), where q is the pixel in the search window; P(p) and P(q) represent the image blocks centered at p and q, respectively, and h is the adaptive smoothing parameter; Based on the calculated weights, the pixel values ​​are weighted averaged to obtain the denoised pixel value I'(p)=Σw(p,q)*I(q) / Σw(p,q); After processing all pixels, the denoised white light image sequence I'_w and fluorescence image sequence I'_f are obtained; Step S13, obtaining the denoised white light image sequence I'_w and the fluorescence image sequence I'_f; The local binary pattern algorithm is used to extract texture features T(x, y); Calculate the local gradient direction histogram to obtain edge information E(x, y); Extract the fluorescence intensity distribution F(x, y) from the denoised fluorescence image sequence; Wavelet scattering network is used to extract multi-scale features and construct the feature S_j[p]x = |x * ψ_j,p| * φ_J; where ψ_j,p is the wavelet function and φ_J is the smoothing function; Combine to form a multidimensional feature map C(x, y) = [T(x, y), E(x, y), F(x, y), S_j[p]x].

3. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 2, characterized in that: The step S2 is specifically as follows: Step S21, obtaining a denoised white light image sequence I'_w; Construct a Gaussian pyramid {G_0, G_1, ..., G_n}, where G_0 is the denoised white light image sequence I'_w, and n is a natural number; Starting from the top layer of the pyramid, estimate the depth layer by layer; for each layer, construct the energy minimization equation E(D_i) =E_data(D_i) + λE_smooth(D_i); E(D_i) is the total energy function; D_i is the depth estimate in the i-th layer of the pyramid; E_data(D_i) is the data term or reconstruction error term; E_smooth(D_i) is the smoothing term or regularization term; λ is the weight coefficient; Solve the energy minimization equation through an iterative optimization algorithm to obtain the depth estimate of the current layer; Use this depth estimate as the initial value for the next layer and repeat the optimization process until the bottom layer is processed; Get the initial depth map D_init(x, y); Step S22, obtaining left and right views of the denoised white light image sequence I'_w; Calculate the pixel matching cost C(p, d) between the left and right views; where d is the direction; Perform cost aggregation along multiple directions and calculate the aggregate matching cost S(p, d); For each pixel, select the disparity value that minimizes the aggregate matching cost S(p, d) as the optimal disparity of the pixel; Solve the optimization problem through dynamic programming algorithm to obtain a globally consistent disparity map; Apply left-right consistency check and sub-pixel refinement processing to obtain the optimized disparity map D_stereo; Step S23, obtaining an initial depth map D_init(x, y) and an optimized disparity map D_stereo; Convert the initial depth map D_init(x, y) and the optimized disparity map D_stereo to the same depth representation; Construct the energy function E(D) = Σψ_u(D_a) + Σψ_p(D_a, D_b); where ψ_u(D_a) is the single pixel potential energy term; ψ_p(D_a, D_b) is the paired pixel potential energy term; D_a is the depth value of the a-th pixel in the depth map; D_b is the depth value of the b-th pixel in the depth map; Use the mean field approximation algorithm to iteratively solve the conditional random field CRF; Each iteration updates the depth expectation and variance of each pixel. After the iteration converges, a fused and refined high-precision depth map Z(x, y) is obtained. Step S24, obtaining a high-precision depth map Z(x, y) and a pre-calibrated camera parameter matrix K; For each pixel (u, v), calculate its normalized coordinates; Combined with the depth information Z(u, v), calculate the 3D coordinates corresponding to each pixel; Generate an initial 3D point cloud P.

4. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 3, characterized in that: The step S3 is specifically as follows: Step S31, acquiring the denoised white light image sequence I'_w and the fluorescence image sequence I'_f, and the pre-stored nonlinear deformation field Φ*(x, y, z, t); The denoised white light image sequence and fluorescence image sequence are represented as probability distributions μ and ν'; the Wasserstein distance W(μ,ν') of the white light image sequence and the fluorescence image sequence is constructed; and the energy functional E[T] = W(T#μ,ν') + R(T); where T is the transformation to be determined, T#μ represents the distribution of μ after transformation by T, and R(T) is a regularization term designed based on the nonlinear deformation field; The energy functional is transformed into the Euler-Lagrange equation by the variational method; the alternating direction multiplier method is used to solve the equation; a multi-resolution strategy is introduced to gradually optimize the transformation field from coarse to fine; the optimization is iterated until convergence to obtain the optimal transformation field T*(x, y, t); Step S32, obtaining the denoised white light image sequence I'_w and the fluorescence image sequence I'_f, and the optimal transformation field T*(x, y, t); Based on T*(x, y, t), a time-varying adaptive Delaunay triangulated mesh G(t) is constructed. Cubic B-spline interpolation is applied on G(t) to calculate the resampled image value I_r(x, y, t) =∑∑∑c_ijkB_i(x)B_j(y)B_k(t). Where c_ijk is the control point coefficient, and B_i, B_j and B_k are the B-spline basis functions in the x, y and t directions respectively. Adaptive tension spline method is used to process image boundaries and discontinuous areas; motion-compensated time interpolation strategy is used to ensure time continuity; Resample the denoised white light image sequence I'_w and the fluorescence image sequence I'_f respectively to obtain the resampled white light image sequence I_r_w(x, y, t) and the fluorescence image sequence I_r_f(x, y, t); Step S33, obtaining the resampled white light image sequence I_r_w (x, y, t) and the fluorescence image sequence I_r_f (x, y, t); Multi-scale wavelet decomposition is performed on the two image sequences respectively, using biorthogonal wavelet basis; For each scale j and direction d, the corresponding wavelet coefficients W_j,dw(x, y, t) and W_j,df(x, y, t) are obtained; Design an adaptive fusion rule to calculate the fused wavelet coefficients W_j,dfused(x, y, t) =ω_w(x, y,t)·W_j,dw(x, y, t)+ω_f(x, y, t)·W_j,df(x, y, t); where the weights ω_w and ω_f are dynamically calculated based on local contrast, directional consistency, and temporal stability; Perform inverse wavelet transform on the fused wavelet coefficients; apply cycle consistency constraints to ensure that the fusion result is consistent with the original input in each modality; and obtain the fused image sequence I_fused(x, y, t).

5. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 4, characterized in that: In step S12, the original white light image sequence I_w and the fluorescence image sequence I_f are subjected to denoising, which may also be: Step S121, obtaining the original white light image sequence I_w and the fluorescence image sequence I_f; constructing an image pyramid to decompose the image from coarse to fine; Step S122: For each scale, perform the following operations: Calculate the local structure tensor, estimate the edge direction and strength, construct an adaptive diffusion coefficient, suppress diffusion at the edge, enhance diffusion in smooth areas, and implement anisotropic diffusion filtering; For each pixel p to be processed, similar pixel blocks are found in the search window; weights between pixel blocks are calculated, taking into account structural similarity and scale information; Based on the calculated weights, the pixel values ​​are weighted averaged; Perform wavelet decomposition to obtain coefficients of different scales and directions, apply adaptive soft thresholding, and dynamically adjust the threshold based on local variance estimation to remove residual noise; Step S123, reconstructing step by step from coarse scale to fine scale, using cross-scale consistency constraints to ensure the consistency of the results; Step S124: After all pixels are processed, a denoised white light image sequence I'_w and a fluorescence image sequence I'_f are obtained.

6. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 4, characterized in that: In step S23, the process of iteratively solving the conditional random field CRF using the mean field approximation algorithm is specifically as follows: Step S231: Based on the initial depth map and image data, a random variable including a pixel-level depth label and a potential function considering image features and spatial relationships are established to obtain a conditional random field model; Step S232, assigning an initial probability distribution to each possible depth label of each pixel, and setting a maximum number of iterations and a convergence threshold; Step S233: using the constructed compatibility function and comprehensively considering the spatial and intensity relationships between pixels, updating the depth label probability distribution of each pixel, and obtaining the message and edge probability distribution of each pixel; Step S234: Calculate the distribution difference between adjacent iterations, stop when the difference is less than a threshold or reaches the maximum number of iterations, select the depth label with the highest probability for each pixel, and obtain the optimized conditional random field solution, that is, the optimized depth map.

7. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 4, characterized in that: In step S31, the calculation process of the nonlinear deformation field is specifically as follows: Step S311, obtaining a pre-stored multi-dimensional feature map C(x, y) and a high-precision depth map Z(x, y); Generate a triangular mesh M based on the multi-dimensional feature map C(x, y) and the high-precision depth map Z(x, y); Using the texture and fluorescence information in the multidimensional feature map C(x, y), material properties E and ν" are assigned to the grid cells; Establish a nonlinear hyperelastic model σ = f(E, ν", ε); where σ is the stress tensor, ε is the strain tensor, and f is a nonlinear function; use the improved Mooney-Rivlin model or Ogden model as the function f; solve the nonlinear equations to obtain the mechanical response of the tissue under different loads; obtain the tissue physical model Φ(M, E, ν", f); Step S312, obtaining the denoised white light image sequence I'_w and the tissue physical model Φ; Apply the improved optical flow algorithm to the continuous white light image frames to calculate the dense optical flow field V*(x, y, t); The dense optical flow field V* is regularized using the tissue physics model Φ to obtain a physically consistent motion field V*'; Perform time-frequency analysis on the physically consistent motion field V*' to extract the main frequency and amplitude information of breathing; The motion field is decomposed by wavelet transform method to identify the scale corresponding to the respiratory frequency; the respiratory period T is estimated; the amplitude distribution of the motion field is analyzed to obtain the spatial distribution and amplitude A of the respiratory motion; the respiratory motion model R(V*', T,A) is obtained and stored; Step S313, obtaining a tissue physical model Φ and a respiratory motion model R; Based on the geometric information of the tissue physical model Φ, the Riemannian metric g_ij is defined on the tissue surface; Taking the respiratory motion model R as the boundary condition, the modified Navier-Stokes equations are constructed: dv* / dt + ▽_gv*·v* = -▽_gp + μΔ_gv* + f; where v* is the velocity field, p is the pressure, μ is the viscosity coefficient, and f is the external force term; d is the partial differential symbol; ▽_g and Δ_g are the gradient and Laplace operator based on the Riemann metric, respectively; the Navier-Stokes equations are discretized using the finite element method; the equations are solved using a semi-implicit time integration scheme to obtain the velocity field and stress distribution inside the tissue; the displacement field is obtained by integrating the velocity field, and the nonlinear deformation field Φ*(x, y, z, t) is obtained and output.

8. The 3D ultra-high definition fluorescence medical endoscope imaging method according to claim 4, characterized in that: In step S42, the equation is discretized using the spectral element method, and the multigrid method is used to accelerate the solution process, specifically: Step S421, discretize the 3D surface model into hexahedral or tetrahedral elements; use high-order Legendre or Chebyshev polynomial basis functions in each element; construct a global stiffness matrix and a mass matrix; adopt appropriate boundary conditions to form a final set of linear equations; Step S422, constructing a grid hierarchy, coarsening from the finest grid layer by layer; applying several Gauss-Seidel smoothing iterations on the finest grid; calculating the residual and interpolating it to the coarse grid; solving the error equation on the coarse grid; interpolating the error solution of the coarse grid back to the fine grid and correcting it; applying smoothing iterations again on the fine grid; until convergence or reaching the maximum number of iterations; Step S423, converting the solution obtained by the multi-grid method into node displacements; using the node displacements to update the geometric shape of the 3D surface model; and using the updated geometric information in a subsequent dynamic texture mapping process.

9. A 3D ultra-high definition fluorescent medical endoscope imaging device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the 3D ultra-high-definition fluorescence medical endoscope imaging method according to any one of claims 1 to 8.

Images