An active anti-shake method based on endoscope image sequence cooperation

Abstract

The application provides an active anti-shake method based on endoscopic image sequence cooperation, comprising: acquiring a time sequence image window containing a current frame to be stabilized, and extracting a dense space-time feature set of each frame in the time sequence image window and a semantic segmentation mask of the current frame to be stabilized; based on a preset image jitter analysis model, synchronously combining the dense space-time feature set and the semantic segmentation mask for cooperative reasoning to obtain a multi-dimensional image jitter data set; based on the multi-dimensional image jitter data set, optimizing a real-time endoscopic posture data set in the time sequence image window to obtain an optimized endoscopic posture data set; according to the multi-dimensional image jitter data set and the optimized endoscopic posture data set, performing primary stabilization repair on the current frame to be stabilized to obtain a primary stabilized frame; and performing time domain consistency fusion on the primary stabilized frame and a stabilized time sequence adjacent frame to generate a final stabilized frame corresponding to the current frame to be stabilized, thereby realizing enhancement of endoscopic image quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an active image stabilization method based on endoscopic image sequence coordination. Background Technology

[0002] In the field of medical endoscopic surgery, complex and compound jitter often exists in the video due to factors such as manual operation by doctors, physiological activities of patients and peristalsis of organs, which poses a challenge to surgical navigation, remote consultation and automated analysis. However, traditional methods often have certain limitations in dealing with this challenge.

[0003] Specifically, traditional methods mostly perform image stabilization by assuming the scene is static or that the whole scene is in rigid motion. However, these methods often fail to accurately distinguish between dynamic objects and non-rigid motion in the scene, resulting in low image stabilization quality. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an active image stabilization method based on endoscopic image sequence collaboration. The method includes: S1: acquiring a temporal image window containing the current frame to be stabilized, and extracting a dense spatiotemporal feature set of each frame within the temporal image window and a semantic segmentation mask of the current frame to be stabilized; S2: based on a preset image jitter analysis model, simultaneously combining the dense spatiotemporal feature set and the semantic segmentation mask for collaborative inference to obtain a multidimensional image jitter dataset; S3: based on the multidimensional image jitter dataset, optimizing the real-time endoscopic posture dataset within the temporal image window to obtain an optimized endoscopic posture dataset; S4: performing initial stabilization repair on the current frame to be stabilized according to the multidimensional image jitter dataset and the optimized endoscopic posture dataset to obtain a primary stabilized frame; S5: performing temporal consistency fusion between the primary stabilized frame and the stabilized temporal nearest neighbor frames to generate a final stabilized frame corresponding to the current frame to be stabilized.

[0005] As a further aspect of the present invention, obtaining a temporal image window containing the current frame to be stabilized, and extracting the dense spatiotemporal feature set of each frame within the temporal image window and the semantic segmentation mask of the current frame to be stabilized, includes: the temporal image window being represented as a sliding window in the original endoscopic video stream, consisting of the current frame to be stabilized and n frames before and after the frame to be stabilized; using a pre-trained spatiotemporal feature encoder to extract features from the image sequences corresponding to each frame within the temporal image window to obtain the dense spatiotemporal feature set corresponding to each frame; using a semantic segmentation network to perform forward inference on the current frame to be stabilized to generate an initial semantic segmentation mask; and using a cross-attention mechanism to optimize the initial semantic segmentation mask based on the dense spatiotemporal feature set to generate a semantic segmentation mask and a corresponding semantic confidence map.

[0006] As a further aspect of the present invention, based on a preset image jitter analysis model, collaborative reasoning is performed simultaneously with the dense spatiotemporal feature set and the semantic segmentation mask to obtain a multidimensional image jitter dataset. This includes: matching and connecting feature points based on the dense spatiotemporal feature set corresponding to adjacent frames within a temporal image window of the frame to be stabilized, constructing a short-range trajectory set; classifying the short-range trajectory set based on the semantic segmentation mask, dividing it into subsets of tissue motion trajectories, tool motion trajectories, and background interference trajectories; inputting the tissue motion trajectory subset as observation data into the image jitter analysis model for collaborative reasoning to obtain corresponding global jitter motion data and non-rigid physiological motion data; performing fitting analysis on the tool motion trajectory subset using a robust fitting method to obtain corresponding tool motion transformation data; and encapsulating the global jitter motion data, non-rigid physiological motion data, tool motion transformation data, and corresponding motion estimation uncertainty data into a multidimensional image jitter dataset for output.

[0007] As a further aspect of the present invention, the image shake analysis model includes: the image shake analysis model is represented as a generative probabilistic graphical model, used to decompose the motion patterns of a subset of tissue motion trajectories to generate global shake motion data and non-rigid physiological motion data; the global shake motion data is represented as an affine transformation sequence describing the global shake of the lens; the non-rigid physiological motion data is represented as a non-rigid deformation field describing the autonomous deformation of the organ; the image shake analysis model is trained using a variational expectation-maximization algorithm with the goal of maximizing the variational lower bound of the observed data until a preset training termination condition is reached.

[0008] As a further aspect of the present invention, based on the multidimensional image jitter dataset, the real-time endoscope pose dataset is optimized within the time-series image window to obtain an optimized endoscope pose dataset. This includes: integrating global jitter motion data within the multidimensional image jitter dataset to obtain a preliminary camera projection matrix for each frame within the time-series image window relative to the starting frame; performing uncertainty quantization based on the preliminary camera projection matrix, and selecting keyframes based on the uncertainty quantization results; the keyframes are image frames whose geometric uncertainty is below a preset threshold and whose tissue region visibility meets a preset condition; using the selected keyframes as nodes, optimizing the camera pose of the keyframes in conjunction with the multidimensional image jitter dataset; optimizing the camera pose of all non-keyframes within the time-series image window based on the optimized camera pose of the keyframes to obtain an optimized camera pose sequence; performing content-aware smoothing path planning based on the semantic segmentation mask of the current frame to be stabilized to obtain a smooth target pose; and encapsulating the optimized camera pose sequence and the smooth target pose into an optimized endoscope pose dataset for output.

[0009] As a further aspect of the present invention, based on the multidimensional image jitter dataset and the optimized endoscope pose dataset, the current frame to be stabilized is subjected to initial stabilization repair to obtain a primary stabilized frame, including: solving for a homography transformation matrix describing the global basic motion based on the optimized endoscope pose dataset, and obtaining a global basic deformation field based on the homography transformation matrix; generating a non-rigid correction field for compensating for physiological deformation based on non-rigid physiological motion data in the multidimensional image jitter dataset; obtaining a motion compensation field for the tool region based on tool motion transformation data in the multidimensional image jitter dataset; generating an image deformation repair field by weighted fusion of the global basic deformation field, the non-rigid correction field, and the motion compensation field based on a semantic segmentation mask; and performing deformation repair on the current frame to be stabilized based on the image deformation repair field to obtain a primary stabilized frame.

[0010] As a further aspect of the present invention, the primary stable frame is fused with the already stabilized temporally nearest neighbor frames in a temporal domain to generate the final stable frame corresponding to the current frame to be stabilized. This includes: using a numerical optimization algorithm to solve for the energy function corresponding to the primary stable frame and outputting the corresponding final stable frame; the energy function includes at least a data stabilization term, a temporal smoothing term, and a spatial smoothing term; while solving for the energy function, uncertainty assessment is performed by combining the semantic confidence map corresponding to the semantic segmentation mask with the motion estimation uncertainty data in the multidimensional image jitter dataset; the corresponding uncertainty assessment result is obtained, and the corresponding operations of S1 to S4 are adaptively adjusted according to the uncertainty assessment result.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] A temporal image window containing the current frame to be stabilized is obtained, and the dense spatiotemporal feature set of each frame in the temporal image window and the semantic segmentation mask of the current frame to be stabilized are extracted. By performing the extraction of dense spatiotemporal features and semantic segmentation in parallel, a data foundation is provided for subsequent processing.

[0013] Based on the preset image jitter analysis model, the dense spatiotemporal feature set and semantic segmentation mask are combined simultaneously for collaborative reasoning to obtain a multidimensional image jitter dataset. The image jitter analysis model uses a hierarchical probability model to perform collaborative reasoning on the movement of tissue regions, which helps to more clearly distinguish the global rigid motion caused by camera jitter, the non-rigid deformation of the organ itself, and the noise residual.

[0014] Based on the multidimensional image jitter dataset, the real-time endoscope pose dataset is optimized within the time-series image window to obtain an optimized endoscope pose dataset. The camera is then incrementally adjusted in conjunction with the global motion patterns contained in the multidimensional image jitter dataset, thereby improving the global consistency and geometric accuracy of camera pose estimation.

[0015] Based on the multidimensional image jitter dataset and the optimized endoscope pose dataset, the current frame to be stabilized is initially stabilized and repaired to obtain a primary stabilized frame. By performing targeted processing on different motion modes, it helps to suppress global jitter and maintain the visual coherence of surgical tool operation, thereby making the synthesized image more natural.

[0016] The primary stabilized frame is fused with the already stabilized temporally nearest neighbor frame in a temporal domain to generate the final stabilized frame corresponding to the current frame to be stabilized, thereby enhancing the quality of the endoscopic image. Attached Figure Description

[0017] Figure 1 This is a flowchart of the steps of an active image stabilization method based on endoscopic image sequence collaboration according to the present invention.

[0018] Figure 2 This is a flowchart of step S1 in an active image stabilization method based on endoscopic image sequence collaboration according to the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating the steps of an active image stabilization method based on endoscopic image sequence collaboration according to the present invention. Figure 2 This is a flowchart of step S1 in an active image stabilization method based on endoscopic image sequence collaboration according to the present invention. The active image stabilization method based on endoscopic image sequence collaboration will be described in detail below.

[0020] Step S1: Obtain a temporal image window containing the current frame to be stabilized, and extract the dense spatiotemporal feature set of each frame in the temporal image window and the semantic segmentation mask of the current frame to be stabilized.

[0021] It should be noted that the time-series image window represents the sequence of images in the original endoscopic video stream, consisting of the current frame to be stabilized and the frames before and after it. A sliding window composed of frames, specifically, maintains a fixed length of The first-in-first-out frame queue is used as a buffer for each time step. The system will capture the currently captured frame. Insert a buffer and remove the oldest frame. This preserves the content in the time-series image window. Dynamic updates, among which The value is usually set according to the motion frequency of the target scene and the real-time requirements of the calculation. For example, assuming the system runs at 30fps, the parameter... =15, at the time point At time 1000, the sliding window will contain 31 frames of images from frame 985 to frame 1015. When frame 1016 arrives, the window will be updated to contain frames 986 to 1016, and so on, thus ensuring that the analysis is always based on a continuous video segment around the current frame.

[0022] In this embodiment, step S1 includes:

[0023] Step S1-1: Obtain a dense spatiotemporal feature set.

[0024] Specifically, a pre-trained spatiotemporal feature encoder is used to extract features from the image sequences corresponding to each frame within the temporal image window, thereby obtaining a dense spatiotemporal feature set corresponding to each frame.

[0025] In one possible embodiment, a 3D convolutional neural network or spatiotemporal Transformer that has been pre-trained on a large scale without supervision is used as a spatiotemporal feature encoder. This encoder is used to process a small video block consisting of consecutive frames, that is, a sequence of consecutive image frames consisting of the temporal image windows. It captures the joint information of inter-frame motion and intra-frame structure by simultaneously performing convolution or self-attention operations on the video block in the spatiotemporal dimension, thereby outputting a multi-scale feature map corresponding to each frame. This multi-scale feature map is the dense spatiotemporal feature set corresponding to each frame.

[0026] For example, a lightweight 3D ResNet-18 is used as the backbone network. The input is a video block of size T×H×W×3 extracted from a temporal image window, where T is the temporal depth, for example, 8 frames; H and W correspond to the spatial dimensions. The backbone network performs forward propagation through a series of 3D convolutional, pooling, and non-linear activation layers. Output points are set in multiple intermediate layers of the network to extract the corresponding feature map for each frame in the video block. Assuming output points are set at downsampling rates of 4, 8, and 16 times, for a layer with a downsampling rate of 8 times, if the input frame spatial size is 640x480, the output feature map size of this layer is T×80×60×256, where 256 is the number of feature channels. That is, for each frame within the window, a dense feature map with a spatial resolution of 80x60 and a 256-dimensional feature vector is obtained. .

[0027] Step S1-2: Generate a semantic segmentation mask.

[0028] Specifically, a semantic segmentation network is used to perform forward reasoning on the current frame to be stabilized to generate an initial semantic segmentation mask. Based on the dense spatiotemporal feature set, a cross-attention mechanism is used to optimize the initial semantic segmentation mask to generate a semantic segmentation mask and a corresponding semantic confidence map.

[0029] In one possible embodiment, the semantic segmentation network adopts an encoder-decoder structure, wherein the encoder is used to perform fast downsampling to extract the semantic features contained in the current frame, such as using models such as MobileNetV3 or EfficientNet-Lite as the encoder; the decoder is used to progressively upsample and perform pointwise convolution to output the class probability of each pixel, and obtain an initial semantic segmentation mask through argmax operation. The initial semantic segmentation mask includes at least four categories: background, tissue, surgical tools, and transient interference such as blood or bubbles.

[0030] For example, the semantic segmentation network uses the current frame to be stabilized. Assuming the input is 640x480x3, the encoder first downsamples it to a low resolution of 1 / 32, 1 / 16, 1 / 8 of the original image, generating feature maps at multiple scales. The decoder then fuses features from different scale feature maps in the encoder through skip connections and gradually upsamples them to finally output a probability map of the same size as the input, i.e., 640x480. Each pixel in this probability map corresponds to a 4-dimensional vector, representing the probability of belonging to one of the four initial semantic segmentation mask categories. The dimension with the largest probability value in this 4-dimensional vector is the corresponding initial semantic segmentation mask category.

[0031] From the current frame to be stabilized Features of the corresponding region are extracted from the feature map of a certain scale in the dense spatiotemporal feature set. Then, through a cross-modal attention mechanism, the correlation weight map between the spatiotemporal features of the dense spatiotemporal feature set and the features extracted by the decoder of the semantic segmentation network is calculated. Based on this weight map, the boundary ambiguity or instantaneous misjudgment that may exist in the initial semantic segmentation mask is corrected. For example, at the boundary where the tissue and the tool are in contact, the movement of the tool may cause the segmentation to be blurred in a single frame, but the spatiotemporal features may show that the region belongs to the stable tissue in the previous and next frames. Through attention weighting, the final segmentation result will be corrected to the "tissue" category, thereby obtaining a more accurate boundary.

[0032] For example, suppose the decoder of a semantic segmentation network has a feature map size of 160x120 at a certain layer. First, it starts from the current frame to be stabilized. Feature maps with a spatial resolution of 160x120 are obtained from the corresponding dense spatiotemporal feature set. Next, the two types of features from the decoder and the dense spatiotemporal feature set are concatenated along the channel dimension, or an attention weight matrix is ​​calculated using a small convolutional network. This matrix has dimensions of 160x120x1 and values ​​between 0 and 1. In regions where spatiotemporal features and segmentation features are highly consistent, such as stable tissue texture areas, the attention weight value is close to 1. In inconsistent regions, such as motion-blurred instrument edges or newly appearing interference edges, the attention weight value is lower. Then, the decoder features are reweighted based on this attention weight matrix, i.e., attention weight * decoder feature + (1 - attention weight) * spatiotemporal features of the dense spatiotemporal feature set. Finally, the weighted and fused features are fed into a segmentation head to generate the final semantic segmentation mask with clearer boundaries. And a corresponding semantic confidence graph .

[0033] It should be noted that the semantic confidence graph... The confidence value for each pixel position is the maximum probability value of the initial semantic segmentation mask category corresponding to that pixel position. For example, assuming the classification output probability of a pixel in the image is {organization: 0.85, tool: 0.10, background: 0.03, transient interference: 0.02}, the maximum probability of 0.85 is taken as the confidence of that pixel, indicating that the semantic segmentation network is very confident that it belongs to the organization. Conversely, if the probability of another pixel is: {organization: 0.45, tool: 0.50, background: 0.03, transient interference: 0.02}, its maximum probability is only 0.50, and the confidence is very low, indicating that the network has difficulty distinguishing between organization and tool in this case, and the segmentation result is highly uncertain.

[0034] Step S2: Based on the preset image jitter analysis model, the dense spatiotemporal feature set and semantic segmentation mask are combined simultaneously for collaborative reasoning to obtain a multidimensional image jitter dataset.

[0035] Specifically, feature point matching and connection are performed based on the dense spatiotemporal feature set corresponding to adjacent frames of the frame to be stabilized within the temporal image window to construct a short-range trajectory set. Based on the semantic segmentation mask, the short-range trajectory set is classified into subsets of organization motion trajectory, tool motion trajectory, and background interference trajectory.

[0036] Furthermore, the subset of tissue motion trajectories is input as observation data into the image jitter analysis model for collaborative reasoning to obtain the corresponding global jitter motion data and non-rigid physiological motion data. A robust fitting method is then used to fit and analyze the subset of tool motion trajectories to obtain the corresponding tool motion transformation data.

[0037] It should be noted that the image jitter analysis model is represented as a generative probabilistic graphical model, which uses the 2D pixel coordinates of the trajectory points in the organized motion trajectory in each frame. For the observed variables, where For trajectory indexing, For frame indexing; and for each observation point Introduce a discrete ternary latent variable As a latent variable, it is used to indicate which motion layer the motion of the observation point mainly belongs to, where 1 indicates that it belongs to the global rigid motion layer, 2 indicates that it belongs to the non-rigid physiological motion layer, and 3 indicates that it belongs to the local residual or noise layer.

[0038] The image shake analysis model is used to decompose the motion patterns of a subset of tissue motion trajectories, generating global shake motion data and non-rigid physiological motion data; the global shake motion data is represented as an affine transformation sequence describing the global lens shake, i.e., using an affine transformation matrix. Modeling the inter-frame global transformation caused by camera shake; the non-rigid physiological motion data is represented as a non-rigid deformation field describing the autonomous deformation of organs. Specifically, a U×V sparse control grid is defined on the image plane, with each grid vertex... Associate a 2D displacement vector The entire parameter set The displacement of all vertices is defined for any point in the image. The corresponding displacement can be obtained by bilinear interpolation of the displacements of its four surrounding vertices, thus generating a parameterized dense displacement field; a residual noise parameter is obtained. This parameter represents the variance of residual Gaussian noise that is not explained by global jitter motion data and non-rigid physiological motion data.

[0039] The variational expectation-maximization algorithm is used to train the image jitter analysis model with the goal of maximizing the variational lower bound of the observed data until the preset training termination condition is reached.

[0040] That is, within the time series image window, the variational expectation-maximization algorithm is used to process the latent variables. posterior distribution and model parameters The maximum a posteriori estimate is used for approximate inference. Specifically, in the variational E-step, the model parameters are fixed. Optimize a latent variable variational distribution To make it approximate the true posterior, for example, assuming It is a mean field distribution, i.e. Each of them Represented as a classification distribution on {1,2,3}, by maximizing the lower bound of evidence, the output is the soft assignment probability of each trajectory point belonging to each motion layer. , .

[0041] In the variational M-step, the variational distribution is fixed. Update model parameters To maximize the lower bound of evidence, specifically, the points belonging to the global rigid motion layer, which are crucial for estimation Its contribution is large, and its corresponding weight is By solving the normal equations, the affine transformation matrix for each frame can be obtained. Highly efficient updates; updates The linear least squares problem, which involves a data term and a regularization term, can be quickly solved using the preconditional conjugate gradient method to obtain the optimal displacements of all control vertices. The data term is used to fit the observed displacements, and its weights are determined by… The decision is made; regularization terms such as Laplace smoothing can be used to penalize differences in the displacements of adjacent control vertices; the residual noise parameter is determined based on the weighted sum of squared residuals at all points. Update it, its weight is .

[0042] The variational E-step and M-step are performed alternately until the change in the lower bound of evidence is less than a preset threshold or the maximum number of iterations is reached. After convergence, the model parameters are obtained. The estimate and the probability of belonging to each point .

[0043] For example, for a trajectory point on a flexible tissue, observations over 5 frames within a window, in the early stages of inference, It may be uniformly distributed, but after several iterations, because this point exhibits a low-frequency, smooth displacement pattern consistent with its surrounding points, its... That is, the probability of belonging to the non-rigid motion layer may become very high, let's say 0.8. and The value is relatively low; in the variational M-step, this point contributes to the non-rigid displacement field with a weight of 0.8. The fit, at the same time, It mainly consists of points whose positions in the image are fixed and whose motion patterns are consistent throughout the entire frame, such as their High-resolution anatomical landmarks are used for estimation; ultimately, for the current frame... A set of global motion parameters is obtained { A non-rigid deformation parameter defined on the control mesh. Interpolation can be used to obtain any point on the image. Physiological exercise correction amount The entropy value of the probability of belonging to each trajectory point can be used as a measure of the uncertainty of motion estimation.

[0044] Furthermore, the global jitter motion data, non-rigid physiological motion data, tool motion transformation data, and corresponding motion estimation uncertainty data are encapsulated into a multi-dimensional image jitter dataset for output.

[0045] In one possible embodiment, for any two adjacent frames within a time-series image window and First, a feature map of a fixed level is selected from its corresponding dense spatiotemporal feature set, such as a feature map with a resolution of 1 / 8; then, in the reference frame... Regular grid sampling is performed on the feature map, or sampling is performed at corners where the image gradient is significant, to obtain a set of sparse source feature points. Then, for each source feature point... Extract a local feature descriptor from the feature map of its location in the target frame. In the feature map, The search is performed within a local search window centered on the coordinates of the descriptor, and k nearest neighbor candidate matching points are found by calculating the cosine distance or L2 distance between descriptors. Then, a ratio test and a bidirectional consistency check are used to filter these k nearest neighbor candidate matching points, retaining only bidirectionally matched feature point pairs. Recorded as a trajectory at time and According to the observation, if a trajectory can be successfully matched and satisfy geometric constraints in multiple consecutive frames within the time-series image window, then it is linked into a short-range trajectory.

[0046] For example, suppose that in a feature map at a certain level, in a frame... A point is sampled at coordinates (40, 30), and a 256-dimensional feature vector is extracted from the corresponding position in the feature map. This feature vector is then defined as the source vector. In the frame... Within a 10x10 local search window centered at (40,30), the distance between the feature vector and the source vector at all locations is calculated. Assuming the distance to the nearest neighbor candidate matching point is 0.1 and the distance to the second nearest neighbor candidate matching point is 0.5, the ratio between them is 0.2, which is lower than the preset threshold of 0.8. Therefore, the ratio test is passed. Furthermore, assuming the ratio is lower than the preset threshold of 0.8, the process continues... (40,30) inverse matching of feature map back to frame When examining the feature map, the original point was also found. That is, through bidirectional checking, this matching pair was identified as a short trajectory and may become part of a longer trajectory.

[0047] For each short-range trajectory, take its current frame. For each observation point, its coordinates are mapped back to the original image resolution, and the semantic segmentation mask is queried. At this coordinate location, the category label assigns the short-range trajectory to one of three different processing channels: Tissue Motion Trajectory Channel: If the category label is "tissue," the trajectory is marked as a tissue motion observation. All tissue motion observations together constitute a subset of tissue motion trajectories. Tissue motion observations are observation points attached to the surface of flexible tissue, and their motion patterns are a mixture of camera shake, organ physiological deformation, and noise. Tool Motion Trajectory Channel: If the category label is "tool," the trajectory is marked as a tool motion observation. All tool motion observations together constitute a subset of tool motion trajectories. Tool motion observations are modeled as rigid bodies or near-rigid bodies, and their motion is independent of the tissue, requiring separate processing to avoid contaminating the tissue motion pattern. Background Interference Trajectory Channel: For trajectories with the category label "background" or "interference," they are classified as a subset of background interference trajectories and discarded directly because they may originate from static scenes or unpredictable transient interference, contributing little to the estimation of meaningful motion patterns or easily introducing noise.

[0048] The subset of tissue motion trajectories is input as observation data into an image jitter analysis model for collaborative inference, thereby obtaining corresponding global jitter motion data, non-rigid physiological motion data, and the soft-assignment probability of each trajectory point belonging to each motion layer. .

[0049] A robust fitting method is used to fit and analyze the subset of tool motion trajectories to obtain the corresponding tool motion transformation data. Specifically, for all trajectory points marked as tools, a rigid body transformation matrix is ​​iteratively estimated using the RANSAC algorithm between each pair of adjacent frames in the sliding window. RANSAC fits the model by randomly sampling the minimum point set and counting the number of inliers. Finally, the model with the most inliers and the smallest fitting error is selected as the robust estimate of the tool motion between frames. For the current frame... The tool motion is usually represented as a transformation relative to the previous frame or a reference frame.

[0050] For example, suppose there are 20 points classified as belonging to the same surgical forceps. When estimating the tool motion from frame t-1 to frame t, RANSAC randomly selects 2 points to calculate a rigid body transformation, and then checks the reprojection error of the other 18 points under the transformation. If the error is less than a threshold, such as 2 pixels, it is counted as an inlier. After multiple random samplings, the transformation model with the most inliers is selected as the final robust estimate of the tool motion.

[0051] The soft-assignment probability of each trajectory point belonging to each motion layer is used. The mean or distribution statistics of the entropy are used to represent the corresponding motion estimation uncertainty data. Finally, the global jitter motion data, non-rigid physiological motion data, tool motion transformation data and the corresponding motion estimation uncertainty data are encapsulated into a multi-dimensional image jitter dataset for output.

[0052] Step S3: Based on the multidimensional image jitter dataset, optimize the real-time endoscope posture dataset within the time-series image window to obtain an optimized endoscope posture dataset.

[0053] Specifically, the global jitter motion data in the multidimensional image jitter dataset is integrated to obtain the preliminary camera projection matrix of each frame in the time-series image window relative to the starting frame. Uncertainty quantization is performed based on the preliminary camera projection matrix, and key frames are selected according to the result of uncertainty quantization. The key frames are image frames whose geometric uncertainty is lower than a preset threshold and whose tissue area visibility meets the preset conditions.

[0054] Furthermore, using the selected keyframes as nodes, the camera pose of the keyframes is optimized in conjunction with the multidimensional image jitter dataset. Based on the optimized camera pose of the keyframes, the camera pose of all non-keyframes within the temporal image window is optimized to obtain an optimized camera pose sequence.

[0055] Simultaneously, based on the semantic segmentation mask of the current frame to be stabilized, content-aware smooth path planning is performed to obtain a smooth target pose.

[0056] Furthermore, the optimized camera pose sequence and the smoothed target pose are encapsulated into an optimized endoscope pose dataset for output.

[0057] In one possible embodiment, the first frame of the selected time-series image window is... As the initial reference frame, its camera pose is defined as the unit pose, and for each subsequent frame... By continuously left-multiplying its relative motion transformation with the preceding frame The absolute pose is calculated recursively to obtain an initial camera pose sequence. }, which can be achieved by adopting The recursion is performed in the form of , where Represented as The camera pose corresponding to the frame, i.e., the corresponding preliminary camera projection matrix; It is a mapping function used to map global jitter motion data in a 2D image plane to camera pose changes in 3D space. For example, assuming It is a 2x3 affine matrix, representing the transition from frame... to frame The image transformation, by assuming a virtual calibration matrix and the average depth of the scene, backprojects this 2D transformation into a relative rotation and translation motion of the camera in 3D space. , and Let these be the corresponding rotation matrices and translation vectors, respectively. ,but This allows for the preliminary reconstruction of the corresponding camera motion sequence from the inter-frame image motion sequence.

[0058] For each initial camera projection matrix Assess the geometric uncertainty of its estimate. Specifically, using generation The two-dimensional feature matching points on which the motion estimation depends are used to calculate the reprojection error of these points under the current motion model. By analyzing the statistical distribution of these errors, such as calculating the trace or maximum eigenvalue of the covariance matrix, the reliability of the motion estimation can be quantified. That is, the more dispersed the error distribution and the larger the covariance, the better. The higher the value, It is eventually normalized to [0, 1], and the higher the value, the less reliable the initial pose estimation of the frame.

[0059] Semantic segmentation mask based on the current frame And by propagating it to other frames within the window through a motion model, the value of each frame is estimated. Organizational area visibility Specifically, it calculates the proportion of pixels with the mask category of "tissue" out of the total number of pixels in the image. The higher the proportion, the richer the target tissue content contained in the frame; calculate the normalized Euclidean distance from the weighted centroid of the tissue region to the image center. The smaller the distance, the closer the key content is to the center of the image; using , It is also normalized to [0, 1], and the higher the value, the better the visibility of the target organization in the frame.

[0060] Will Below the corresponding preset threshold and All frames exceeding a preset threshold are encapsulated as a candidate set of keyframes. From this candidate set, keyframes are selected... As the final keyframe, the selection strategy can adopt a priority selection approach. Lowest and The highest frame; if the number of candidate frames exceeds For frames, methods such as non-maximum suppression can be used to ensure that keyframes are evenly distributed in the time dimension to cover the motion information of the entire window.

[0061] Using the selected keyframes and their associated 2D feature trajectories from their short-range trajectory sets as input, the pose of the keyframes and the coordinates of 3D map points are jointly optimized by minimizing the reprojection error. Specifically, for 2D feature trajectory points observed in any two or more keyframes, their 3D spatial coordinates are initialized using their corresponding keyframe initial poses through linear triangulation or minimizing the reprojection error. Define the variable to be optimized as the set of poses from all keyframes. } and the set of all triangulated 3D points { The objective function is all observed two-dimensional feature points. The sum of squares of the reprojection errors, Represented as in the first The first keyframe observed A three-dimensional point cloud is obtained. An incremental Bayesian (BA) solver is used, which incrementally adds new keyframes and their observed feature points to the factor graph. The solver solves for the maximum a posteriori (MAP) estimate by maintaining and updating the information matrix of the Bayesian network. Furthermore, it optimizes only the local subgraph within the current window and retains the old state information of elements removed from the window as priors through marginalization, thus controlling computational load while maintaining accuracy. After optimization, the optimized keyframe pose and a more accurate three-dimensional point cloud are obtained.

[0062] For example, suppose keyframes KF1, KF2, and KF3 are selected, corresponding to frames t-5, t, and t+3 respectively. After selecting the corresponding feature trajectories observed in all three frames, firstly, based on... , , Triangulate a 3D point In BA optimization, The poses of the three keyframes are all variables, and the optimization process will continuously adjust them. , , and This makes it possible to The prediction positions are projected onto the corresponding positions of these three keyframes respectively.

[0063] The precise pose of all non-key frames is optimized using the optimized keyframe pose and the corresponding global jitter motion data. Specifically, for any non-key frame... Find the two adjacent keyframes before and after it. and ,in Non-keyframe The pose can be obtained by motion interpolation between the optimized poses of two keyframes, i.e., by calculating... Compared to The cumulative relative motion, and the cumulative relative motion through Function conversion to relative attitude transformation ,but The optimized posture can be expressed as ,in for The optimized pose is constructed by building a global pose graph based on the poses of all frames, including keyframes and non-keyframes. The poses of the keyframes optimized by BA are used as strong constraints, and the relative motion between frames included in the global jitter motion data is used as the basis for the optimization. Using edge constraints, a fast pose graph optimization is performed on all poses within the entire window with the goal of minimizing the constraint error between adjacent poses, thereby obtaining a globally consistent and smooth final camera pose sequence, which is the optimized camera pose sequence.

[0064] For example, frames It is a non-keyframe, preceded by a keyframe. Keyframes follow At this point, the output of step S2 is used. , , Calculate from arrive Cumulative transformation And convert it into a relative pose; then, use the optimized Multiplying by this relative posture, we get... Subsequently, in the global pose graph optimization, Will be affected by , and its adjacent non-keyframes , The combined influence of relative motion constraints between them allows for fine-tuning to achieve global optimum.

[0065] A Kalman smoother or spline fitting is used to smooth the position and rotation components of the optimized camera pose sequence in the time domain, resulting in a smoothed base path; based on the semantic segmentation mask of the current frame... Generate a semantic importance weight graph. For example, the weight of the lesion area is set to 1.0, the weight of healthy tissue is 0.5, and the weight of background and tools is 0.1. Calculate the visual center of the high-weighted area under the current optimized posture, calculate a desired image plane offset vector to make the visual center closer to the image center, offset the entire 2D image by the offset amount corresponding to the image plane offset vector, and back-project it into a virtual camera translation adjustment amount in 3D space through the camera intrinsic parameter matrix and the currently estimated scene depth. Fine-tune based on the virtual camera translation adjustment amount.

[0066] For example, suppose that after Kalman smoothing, the base smoothed translation of the current frame is... At the same time, according to Calculations revealed that the visual center of the lesion region was at image coordinates (300, 200), while the actual image center was (320, 240). Setting β=0.5 (β being a gain factor less than 1 to prevent over-adjustment), the image plane offset vector... Pixels, assuming a camera focal length of 500 pixels and an average depth of 0.1 meters, then rice, Meters. Therefore, the corresponding virtual camera translation adjustment amount. If the confidence level of the current semantic segmentation mask is 0.8, then the smoothed target pose is: .

[0067] Step S4: Based on the multidimensional image jitter dataset and the optimized endoscope pose dataset, perform initial stabilization repair on the current frame to be stabilized to obtain a primary stabilized frame.

[0068] Specifically, a homography transformation matrix describing the global basic motion is obtained by solving the optimized endoscope posture dataset, and a global basic deformation field is obtained based on the homography transformation matrix; a non-rigid correction field for compensating for physiological deformation is generated based on the non-rigid physiological motion data in the multidimensional image jitter dataset; and a motion compensation field for the tool region is obtained based on the tool motion transformation data in the multidimensional image jitter dataset.

[0069] Furthermore, based on the semantic segmentation mask, the global basic deformation field, non-rigid correction field, and motion compensation field are weighted and fused to generate an image deformation repair field. The current frame to be stabilized is then repaired according to the image deformation repair field to obtain a primary stable frame.

[0070] In one possible embodiment, firstly, the camera is calculated from the optimized camera pose sequence. Transform to smooth target pose Required global geometry mapping, assuming and All are 3x4 camera projection matrices, and all are in the form of ,in It is a 3x3 rotation matrix. It is a 3x1 translation vector. and Together they define the camera's position and orientation in the world coordinate system; specifically, they set... and The desired transformation is to make a three-dimensional point exist The projection below, and its in The projection of the image onto the plane is transformed by a homography transformation matrix on a two-dimensional image plane. Relatedly, assuming the scene primarily lies on an approximate plane, then this homography transformation matrix... A set of three-dimensional points located on an imaginary plane can be defined in a normalized camera coordinate system. This imaginary plane is represented as... In the camera coordinate system, using the average depth of the scene as the corresponding depth and with the normal vector along the optical axis, project this set of 3D points in the imaginary plane onto... and On the corresponding image plane, two sets of two-dimensional points are obtained. and Solving for the condition using the direct linear transformation algorithm 3x3 homography matrix ,in This is represented as equality in homogeneous coordinates. Used for encoding as compensation camera from arrive The pose changes of the image, and the overall, rigid projection transformation that the image pixels need to undergo.

[0071] Given For each integer pixel coordinate in the target image (i.e., the stabilized image) Calculate its value in the original image Corresponding coordinates in Specifically, calculate the inverse transformation in homogeneous coordinates: , obtain in Displacement vector required for position By traversing all target pixels, a dense displacement field of the same size as the target image is obtained. This dense displacement field is the global fundamental deformation field, and it stores a two-dimensional displacement vector at each pixel position.

[0072] Based on non-rigid physiological motion data, a free deformation model is used for linear interpolation to generate a non-rigid correction field. Specifically, suppose there is a UxV regular grid overlaid on the original image, with each grid vertex... There is a two-dimensional displacement For any point on the original image The corresponding non-rigid displacement can be obtained by bilinear interpolation of the displacements of the four vertices of the grid cell in which it resides. By traversing all pixels, a dense displacement field of the same size as the original image is generated, thereby generating the corresponding non-rigid correction field. , This is used to characterize the locally non-uniform pixel displacement required to counteract the peristalsis or deformation of the organ itself; it calculates the position of the pixels in the tool region in the original image after the tool motion transformation data transformation, and then calculates the difference between these positions and those after only the transformation data transformation. The differences between the transformed positions are then used to obtain a dense compensation field for the tool region through fitting or interpolation. , This is the motion compensation field of the tool area.

[0073] For example, suppose a tissue point has coordinates (200, 150) in the original image. The displacement given at this point is (5, -3), used to compensate for camera shake. Simultaneously, step S2 estimates that this point has a displacement of (1, 2) due to organ peristalsis, representing a downward and rightward movement. Therefore, to eliminate the influence of organ peristalsis in a stable image, a reverse correction needs to be applied, i.e. At that point, it should be (-1, -2); for a single surgical forceps tip in the same image, The given displacement is also (5, -3), but step S2 estimates that the pliers themselves undergo a translation, assuming its effect is to move the tip of the pliers to (202, 148). Therefore, An additional displacement needs to be calculated so that the position of the tip of the surgical forceps in the stabilized image is consistent with the motion logic of the forceps itself and coordinated with the globally stable background.

[0074] semantic segmentation mask Morphological dilation operations are performed on the "tissue" and "tool" regions in the image to obtain dilated tissue region masks and tool region masks. The intersection region of the two is defined as the transition zone, and the non-intersection region is the pure region of each region. For each pixel position on the target image... If it belongs to the pure region corresponding to the tissue region mask, then The final deformation field If it belongs to the pure region corresponding to the tool region mask, then the final deformation field of q is ;like If it is located in the transition zone, then its final deformation field is ,in , The weighting coefficients for the corresponding terms can be based on pixels. The distances to the pure organizational boundary and the pure tool boundary are set, i.e. , ,in and They are respectively Distance to the nearest non-transition band organization pixel and Distance to the nearest non-transition band tool pixel; for semantic segmentation masks The background and transient interference areas in the image are directly reused. .

[0075] For each pixel coordinate in the target image ,pass Find it in the original image The corresponding position in, i.e. ,according to The colors of the four nearest pixels are weighted and averaged to obtain the corresponding color value, thereby resampling the image to generate the corresponding primary stable frame.

[0076] It should be noted that during the resampling process, you may encounter... In cases where the image falls outside the original image boundary or where interpolation quality is poor in areas such as heavily deformed edges, an effective pixel mask label is simultaneously generated during the resampling process. It is used to identify which pixels in the target image were obtained through effective and reliable deformation and interpolation, that is, in In the middle, corresponding Pixels within the valid area of ​​the original image and with reasonable interpolation weights are marked as valid; otherwise, they are marked as invalid.

[0077] For example, suppose there is a pixel near the boundary between the organization and the tool. The calculated distance to the pure tissue boundary is 2 pixels, and the distance to the pure tool boundary is 8 pixels. Therefore... , Suppose at this point, The given displacement is , The given displacement is The displacement after fusion is During resampling, for =(100,200), search in the original image The color at a given location is obtained through bilinear interpolation and then filled into the stabilized image. The location, and set A value of 1 indicates that resampling of this pixel is valid.

[0078] Step S5: Perform temporal consistency fusion between the primary stable frame and the already stabilized temporal neighbor frame to generate the final stable frame corresponding to the current frame to be stabilized.

[0079] Specifically, a numerical optimization algorithm is used to solve the energy function corresponding to the primary stable frame and output the corresponding final stable frame. The energy function includes at least a data stabilization term, a temporal smoothing term, and a spatial smoothing term.

[0080] It should be noted that the data stabilization term is used to ensure the final output image. exist Within the effective deformation region of the marker, it should be as close as possible to the primary stable frame. It can be represented using a weighted L2 norm, for example, ,in, Represents the positions of all pixels in the image domain. for In position The value at the specified location; the temporal smoothing term is used to force the current frame... Stable frames adjacent to it , Maintaining continuity in content, it can be calculated. This is achieved by the difference between the optically flow-aligned neighboring frames, for example, ,in, It is a robust kernel function, such as Charbonnier loss or Huber loss, used to mitigate the effects of optical flow estimation errors or occlusion. Indicated from the current frame arrive The optical flow field of the frame; the spatial smoothing term is used to encourage the output image. It inherently possesses spatial smoothness to suppress noise or unnatural textures that may be introduced by deformation or fusion, for example, by employing a penalty on the image gradient, i.e. ),in, and These represent the gradient operators in the horizontal and vertical directions, respectively. Represented as a small constant The Charbonnier norm; ultimately, the energy function can be expressed as a weighted sum of the data stability term, the time-domain smoothing term, and the spatial smoothing term.

[0081] In one possible embodiment, the energy function is first iteratively solved using methods such as gradient descent, conjugate gradient, or preconditional conjugate gradient. This is combined with iterative reweighted least squares, transforming each iteration into a weighted least squares problem. The iteration stops when the change in the energy function value is less than a preset threshold or when the maximum number of iterations is reached. The final result is... This is the final stable frame after temporal smoothing. For example, suppose at a certain pixel position... , The color is red (255,0,0), and according to optical flow... In the previous stable frame The corresponding position in the image is dark red (240, 10, 10). The data stabilization term of the energy function tends to keep the pixel red, the temporal smoothing term tends to make it dark red to match the previous frame, and the spatial smoothing term references the colors of surrounding pixels. The iterative solution process of the energy function will find a balance point, such as a compromise red (248, 3, 5), which minimizes the sum of the three losses for effective pixel mask labels. In the invalid region, the data items have no effect, and the color of the pixel is mainly propagated from its valid neighborhood by the temporal and spatial smoothing items.

[0082] Next, uncertainty assessment is performed by combining the semantic confidence map corresponding to the semantic segmentation mask with the motion estimation uncertainty data in the multidimensional image jitter dataset to obtain the corresponding uncertainty assessment results. Based on these results, the corresponding operations in steps S1 to S4 are adaptively adjusted. Specifically, when the semantic confidence in the semantic confidence map is lower than a preset threshold, in step S3, which calculates a desired image plane offset vector, the smoothed base path is directly used as the corresponding image plane offset vector. Furthermore, the process in step S4 involving... The radius of morphological dilation is adjusted; when the motion estimation uncertainty data is lower than a preset threshold, the length of the temporal image window in steps S1 and S2 is increased, for example, from the default 15 frames to 31 frames. This provides longer historical information, enabling noise smoothing and outlier suppression. The increased window will gradually return to the default size after the uncertainty decreases.

[0083] This embodiment provides an electronic device, which may include: at least one processor, at least one network interface, a user interface, a memory, and at least one communication bus.

[0084] The following is a detailed introduction to the various components of the electronic device:

[0085] The communication bus can be used to enable communication between the various components mentioned above.

[0086] The user interface may include buttons, and optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0087] The network interface may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0088] The processor may include one or more processing cores. It connects various parts of the electronic device via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various functions and process data. Optionally, the processor can be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor can integrate one or more of the following: CPU, GPU, and modem, for example, one or more digital signal processors (DSPs) or one or more field-programmable gate arrays (FPGAs). The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0089] The memory may include RAM or ROM. Optionally, the memory may include a non-transitory computer-readable medium. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The memory, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an evaluation application. The processor may be used to call the evaluation application stored in the memory and execute the method steps mentioned in the foregoing embodiments.

[0090] It should be noted that the above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0091] The above embodiments can be implemented, in whole or in part, through software, hardware (such as circuits), firmware, or any other combination thereof.

[0092] When implemented using software, the above embodiments can be implemented in whole or in part as a computer program product, which includes one or more computer instructions or computer programs; when the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part.

[0093] It is understood that the computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device; the computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via transmission methods such as infrared, wireless, or microwave; the computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0094] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0095] It should be understood that, in the embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0096] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An active image stabilization method based on endoscopic image sequence coordination, characterized in that, It includes the following steps: S1: Obtain a temporal image window containing the current frame to be stabilized, and extract the dense spatiotemporal feature set of each frame in the temporal image window and the semantic segmentation mask of the current frame to be stabilized. The time-series image window is represented as a sliding window in the original endoscopic video stream, consisting of the current frame to be stabilized and n frames before and after the frame to be stabilized. The extraction of the dense spatiotemporal feature set is represented by using a pre-trained spatiotemporal feature encoder to extract features from the image sequence corresponding to each frame within the temporal image window, thereby obtaining the dense spatiotemporal feature set corresponding to each frame; The generation of the semantic segmentation mask for the current frame to be stabilized is represented by using a semantic segmentation network to perform forward reasoning on the current frame to be stabilized to generate an initial semantic segmentation mask. Based on the dense spatiotemporal feature set, the initial semantic segmentation mask is optimized using a cross-attention mechanism to generate a semantic segmentation mask and a corresponding semantic confidence map. S2: Based on the preset image jitter analysis model, the dense spatiotemporal feature set and semantic segmentation mask are combined synchronously to perform collaborative reasoning to obtain a multidimensional image jitter dataset; The collaborative reasoning is represented as performing feature point matching and connection based on the dense spatiotemporal feature set corresponding to the adjacent frames of the frame to be stabilized within the temporal image window, and constructing a short-range trajectory set. Based on the semantic segmentation mask, the short-range trajectory set is classified into a subset of organization motion trajectory, tool motion trajectory, and background interference trajectory. The subset of tissue motion trajectories is input as observation data into the image jitter analysis model for collaborative reasoning to obtain the corresponding global jitter motion data and non-rigid physiological motion data. A robust fitting method is used to fit and analyze the subset of the tool's motion trajectory to obtain the corresponding tool motion transformation data. The global jitter motion data, non-rigid physiological motion data, tool motion transformation data, and corresponding motion estimation uncertainty data are encapsulated into a multi-dimensional image jitter dataset for output. S3: Based on the multidimensional image jitter dataset, optimize the real-time endoscope posture dataset within the time-series image window to obtain an optimized endoscope posture dataset; The acquisition of the optimized endoscope pose dataset is represented by integrating the global jitter motion data in the multidimensional image jitter dataset to obtain the preliminary camera projection matrix of each frame in the time-series image window relative to the starting frame. Uncertainty quantization is performed based on the preliminary camera projection matrix, and keyframes are selected based on the results of uncertainty quantization. The keyframe is represented as an image frame whose geometric uncertainty is below a preset threshold and whose tissue region visibility meets a preset condition. Using the selected keyframes as nodes, the camera pose of the keyframes is optimized in conjunction with the multidimensional image jitter dataset; The camera pose of all non-key frames within the temporal image window is optimized based on the optimized keyframes to obtain an optimized camera pose sequence. Based on the semantic segmentation mask of the current frame to be stabilized, perform content-aware smooth path planning to obtain a smooth target pose; The optimized camera pose sequence and the smoothed target pose are encapsulated into an optimized endoscope pose dataset for output. S4: Based on the multidimensional image jitter dataset and the optimized endoscope pose dataset, perform initial stabilization repair on the current frame to be stabilized to obtain a primary stabilized frame; The acquisition of the primary stable frame is represented by solving the homography transformation matrix describing the global basic motion based on the optimized endoscope pose dataset, and obtaining the global basic deformation field based on the homography transformation matrix. A non-rigid correction field is generated to compensate for physiological deformation based on non-rigid physiological motion data in a multidimensional image jitter dataset. Based on the tool motion transformation data in the multidimensional image jitter dataset, obtain the motion compensation field of the tool region; Based on the semantic segmentation mask, the global basic deformation field, non-rigid correction field and motion compensation field are weighted and fused to generate an image deformation repair field. Based on the image deformation repair field, the current frame to be stabilized is deformed and repaired to obtain a primary stable frame; S5: Perform temporal consistency fusion between the primary stabilized frame and the already stabilized temporal nearest neighbor frame to generate the final stabilized frame corresponding to the current frame to be stabilized.

2. The active image stabilization method based on endoscopic image sequence coordination according to claim 1, characterized in that, Image jitter analysis models include: The image jitter analysis model is represented as a generative probabilistic graphical model, which is used to decompose the motion patterns of a subset of tissue motion trajectories to generate global jitter motion data and non-rigid physiological motion data. The global jitter motion data is represented as an affine transformation sequence used to describe the global jitter of the lens; The non-rigid physiological motion data is represented as a non-rigid deformation field describing the autonomous deformation of the organ; The variational expectation-maximization algorithm is used to train the image jitter analysis model with the goal of maximizing the variational lower bound of the observed data until the preset training termination condition is reached.

3. The active image stabilization method based on endoscopic image sequence coordination according to claim 1, characterized in that, The primary stabilized frame is fused with its temporally nearest-neighbor frames using temporal consistency fusion to generate the final stabilized frame corresponding to the current frame to be stabilized, including: A numerical optimization algorithm is used to solve the energy function corresponding to the primary stable frame, and the corresponding final stable frame is output. The energy function includes at least a data stability term, a time-domain smoothing term, and a spatial smoothing term; While solving the energy function, uncertainty assessment is performed by combining the semantic confidence map corresponding to the semantic segmentation mask with the motion estimation uncertainty data in the multidimensional image jitter dataset. Obtain the corresponding uncertainty assessment results, and adaptively adjust the corresponding operations of S1 to S4 based on the uncertainty assessment results.

Images