Instant image splicing method and device for scanning endoscope and storage medium

Abstract

The invention discloses an instant image splicing method and device for a scanning endoscope and a storage medium, and belongs to the technical field of endoscope image processing. The method comprises the steps that continuously-collected images are input one by one; judging a motion state, and dividing the input image into a static image and a motion image; judging whether visual continuity is satisfied if the static image is used as the next spliced image, if not, inserting the corrected moving image as supplement, and generating a to-be-spliced image sequence without motion distortion; and registering each to-be-spliced image in the sequence with the spliced image in the process to obtain an optimal projection position, projecting the to-be-spliced image to the panoramic image output at the last time according to the position, and fusing to obtain the panoramic image output at the time. According to the invention, the visual fault caused by motion distortion can be solved in the continuous imaging process, and the visual continuous panoramic image can be output in real time, so that real-time diagnosis and focus positioning can be conveniently carried out in an endoscopic operation.

Description

Technical Field

[0001] The present invention belongs to the technical field of endoscopic image processing, and more specifically, relates to an instant image stitching method, device, and storage medium for a scanning endoscope. Background Art

[0002] Scanning endoscopes are a type of microscopic endoscope based on the principle of point-scanning imaging, mainly including confocal endoscopes, two-photon endoscopes, multi-photon endoscopes, etc. Scanning endoscopes can be used as real-time optical biopsy tools, capable of obtaining images with cell resolution in real-time, in-situ, and non-invasively, and have broad clinical application prospects. For example, confocal endoscopes have been approved for clinical use in early cancer screening in China. However, when using a scanning endoscope for large-area examinations, the probe movement and data acquisition occur simultaneously, and it is difficult for the operator to obtain reliable and sufficient diagnostic information within the constantly changing limited field of view. Therefore, an image stitching method is required to register and fuse the overlapping parts of the currently acquired continuous image sequence into a panoramic image to instantaneously provide the entire examined field of view.

[0003] The core step of image stitching is the registration of the overlapping parts. Existing image registration methods are not applicable to scanning endoscope data because scanning endoscopes image by point scanning, and only one point of data is acquired at each moment, that is, different pixels within the same image are obtained at different times. When the probe moves during imaging, the transformation between adjacent images includes not only rigid displacement and rotation but also elastic motion distortion, and the latter will lead to the registration difficulty of inconsistent structural features in the overlapping parts, that is, the motion distortion problem brings additional challenges to the stitching of scanning endoscope images. Summary of the Invention

[0004] In view of the above defects or improvement requirements of the prior art, the present invention provides an instant image stitching method, device, and storage medium for a scanning endoscope, aiming to solve the technical problem of motion distortion in the instant image stitching process of existing scanning endoscopes.

[0005] To achieve the above object, according to one aspect of the present invention, an instant image stitching method for a scanning endoscope is provided, including:

[0006] S1: Input images I obtained by the image acquisition module of the scanning endoscope one by one k ;

[0007] S2: Determine whether the image acquisition module is in a static state or a moving state when acquiring the image I k ; if it is in a static state, then use the image I k as a static image F s, and enter S3; if in the motion state, then use the image I k as the motion image F d and store it in the motion image list {F d}, and return to S1;

[0008] S3: Obtain the overlap rate between the static image F s and the last image F in the key image sequence i-1 ; the key image sequence {F1,..., F i-1} is used to splice and fuse to generate the panoramic image output last time; if the overlap rate is lower than the first threshold, perform motion correction on all the stored motion images F d to obtain the restored image F c , select a part from the set {F c} as elements in the to-be-spliced image sequence {F key} according to the first threshold, and enter S4; if it is higher than the first threshold and lower than the second threshold, then use the static image F s as an element in the to-be-spliced image sequence {F key}, and directly enter S4; if it is higher than the second threshold, return to S1;

[0009] S4: For each element F in the to-be-spliced image sequence {F key}, perform the following steps:

[0010] S41: One by one, use the to-be-spliced image F key as the current splicing image F key , and register the current splicing image F i with each image in the key image sequence {F1,..., F i} that has an overlapping relationship with F i-1 respectively to obtain the possible projection positions {u i} corresponding to F i , and select the best projection position corresponding to F i from them i ;

[0011] S42: On the panoramic image output last time, calculate the projection mask of F according to the best projection position i , project according to the projection mask to obtain the projection image, fuse both sides of the projection mask boundary in the projection image, and obtain the panoramic image output immediately this time; add the current splicing image F i to the end of the key image sequence {F1,..., F i-1};

[0012] until the entire sequence of images to be stitched {F key} is traversed, then return to S1.

[0013] Further, in S2, determining whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k includes:

[0014] Obtaining the similarity metric value between the image I k and the previous image I k-1 . When the similarity metric value meets the preset condition, it is considered that the image acquisition module is in a stationary state; otherwise, it is considered that the image acquisition module is in a moving state.

[0015] Further, in S2, determining whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k includes:

[0016] Cropping a matching template at the center position of the previous image I k-1 , searching for the matching window with the highest similarity to the matching template within the image I k , and obtaining the displacement estimation parameter and the similarity metric value V representing the similarity between the matching template and the matching window;

[0017] When and V > V m , it is determined that the image acquisition module is in a stationary state when acquiring I k ; otherwise, it is determined that the image acquisition module is in a moving state; where T e is the drift threshold of the stationary field of view, and V m is the background noise threshold.

[0018] Further, in S3, performing motion correction on all the stored motion images F d to obtain the restored image F c , and selecting a part from the set {F c} as elements in the sequence of images to be stitched {F key} according to the first threshold, including:

[0019] Step a: Using the end image F i-1 as the reference image I f , and using the first image in the list of motion images {F d} as the moving image I m , where the image coordinate system of the end image F i-1 is the global coordinate system G of the motion correction;

[0020] Step b: Extracting the reference image If and the moving image I m of feature points to obtain an initial feature matching set {M} between the feature points;

[0021] Step c: Convert the initial feature matching set {M} into local motion state parameters of the image acquisition module of the scanning endoscope, and use a first fitting method to fit a local motion model based on the local motion state parameters, and delete the initial feature matches that do not conform to the local motion model from the initial feature matching set {M} to obtain a robust feature matching set {M*};

[0022] Step d: Perform sampling time interpolation in the overlapping regions corresponding to the robust feature matching set {M*}, perform linear extrapolation in the non-overlapping regions, and combine the results of sampling time interpolation and linear extrapolation to generate a local dense optical flow field that completely covers the moving image I m field of view

[0023] Step e: Set the current moving image as the new reference image I f , set the next image in the list of motion images as the new moving image I m , and for the new reference image I f and the new moving image I m execute steps b - d to obtain the local dense optical flow field sets of each motion image in the list of motion images {F d} with respect to the previous motion image

[0024] Step f: For all the moving image feature points within all the robust feature matching sets {M*} search forward for each according to the local dense optical flow field sets the position P in the global coordinate system G G , generate a globally ordered observation value list {z G} and calculate the corresponding global motion state parameters; use a second fitting method to fit a global motion model based on the global motion state parameters, and generate a global dense optical flow field set of all the motion image sequences {F d} according to the global motion model

[0025]

[0026] Step g: Use each of the global dense optical flow fields to remap the corresponding motion image F d to obtain the corresponding restored image F c , and according to the first threshold of image overlap, from the set of restored images {F cSelect one or more from them as the image sequence {F} to be stitched key} as elements in

[0027] Further, the step c includes:

[0028] Using the acquisition time t m of the feature points of the moving image I k as the independent variable, and using the pixel coordinate displacement T f of the feature points of the reference image I corresponding to the feature points k as the dependent variable, taking [t k , T k as the k-th observation value z k , and then sorting according to the acquisition time t k to generate an observation value sequence: Calculate the corresponding local motion state parameters according to the observation value sequence {z k};

[0029] Take the local motion state parameters as the input of the first fitting algorithm, and fit the local motion model of the relative motion between the image acquisition module when acquiring the moving image I m and the reference image I f ;

[0030] For all acquisition times {t k} in the observation value sequence, generate the fitting value of the pixel coordinate displacement by the local motion model k Delete the initial feature match corresponding to the observation value T of the pixel coordinate displacement with too large deviation from the fitting value from the initial feature match set {M}, and obtain the robust feature match set {M*}.

[0031] Further, the step f includes:

[0032] For all moving image feature points in all robust feature match sets {M*} Search forward according to the local dense optical flow field set to find the position P of each G in the global coordinate system G, and generate a global ordered observation value list: And calculate the corresponding global motion state parameters; where is the global sampling time of the k-th feature point;

[0033] Fitting the motion image sequence {F based on the global motion state parameters by using the second fitting method d} to the corresponding global motion model; continuously sampling the non-overlapping regions according to the global motion model, and interpolating the overlapping regions corresponding to the global ordered observation value list {z G}, and combining the results of sampling and interpolation to generate the global dense optical flow field set of all motion images F d to the end image F i-1 . Further, the S42 includes:

[0034] S421: Extracting the panoramic sub-field of view FOV corresponding to the current stitching image F i within the field of view of the previously output panoramic image according to the optimal projection position i , where the center position of the effective region of the panoramic sub-field of view FOV i is C i , and the center positions of the effective regions of the field of view FOV i of the key images {F j} overlapping with the panoramic sub-field of view FOV j are {C j}; generating a convex polygon inside the FOV i according to the relative positions of C j and {C i};

[0035] S422: Using the convex polygon as a projection mask, projecting the current stitching image F i only inside the projection mask of the panoramic sub-field of view FOV i to obtain a projection image; fusing both sides of the projection mask boundary in the projection image to obtain the panoramic image output immediately this time; adding the current stitching image F i to the end of the key image sequence {F1,..., F i-1}.

[0036] According to another aspect of the present invention, there is provided an instant image stitching device for a scanning endoscope, including:

[0037] An image input module for sequentially inputting the images I k acquired by using the image acquisition module of the scanning endoscope;

[0038] A motion judgment module for judging whether the image acquisition module is in a stationary state or a motion state when acquiring the image I k ; if in a stationary state, taking the image I k as a static image F s, and enter the sequence generation module to be stitched; if in the motion state, the image I k is used as the motion image F d and stored in the motion image list {F d}, and then return to the image input module;

[0039] The sequence generation module to be stitched is used to obtain the overlap rate between the static image F s and the end image F in the key image sequence i-1 ; the key image sequence {F1,..., F i-1} is used to splice and fuse to generate the panoramic image output last time; if the overlap rate is lower than the first threshold, all the stored motion images F d are motion-corrected to obtain the restored image F c , and a part is selected from the set {F c} according to the first threshold and regarded as an element in the sequence {F key} to be stitched, and then enter the panoramic image generation module; if the overlap rate is higher than the first threshold and lower than the second threshold, the static image F s is used as an element in the sequence {F key} to be stitched, and directly enter the panoramic image generation module; if the overlap rate is higher than the second threshold, return to the image input module;

[0040] The panoramic image generation module is used to perform the following steps for each element F key in the sequence {F key} to be stitched:

[0041] One by one, the image F to be stitched key is used as the current stitched image F i , and the current stitched image F i is registered with each image in the key image sequence {F1,..., F i-1} that has an overlapping relationship with F i respectively to obtain the possible projection positions {u i} corresponding to F i}, and the best projection position corresponding to F i is preferentially obtained from them

[0042] On the panoramic image output last time, calculate the projection mask of F according to the best projection position i , project according to the projection mask to obtain the projection image, and fuse both sides of the projection mask boundary in the projection image to obtain the panoramic image output immediately this time; add the current stitched image F i to the key image sequence {F1,..., Fi-1} end;

[0043] until the entire sequence of images to be stitched {F has been traversed key} then return to the image input module.

[0044] According to another aspect of the present invention, there is provided a scanning endoscope, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0045] According to another aspect of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0046] Generally speaking, compared with the prior art by the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0047] (1) The present invention provides an instant image stitching method for a scanning endoscope. After inputting images one by one and making motion judgments, static images that meet the overlap threshold or restored images after motion correction are selected for stitching and fusion, and a panoramic image with an extended field of view is instantaneously output. First, considering that the scanning endoscope is based on the point-scanning imaging principle, a motion model fitting method is used to correct images with motion distortion, innovatively realizing the function of correcting images with large-scale motion distortion. At the same time, considering the high-time-efficiency requirement for field-of-view expansion, a motion judgment module and a sequence generation module of images to be stitched are designed to avoid repeated calculations of the same field of view, and an innovative projection mask generation method is proposed in the stitching and fusion step, only fusing near the boundary of the projection mask to improve efficiency. Finally, the method provided by the present invention realizes instantaneously providing a visually continuous panoramic image when using a scanning endoscope for large-area examinations, avoiding visual tomograms introduced by motion distortion, and being able to synchronously locate the position of the image acquisition module, facilitating real-time diagnosis and lesion location during endoscopic surgery.

[0048] (2) The reason for calculating the similarity of continuously input images to judge the motion state in this solution is that considering the fixed structure of the object to be imaged, the similarity between continuously acquired input images is negatively correlated with the moving amplitude of the corresponding image acquisition module, with low computational complexity and high accuracy.

[0049] (3) Considering that during the acquisition process, there may be jumps in image quality caused by unstable contact forces between the image acquisition module and the object to be imaged, such as changes in contrast, signal-to-noise ratio, brightness, etc., which will introduce background noise and cause field-of-view drift. By using the method of central template matching and setting a threshold, interference can be excluded to achieve a robust motion state judgment.

[0050] (4) When performing motion distortion correction on the motion image in this solution, the first fitting method is first used to locally filter out incorrect matches, and then the second fitting method is used to globally complete the information of non-overlapping regions; because it is considered that the acquisition times of each pixel in the image acquired by the scanning endoscope are different, so this technical solution not only performs registration of overlapping regions in the spatial dimension, but also compensates for the lack of spatial information in non-overlapping regions through motion model fitting in the time dimension; this technical solution combines spatio-temporal information to generate an optical flow field and obtains a corrected restored image through reprojection, innovatively realizing the image correction function for images with large motion distortion, ensuring the output stability of the instant image stitching system of the scanning endoscope, thereby breaking the moving speed limit of the image acquisition module existing in the existing stitching methods.

[0051] (5) This solution considers that the scanning endoscope has a high pixel acquisition frequency, and the scanning start point of the moving image is relatively stationary with respect to the scanning end point of the reference image. Therefore, the displacement between each pair of features in the initial feature matching set can be converted into local motion parameters of the image acquisition module, and the first fitting method is used to generate a local motion model, and incorrect matches in the initial feature matching set are deleted according to this model. This technical method solves the problem that the proportion of incorrect matches in the results is too large when the existing feature matching method is applied to the motion image of the scanning endoscope, and has strong interpretability and an expandable application range.

[0052] (6) The local dense optical flow field in this solution contains results obtained by linear speculation in non-overlapping regions, which does not conform to the real motion model and cannot be directly used for image correction; to solve the error introduced by linear speculation, the second fitting method is used to generate a global motion model corresponding to the entire motion image list, and then the optical flow field corresponding to the non-overlapping region is corrected through continuous sampling. This technology makes full use of the spatio-temporal continuity of motion and realizes the correction of motion images lacking overlapping information.

[0053] (7) In this solution, it is considered that image registration has been achieved and the panoramic image only shows a single field of view, so it is not necessary to operate on all pixels in the panoramic sub-field of view; the present invention generates a projection mask through the relative positions of overlapping images, projects the current stitched image inside the mask and performs fusion only near the boundary of the projection mask, thereby improving the efficiency of projection and fusion and helping to realize the instant output of a panoramic image without stitching traces. Description of the Drawings

[0054] Figure 1 It is a flowchart of an instant image stitching method for a scanning endoscope provided in Embodiment 1 of the present invention;

[0055] Figure 2It is a schematic diagram of the effect of an instant image stitching method for a scanning endoscope provided in Embodiment 1 of the present invention. Detailed implementation manners

[0056] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0057] Embodiment 1

[0058] As Figure 1 shown, this embodiment provides an instant image stitching method for a scanning endoscope, including: S1-S4.

[0059] S1: Input the images I obtained by the image acquisition module of the scanning endoscope one by one k .

[0060] S2: Determine whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k ; if it is in a stationary state, then take the image I k as the static image F s , and enter S3; if it is in a moving state, then take the image I k as the moving image F d and store it in the moving image list {F d}, and return to S1.

[0061] S3: Obtain the overlap rate between the static image F s and the end image F i-1 in the key image sequence; the key image sequence {F1,..., F i-1} is used to splice and fuse to generate the panoramic image output last time; if the overlap rate is lower than the first threshold, then perform motion correction on all the stored moving images F d to obtain the restored image F c , select a part from the set {F c} as the elements in the to-be-stitched image sequence {F key} according to the first threshold, and enter S4; if the overlap rate is higher than the first threshold and lower than the second threshold, then take the static image F s as the element in the to-be-stitched image sequence {F key}, and directly enter S4; if the overlap rate is higher than the second threshold, return to S1.

[0062] S4: For each element F in the to-be-stitched image sequence {F key}key , perform the following steps: S41: One by one, take the image F to be stitched key as the current stitched image F i , and for each image in the key image sequence that overlaps with F i , perform registration respectively to obtain the possible projection positions {u i} corresponding to F, and select the best projection position corresponding to F i from them i ; S42: On the previously output panoramic image, calculate the projection mask of F according to the best projection position i , project according to the projection mask to obtain a projection image, fuse both sides of the projection mask boundary in the projection image, and obtain the panoramic image output immediately this time; add the current stitched image F to the end of the key image sequence. Until the image sequence {F } to be stitched is traversed, then return to S1 i . i The execution process of the present invention is as follows: First, input the continuously acquired images I key one by one; then, match I

[0063] with I to judge the motion state, and classify I k as a static image F k or a motion image F k-1 ; Second, when the overlap between the static image F k and the last key image F s is insufficient, perform motion correction on all the motion images F d between them to generate a restored image F s , select a part from the static image F i-1 and the optional restored image set {F d} according to the visual continuity requirement to generate a sequence of images to be stitched {F c} without motion distortion; Finally, for each element F s in the sequence of images to be stitched {F c}, register F key with all the images in the key image sequence {F1,..., F key} that overlap with F key to obtain the possible projection positions {u key} corresponding to F i-1 , select the best projection position corresponding to F key from them i ; extract the panoramic sub-field of view according to i , perform zonal projection and fusion on it, and immediately output a new panoramic image, and add F i to the key image sequence Extract the panoramic sub-field of view, perform zonal projection and fusion on it, and immediately output a new panoramic image, and add F to the key image sequencei to the end of the key image sequence {F1, ..., F i-1}, and until after traversing {F key}, a next input is performed. Repeat the above steps until there is no new input.

[0064] Among them, the optimal projection position is the coordinate transformation that maximizes the similarity of all overlapping regions related to F i : In the formula, F j is the key image in the key image sequence {F1, ..., F i-1} that has an overlapping relationship with F i , u i is the coordinate transformation implemented on F i , the operator is the operation of implementing the coordinate transformation of the left term on the right term, Ω(·) is the operation of taking the overlapping region of the two images in the parentheses, sim(·) is the calculation of image similarity, and w ij is the weight factor related to the overlapping rate.

[0065] As an optional implementation manner, in S2, it is judged whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k , including: obtaining the similarity metric value between two consecutive input images I k-1 and I k . When the similarity metric value meets the preset condition, it is considered that the image acquisition module is in a stationary state, otherwise it is considered that the image acquisition module is in a moving state.

[0066] Furthermore, in S2, it is judged whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k , including: using the template matching method to calculate the image similarity between two consecutive images I k-1 and I k , outputting the similarity metric value of the matching template and the most similar matching window, and judging whether the image acquisition module is in a stationary state or a moving state when acquiring the image I k according to the preset condition.

[0067] Specifically, for two consecutive input images: I k-1 and I k , the matching template is cropped at the center position of I k-1 , the matching window with the largest normalized correlation coefficient related to the template is searched within I k , and the displacement and the normalized correlation coefficient V of this window and the matching template are obtained. When is satisfied and V > V m , it is determined that I is acquired.k When the image acquisition module is stationary, assume I k For a static image F s , the process enters S3; if the condition is not met, it is determined that the collection I k When the image acquisition module moves, the register I k To the motion picture list d} and returns S1. Among them, T e is the drift threshold of the stationary field of view, V m is the background noise threshold, and both are empirical thresholds related to the template size.

[0068] As an optional implementation, considering the point scanning imaging principle of the scanning endoscope, a pixel acquisition time variable is added to the motion distortion correction process. By combining the temporal continuity of the image acquisition module movement and the spatial continuity of the imaging target, the free motion process of the image acquisition module is fitted, thereby restoring the underlying position of the pixels in the motion image. That is, in S3, all the motion images F stored are d Perform motion correction to obtain the restored image F c , according to the first threshold from the set {F c} select a part as the image sequence to be stitched key}, the specific steps are as follows: step a-step g.

[0069] It should be noted that Figure 2 This is a schematic diagram of the effect of a real-time image stitching method for a scanning endoscope provided in Example 1 of the present invention. The letter labels therein only indicate the order of the front and back, and have no corresponding relationship with the step numbers ag below. al is a partial ordered output of the embodiment. The blue dot marks the field of view center of the starting stitched image, the red dot marks the field of view center of the current stitched image, the green dot marks the field of view center of other stitched images, the dark green line and the purple arrow indicate the scanning trajectory and direction, and the red box is the current stitched image. The results show that this method can instantly provide a panoramic field of view that has been scanned by a scanning endoscope, and overcome motion distortion to maintain visual continuity.

[0070] Step a: Transform the final image F i-1 As the reference image I f , the motion image list {F d The first image in the} is used as the moving image I m , where the terminal image F i-1 The image coordinate system is the global coordinate system G for motion correction.

[0071] Step b: Extract reference images I separately f and moving image I mFeature points to obtain an initial set of feature matches {M} between the feature points, where each feature match M includes the feature point P f belonging to I f and the feature point P m belonging to I m .

[0072] Step c: Convert the initial set of feature matches {M} into local motion state parameters of the image acquisition module of the scanning endoscope. Use the first fitting method to fit a local motion model based on the local motion state parameters, and delete the initial feature matches that do not conform to the local motion model from the initial set of feature matches {M} to obtain a robust set of feature matches {M*}. Among them, according to the point scanning imaging principle, the acquisition time of the feature point P m and the displacement between the matching features are used as observation values to calculate the local motion state parameters.

[0073] Step d: Perform sampling time interpolation in the overlapping regions corresponding to the robust set of feature matches {M*}, perform linear extrapolation in the non-overlapping regions, and combine the results of sampling time interpolation and linear extrapolation to generate a local dense optical flow field that completely covers the moving image I m field of view

[0074] Step e: Set the current moving image as the new reference image I f , set the next image in the list of moving images as the new moving image I m , and perform steps b - d for the new reference image I f and the new moving image I m to obtain the local dense optical flow field sets of each moving image in the list of moving images {F d} with respect to the previous moving image

[0075] Step f: For all the moving images I m in the robust set of feature matches {M*}, for all the moving image feature points in accordance with the local dense optical flow field sets look forward to find the position P of each G in the global coordinate system G, generate a list of globally ordered observations {z G} and calculate the corresponding global motion state parameters. Use the second fitting method to fit a global motion model based on the global motion state parameters, and generate the global dense optical flow field sets of all the moving images F d to the end image F i-1

[0076] Step g: Utilize each global dense optical flow field​ Remap the corresponding motion image F d to obtain a restored image F c , and select one or more from the set of restored images {F c} as elements in the sequence of images to be stitched {F key} according to the first threshold of image overlap.

[0077] As an alternative implementation, step c includes: taking the acquisition time t m of the feature points of the moving image I k as the independent variable, and taking the pixel coordinate displacement T f of the feature points of the reference image I corresponding to the feature points k as the dependent variable, and taking [t k , T k as the k-th observation value z k , then sort by the acquisition time t k to generate an observation value sequence: and calculate the corresponding local motion state parameters; use the local motion state parameters as the input of the first fitting algorithm to fit the local motion model of the relative motion between the image acquisition module and the reference image I m when acquiring the moving image I f ; for all acquisition times {t k}, generate the fitted value of the pixel coordinate displacement from the local motion model Delete the initial feature matches corresponding to the large deviation between the observation value T k and the fitted value from the initial feature match set {M} to obtain a robust feature match set {M*}.

[0078] It should be noted that the first fitting algorithm can be a non-linear fitting algorithm such as Kalman filter, extended Kalman filter, particle filter, etc. For example, the extended Kalman filter can be used to simulate the non-linear state transition process of the free movement of the image acquisition module while the scanning endoscope is imaging, and filter out the outliers in the initial feature match set {M} between two continuously acquired images to generate a robust feature match set {M*}. Specifically, the process equation and parameter settings of the extended Kalman filter algorithm are as follows:

[0079] Prediction process:

[0080] Calculate the Kalman gain K:

[0081] Update process:

[0082] Among them, the motion state of the image acquisition module at the k-th observation point is: x k = [p k , v k , a k , that is, the displacement vector p, the velocity vector v, and the acceleration vector a. The external control vector u is the time derivative of a; the observation matrix H, the state transition matrix F, and the control matrix B are all derived from the motion equation; the initial state distribution P0, the process noise Q0, and the observation noise R are all measured from the static image.

[0083] As an optional implementation manner, step f includes: for all moving image feature points within all robust feature matching sets {M*} Search forward according to the local dense optical flow field set Find the position P in the global coordinate system G G , and generate a global ordered observation value list: And calculate the corresponding global motion state parameters; where is the sampling time of the k-th feature point in the global coordinate system G; use the second fitting method to fit the global motion model corresponding to the motion image sequence {F d}, continuously sample the non-overlapping area according to the global motion model, and interpolate the overlapping area corresponding to the global ordered observation value list {z G}, and combine the results of sampling and interpolation to generate the set of global dense optical flow fields of all motion images F d to the end image F i-1

[0084] Among them, it should be noted that the second fitting algorithm can be a nonlinear fitting algorithm such as Kalman filter, extended Kalman filter, particle filter, etc. For example, the extended Kalman filter can be used and the boundary conditions can be set to fit the complete "stationary - moving - stationary" process corresponding to the motion image sequence {F d}, solve the problem of motion discontinuity caused by linear speculation in the non-overlapping area during the calculation of the local dense optical flow field , and generate the smooth global dense optical flow field set of all motion images F d to the end image F i-1

[0085] As an optional implementation manner, S42 includes: S421 - S422.

[0086] Among them, S421: Extract the current stitching image F within the field of view of the previously output panoramic image according to the best projection position i ​​The corresponding panoramic sub - field of view FOV i , where the center position of the effective area of the panoramic sub - field of view FOV i is C i , and the key image {F i} that overlaps with the panoramic sub - field of view FOV j has the center position of the effective area of the corresponding panoramic sub - field of view FOV j as {C j}; According to the perpendicular bisector of the line segment between C i and {C j}, cut the FOV i , and each time of cutting, only keep the area on the same side of the tangent as C i , and finally cut to generate a convex polygon. S422: Use the convex polygon as a projection mask, and project F i only inside the projection mask of the FOV i to obtain a projected image; perform weighted fusion on the projected images on both sides of the boundary near the projection mask boundary to obtain the panoramic image output immediately this time; add the current stitched image F i to the end of the key image sequence {F1,..., F i-1}.

[0087] Embodiment 2

[0088] This embodiment provides an instant image stitching device for a scanning endoscope, including: an image input module, a motion judgment module, a sequence - to - be - stitched generation module, and a panoramic image generation module.

[0089] The image input module is used to input the images I k acquired by the image acquisition module of the scanning endoscope one by one.

[0090] The motion judgment module is used to judge whether the image acquisition module is in a static state or a moving state when acquiring the image I k ; if it is in a static state, take the image I k as the static image F s , and enter the sequence - to - be - stitched generation module; if it is in a moving state, take the image I k as the moving image F d and store it in the moving image list {F d}, and return to the image input module.

[0091] The sequence - to - be - stitched generation module is used to obtain the overlap rate between the static image F s and the end - image F i-1 in the key image sequence; the key image sequence {F1,..., F i-1} is used to stitch and fuse the panoramic image outputted last time; if the overlap rate is lower than the first threshold, all the stored motion images F d Perform motion correction to obtain the restored image F c , according to the first threshold from the set {F c} select a part as the image sequence to be stitched key} and enter the panoramic image generation module; if the overlap rate is higher than the first threshold and lower than the second threshold, the static image F s As the image sequence to be stitched key} and directly enter the panoramic image generation module; if the overlap rate is higher than the second threshold, return to the image input module.

[0092] Panoramic image generation module, used for the image sequence to be stitched key Each element F in key , perform the following steps: First, the images to be stitched F are key As the current stitched image F i , the current stitched image F i With the key image sequence {F1, ..., F i-1 Same as F i The images with overlapping relationship are registered to obtain F i The corresponding possible projection position {u i}, select the best one to get F i The corresponding optimal projection position Then, on the last exported panoramic image, press the best projection position Calculate F i The projection mask is projected according to the projection mask to obtain the projection image, and the two sides of the projection mask boundary in the projection image are fused to obtain the panoramic image output in this instant; add the current stitching image F i To the key image sequence {F1, ..., F i-1}End; until the sequence of images to be spliced ​​is traversed {F key} then returns to the image input module.

[0093] Example 3

[0094] This embodiment provides a scanning endoscope, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0095] Example 4

[0096] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0097] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A real-time image stitching method for a scanning endoscope, characterized in that: include: S1: Input the images I acquired by the image acquisition module of the scanning endoscope one by one k ; S2: Determine and collect the image I k When the image acquisition module is in a stationary state or a moving state; if it is in a stationary state, the image I k As a static image F s , and enter S3; if it is in motion, the image I k As a moving image F d Save to the motion image list {F d }, and return to S1; S3: Acquire the static image F s and the terminal image F in the key image sequence i-1 The overlap ratio of the key image sequence {F1, ..., F i-1 } is used to stitch and fuse the panoramic image outputted last time; if the overlap rate is lower than the first threshold, all the stored motion images F d Perform motion correction to obtain the restored image F c , according to the first threshold from the set {F c } select a part as the image sequence to be stitched key }, and enter S4; if the overlap rate is higher than the first threshold and lower than the second threshold, the static image F s As the image sequence to be stitched key } and directly enter S4; if the overlap rate is higher than the second threshold, return to S1; S4: for the image sequence to be stitched {F key Each element F in key , perform the following steps: S41: The images to be stitched F are stitched one by one key As the current stitched image F i , the current stitched image F i With each of the key image sequences {F1, ..., F i-1 Same as F i The images with overlapping relationship are registered to obtain F i The corresponding possible projection position {u i }, select the best one to get F i The corresponding optimal projection position S42: On the panoramic image output last time, according to the optimal projection position Calculate F i The projection mask is projected according to the projection mask to obtain a projection image, and the two sides of the projection mask boundary in the projection image are fused to obtain the panoramic image output in this instant; the current stitching image F is added i To the key image sequence {F1, ..., F i-1 }end; Until the sequence of images to be stitched is traversed, key } then returns to S1.

2. The real-time image stitching method of a scanning endoscope according to claim 1, characterized in that: In S2, it is determined that the image I is collected k When the image acquisition module is in a stationary state or a moving state, it includes: Get Image I k With the previous image I k-1 When the similarity measurement value meets the preset condition, the image acquisition module is regarded as being in a stationary state; otherwise, the image acquisition module is regarded as being in a moving state.

3. The real-time image stitching method of a scanning endoscope as claimed in claim 2, characterized in that: In S2, it is determined that the image I is collected k When the image acquisition module is in a stationary state or a moving state, it includes: In the previous image I k-1 The matching template is cut out at the center of the image I k Search for the matching window with the greatest similarity to the matching template to obtain the displacement estimation parameter and the similarity metric value V characterizing the similarity between the matching template and the matching window; When satisfied And V>V m When I k When the image acquisition module is in a stationary state, otherwise it is determined that the image acquisition module is in a moving state; wherein, T e is the drift threshold of the static field of view, V m is the background noise threshold.

4. The real-time image stitching method of a scanning endoscope as claimed in claim 1, characterized in that: In S3, all the motion images F stored are d Perform motion correction to obtain the restored image F c , according to the first threshold from the set {F c } select a part as the image sequence to be stitched key }, including: Step a: Transform the terminal image F i-1 As the reference image I f , the motion image list {F d The first image in the} is used as the moving image I m , where the terminal image F i-1 The image coordinate system is the global coordinate system G for motion correction; Step b: Extract reference images I separately f and moving image I m feature points to obtain an initial feature matching set {M} between the feature points; Step c: converting the initial feature matching set {M} into local motion state parameters of the image acquisition module of the scanning endoscope; fitting a local motion model based on the local motion state parameters using a first fitting method, deleting initial feature matching that does not conform to the local motion model from the initial feature matching set {M}, and obtaining a robust feature matching set {M*}; Step d: Perform sampling time interpolation in the overlapping area corresponding to the robust feature matching set {M*}, perform linear inference in the non-overlapping area, and combine the results of sampling time interpolation and linear inference to generate a complete coverage of the moving image I m Local dense optical flow field of the field of view Step e: Set the current moving image as the new reference image I f , set the next one in the motion image list as the new moving image I m , for the new reference image I f and new moving images I m Execute steps b to d to obtain the motion image list {F d The local dense optical flow field set of each moving image and the previous moving image in} Step f: For all robust feature matching sets {M*}, match all moving image feature points According to the local dense optical flow field collection Look forward for each The position P in the global coordinate system G G , generate a global ordered observation list {z G } and calculate the corresponding global motion state parameters, use the second fitting method to fit the global motion model based on the global motion state parameters, and generate all the motion image sequences according to the global motion model {F d }'s global dense optical flow field set Step g: Using each of the global dense optical flow fields The corresponding motion image F d Remap to get the corresponding restored image F c , according to the first threshold of image overlap, from the restored image set {F c } select one or more images as the image sequence to be stitched {F key } in the element.

5. The real-time image stitching method of a scanning endoscope as claimed in claim 4, characterized in that: The step c comprises: With the moving image I m Feature points The acquisition time t k is the independent variable, and the reference image I f Feature points With feature points The corresponding pixel coordinate displacement T k As the dependent variable, [t k ,T k ] as the kth observation z k , then press the acquisition time t k Sorting generates a sequence of observations: According to the observation value sequence {z k } Calculate the corresponding local motion state parameters; The local motion state parameters are used as the input of the first fitting algorithm to fit the image acquisition module when acquiring the moving image I m When the reference image I f The local motion model in which the relative motion occurs; For all acquisition times {t k } Generate pixel coordinate displacement fitting values ​​from the local motion model Delete the observed value T of the pixel coordinate displacement from the initial feature matching set {M} k With the fitted value The initial feature matching corresponding to the excessive deviation obtains the robust feature matching set {M*}.

6. The real-time image stitching method of a scanning endoscope as claimed in claim 4, characterized in that: The step f comprises: For all moving image feature points in the robust feature matching set {M*} According to the local dense optical flow field collection Look forward for each The position P in the global coordinate system G G , which generates a globally ordered list of observations: And calculate the corresponding global motion state parameters; is the global sampling time of the kth feature point; Using the second fitting method to fit the motion image sequence based on the global motion state parameters d } corresponding to the global motion model; continuously sampling the non-overlapping area according to the global motion model, and performing a global ordered observation list {z G } The corresponding overlapping areas are interpolated, and the results of sampling and interpolation are combined to generate all motion images F d To the end image F i-1 The global dense optical flow field collection 7. The real-time image stitching method of a scanning endoscope as claimed in claim 1, characterized in that: The S42 includes: S421: According to the optimal projection position The current stitched image F is extracted within the field of view of the last output panoramic image. i Corresponding panoramic sub-field of view FOV i , where the panoramic sub-field of view FOV i The center position of the effective area is C i , and panoramic sub-field of view FOV i There are overlapping key images {F j }Corresponding field of view FOV j The center position of the effective area is {C j }; According to C i and {C j }'s relative position in FOV i Generate a convex polygon inside; S422: Using the convex polygon as a projection mask, only in the panoramic sub-field of view FOV i The projection mask internally projects the current stitched image F i Get the projection image; fuse the two sides of the projection mask boundary in the projection image to get the panoramic image output in real time; add the current stitching image F i To the key image sequence {F1, ..., F i-1 }end.

8. A real-time image stitching device for a scanning endoscope, characterized in that: include: An image input module is used to input images I acquired by the image acquisition module of the scanning endoscope one by one. k ; A motion judgment module is used to judge the acquisition of the image I k When the image acquisition module is in a stationary state or a moving state; if it is in a stationary state, the image I k As a static image F s , and enter the sequence generation module to be spliced; if it is in motion, the image I k As a moving image F d Save to the motion image list {F d }, and return to the image input module; The module for generating the sequence to be spliced ​​is used to obtain the static image F s and the terminal image F in the key image sequence i-1 The overlap ratio of the key image sequence {F1, ..., F i-1 } is used to stitch and fuse the panoramic image outputted last time; if the overlap rate is lower than the first threshold, all the stored motion images F d Perform motion correction to obtain the restored image F c , according to the first threshold from the set {F c } select a part as the image sequence to be stitched key }, and enter the panoramic image generation module; if the overlap rate is higher than the first threshold and lower than the second threshold, the static image F s As the image sequence to be stitched key } and directly enter the panoramic image generation module; if the overlap rate is higher than the second threshold, return to the image input module; Panoramic image generation module, used for the image sequence to be stitched key Each element F in key , perform the following steps: The images to be stitched F are stitched one by one key As the current stitched image F i , the current stitched image F i With each of the key image sequences {F1, ..., F i-1 Same as F i The images with overlapping relationship are registered to obtain F i The corresponding possible projection position {u i }, select the best one to get F i The corresponding optimal projection position On the panoramic image output last time, according to the optimal projection position Calculate F i The projection mask is projected according to the projection mask to obtain a projection image, and the two sides of the projection mask boundary in the projection image are fused to obtain the panoramic image output in this instant; the current stitching image F is added i To the key image sequence {F1, ..., F i-1 }end; Until the sequence of images to be stitched is traversed, key } then returns to the image input module.

9. A scanning endoscope, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images