Method and apparatus for processing magnifying endoscopy image, and storage medium

Abstract

Disclosed in the present invention are a method and apparatus for processing a magnifying endoscopy image, and a storage medium. The apparatus comprises: a reference image acquisition module; a magnified image acquisition module; a feature extraction and matching module, which is used for analyzing feature points and feature descriptors of a conventional-magnification image and a high-magnification image, and performing feature point matching, so as to obtain a set of matched feature point pairs; a geometric transformation computation and image registration module, which is used for estimating a geometric transformation from the high-magnification image to a reference-magnification image, so as to obtain an optimal homography matrix between the high-magnification image and the reference-magnification image; an image fusion module, which is used for aligning the high-magnification image, and seamlessly fusing the reference-magnification image and the aligned high-magnification image, so as to obtain a panoramic image; and a user interaction and real-time display module, which is used for parsing a user interaction instruction and displaying the panoramic image on the basis of the user interaction instruction. The present invention requires a small computation amount, and can provide in real time a seamless panoramic image under a conventional magnification and a high magnification, thereby facilitating a clinical endoscopic examination performed by a doctor.

Description

An endoscope magnification image processing method, device and storage medium

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to a Chinese patent application filed at the earliest on September 10, 2024 with the China Patent Office, application number 202411258947.2, and entitled "An endoscope magnification image processing method, device and storage medium", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of image analysis and processing, and particularly relates to an endoscope magnification image processing method, device and storage medium. BACKGROUND

[0004] At present, endoscopy has become one of the important means for diagnosing digestive tract lesions, and plays a key role in the early detection and diagnosis of gastrointestinal diseases. With the continuous progress of endoscopy technology, doctors can obtain clearer mucosal detail images through magnifying endoscopy. However, the existing endoscopy technology has the problem of limited image field of view during operation. Magnifying endoscopy can provide high-resolution local images, but its field of view is small and it is difficult to fully display the overall view of the entire examination area. The existing endoscopy technology cannot realize real-time alignment and splicing of images at different magnifications during the switching between conventional magnification and high magnification. These problems cause the doctor to lose the overall view during the examination process, increasing the difficulty of diagnosis during the operation process, and important lesion areas are easily missed, so image splicing is needed.

[0005] The image spliced by the doctor manually needs the doctor to have high operation skills and experience, and at the same time, splicing errors are easily caused by improper operation, leading to misjudgment of the lesion position. In addition, manual splicing is very time-consuming and labor-intensive, so intelligent splicing is usually used. However, in the prior art, the image processing method usually needs a large amount of calculation and post-processing calculation steps when implementing image splicing, which makes it difficult to realize real-time splicing and display in the clinical examination process, affecting the examination efficiency. Therefore, in view of the problem of slow real-time splicing speed in the prior art, it is urgent to develop an endoscope magnification image processing method, device and storage medium. SUMMARY

[0006] In view of the problems of large calculation amount and insufficient real-time performance of magnifying endoscope image splicing processing, the present application provides an endoscope magnification image processing method, device and storage medium to provide doctors with more intuitive and detailed panoramic images in real time.

[0007] The technical solution adopted by the present application to solve the technical problems is as follows:

[0008] In a first aspect, the present application provides an endoscope image processing device, comprising:

[0009] a reference image acquisition module, configured to decode the normal magnification video stream output by the endoscope device into static images frame by frame and cache, to obtain a normal magnification image sequence;

[0010] a magnified image acquisition module, configured to decode the high magnification video stream output by the endoscope device into static images frame by frame and cache, to obtain a magnified magnification image sequence;

[0011] a feature extraction and matching module, configured to analyze the feature points and the corresponding feature descriptors of the normal magnification images and the magnified magnification images, to perform feature point matching according to the feature descriptors of the normal magnification images and the feature descriptors of the magnified magnification images, to obtain a matched feature point pair set;

[0012] a geometric transformation calculation and image registration module, configured to estimate the geometric transformation of the magnified magnification image to the reference magnification image, to obtain an optimal homography matrix between the magnified magnification image and the reference magnification image;

[0013] an image fusion module, configured to align the magnified magnification image using the optimal homography matrix, to seamlessly fuse the reference magnification image and the aligned magnified magnification image to obtain a panoramic image;

[0014] a user interaction and real-time display module, configured to obtain and parse a user interaction instruction, and to display the panoramic image according to the parsing result.

[0015] In a preferred embodiment, the device further comprises an image preprocessing module, configured to pre-process the normal magnification image sequence and the magnified magnification image sequence to optimize the image quality.

[0016] In a preferred embodiment, the feature extraction and matching module comprises a scale-invariant feature transform subunit and a feature descriptor generation subunit; the scale-invariant feature transform subunit is configured to perform multi-scale analysis on the normal magnification images and the magnified magnification images, the multi-scale analysis comprising finding feature points and calculating the directions of the feature points, and generating a Gaussian difference pyramid of the images; the feature descriptor generation subunit is configured to calculate a histogram of oriented gradients in the neighborhood of each feature point in the Gaussian difference pyramid of the images, to generate a feature descriptor.

[0017] In a preferred embodiment, the feature extraction and matching module comprises a FLANN feature matching subunit and a matching filtering subunit; the FLANN feature matching subunit is configured to construct an index tree by a FLANN algorithm, match feature descriptors to obtain matched feature point pairs; the matching filtering subunit is configured to filter out mismatched point pairs by using a bidirectional nearest neighbor matching strategy, retain matched feature point pairs with the smallest matching distance, and obtain a matched feature point pair set.

[0018] In a preferred embodiment, the geometric transformation calculation and image registration module comprises a homography matrix solving subunit and a random sample consensus subunit; the homography matrix solving subunit is configured to solve a homography matrix between the magnified image and the reference magnification image by using a least square method; the random sample consensus subunit is configured to iteratively sample in the matched feature point pairs by using a random sample consensus algorithm, and estimate an optimal homography matrix.

[0019] In a preferred embodiment, the image fusion module comprises an image alignment unit and a multi-scale image fusion unit; the image alignment unit is configured to map the magnified image into a coordinate system of the reference magnification image based on the optimal homography matrix; the multi-scale image fusion unit is configured to perform Gaussian blur and down-sampling operations on the reference magnification image and the aligned magnified image to generate a Gaussian pyramid, perform Laplacian pyramid generation by Gaussian difference calculation, perform weighted fusion in each layer of the Laplacian pyramid, and generate a seamless fusion image, which is the panoramic image.

[0020] In a preferred embodiment, the user interaction and real-time display module is specifically configured to obtain the panoramic image and a user interaction instruction, parse the user interaction instruction, and update the display of the panoramic image in real time according to the user interaction instruction.

[0021] In a preferred embodiment, the device further comprises a lesion identification module configured to automatically identify a lesion from the panoramic image by using a deep learning model.

[0022] In a second aspect, the present application provides a magnified endoscope image processing method, comprising:

[0023] frame-by-frame decoding and caching a regular magnification video stream output by an endoscope device into static images to obtain a regular magnification image sequence, and frame-by-frame decoding and caching a high magnification video stream output by the endoscope device into static images to obtain a magnified magnification image sequence;

[0024] analyzing feature points and corresponding feature descriptors of the regular magnification images and the magnified magnification images, performing feature point matching according to the feature descriptors of the regular magnification images and the feature descriptors of the magnified magnification images to obtain a matched feature point pair set;

[0025] Estimate the geometric transformation of the magnification image to the reference image to obtain the optimal homography matrix between the magnification image and the reference image;

[0026] Align the magnification image using the optimal homography matrix, seamlessly fuse the reference image and the aligned magnification image to obtain the panoramic image;

[0027] Obtain and parse the user interaction instruction, and display the panoramic image according to the parsing result.

[0028] In a third aspect, the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the magnification endoscope image processing method of the second aspect.

[0029] The magnification endoscope image processing method, device and storage medium of the present application match feature points through the feature extraction and matching module, obtain the optimal homography matrix between the magnification image and the reference image through the geometric transformation calculation and image registration module, obtain the seamless fusion of the magnification image to the panoramic image through the image fusion module, and realize the display of the panoramic image according to the user's intention through the user interaction and real-time display module. Compared with the prior art, the present application has small calculation amount, fast image processing, and can display the panoramic image without post-processing, can realize real-time endoscope image alignment and splicing under normal magnification and high magnification, can provide real-time panoramic image, and is a seamless panoramic inspection view, improves the visual effect, can display according to the user interaction instruction of the doctor, is beneficial to the endoscope examination work of the doctor, and based on the present application, the accuracy and efficiency of endoscope examination can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0031] Fig. 1 is a framework diagram of a magnification endoscope image processing system;

[0032] Fig. 2 is a flowchart of a magnification endoscope image processing method. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0034] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0035] It should be noted that the terms "first", "second", and the like in the description of the present application are intended to only describe different instances and thus cannot be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. The "and / or" mentioned in the present application means any and all combinations of one or more related listed items. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of the technical personnel in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0036] The existing magnifying endoscope image stitching has large calculation amount and insufficient real-time performance. Therefore, the present application provides a magnifying endoscope image processing device, which comprises:

[0037] A reference image acquisition module 10 is configured to decode a regular magnification video stream output by an endoscope device into static images frame by frame and cache to obtain a regular magnification image sequence;

[0038] A magnifying image acquisition module 20 is configured to decode a high magnification video stream output by the endoscope device into static images frame by frame and cache to obtain a magnifying magnification image sequence;

[0039] A feature extraction and matching module 30 is configured to analyze feature points (a feature point set) and corresponding feature descriptors of a regular magnification image (in the regular magnification image sequence) and a magnifying magnification image (in the magnifying magnification image sequence), perform feature point matching according to the feature descriptors of the regular magnification image and the feature descriptors of the magnifying magnification image, and obtain a matched feature point pair set;

[0040] A geometric transformation calculation and image registration module 40 is configured to estimate a geometric transformation of the magnifying magnification image to the reference magnification image to obtain an optimal homography matrix between the magnifying magnification image and the reference magnification image;

[0041] An image fusion module 50 is configured to align the magnifying magnification image by using the optimal homography matrix, seamlessly fuse the reference magnification image and the aligned magnifying magnification image to obtain a panoramic image;

[0042] A user interaction and real-time display module 60 is configured to obtain and analyze a user interaction instruction, and display the panoramic image according to the analysis result.

[0043] In a preferred embodiment, the regular magnification image sequence is a pre-processed regular magnification image sequence, and the magnified magnification image sequence is a pre-processed magnified magnification image sequence, and the pre-processing is to optimize the image quality.

[0044] The implemented embodiment can be to set an image pre-processing module, which is used to pre-process the regular magnification image sequence and the magnified magnification image sequence to optimize the image quality, to obtain the pre-processed regular magnification image sequence and the pre-processed magnified magnification image sequence.

[0045] Specifically, the image pre-processing module includes an image pre-processing unit. The image pre-processing unit inputs the obtained regular magnification image sequence and the magnified magnification image sequence; and internally includes a denoising and enhancement subunit, which is used to optimize the image quality, and specifically, through standard filtering and equalization processing, to obtain the pre-processed regular magnification image sequence and the pre-processed magnified magnification image sequence.

[0046] The implemented embodiment can also be to set a first pre-processing unit for pre-processing the regular magnification image sequence in the reference image acquisition module 10, and set a second pre-processing unit for pre-processing the magnified magnification image sequence in the magnified image acquisition module 20.

[0047] It can be understood that the regular magnification image and the magnified magnification image involved in the feature extraction and matching module 30, the geometric transformation calculation and image registration module 40, and the image fusion module 50 are respectively the pre-processed regular magnification image sequence and the pre-processed magnified magnification image sequence.

[0048] The reference image acquisition module 10 includes an image acquisition unit. The image acquisition unit inputs the regular magnification video stream output by the endoscope device; and internally includes a first video decoding and caching subunit, which is used to decode the video stream into frame-by-frame regular magnification images and cache, that is, to decode the video signal into static images; through decoding and caching operations, to obtain the regular magnification image sequence composed of frame-by-frame regular magnification images.

[0049] The magnified image acquisition module 20 includes a magnified image acquisition unit. The magnified image acquisition unit inputs the high magnification video stream output by the endoscope device; and internally includes a second video decoding and caching subunit, which is used to decode the high magnification video stream into frame-by-frame magnified magnification images and cache; through decoding and caching operations, to obtain the magnified magnification image sequence composed of frame-by-frame magnified magnification images.

[0050] The feature extraction and matching module 30 is used to analyze the regular magnification images in the regular magnification image sequence to obtain a set of feature points, which can be called the set of feature points one, and to analyze the magnification images in the magnification image sequence to obtain a set of feature points, which can be called the set of feature points two. Then, feature points one and feature points two are matched, and all the matched feature point pairs constitute the set of matched feature point pairs. The feature extraction and matching module 30 includes a feature extraction unit and a feature matching unit.

[0051] The feature extraction unit takes as input a preprocessed sequence of images at normal magnification and a preprocessed sequence of images at magnification; it contains a scale-invariant feature transform (SIFT) subunit and a feature descriptor generation subunit, used to extract key feature points and descriptors from the images; through multi-scale analysis and directional gradient calculation, it obtains a set of feature points and corresponding feature descriptors.

[0052] The Scale Invariant Feature Transform (SIFT) subunit is used to perform multi-scale analysis on the input image. This multi-scale analysis includes finding feature points and calculating their orientations to generate a Difference of Gaussian (DoG) pyramid for the image. The feature point detection formula is as follows: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y) (1)

[0053] Where D(x,y,σ) represents the Gaussian difference in scale space, G() represents the Gaussian blur function, I(x,y) represents the input image, (x,y) represents the coordinates of the pixels in the input image, σ represents the variable kernel, and k is a positive integer representing the k-th cascaded filter through which the input image passes in the Gaussian difference pyramid.

[0054] The feature descriptor generation subunit is used to calculate the Histogram of Oriented Gradients (HOG) in the neighborhood of each feature point in the Gaussian difference pyramid of the image, generating a 128-dimensional feature descriptor. The formula for calculating the feature descriptor is:

[0055] Where Descriptor represents the feature descriptor, m(θ) is the magnitude of the directional gradient, θ represents the gradient direction, and bin r Let be the r-th directional interval of the directional gradient histogram, where r is a positive integer.

[0056] The feature matching unit inputs feature descriptors of the regular magnification image and feature descriptors of the magnified magnification image; internally contains a FLANN feature matching subunit and a matching filtering subunit, and is used for matching feature points between different magnification images to obtain matched feature point pairs; through FLANN (Fast Library for Approximate Nearest Neighbors) approximate nearest neighbor search and a bidirectional filtering strategy, finally outputs a matched feature point pair set.

[0057] The FLANN feature matching subunit is used for matching the feature descriptors of the regular magnification image and the feature descriptors of the magnified magnification image to obtain matched feature point pairs; and the matching filtering subunit is used for filtering mismatched feature point pairs.

[0058] The FLANN feature matching subunit uses a FLANN (Fast Library for Approximate Nearest Neighbors) algorithm to construct an index tree, quickly matches the feature descriptors, and obtains matched feature point pairs. The search parameters are set as the number of trees t = 8 and the number of candidate points n = 2.

[0059] The matching filtering subunit adopts a bidirectional nearest neighbor matching strategy to filter mismatched point pairs, retains matched feature point pairs with the smallest matching distance, and obtains a matched feature point pair set. Specifically, the matched feature point pairs are filtered according to matching score calculation, and the matching score calculation formula is: Δ (p i ,p j ) = ||d i -d j || 2 (3)

[0060] wherein d i , d j represent feature descriptors, i and j represent feature descriptor index serial numbers, Δ (p i ,p j ) represents the Euclidean distance of feature descriptors d i and d j , and (p i ,p j ) represents a matched feature point pair, wherein p i represents a feature point in the reference magnification image, and p j represents a feature point in the magnified magnification image.

[0061] The geometric transformation calculation and image registration module 40 specifically estimates a geometric transformation of the magnified magnification image to the reference magnification image according to the matched feature point pair set.

[0062] The geometry transformation calculation and image registration module 40 comprises a geometry transformation estimation unit.

[0063] The geometry transformation estimation unit takes the matched feature point pair set as input, and comprises a homography matrix solving subunit and a random sample consensus (RANSAC) subunit, which are used to estimate the geometry transformation from the magnification image to the reference magnification image, and obtain the optimal homography matrix by using the least square method and the RANSAC algorithm.

[0064] The homography matrix solving subunit is used to solve the homography matrix H between the magnification image and the reference magnification image by using the least square method, and the formula is as follows:

[0065] The random sample consensus subunit is used to estimate the optimal homography matrix by iteratively sampling in the matched feature point pairs by using the RANSAC algorithm, and the optimal homography matrix is taken as the geometry transformation from the magnification image to the reference magnification image. The optimization objective function of the RANSAC algorithm is as follows:

[0066] wherein, H best represents the optimal homography matrix, M represents the matched feature point pair set obtained by the feature extraction and matching module 30, II() is an indicator function, and ε is an inlier distance threshold value;

[0067] The image fusion module 50 comprises an image alignment unit and a multi-scale image fusion unit,

[0068] The image alignment unit takes the magnification image and the optimal homography matrix as input, and comprises a perspective transformation subunit, which is used to accurately map the magnification image into the coordinate system of the reference magnification image. By using the perspective transformation formula based on the optimal homography matrix, the aligned magnification image is obtained, that is, the image alignment technology is used to realize the alignment of the magnification image and the reference magnification image.

[0069] The perspective transformation subunit: performs perspective transformation on the magnification image based on the optimal homography matrix, and the transformation formula is as follows: J J

[0070] wherein, J”() represents the aligned magnification image, (x', y') represents the target coordinate, J'() represents the magnification image, (x J J ) represents the original coordinate of the magnification image, and H() represents the transformation function of the optimal homography matrix perspective transformation.

[0071] ​​​The multi-scale image fusion unit inputs the reference magnification image and the aligned magnification image; it internally contains a Gaussian-Laplacian pyramid generation subunit and an image fusion subunit, which are used to generate multi-scale representations and perform seamless fusion; through multi-scale fusion technology, the final panoramic image is obtained.

[0072] The Gaussian-Laplacian pyramid generation subunit is used to perform Gaussian blur and down-sampling operations on the image (input of the multi-scale image fusion unit) to generate a Gaussian pyramid, and a Laplacian pyramid is formed through Gaussian difference calculation. The formula of the Gaussian-Laplacian pyramid generation subunit is: G L(k')=I G (k'-1)), L(k')=I G (k'-1)), L(k')=I G (k'+1)) (7)

[0073] wherein k' represents the layer of the Gaussian pyramid, i.e., the k'th layer; I G () is the Gaussian pyramid representation, which specifically represents the output image of a certain layer of the Gaussian pyramid; I G (0) represents the input image of the Gaussian pyramid (reference magnification image or aligned magnification image), and L() is the Laplacian pyramid representation.

[0074] The image fusion subunit is used to perform weighted fusion in the Laplacian pyramid of each layer to generate a seamless fusion image and obtain the final panoramic image. The fusion formula is:

[0075] wherein I F represents the fusion result, i.e., the panoramic image; L J (k') is the Laplacian pyramid representation of the magnification image; L I (k') is the Laplacian pyramid representation of the reference magnification image; a k' represents a weight factor, which controls the fusion proportion of images at different scales.

[0076] The user interaction and real-time display module 60 includes a user interface unit. The user interface unit inputs the panoramic image and user interaction instructions; it internally contains an interaction analysis subunit and a display update subunit, which are used to analyze user input and update the display of the panoramic image in real time; through interaction analysis and rendering operations, the display image of the panoramic image is obtained.

[0077] In the embodiment, the device further comprises a lesion recognition module, the input is a panoramic image, and the output is a lesion recognition result, including but not limited to a lesion position and a lesion type. The lesion recognition result is obtained by automatically recognizing the lesion by using a deep learning model. The lesion recognition and evaluation are performed by fully utilizing the detailed information in the high magnification image. The lesion recognition and evaluation are accurate. The problem that the existing endoscope device mainly provides a simple image magnification function, lacks intelligent analysis and diagnosis support, and cannot fully utilize the detailed information in the high magnification image for lesion recognition and evaluation is solved.

[0078] The application provides a magnifying endoscope image processing method.

[0079] S1, decode the regular magnification video stream output by the endoscope device into static images frame by frame and cache to obtain a regular magnification image sequence, and decode the high magnification video stream output by the endoscope device into static images frame by frame and cache to obtain a magnifying image sequence;

[0080] S2, analyze feature points and corresponding feature descriptors of the regular magnification image and the magnifying image, perform feature point matching according to the feature descriptors of the regular magnification image and the feature descriptors of the magnifying image, and obtain a matched feature point pair set;

[0081] S3, estimate a geometric transformation from the magnifying image to the reference magnification image to obtain an optimal homography matrix between the magnifying image and the reference magnification image;

[0082] S4, align the magnifying image by using the optimal homography matrix, seamlessly fuse the reference magnification image and the aligned magnifying image to obtain a panoramic image;

[0083] S5, obtain and analyze a user interaction instruction, and display the panoramic image according to the analysis result.

[0084] In the embodiment, the method further comprises a step of preprocessing the regular magnification image sequence and the magnifying image sequence to optimize image quality. Specifically, the step is located before analyzing the feature points of the regular magnification image and the magnifying image, and the preprocessed images are used in subsequent steps.

[0085] In the embodiment, S2 specifically comprises: performing multi-scale analysis on the regular magnification image and the magnifying image, the multi-scale analysis comprises finding feature points and calculating the directions of the feature points, and generating a Gaussian difference pyramid of the image; calculating a histogram of oriented gradients in the neighborhood of each feature point in the Gaussian difference pyramid of the image to generate a feature descriptor; constructing an index tree by using an FLANN algorithm, matching the feature descriptors to obtain matched feature point pairs; filtering the mismatched point pairs by using a bidirectional nearest neighbor matching strategy, retaining the matched feature point pairs with the smallest matching distance, and obtaining a matched feature point pair set.

[0086] In the embodiment, the S3 is specifically: using a least square method to solve a homography matrix between the magnified image and the reference magnification image; and estimating an optimal homography matrix by iterative sampling in the matched feature point pairs through a random sample consensus algorithm.

[0087] In the embodiment, the S4 is specifically: mapping the magnified image into a coordinate system of the reference magnification image based on the optimal homography matrix; performing a Gaussian blur and a down-sampling operation on the reference magnification image and the aligned magnified image to generate a Gaussian pyramid, generating a Laplacian pyramid through Gaussian difference calculation, performing weighted fusion in each layer of the Laplacian pyramid to generate a seamless fusion image, and the seamless fusion image being the panoramic image.

[0088] In the embodiment, the S5 is specifically: obtaining the panoramic image and a user interaction instruction, analyzing the user interaction instruction and updating the display of the panoramic image in real time according to the user interaction instruction.

[0089] In the embodiment, the method further includes a step of automatically identifying a lesion in the panoramic image by using a deep learning model.

[0090] The magnified endoscope image processing method can be implemented by referring to the magnified endoscope image processing system in any of the above embodiments, and the specific implementation steps are not described again.

[0091] The application further provides a storage medium storing a computer program, and the computer program is executed by a processor to implement the steps of the magnified endoscope image processing method.

[0092] The magnified endoscope image processing method, device and storage medium have the following effects:

[0093] The application discloses an enlarged endoscope image processing method and device and a storage medium, which match feature points through a feature extraction and matching module 30, obtain an optimal homography matrix between an enlarged multiple image and a reference multiple image through a geometric transformation calculation and image registration module 40, obtain a seamless multiple image to a panoramic image through an image fusion module 50, and realize the display of the panoramic image according to the user's will through a user interaction and real-time display module 60. Compared with the prior art, the application has small calculation amount, fast image processing, and can display the panoramic image without post-processing, can realize real-time endoscope image alignment and splicing under normal multiple and high multiple, and can provide a seamless panoramic inspection view in real time, improve the visual effect, and can display according to the user interaction instruction of the doctor, for example, display the non-partition area that the doctor wants to see, and is beneficial to the endoscope inspection work of the doctor. Based on the application, the accuracy and efficiency of the endoscope inspection can be improved, especially when detailed observation is performed under the enlarged multiple, the overall view and detail information can be reserved. The application provides an efficient real-time image splicing and fusion method, and is suitable for various endoscopes. Further, the panoramic image obtained based on the application can be automatically recognized by a deep learning model, so that the lesion recognition is accurate, and auxiliary support is provided for the work of the clinician.

[0094] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0095] Although the preferred embodiments of the application have been described, those skilled in the art, once they know the basic inventive concept, can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the application. Obviously, those skilled in the art can make various modifications and changes to the application without departing from the spirit and scope of the application. Thus, if these modifications and changes of the application fall within the scope of the claims of the application and their equivalent technologies, the application also intends to include these modifications and changes.

Claims

1. An amplified endoscope image processing apparatus characterized by comprising: The device comprises: a reference image acquisition module for decoding the regular magnification video stream output by the endoscope device frame by frame into static images and caching to obtain a regular magnification image sequence; a magnified image acquisition module for decoding the high magnification video stream output by the endoscope device frame by frame into static images and caching to obtain a magnified magnification image sequence; a feature extraction and matching module for analyzing the feature points and their corresponding feature descriptors of the regular magnification images and the magnified magnification images, matching the feature points according to the feature descriptors of the regular magnification images and the feature descriptors of the magnified magnification images to obtain a matched feature point pair set; a geometric transformation calculation and image registration module for estimating the geometric transformation of the magnified magnification image to the reference magnification image to obtain the optimal homography matrix between the magnified magnification image and the reference magnification image; an image fusion module for aligning the magnified magnification image using the optimal homography matrix, seamlessly fusing the reference magnification image and the aligned magnified magnification image to obtain a panoramic image; a user interaction and real-time display module for obtaining and analyzing user interaction instructions and displaying the panoramic image according to the analysis result.

2. The magnifying endoscope image processing apparatus according to claim 1, wherein The device further comprises an image preprocessing module for preprocessing the regular magnification image sequence and the magnified magnification image sequence to optimize the image quality.

3. The magnifying endoscope image processing apparatus according to claim 1, wherein The feature extraction and matching module comprises a scale-invariant feature transform subunit and a feature descriptor generation subunit; the scale-invariant feature transform subunit is configured to perform multi-scale analysis on the regular magnification images and the magnified magnification images, the multi-scale analysis comprising finding feature points and calculating the directions of the feature points, and generating a Gaussian difference pyramid of the images; the feature descriptor generation subunit is configured to calculate a histogram of oriented gradients in the neighborhood of each feature point in the Gaussian difference pyramid of the images to generate a feature descriptor.

4. The magnifying endoscope image processing apparatus of claim 1, wherein The feature extraction and matching module comprises a FLANN feature matching subunit and a matching filtering subunit; the FLANN feature matching subunit is configured to construct an index tree by using the FLANN algorithm and match the feature descriptors to obtain matched feature point pairs; the matching filtering subunit is configured to filter the mis-matched point pairs by using a bidirectional nearest neighbor matching strategy, retain the matched feature point pairs with the smallest matching distance, and obtain a matched feature point pair set.

5. The magnifying endoscope image processing apparatus of claim 1, wherein The geometric transformation calculation and image registration module comprises a homography matrix solving subunit and a random sample consensus subunit; the homography matrix solving subunit is configured to solve the homography matrix between the magnified magnification image and the reference magnification image by using a least squares method; the random sample consensus subunit is configured to iteratively sample in the matched feature point pairs by using a random sample consensus algorithm to estimate the optimal homography matrix.

6. The magnifying endoscope image processing apparatus of claim 1, wherein The image fusion module comprises an image alignment unit and a multi-scale image fusion unit; the image alignment unit is configured to map the magnified magnification image to the coordinate system of the reference magnification image based on the optimal homography matrix; The multi-scale image fusion unit is configured to perform Gaussian blur and down-sampling on the reference magnification image and the aligned high magnification image to generate a Gaussian pyramid, perform Laplacian pyramid generation by Gaussian difference calculation, perform weighted fusion in each layer of the Laplacian pyramid, and generate a seamless fusion image, which is the panoramic image.

7. The magnifying endoscope image processing apparatus of claim 1, wherein The user interaction and real-time display module is configured to obtain the panoramic image and a user interaction instruction, analyze the user interaction instruction, and update the display of the panoramic image in real time according to the analysis result.

8. The magnifying endoscope image processing apparatus of claim 1, wherein The device further comprises a lesion recognition module configured to automatically recognize a lesion from the panoramic image by using a deep learning model.

9. A magnifying endoscope image processing method characterized by comprising: The method comprises the following steps: The conventional magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a conventional magnification image sequence, and the high magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a high magnification image sequence; Feature points and corresponding feature descriptors of the conventional magnification images and the high magnification images are analyzed, and feature point matching is performed according to the feature descriptors of the conventional magnification images and the high magnification images to obtain a set of matched feature point pairs; An optimal homography matrix between the high magnification image and the reference magnification image is estimated; The high magnification image is aligned by using the optimal homography matrix, and the reference magnification image and the aligned high magnification image are seamlessly fused to obtain the panoramic image; A user interaction instruction is obtained and analyzed, and the panoramic image is displayed according to the analysis result.

10. A storage medium storing a computer program, characterized by The computer program is executed by a processor to implement the steps of the magnified endoscope image processing method of claim 9.

Images