A magnifying endoscope image processing method, device and storage medium

Abstract

The present invention discloses a magnifying endoscopic image processing method, device and storage medium, the method comprising: a reference image acquisition module; a magnified image acquisition module; a feature extraction and matching module, used to analyze the feature points and feature descriptors of a conventional magnification image and a magnified image, and perform feature point matching to obtain a set of matched feature point pairs; a geometric transformation calculation and image registration module, used to estimate the geometric transformation from the magnified image to the reference magnification image, and obtain the optimal homography matrix between the magnified image and the reference magnification image; an image fusion module, used to align the magnified image, seamlessly fuse the reference magnification image and the aligned magnification image to obtain a panoramic image; a user interaction and real-time display module, used to parse user interaction instructions and display the panoramic image accordingly. The present invention has a small amount of calculation, can provide seamless panoramic images in real time at conventional magnifications and high magnifications, and is beneficial to the clinical endoscopic examination work of doctors.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis and processing, and in particular to a magnifying endoscope image processing method, device and storage medium. Background Art

[0002] At present, endoscopic examination has become one of the important means of diagnosing gastrointestinal lesions, especially playing a key role in the early detection and diagnosis of gastrointestinal diseases. With the continuous advancement of endoscopic technology, doctors can obtain clearer mucosal detail images through magnifying endoscopes. However, the existing endoscopic technology has the problem of limited image field of view during operation. Magnifying endoscopes can provide high-resolution local images, but their field of view is small, and it is difficult to fully display the entire inspection area. During the operation of existing endoscopic technology, when switching between conventional magnification and high magnification, it is impossible to achieve real-time alignment and stitching of images at different magnifications. These problems cause doctors to lose their overall field of view during the examination, increase the difficulty of diagnosis during the operation, and easily miss important lesion areas, so image stitching is required.

[0003] Manual stitching of images by doctors requires doctors to have high operating skills and experience. At the same time, it is easy to cause stitching errors due to improper operation, leading to misjudgment of the location of the lesion. In addition, manual stitching is very time-consuming and energy-consuming, so intelligent stitching is usually used. However, in the prior art, image processing methods usually require a large number of calculations and post-processing calculation steps when realizing image stitching, which makes it difficult to achieve real-time stitching and display during clinical examinations, affecting the inspection efficiency. Therefore, in order to solve the problem that the real-time stitching speed in the prior art is slow and difficult to meet the needs, it is urgent to develop a magnifying endoscope image processing method, device and storage medium. Summary of the invention

[0004] In view of the problems of large amount of calculation and insufficient real-time performance in magnifying endoscopic image stitching processing, the present invention provides a magnifying endoscopic image processing method, device and storage medium to provide doctors with more intuitive and detailed panoramic images in real time.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] In a first aspect, the present invention provides a magnifying endoscope image processing device, comprising:

[0007] A reference image acquisition module is used to decode the normal magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a normal magnification image sequence;

[0008] The magnified image acquisition module is used to decode the high-magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a magnified image sequence;

[0009] A feature extraction and matching module is used to analyze the feature points of the normal magnification image and the magnification image and their corresponding feature descriptors, and to match the feature points according to the feature descriptors of the normal magnification image and the feature descriptors of the magnification image to obtain a set of matched feature point pairs;

[0010] A geometric transformation calculation and image registration module is used to estimate the geometric transformation from the magnification image to the reference magnification image, and obtain the optimal homography matrix between the magnification image and the reference magnification image;

[0011] An image fusion module, used to align the magnification images using an optimal homography matrix, and seamlessly fuse the reference magnification image and the aligned magnification image to obtain a panoramic image;

[0012] The user interaction and real-time display module is used to obtain and analyze user interaction instructions and display the panoramic image according to the analysis results.

[0013] In a preferred embodiment, the device further comprises an image preprocessing module, wherein the image preprocessing module is used to preprocess the conventional magnification image sequence and the magnification image sequence to optimize the image quality.

[0014] In a preferred embodiment, the feature extraction and matching module includes a scale-invariant feature transformation subunit and a feature descriptor generation subunit; the scale-invariant feature transformation subunit is used to perform multi-scale analysis on conventional magnification images and magnification images, and the multi-scale analysis includes finding feature points and calculating the directions of the feature points to generate a Gaussian difference pyramid of the image; the feature descriptor generation subunit is used to calculate a directional gradient histogram in the neighborhood of each feature point in the Gaussian difference pyramid of the image to generate a feature descriptor.

[0015] In a preferred embodiment, the feature extraction and matching module includes a FLANN feature matching subunit and a matching filtering subunit; the FLANN feature matching subunit is used to construct an index tree through the FLANN algorithm, and match feature descriptors to obtain matching feature point pairs; the matching filtering subunit is used to use a bidirectional nearest neighbor matching strategy to filter out mismatched point pairs, retain the matching feature point pairs with the smallest matching distance, and obtain a set of matched feature point pairs.

[0016] In a preferred embodiment, the geometric transformation calculation and image registration module includes a homography matrix solving subunit and a random sampling consistency subunit; the homography matrix solving subunit is used to use the least squares method to solve the homography matrix between the magnified magnification image and the reference magnification image; the random sampling consistency subunit is used to iteratively sample in matching feature point pairs through a random sampling consistency algorithm to estimate the optimal homography matrix.

[0017] In a preferred embodiment, the image fusion module includes an image alignment unit and a multi-scale image fusion unit; the image alignment unit is used to map the magnification image to the coordinate system of the reference magnification image based on the optimal homography matrix; the multi-scale image fusion unit is used to perform Gaussian blur and downsampling operations on the reference magnification image and the aligned magnification image to generate a Gaussian pyramid, generate a Laplacian pyramid through Gaussian difference calculation, perform weighted fusion in each layer of the Laplacian pyramid, and generate a seamless fused image, wherein the seamless fused image is the panoramic image.

[0018] In a preferred embodiment, the user interaction and real-time display module is specifically used to obtain the panoramic image and the user interaction instruction, parse the user interaction instruction and update the display of the panoramic image in real time accordingly.

[0019] In a preferred embodiment, the device further includes a lesion recognition module for automatically identifying lesions based on the panoramic image using a deep learning model.

[0020] In a second aspect, the present invention provides a magnifying endoscopic image processing method, comprising:

[0021] The normal magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a normal magnification image sequence, and the high magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a magnification image sequence;

[0022] Analyze the feature points of the normal magnification image and the magnified magnification image and their corresponding feature descriptors, perform feature point matching based on the feature descriptors of the normal magnification image and the feature descriptors of the magnified magnification image, and obtain a set of matched feature point pairs;

[0023] Estimate the geometric transformation from the magnified image to the reference image, and obtain the optimal homography matrix between the magnified image and the reference image;

[0024] The magnification images are aligned using the optimal homography matrix, and the reference magnification image and the aligned magnification image are seamlessly fused to obtain a panoramic image.

[0025] Obtain and parse the user interaction instruction, and display the panoramic image according to the parsing result.

[0026] In a third aspect, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of a magnifying endoscopic image processing method described in the second aspect.

[0027] The present invention provides a magnifying endoscopic image processing method, device and storage medium, which match feature point pairs through a feature extraction and matching module, obtain the optimal homography matrix between the magnification image and the reference magnification image through a geometric transformation calculation and image registration module, obtain a seamless fusion of the magnification image into a panoramic image through an image fusion module, and display the panoramic image according to the user's wishes through a user interaction and real-time display module. Compared with the prior art, the present invention has a small amount of calculation and fast image processing. The obtained panoramic image can be displayed without post-processing, and real-time endoscopic image alignment and splicing at conventional magnifications and high magnifications can be achieved. The panoramic image can be provided in real time, and it is a seamless panoramic inspection view, which improves the visual effect and can be displayed according to the doctor's user interaction instructions, which is beneficial to the doctor's endoscopic inspection work. Based on the present invention, the accuracy and efficiency of endoscopic inspection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a framework diagram of a magnifying endoscope image processing system;

[0030] Figure 2 The present invention is a flowchart of a magnifying endoscopy image processing method. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0033] It should be noted that the descriptions of "first", "second", etc. in the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. The "and / or" mentioned in the present invention refers to any and all combinations of one or more of the relevant listed items. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0034] The existing magnifying endoscopic image stitching requires large computational complexity and lacks real-time performance. To this end, the present invention provides a magnifying endoscopic image processing device, comprising:

[0035] The reference image acquisition module 10 is used to decode the normal magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a normal magnification image sequence;

[0036] The magnified image acquisition module 20 is used to decode the high-magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a magnified image sequence;

[0037] The feature extraction and matching module 30 is used to analyze the feature points (feature point set) of the normal magnification image (in the normal magnification image sequence) and the magnification image (in the magnification image sequence) and their corresponding feature descriptors, and perform feature point matching based on the feature descriptors of the normal magnification image and the feature descriptors of the magnification image to obtain a matched feature point pair set;

[0038] A geometric transformation calculation and image registration module 40 is used to estimate the geometric transformation from the magnification image to the reference magnification image, and obtain the optimal homography matrix between the magnification image and the reference magnification image;

[0039] An image fusion module 50 is used to align the magnification images using an optimal homography matrix, and seamlessly fuse the reference magnification image and the aligned magnification image to obtain a panoramic image;

[0040] The user interaction and real-time display module 60 is used to obtain and analyze the user interaction instruction, and display the panoramic image according to the analysis result.

[0041] In a preferred embodiment, the normal magnification image sequence is a normal magnification image sequence after preprocessing, and the magnification image sequence is a magnification image sequence after preprocessing, and the preprocessing is to optimize image quality.

[0042] The implementation embodiment may be to set an image preprocessing module, wherein the image preprocessing module is used to preprocess the normal magnification image sequence and the magnification image sequence to optimize the image quality, and obtain the preprocessed normal magnification image sequence and the preprocessed magnification image sequence;

[0043] Specifically, the image preprocessing module includes an image preprocessing unit. The image preprocessing unit has the acquired normal magnification image sequence and magnification image sequence as input, and contains a denoising and enhancement subunit for optimizing image quality. Specifically, the preprocessed normal magnification image sequence and the preprocessed magnification image sequence are obtained through standard filtering and equalization processing.

[0044] The implemented embodiment may also be that a first preprocessing unit for preprocessing a normal magnification image sequence is provided in the reference image acquisition module 10 , and a second preprocessing unit for preprocessing a magnification image sequence is provided in the magnified image acquisition module 20 .

[0045] It can be understood that the normal magnification images and magnification images involved in the feature extraction and matching module 30, the geometric transformation calculation and image registration module 40, and the image fusion module 50 are respectively pre-processed normal magnification images and pre-processed magnification image sequences.

[0046] The reference image acquisition module 10 includes an image acquisition unit. The image acquisition unit inputs a normal magnification video stream output by an endoscope device; the image acquisition unit internally includes a first video decoding and caching subunit, which is used to decode the video stream into a normal magnification image frame by frame and cache it, that is, to decode the video signal into a static image; through the decoding and caching operations, a normal magnification image sequence is obtained, and the sequence is composed of normal magnification images frame by frame.

[0047] The magnified image acquisition module 20 includes a magnified image acquisition unit. The magnified image acquisition unit is input with a high-magnification video stream output by the endoscope device; it contains a second video decoding and caching subunit, which is used to decode the high-magnification video stream frame by frame into magnified images and cache them; through the decoding and caching operations, a magnified image sequence is obtained, which is composed of magnified images frame by frame.

[0048] The feature extraction and matching module 30 is used to analyze the normal magnification images in the normal magnification image sequence to obtain a feature point set, which can be called the feature point set one, and to analyze the magnification images in the magnification image sequence to obtain a feature point set, which can be called the feature point set two, and then match the feature point one with the feature point two, and all the matched feature point pairs obtained constitute the matched feature point pair set. The feature extraction and matching module 30 includes a feature extraction unit and a feature matching unit.

[0049] The feature extraction unit is input with a preprocessed normal magnification image sequence and a preprocessed magnification image sequence; it contains a scale invariant feature transform (SIFT) subunit and a feature descriptor generation subunit, which are used to extract key feature points and descriptors from the image; and obtains a feature point set and corresponding feature descriptors through multi-scale analysis and directional gradient calculation.

[0050] The scale-invariant feature transform (SIFT) subunit is used to perform multi-scale analysis on the input image, which includes finding feature points and calculating the directions of these feature points to generate a Gaussian difference pyramid (Difference of Gaussian, DoG) of the image. The feature point detection formula is as follows:

[0051] (1)

[0052] in, represents the Gaussian difference in scale space, represents the Gaussian blur function, represents the input image, ( x,y ) represents the coordinates of the pixels in the input image, represents a variable core, k is a positive integer, indicating the first k Cascaded filters.

[0053] The feature descriptor generation subunit is used to calculate the Histogram of Oriented Gradients (HOG) in the neighborhood of each feature point in the Gaussian difference pyramid of the image and generate a 128-dimensional feature descriptor. The feature descriptor calculation formula is:

[0054] (2)

[0055] in, represents the feature descriptor, is the directional gradient amplitude, represents the gradient direction, is the oriented gradient histogram r Direction intervals, r Is a positive integer.

[0056] The feature matching unit is input with the feature descriptor of the normal magnification image and the feature descriptor of the magnified magnification image; it contains a FLANN feature matching subunit and a matching filtering subunit, which are used to match feature points between images of different magnifications to obtain matching feature point pairs; through the FLANN (fast approximate nearest neighbor search library function) approximate nearest neighbor search and bidirectional filtering strategy, finally outputs a set of matching feature point pairs.

[0057] The FLANN feature matching subunit is used to match the feature descriptor of the normal magnification image with the feature descriptor of the magnification image to obtain a matched feature point pair; the matching filtering subunit is used to filter out the erroneously matched feature point pairs.

[0058] The FLANN feature matching subunit uses the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm to build an index tree, quickly match feature descriptors, and obtain matching feature point pairs. The search parameters are set to the number of trees t = 8 and the number of candidate points to be checked n = 2.

[0059] The matching filtering subunit uses a bidirectional nearest neighbor matching strategy to filter mismatched point pairs, retain the matching feature point pairs with the smallest matching distance, and obtain a set of matched feature point pairs. The matching score is calculated and filtered according to the matching score. The matching score calculation formula is:

[0060] (3)

[0061] in, represents the feature descriptor, i and j Indicates the feature descriptor index number, Represents feature descriptor and The Euclidean distance of Represents the matched feature point pairs, where represents the feature points in the reference magnification image, Represents feature points in the magnified image.

[0062] The geometric transformation calculation and image registration module 40 estimates the geometric transformation from the magnification image to the reference magnification image based on the set of matched feature point pairs.

[0063] The geometric transformation calculation and image registration module 40 includes a geometric transformation estimation unit.

[0064] The geometric transformation estimation unit takes as input a set of matched feature point pairs; it contains a homography matrix solving subunit and a random sampling consensus (RANSAC) subunit, which are used to estimate the geometric transformation from the magnification image to the reference magnification image, and obtain the optimal homography matrix using the least squares method and the RANSAC algorithm.

[0065] The homography matrix solving subunit is used to use the least square method to solve the homography matrix between the magnified magnification image and the reference magnification image. H , the formula is:

[0066] (4)

[0067] The random sampling consistency subunit is used to iteratively sample in the matched feature point pairs through the RANSAC algorithm to estimate the optimal homography matrix as the geometric transformation of the magnification image to the reference magnification image. The optimization objective function of the RANSAC algorithm is:

[0068] (5)

[0069] in, H best represents the optimal homography matrix, M represents the set of matched feature point pairs finally obtained by the feature extraction and matching module 30, is the indicator function, is the inlier distance threshold;

[0070] The image fusion module 50 includes an image alignment unit and a multi-scale image fusion unit.

[0071] The image alignment unit takes as input a magnification image and an optimal homography matrix; it contains a perspective transformation subunit for accurately mapping the magnification image to the coordinate system of the reference magnification image; the aligned magnification image is obtained by a perspective transformation formula based on the optimal homography matrix, that is, the alignment of the magnification image and the reference magnification image is achieved by using the image alignment technology.

[0072] The perspective transformation subunit: performs perspective transformation on the magnification image based on the optimal homography matrix, and the transformation formula is:

[0073] (6)

[0074] in, represents the magnification image after alignment, represents the target coordinates, represents the magnification image, Indicates the original coordinates of the magnified image, Transformation function representing the optimal homography viewpoint transformation.

[0075] The multi-scale image fusion unit is input with a reference magnification image and an aligned magnification image; it contains a Gaussian-Laplacian pyramid generation subunit and an image fusion subunit, which are used to generate multi-scale representations and perform seamless fusion; and the final panoramic image is obtained through the multi-scale fusion technology.

[0076] The Gaussian-Laplacian pyramid generation subunit is used to generate a Gaussian pyramid by performing Gaussian blur and downsampling operations on the image (the input of the multi-scale image fusion unit), and to form a Laplacian pyramid by Gaussian difference calculation. The formula of the Gaussian-Laplacian pyramid generation subunit is:

[0077] (7)

[0078] in, Represents the layer of the Gaussian pyramid, i.e. layer; It is a Gaussian pyramid representation, specifically representing an output image of a certain layer of the Gaussian pyramid; represents the Gaussian pyramid input image (baseline magnification image or aligned magnification image), It is represented by the Laplace pyramid.

[0079] The image fusion subunit is used to perform weighted fusion in each layer of the Laplacian pyramid to generate a seamless fusion image and obtain the final panoramic image. The fusion formula is:

[0080] (8)

[0081] in, represents the fusion result, i.e., the panoramic image; is the Laplacian pyramid representation of the magnification image; is the Laplacian pyramid representation of the reference magnification image; Represents the weight factor, which controls the fusion ratio of images at different scales.

[0082] The user interaction and real-time display module 60 includes a user interface unit. The user interface unit receives the panoramic image and the user interaction instruction as input; it includes an interaction parsing subunit and a display updating subunit, which are used to parse the user input and update the display of the panoramic image in real time; and obtains the display image of the panoramic image through the interaction parsing and rendering operations.

[0083] In this embodiment, the device also includes a lesion recognition module, the input of which is a panoramic image, and the output is a lesion recognition result, including but not limited to the lesion location and lesion type. The lesion is automatically identified using a deep learning model to obtain a lesion recognition result, and the detailed information in the high-magnification image is fully utilized for lesion recognition and evaluation. The lesion recognition and evaluation are accurate, which solves the problem that the existing endoscopic equipment mainly provides a simple image magnification function, lacks intelligent analysis and diagnostic support, and cannot fully utilize the detailed information in the high-magnification image for lesion recognition and evaluation.

[0084] The present invention provides a magnifying endoscope image processing method, the method comprising:

[0085] S1, decoding the normal magnification video stream output by the endoscope device into static images frame by frame and caching them to obtain a normal magnification image sequence, and decoding the high magnification video stream output by the endoscope device into static images frame by frame and caching them to obtain a magnification image sequence;

[0086] S2, analyzing the feature points of the normal magnification image and the magnification image and their corresponding feature descriptors, performing feature point matching according to the feature descriptors of the normal magnification image and the feature descriptors of the magnification image, and obtaining a set of matched feature point pairs;

[0087] S3, estimating the geometric transformation from the magnification image to the reference magnification image, and obtaining the optimal homography matrix between the magnification image and the reference magnification image;

[0088] S4, aligning the magnification images using the optimal homography matrix, and seamlessly fusing the reference magnification image and the aligned magnification image to obtain a panoramic image;

[0089] S5. Obtain and analyze the user interaction instruction, and display the panoramic image according to the analysis result.

[0090] In this embodiment, the method further includes a step of preprocessing the normal magnification image sequence and the magnification image sequence to optimize the image quality. Specifically, this step is located before analyzing the feature points of the normal magnification image and the magnification image, and the preprocessed images are used in subsequent steps.

[0091] In this embodiment, S2 specifically includes: performing multi-scale analysis on the conventional magnification image and the magnification image, the multi-scale analysis including finding feature points and calculating the directions of the feature points to generate a Gaussian difference pyramid of the image; calculating the directional gradient histogram in the neighborhood of each feature point in the Gaussian difference pyramid of the image to generate a feature descriptor; constructing an index tree through the FLANN algorithm, matching the feature descriptors to obtain matching feature point pairs; using a bidirectional nearest neighbor matching strategy to filter out mismatched point pairs, retaining the matching feature point pairs with the smallest matching distance, and obtaining a set of matching feature point pairs.

[0092] In this embodiment, S3 specifically includes: using the least square method to calculate the homography matrix between the magnified image and the reference image; iteratively sampling in the matching feature point pairs through a random sampling consistency algorithm to estimate the optimal homography matrix.

[0093] In this embodiment, S4 specifically includes: mapping the magnification image to the coordinate system of the reference magnification image based on the optimal homography matrix; performing Gaussian blur and downsampling operations on the reference magnification image and the aligned magnification image to generate a Gaussian pyramid, generating a Laplacian pyramid through Gaussian difference calculation, performing weighted fusion in each layer of the Laplacian pyramid to generate a seamless fusion image, and the seamless fusion image is the panoramic image.

[0094] In this embodiment, S5 specifically includes: obtaining a panoramic image and a user interaction instruction, parsing the user interaction instruction and updating the display of the panoramic image in real time accordingly.

[0095] In this embodiment, the method also includes a step of automatically identifying lesions based on the panoramic image using a deep learning model.

[0096] When the magnifying endoscopic image processing method is implemented, the magnifying endoscopic image processing system in any of the above embodiments may be referred to to implement the magnifying endoscopic image processing, and the specific implementation steps will not be repeated.

[0097] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the magnifying endoscope image processing method.

[0098] The effects of the magnifying endoscope image processing method, device and storage medium of the present invention are:

[0099] The present invention provides a magnifying endoscopic image processing method, device and storage medium. The feature extraction and matching module 30 matches feature point pairs, the geometric transformation calculation and image registration module 40 obtains the optimal homography matrix between the magnification image and the reference magnification image, the image fusion module 50 obtains seamless fusion magnification image to panoramic image, and the user interaction and real-time display module 60 realize the display of panoramic image according to the user's wishes. Compared with the prior art, the present invention has a small amount of calculation, fast image processing, and the obtained panoramic image can be displayed without post-processing. It can realize real-time endoscopic image alignment and splicing at conventional magnification and high magnification, provide seamless panoramic inspection view in real time, improve visual effect, and can be displayed according to the user interaction instructions of the doctor, such as displaying the doctor's desired area, etc., which is beneficial to the doctor's endoscopic inspection work. Based on the present invention, the accuracy and efficiency of endoscopic inspection can be improved, especially when detailed observation is performed under magnification, the overall view and detail information can be retained. The present invention provides an efficient real-time image splicing and fusion method, which is applicable to a variety of endoscopes. Furthermore, based on the panoramic image obtained by the present invention, the lesion area can be automatically identified through a deep learning model, so that the lesion identification is accurate and auxiliary support is provided for the work of clinicians.

[0100] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concept is known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A magnifying endoscope image processing device, characterized in that: include: A reference image acquisition module is used to decode the normal magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a normal magnification image sequence; The magnified image acquisition module is used to decode the high-magnification video stream output by the endoscope device into static images frame by frame and cache them to obtain a magnified image sequence; A feature extraction and matching module is used to analyze the feature points of the conventional magnification image and the magnification image and their corresponding feature descriptors, and to match the feature points according to the feature descriptors of the conventional magnification image and the feature descriptors of the magnification image to obtain a set of matched feature point pairs; the feature extraction and matching module includes a FLANN feature matching subunit and a matching filtering subunit; the FLANN feature matching subunit is used to construct an index tree through the FLANN algorithm, and to match the feature descriptors to obtain matched feature point pairs; the matching filtering subunit is used to filter mismatched point pairs using a bidirectional nearest neighbor matching strategy, retain the matched feature point pairs with the smallest matching distance, and obtain a set of matched feature point pairs; A geometric transformation calculation and image registration module is used to estimate the geometric transformation from the magnification image to the reference magnification image, and obtain the optimal homography matrix between the magnification image and the reference magnification image; the geometric transformation calculation and image registration module includes a homography matrix solving subunit and a random sampling consistency subunit; the homography matrix solving subunit is used to use the least squares method to calculate the homography matrix between the magnification image and the reference magnification image; the random sampling consistency subunit is used to iteratively sample in the matching feature point pairs through a random sampling consistency algorithm to estimate the optimal homography matrix; An image fusion module is used to align the magnification images using the optimal homography matrix, and seamlessly fuse the reference magnification image and the aligned magnification image to obtain a panoramic image; the image fusion module includes an image alignment unit and a multi-scale image fusion unit; the image alignment unit is used to map the magnification image to the coordinate system of the reference magnification image based on the optimal homography matrix; The multi-scale image fusion unit is used to perform Gaussian blur and down-sampling operations on the reference magnification image and the aligned magnification image to generate a Gaussian pyramid, generate a Laplacian pyramid by Gaussian difference calculation, perform weighted fusion in each layer of the Laplacian pyramid, and generate a seamless fusion image, wherein the seamless fusion image is the panoramic image; A user interaction and real-time display module, used to obtain and analyze user interaction instructions, and display the panoramic image according to the analysis results; The feature extraction and matching module includes a scale-invariant feature transformation subunit and a feature descriptor generation subunit; the scale-invariant feature transformation subunit is used to perform multi-scale analysis on the conventional magnification image and the magnification image, and the multi-scale analysis includes finding feature points and calculating the directions of the feature points to generate a Gaussian difference pyramid of the image; the feature descriptor generation subunit is used to calculate a directional gradient histogram in the neighborhood of each feature point in the Gaussian difference pyramid of the image to generate a feature descriptor; The random sampling consistency subunit iteratively samples in the matching feature point pairs through the random sampling consistency algorithm, and estimates the optimization objective function of the optimal homography matrix as: in, H best represents the optimal homography matrix, H represents the homography matrix, M It represents the set of matched feature point pairs finally obtained by the feature extraction and matching module. is the indicator function, is the inlier distance threshold, represents the feature points in the reference magnification image, Represents feature points in the magnified image; The user interaction and real-time display module is specifically used to obtain the panoramic image and the user interaction instruction, analyze the user interaction instruction and update the display of the panoramic image in real time accordingly; The device also includes a lesion recognition module, which is used to automatically identify lesions based on the panoramic image using a deep learning model.

2. A magnifying endoscope image processing device as claimed in claim 1, characterized in that: The device further comprises an image preprocessing module, which is used for preprocessing the normal magnification image sequence and the magnification image sequence to optimize the image quality.

3. A magnifying endoscope image processing method, characterized in that: include: The normal magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a normal magnification image sequence, and the high magnification video stream output by the endoscope device is decoded frame by frame into static images and cached to obtain a magnification image sequence; Analyze the feature points of the regular magnification image and the magnification image and their corresponding feature descriptors, match the feature points according to the feature descriptors of the regular magnification image and the feature descriptors of the magnification image, and obtain a set of matched feature point pairs; specifically: perform multi-scale analysis on the regular magnification image and the magnification image, the multi-scale analysis includes finding feature points and calculating the directions of the feature points, and generating a Gaussian difference pyramid of the image; calculate the directional gradient histogram in the neighborhood of each feature point in the Gaussian difference pyramid of the image, and generate a feature descriptor; construct an index tree through the FLANN algorithm, match the feature descriptors to obtain matched feature point pairs, use a bidirectional nearest neighbor matching strategy to filter out mismatched point pairs, retain the matched feature point pairs with the smallest matching distance, and obtain a set of matched feature point pairs; Estimate the geometric transformation from the magnified image to the reference image, and obtain the optimal homography matrix between the magnified image and the reference image; Specifically, the homography matrix between the magnified image and the reference image is obtained by using the least square method, and the optimal homography matrix is ​​estimated by iterative sampling in the matching feature point pairs through the random sampling consistency algorithm; The magnification images are aligned using an optimal homography matrix, and the reference magnification image and the aligned magnification image are seamlessly fused to obtain a panoramic image; specifically, the magnification image is mapped to the coordinate system of the reference magnification image based on the optimal homography matrix, Gaussian blur and down-sampling operations are performed on the reference magnification image and the aligned magnification image to generate a Gaussian pyramid, a Laplacian pyramid is generated by Gaussian difference calculation, and weighted fusion is performed in each layer of the Laplacian pyramid to generate a seamless fused image, wherein the seamless fused image is the panoramic image; Obtaining and parsing the user interaction instruction, and displaying the panoramic image according to the parsing result; specifically: obtaining the panoramic image and the user interaction instruction, parsing the user interaction instruction and updating the display of the panoramic image in real time accordingly; Automatically identify lesions using deep learning models based on panoramic images; The optimization objective function of iteratively sampling in the matched feature point pairs by the random sampling consistency algorithm and estimating the optimal homography matrix is: in, H best represents the optimal homography matrix, H represents the homography matrix, M It represents the set of matched feature point pairs finally obtained by the feature extraction and matching module. is the indicator function, is the inlier distance threshold, represents the feature points in the reference magnification image, Represents feature points in the magnified image.

4. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a magnifying endoscopy image processing method as described in claim 3 are implemented.

Images