Capsule endoscope self-adaptive frame rate adjusting method based on image motion estimation

Abstract

The invention discloses a capsule endoscope self-adaptive frame rate adjustment method based on image motion estimation. The method comprises the following steps: calculating the motion of a capsule endoscope; searching an optimal solution of motion parameters of the capsule endoscope by using a bee algorithm; local gastrointestinal tract movement is calculated, specifically, the local gastrointestinal tract movement is predicted through SIFT-FLOW, a dense corresponding relation can be established between a neighborhood image and a source image through an SIFT descriptor, and the SIFT descriptor has the excellent performance of local image illumination invariance and local image structure coding; the invalid area is found according to local gastrointestinal tract movement, whether scene change exists between the two images or not is evaluated through the diameter of the maximum inscribed circle of the invalid area, and if DMC exceeds five pixels, the capsule endoscope is controlled to improve the frame rate. According to the invention, problems existing in motion detection and frame rate adjustment in the prior art can be effectively solved.

Description

Technical Field

[0001] The present application relates to the technical field of capsule endoscopes, and in particular to a method for adaptively adjusting the frame rate of a capsule endoscope based on image motion estimation. Background Art

[0002] Capsule endoscope is a miniature wireless endoscope, a medical instrument used to examine the human gastrointestinal tract. After swallowing, the capsule will pass through the gastrointestinal tract, moving with the peristalsis of the body parts or with the help of gravity. By taking a large number of images of the inspected parts per second, the inspected parts can be observed and diseases can be diagnosed based on the images obtained. The application of capsule endoscopy technology has greatly improved the diagnostic efficiency of digestive tract diseases and the comfort of patients.

[0003] Existing capsule endoscopes usually capture images at a fixed frame rate, and their moving speed in the human body is extremely uneven. When the capsule endoscope is stationary or moves slowly in the digestive tract, a large number of redundant images will be generated, which not only wastes battery power but also increases the burden on film readers. When the capsule endoscope moves quickly in the digestive tract, it may miss images due to insufficient frame rate. Summary of the invention

[0004] The embodiment of the present application provides a method for adaptively adjusting the frame rate of a capsule endoscope based on image motion estimation, which solves the problem of uneven movement speed of the capsule endoscope leading to increased power consumption or insufficient frame rate leading to missed shots.

[0005] In order to solve the above technical problems, this application is implemented as follows: A method for adaptive frame rate adjustment of a capsule endoscope based on image motion estimation is provided, which comprises the following steps: calculating capsule endoscope motion: describing the motion of the capsule endoscope as a two-dimensional rigid deformation, and calculating the best solution (BAME) of capsule endoscope motion parameters; calculating local gastrointestinal motion: modeling the gastrointestinal motion as a non-rigid deformation, and predicting the motion of the local gastrointestinal tract based on the modeling. Calculating invalid regions: finding invalid regions based on the local gastrointestinal motion, using the diameter of the maximum inscribed circle (DMC) of the invalid region to evaluate whether there is a scene change between two images, and controlling the capsule endoscope to increase the frame rate if the DMC exceeds a certain number of pixels, wherein the invalid region refers to a region in the source image that contains a gastrointestinal wall scene that cannot be found in the neighboring image.

[0006] In one of the embodiments, in the step of calculating the motion of the capsule endoscope: a honey bee algorithm is used to search for an optimal solution for the motion parameters of the capsule endoscope.

[0007] In one embodiment, in the step of calculating the motion of the capsule endoscope: BAME is described as seeking the minimum error: ; Among them, I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation of searching for deformation parameters in the solution space, error n (I i-1 ,T n (I i )) is the error measure between the neighborhood image and the source image.

[0008] In one embodiment, in the step of calculating the local gastrointestinal tract motion: using SIFT-FLOW to predict the local gastrointestinal tract motion.

[0009] In one of the embodiments, in the SIFT-FLOW model, the SIFT descriptor establishes a dense correspondence between the neighborhood image and the source image, and the SIFT descriptor has excellent performance of local image illumination invariance and encoding local image structure.

[0010] In one embodiment, in the step of calculating the local gastrointestinal tract motion: IFT-FLOW is expressed as an optimization problem based on a corresponding search of a cost function: , ; Among them, ε(p)=(u x (p),u y (p)) is the vector of pixel p=(x,y), s i-1 (p) and s i (p) are the SIFT descriptors extracted from the neighborhood image p in the neighborhood image i−1 and the source image i, respectively. N is the spatial neighborhood of the pixel. The source image is the transformed image using capsule endoscopy motion estimation. R(p,q) constrains the vectors with neighboring pixels, and the vector ε(p) is regarded as the local non-rigid motion component. In one embodiment, the motion estimation between adjacent frame images is described as: .

[0011] Among them, the matrix M is the transformation matrix of the capsule endoscope, ε is a local flow vector that can find the best local match between two consecutive images, and Z i−1 Yes i−1 The approximate value of the corresponding neighborhood image point.

[0012] In one of the embodiments, in the step of calculating the invalid area: if the DMC exceeds 5 pixels, the capsule endoscope is controlled to increase the frame rate.

[0013] In one embodiment, in the step of calculating the invalid area: The first step is backward estimation: according to Z i−1 Estimate the scene deformation from image i to image i−1, and find the points of image i that are not in image i−1 according to the scene deformation. These points are recorded as a set: ; The second step is forward estimation: according to Z i−1 Estimate a set between image i and image i+1: ; Compute the collection: An invalid region of the current frame is a collection: .

[0014] In an embodiment of the present application, capsule endoscope motion and local gastrointestinal motion are calculated separately to achieve hierarchical estimation of capsule endoscope motion and local gastrointestinal motion, and the frame rate adjustment is optimized by combining the comprehensive influence of capsule endoscope motion and local gastrointestinal motion calculation. The present application uses a bee population algorithm to search for the optimal solution of rigid motion parameters, which can extract robustness, stability and significance features between adjacent frame images and establish matching, and more accurately estimate the motion of the capsule. The present application also establishes a dense correspondence through the SIFT-FLOW algorithm, and combines rigid motion estimation to more accurately describe the comprehensive influence of capsule motion and gastrointestinal deformation. The present application evaluates whether there is a significant scene change between the two images by calculating the maximum inscribed circle diameter (DMC) of the invalid area of ​​the image scene change, and then controls the frame rate, which can effectively solve the problems existing in motion detection and frame rate adjustment in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flowchart of the steps of the capsule endoscope adaptive frame rate adjustment method based on image motion estimation of the present application; Figure 2 is a flow chart of the capsule endoscope adaptive frame rate of the present application; Figure 3 It is a schematic diagram of the ineffective area of ​​forward and reverse measurement of the present application; Figure 4 It is a schematic diagram of the colors of similar scenes of this application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The motion model of capsule endoscope includes rigid motion and non-rigid motion. When the capsule endoscope works in the gastrointestinal tract, it is driven by the peristalsis of the gastrointestinal tract. If only the motion of the capsule endoscope is considered, the images of adjacent frames can be described by their secondary coordinates: (1) where Y i T is the image pixel, M is the two-dimensional rigid transformation, including parameters such as translation T, rotation R, scaling s, and perspective p. These parameters describe the motion between adjacent frame images of the capsule endoscope (global rigid motion).

[0018] The motion between adjacent frame images depends not only on the motion of the capsule endoscope, but also on the non-rigid deformation of the gastrointestinal tract due to peristalsis (local non-rigid motion): (2) According to the motion model, the capsule endoscope adaptive frame rate adjustment method based on image motion estimation of the present application can estimate the motion of adjacent frame images in two stages. In the first stage of coarse level estimation, the motion of the capsule endoscope can be regarded as a large displacement estimation of the adjacent frame image scene. In the second stage of fine level estimation, the local gastrointestinal tract deformation is estimated based on the result of the first stage estimation. Please refer to Figure 1 and Figure 2 As shown, it is a flowchart of the steps of the capsule endoscope adaptive frame rate adjustment method based on image motion estimation of the present application; as shown in the figure, the capsule endoscope adaptive frame rate adjustment method based on image motion estimation includes the following steps S1 to S3. Among them: Step S1, calculate the motion of the capsule endoscope: describe the motion of the capsule endoscope as a two-dimensional rigid deformation, and calculate the best solution (BAME) of the motion parameters of the capsule endoscope. Since the capsule endoscope is driven by gastrointestinal peristalsis, the actual gastrointestinal wall is very close to the capsule endoscope, which makes the projection deformation of adjacent frame images small, and the motion of the capsule endoscope can be described as a two-dimensional rigid deformation. Therefore, when estimating the motion of the capsule endoscope, only rigid motion can be focused on, and the projection p and the local gastrointestinal non-rigid displacement ε parameters are ignored. The usual method for this estimation is to extract different features from each image and match the features to establish a global correspondence, and then estimate the transformation between the images (feature-based method). In reality, it is difficult to extract robustness, stability and significance features and establish matching between consecutive adjacent frame images. The main reasons for the above problems are low image resolution, poor structural information, etc. This embodiment uses a honey bee algorithm to search for the best solution (BAME) of the motion parameters of the capsule endoscope. The BA algorithm is a population-based search algorithm used to solve complex multi-objective optimization problems that cannot be accurately solved within a polynomial bounded computing time. In contrast to feature-based methods, this approach is often referred to as a direct (pixel-based) alignment method. In this embodiment, BAME is described as seeking the minimum error: (3); Among them, I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation of searching for deformation parameters in the solution space, error n (I i-1 ,T n (I i )) is the error measure between the neighborhood image and the source image. Mutual information (MI) is used as the error measure. MI is widely used in medical image registration. Since the captured images have similar intensity, color, and tone, grayscale information is used to calculate MI. The pseudo code of BAME is shown in the following table: 01. Input: Image I i-1 and I i 02. • Initialize the bee population m∈Z 03. • Random search solution TS={T n =random(s, R, T): 1≤n≤m, n∈Z} 04. •Evaluate the error of each solution ES={error n (I i-1 ,T n (I i )): 1≤n≤m} 05. •While (Stop criteria not reached) 06. •Select the optimal solution eTS={T n ∈TS: error n ≤threshold, n≤p} 07. for i1=1 to p 08. •In each neighborhood, get new bees to get the best solution eTS =update (eTS) 09. •eES={error n (eTS)} 10. End 11. • Assign the remaining bees to search randomly rTS={T n =random(s, R, T): p <n≤m} 12. • Calculate the error rES={error n (I i-1 ,T n (I i )): p≤n≤m} 13. •TS = eTS∪rTS, ES = eES∪rES 14. • End 15.Output: T * ={T n ∈TS: error n =argmin(ES)}.

[0019] Step S2, calculate the local gastrointestinal movement: model the gastrointestinal movement as a non-rigid deformation, and predict the movement of the local gastrointestinal tract based on the modeling. Use SIFT-FLOW to predict the movement of the local gastrointestinal tract, where the SIFT descriptor can establish a dense correspondence between the neighborhood image and the source image, and the SIFT descriptor has the excellent performance of local image illumination invariance and encoding local image structure. Since the gastrointestinal movement can be modeled as a non-rigid deformation, which is caused by gastrointestinal peristalsis, SIFT-FLOW is used to predict the movement of the local gastrointestinal tract. The SIFT descriptor is able to establish a dense correspondence between the neighborhood image and the source image. The SIFT descriptor has the excellent performance of local image illumination invariance and encoding local image structure, and these properties make the matching more robust. SIFT-FLOW can be expressed as an optimization problem of corresponding search based on a cost function: (4) (5) where ε(p)=(ux (p),u y (p)) is the vector of pixel p=(x,y), s i-1 (p) and s i (p) are the SIFT descriptors extracted from the neighboring image p in the neighboring image i−1 and the source image i, respectively, and N is the spatial neighborhood of the pixel. The source image is the transformed image using capsule endoscopy motion estimation. R(p,q) constrains the vector with neighboring pixels. The vector ε(p) is considered as the local non-rigid motion component.

[0020] Finally, the motion estimation between adjacent frame images can be described as: (6) As mentioned above, the right side of equation (6) contains two terms, where the first term is the global rigid motion estimated by BAME, the matrix M is the transformation matrix of the capsule endoscope, and the second term is the local non-rigid motion estimation of SIFT-FLOW, ε is a local flow vector that can find the best local match between two consecutive images. i−1 Yes i−1 The approximate value of the corresponding neighborhood image point.

[0021] Step S3, calculate the invalid area: find the invalid area according to the local gastrointestinal movement, use the diameter of the maximum inscribed circle (DMC) of the invalid area to evaluate whether there is a scene change between the two images, and if the DMC exceeds a certain number of pixels, the capsule endoscope is controlled to increase the frame rate. In this embodiment, if the DMC exceeds 5 pixels, the capsule endoscope is controlled to increase the frame rate. The invalid area refers to the area in the source image that contains the gastrointestinal wall scene that cannot be found in the neighboring image. Since the scene change in the capsule endoscope means that adjacent frame images have more different contexts, in order to perform robust measurement of the scene change between adjacent frame images, the concept of invalid area is defined. The invalid area refers to the area in the image (source image) that contains the gastrointestinal wall scene that cannot be found in the neighboring image (or key image). As Figure 3 As shown, the invalid area estimation is implemented in two steps. The first step is backward estimation, which estimates the scene deformation from image i to image i−1 according to Eq(6), and finds the points of image i that are not in image i−1 according to the scene deformation. These points are recorded as a set: (7) The second step is forward estimation, recording the points as a set between image i and image i+1: (8) Finally, an invalid region for the current frame is a collection: (9).

[0022] Specifically, the area of ​​the invalid region can reflect the scene changes between adjacent frame images. Figure 4 Shows an invalid area of ​​image i. The green and red areas are the scenes of image i. Figure 4 The green circle is the maximum inscribed circle of the invalid area. Even if the number of points in the invalid area is large, it may still not indicate a large displacement. Since the invalid areas of the image are usually distributed at the edges and are usually narrow. In fact, the maximum diameter of the invalid area can reflect the potential possibility of scene size changes. Therefore, this measurement is simplified, and the diameter of the maximum inscribed circle of the invalid area (DMC) is used to evaluate whether there is a scene change between the two images. If the DMC exceeds 5 pixels, it means that the scene change between the images is large, and the capsule endoscope is controlled to increase the frame rate.

[0023] In summary, the present application provides a method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation, which calculates capsule endoscope motion and local gastrointestinal motion respectively, realizes hierarchical estimation of capsule endoscope motion and local gastrointestinal motion, and optimizes frame rate adjustment by combining the comprehensive influence of capsule endoscope motion and local gastrointestinal motion calculation. The present application uses the bee population algorithm to search for the optimal solution of rigid motion parameters, which can extract robustness, stability and significance features between adjacent frame images and establish matching, and more accurately estimate the motion of the capsule. The present application also establishes dense correspondence through the SIFT-FLOW algorithm, and combines rigid motion estimation to more accurately describe the comprehensive influence of capsule motion and gastrointestinal deformation. The present application evaluates whether there is a significant scene change between the two images by calculating the maximum inscribed circle diameter (DMC) of the invalid area of ​​the image scene change, and then controls the frame rate, which can effectively solve the problems existing in motion detection and frame rate adjustment in the prior art.

[0024] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0025] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation, characterized in that: The following steps are involved: Calculating capsule endoscope motion: describing the motion of the capsule endoscope as a two-dimensional rigid deformation, and calculating the best solution (BAME) of capsule endoscope motion parameters; Computing local gastrointestinal motion: Modeling gastrointestinal motion as a non-rigid deformation and predicting the motion of the local gastrointestinal tract based on the modeling; Calculate the invalid area: find the invalid area according to the local gastrointestinal movement, use the diameter of the maximum inscribed circle (DMC) of the invalid area to evaluate whether there is a scene change between the two images, and if the DMC exceeds a certain number of pixels, control the capsule endoscope to increase the frame rate, wherein the invalid area refers to the area in the source image that contains the gastrointestinal wall scene that cannot be found in the neighboring image.

2. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 1, characterized in that: In the steps of calculating the motion of the capsule endoscope: the honey bee algorithm is used to search for the optimal solution of the motion parameters of the capsule endoscope.

3. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 2, characterized in that: Steps in calculating capsule endoscope motion: BAME is described as seeking minimal error: ； Among them, I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation of searching for deformation parameters in the solution space, error n (I i-1 ,T n (I i )) is the error measure between the neighborhood image and the source image.

4. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 1, characterized in that: In the step of calculating local gastrointestinal motion: SIFT-FLOW is used to predict the motion of the local gastrointestinal tract.

5. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 4, characterized in that: In the SIFT-FLOW model, the SIFT descriptor establishes a dense correspondence between the neighborhood image and the source image. The SIFT descriptor has the excellent performance of local image illumination invariance and encoding local image structure.

6. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 5, characterized in that: Steps in calculating local gastrointestinal motility: IFT-FLOW is expressed as an optimization problem based on the corresponding search of the cost function: ， ； Among them, ε(p)=(u x (p),u y (p)) is the vector of pixel p=(x,y), s i-1 (p) and s i (p) are the SIFT descriptors extracted from the neighborhood image p in the neighborhood image i−1 and the source image i, respectively. N is the spatial neighborhood of the pixel. The source image is the transformed image using capsule endoscopy motion estimation. R(p,q) constrains the vector with neighboring pixels and regards the vector ε(p) as the local non-rigid motion component.

7. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 6, characterized in that: The motion estimation between adjacent frame images is described as: Among them, the matrix M is the transformation matrix of the capsule endoscope, ε is a local flow vector that can find the best local match between two consecutive images, and Z i−1 Yes i−1 The approximate value of the corresponding neighborhood image point.

8. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 7, characterized in that: In the step of calculating the invalid area: if the DMC exceeds 5 pixels, the capsule endoscope is controlled to increase the frame rate.

9. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 7, characterized in that: In the steps of calculating the invalid area: The first step is backward estimation: according to Z i−1估计从图像i到图像i−1的场景变形，并根据场景变形寻找图像i不在图像i−1中的点，这些点被记录为一个集合： ； The second step is forward estimation: according to Z i−1估计从图像i和图像i+1之间的一个集合： ； Compute the collection: An invalid region of the current frame is a collection: 。

Images