Adaptive Frame Rate Adjustment Method for Capsule Endoscope Based on Image Motion Estimation

Abstract

The present application discloses an adaptive frame rate adjustment method for a capsule endoscope based on image motion estimation, which includes the following steps: calculating the motion of the capsule endoscope: using a bee algorithm to search for the optimal solution of the motion parameters of the capsule endoscope; calculating the local gastrointestinal tract motion: using SIFT-FLOW to predict the motion of the local gastrointestinal tract, where the SIFT descriptor can establish dense correspondences between the neighborhood image and the source image, and the SIFT descriptor has excellent performance of local image illumination invariance and encoding the local image structure; calculating the invalid area: finding the invalid area according to the local gastrointestinal tract motion, and using the diameter of the maximum inscribed circle of the invalid area to evaluate whether there is a scene change between two images. If the DMC exceeds 5 pixels, the frame rate of the capsule endoscope is controlled to increase. The present application can effectively solve the problems existing in motion detection and frame rate adjustment in the prior art.

Description

Technical Field

[0001] This application relates to the technical field of capsule endoscopes, and particularly to an adaptive frame rate adjustment method for capsule endoscopes based on image motion estimation. Background Art

[0002] A capsule endoscope is a miniature wireless endoscope, a medical instrument used to examine the human gastrointestinal tract. After being swallowed by a patient, the capsule moves along the gastrointestinal tract, moving with peristalsis of internal body parts or by the action of gravity. By taking a large number of images per second of the examined part, and then based on the obtained images, the examined part can be observed and disease diagnosis can be made. The application of capsule endoscope technology has greatly improved the diagnostic efficiency of digestive tract diseases and the comfort of patients.

[0003] Existing capsule endoscopes usually take images at a fixed frame rate and move at a very uneven speed in the human body. When the capsule endoscope is stationary or moving slowly in the digestive tract, a large number of redundant images will be generated, which not only wastes battery power but also increases the burden on the film readers; when the capsule endoscope moves rapidly in the digestive tract, missed shots may occur due to insufficient frame rate. Summary of the Invention

[0004] An embodiment of this application provides an adaptive frame rate adjustment method for capsule endoscopes based on image motion estimation, which solves the problems of increased power consumption or missed shots due to insufficient frame rate caused by uneven movement speed of the capsule endoscope.

[0005] To solve the above technical problems, this application is implemented as follows:

[0006] An adaptive frame rate adjustment method for capsule endoscopes based on image motion estimation is provided, which includes the following steps: calculating the motion of the capsule endoscope: describing the motion of the capsule endoscope as a two-dimensional rigid deformation, and calculating the optimal solution of the motion parameters of the capsule endoscope (BAME); calculating the local gastrointestinal motion: modeling the gastrointestinal motion as a non-rigid deformation, and predicting the local gastrointestinal motion according to the modeling. Calculating the invalid area: finding the invalid area according to the local gastrointestinal motion, and using the diameter of the maximum inscribed circle of the invalid area (DMC) to evaluate whether there is a scene change between two images. If the DMC exceeds a certain number of pixels, control the capsule endoscope to increase the frame rate, where the invalid area refers to the area of the gastrointestinal wall scene in the source image that cannot be found in the neighboring image.

[0007] In one embodiment, in the step of calculating the motion of the capsule endoscope: a bee algorithm is used to search for the optimal solution of the motion parameters of the capsule endoscope.

[0008] In one embodiment, in the step of calculating the motion of the capsule endoscope: BAME is described as seeking the minimum error:

[0009] ;

[0010] Among them, I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation for searching deformation parameters in the solution space, error n (I i-1 , T n (I i )) is the error metric between the neighborhood image and the source image.

[0011] In one embodiment, in the step of calculating local gastrointestinal motility: SIFT-FLOW is used to predict the motility of the local gastrointestinal tract.

[0012] In one embodiment, in the SIFT-FLOW model, the SIFT descriptor establishes a dense correspondence between the neighborhood image and the source image, and the SIFT descriptor has excellent performance of local image illumination invariance and encoding local image structure.

[0013] In one embodiment, in the step of calculating local gastrointestinal motility: IFT-FLOW is expressed as an optimization problem of a correspondence search based on a cost function:

[0014] ,

[0015] ;

[0016] Among them,, ε(p) = (u x (p), u y (p)) is the vector of pixel p = (x, y), s i-1 (p) and s i (p) are the SIFT descriptors extracted at the position of neighborhood image p in the neighborhood image i−1 and the source image i respectively, N is the spatial neighborhood of the pixel, the source image is the image after transformation using capsule endoscope motion estimation, R(p, q) constrains the vector with adjacent pixels, and the vector ε(p) is regarded as the local non-rigid motion component;

[0017] In one embodiment, the motion estimation between adjacent frame images is described as:

[0018] .

[0019] Among them, the matrix M is the transformation matrix of the capsule endoscope, ε is a local flow vector, the best local match can be found between two consecutive images, Z i−1 is the approximation of the neighborhood image point corresponding to Y i−1 .

[0020] In one embodiment, in the step of calculating the invalid area: if the DMC exceeds 5 pixels, control the capsule endoscope to increase the frame rate.

[0021] In one embodiment, in the step of calculating the invalid area:

[0022] The first step is backward estimation: According to Z i−1 Estimate the scene deformation from image i to image i−1, and find the points in image i that are not in image i−1 according to the scene deformation. These points are recorded as a set: ;

[0023] The second step is forward estimation: According to Z i−1 Estimate a set between image i and image i + 1: ;

[0024] Calculate the union set: An invalid area of the current frame is a set: .

[0025] In the embodiments of the present application, the capsule endoscope movement and the local gastrointestinal movement are calculated separately to achieve the hierarchical estimation of the capsule endoscope movement and the local gastrointestinal movement. The comprehensive influence calculated by combining the capsule endoscope movement and the local gastrointestinal movement is used to optimize the frame rate adjustment. The present application uses the bee colony algorithm to search for the optimal solution of the rigid motion parameters, and can extract robust, stable and significant features between adjacent frame images and establish matches, and more accurately estimate the movement of the capsule. The present application also establishes a dense correspondence relationship through the SIFT-FLOW algorithm, and combines the rigid motion estimation to more accurately describe the comprehensive influence of the capsule movement and the gastrointestinal deformation. The present application calculates the maximum inscribed circle diameter (DMC) of the invalid area of the image scene change to evaluate whether there is a significant scene change between two images, and then controls the frame rate, so that the problems existing in motion detection and frame rate adjustment in the prior art can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0027] Figure 1 is the step flowchart of the capsule endoscope adaptive frame rate adjustment method based on image motion estimation of the present application;

[0028] Figure 2 is the capsule endoscope adaptive frame rate flowchart of the present application;

[0029] Figure 3 is the schematic diagram of the positive and negative measurement of the invalid area of the present application;

[0030] Figure 4 It is a schematic diagram of the colors in similar scenarios of this application. Specific implementation manners

[0031] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, rather than all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without making creative efforts shall fall within the protection scope of this application.

[0032] The motion model of the capsule endoscope includes rigid motion and non-rigid motion. When the capsule endoscope functions in the gastrointestinal tract, the capsule endoscope is pushed by gastrointestinal peristalsis. If only the motion of the capsule endoscope is considered, the images of adjacent frames can be described by homogeneous coordinates:

[0033] (1)

[0034] where Y i T is the image pixel, and M is a two-dimensional rigid transformation, including parameters such as translation T, rotation R, scaling s, and perspective p. These parameters describe the motion (global rigid motion) between the images of adjacent frames of the capsule endoscope.

[0035] The motion between adjacent frame images depends not only on the motion of the capsule endoscope but also on the non-rigid deformation (local non-rigid motion) of the gastrointestinal tract due to peristalsis:

[0036] (2)

[0037] According to this motion model, the method for adaptively adjusting the frame rate of the capsule endoscope based on image motion estimation in this application estimates the motion of adjacent frame images in two stages. In the first stage of coarse-level estimation, the motion of the capsule endoscope can be regarded as a large displacement estimation of the adjacent frame image scene. In the second stage of fine-level estimation, the local gastrointestinal deformation is estimated based on the result of the first-stage estimation. Please refer to Figure 1 and Figure 2 shown, which is the flowchart of the steps of the method for adaptively adjusting the frame rate of the capsule endoscope based on image motion estimation in this application; as shown, the method for adaptively adjusting the frame rate of the capsule endoscope based on image motion estimation includes the following steps S1 to S3. Among them:

[0038] Step S1, calculate the movement of the capsule endoscope: Describe the movement of the capsule endoscope as a two-dimensional rigid deformation, and calculate the best solution (BAME) of the movement parameters of the capsule endoscope. Since the capsule endoscope is propelled by gastrointestinal peristalsis, in fact, the gastrointestinal wall is very close to the capsule endoscope, which makes the projection deformation of adjacent frame images small, and the movement of the capsule endoscope can be described as a two-dimensional rigid deformation. Therefore, when estimating the movement of the capsule endoscope, only the rigid movement needs to be concerned, and the projection p and local gastrointestinal non-rigid displacement ε parameters can be ignored. The common method for such estimation is to extract different features from each image and match the features to establish global correspondence, and then estimate the transformation between the images (feature-based method). In reality, however, it is difficult to extract robust, stable, and significant features and establish matches between consecutive adjacent frame images. The main reasons for the foregoing problems are low image resolution, poor structural information, etc. In this embodiment, the bee algorithm is used to search for the best solution (BAME) of the capsule endoscope movement parameters. The BA algorithm is a population-based search algorithm used to solve complex multi-objective optimization problems that cannot be accurately solved within polynomial bounded computation time. Contrary to the feature-based method, this method is generally referred to as the direct (pixel-based) alignment method. In this embodiment, BAME is described as seeking the minimum error:

[0039] (3);

[0040] where I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation for searching the deformation parameters in the solution space, and error n (I i-1 , T n (I i )) is the error metric between the neighborhood image and the source image. Mutual information (MI) is used as the error metric. MI has been widely used in medical image registration. Since the captured images have similar intensities, colors, and hues, grayscale information is used to calculate MI. The pseudocode of BAME is shown in the following table:

[0041] 01.Input: Images I i-1 and I i

[0042] 02. • Initialize the bee population m ∈ Z

[0043] 03. • Randomly search for solutions TS = {T n = random(s, R, T): 1 ≤ n ≤ m, n ∈ Z}

[0044] 04. • Evaluate the error of each solution ES = {errorn (I i-1 ,T n (I i )): 1 ≤ n ≤ m}

[0045] 05. • While (not reaching the stop criterion)

[0046] 06. • Select the optimal solution eTS = {T n ∈ TS: error n ≤ threshold, n ≤ p}

[0047] 07. for i 1 = 1 to p

[0048] 08. • In each neighborhood, obtain new bees to get the best solution eTS = update(eTS)

[0049] 09. • eES = {error n (eTS)}

[0050] 10. End

[0051] 11. • Assign the remaining bees to random search rTS = {T n = random(s, R, T): p < n ≤ m}

[0052] 12. • Calculate the error rES = {error n (I i-1 ,T n (I i )): p ≤ n ≤ m}

[0053] 13. • TS = eTS ∪ rTS, ES = eES ∪ rES

[0054] 14. • End

[0055] 15.Output: T * ={T n ∈ TS: error n = argmin(ES)}。

[0056] Step S2, calculate local gastrointestinal motility: Model the gastrointestinal motility as a non-rigid deformation and predict the motility of the local gastrointestinal tract according to the model. Use SIFT-FLOW to predict the motility of the local gastrointestinal tract. Among them, the SIFT descriptor can establish dense correspondences between the neighborhood image and the source image. The SIFT descriptor has excellent performance in local image illumination invariance and encoding local image structure. Since the gastrointestinal motility can be modeled as a non-rigid deformation, which is caused by gastrointestinal peristalsis, use SIFT-FLOW to predict the motility of the local gastrointestinal tract. The SIFT descriptor can establish dense correspondences between the neighborhood image and the source image. The SIFT descriptor has excellent performance in local image illumination invariance and encoding local image structure, and these properties make the matching more robust. SIFT-FLOW can be expressed as an optimization problem of correspondence search based on a cost function:

[0057] (4)

[0058] (5)

[0059] where ε(p)=(u x (p),u y (p)) is the vector of pixel p=(x,y), s i-1 (p) and s i (p) are the SIFT descriptors extracted from the position of neighborhood image p in the neighborhood image i−1 and the source image i respectively, and N is the spatial neighborhood of the pixel. The source image is the image transformed using the motion estimation of the capsule endoscope. R(p,q) constrains the vector with adjacent pixels. The vector ε(p) is regarded as the local non-rigid body motion component.

[0060] Finally, the motion estimation between adjacent frame images can be described as:

[0061] (6)

[0062] As described above, the right side of equation (6) contains two terms. The first term is the global rigid body motion estimated by BAME, and the matrix M is the transformation matrix of the capsule endoscope. The second term is the local non-rigid body motion estimation of SIFT-FLOW, and ε is a local flow vector, and the best local match can be found between two consecutive images. Z i−1 is the approximation of the corresponding neighborhood image point of Y i−1 .

[0063] Step S3, calculate the invalid region: Locate the invalid region based on local gastrointestinal motility, and use the diameter (DMC) of the largest inscribed circle of the invalid region to evaluate whether there is a scene change between two images. If the DMC exceeds a certain number of pixels, control the capsule endoscope to increase the frame rate. In this embodiment, if the DMC exceeds 5 pixels, control the capsule endoscope to increase the frame rate. The invalid region refers to the region of the gastrointestinal wall scene in the source image that cannot be found in the neighboring image. Since a scene change in the capsule endoscope means that adjacent frame images have more different contexts, in order to perform a robust measurement of the scene change between adjacent frame images, the concept of the invalid region is defined. The invalid region refers to the region of the gastrointestinal wall scene in the image (source image) that cannot be found in the neighboring image (or key image). As Figure 3 shown, the invalid region estimation is implemented in two steps. The first step is backward estimation. Estimate the scene deformation from image i to image i−1 according to Eq(6), and find the points in image i that are not in image i−1 according to the scene deformation. These points are recorded as a set:

[0064] (7)

[0065] The second step is forward estimation. Record the points as a set between image i and image i+1:

[0066] (8)

[0067] Finally, an invalid region of the current frame is a set:

[0068] (9).

[0069] Specifically, the area of the invalid region can reflect the scene change between adjacent frame images. Figure 4 shows an invalid region of image i. The green and red regions are the scenes of image i ( Figure 4 processed in black and white), which can only be found in images i−1 and i+1 respectively. The yellow region (the scene of image i) can be found in both images i−1 and i+1. The green circle is the largest inscribed circle of the invalid region. Even if the number of points in the invalid region is large, it may still not indicate a large displacement. Since the invalid regions of images are usually distributed at the edges and are usually very narrow. In fact, the maximum diameter of the invalid region can reflect the potential possibility of the scene size change. Therefore, this measurement is simplified, and the diameter (DMC) of the largest inscribed circle of the invalid region is used to evaluate whether there is a scene change between two images. If the DMC exceeds 5 pixels, it indicates that the scene change between the images is large, and control the capsule endoscope to increase the frame rate.

[0070] In summary, the present application provides an adaptive frame rate adjustment method for a capsule endoscope based on image motion estimation, which calculates the motion of the capsule endoscope and the local gastrointestinal motion respectively, realizes the hierarchical estimation of the capsule endoscope motion and the local gastrointestinal motion, and optimizes the frame rate adjustment by combining the comprehensive influence calculated by the capsule endoscope motion and the local gastrointestinal motion. The present application uses the bee colony algorithm to search for the optimal solution of the rigid motion parameters, can extract robust, stable and significant features between adjacent frame images and establish matches, and more accurately estimate the motion of the capsule. The present application also establishes a dense correspondence relationship through the SIFT-FLOW algorithm and combines the rigid motion estimation to more accurately describe the comprehensive influence of the capsule motion and the gastrointestinal deformation. The present application calculates the maximum inscribed circle diameter (DMC) of the invalid area of the image scene change to evaluate whether there is a significant scene change between two images, and then controls the frame rate, so as to effectively solve the problems existing in motion detection and frame rate adjustment in the prior art.

[0071] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0072] The embodiments of the present application have been described above with reference to the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation, characterized in that: The following steps are involved: Calculating capsule endoscope motion: describing the motion of the capsule endoscope as a two-dimensional rigid deformation, and calculating the optimal solution BAME of the capsule endoscope motion parameters; Computing local gastrointestinal motion: Modeling gastrointestinal motion as a non-rigid deformation and predicting the motion of the local gastrointestinal tract based on the modeling; Calculate the invalid area: find the invalid area according to the local gastrointestinal movement, use the diameter DMC of the maximum inscribed circle of the invalid area to evaluate whether there is a scene change between the two images, and if the DMC exceeds a certain number of pixels, control the capsule endoscope to increase the frame rate, wherein the invalid area refers to the area in the source image that contains the gastrointestinal wall scene that cannot be found in the neighboring image; Steps in calculating capsule endoscope motion: BAME is described as seeking minimal error: ； Among them, I i and I i-1 are the neighborhood image and the source image respectively, T n is the nth transformation of searching for deformation parameters in the solution space, error n (I i-1 ,T n (I i )) is the error measure between the neighborhood image and the source image; In the steps of calculating the local gastrointestinal motion: SIFT-FLOW is used to predict the local gastrointestinal motion; In the IFT-FLOW model, the SIFT descriptor establishes a dense correspondence between the neighborhood image and the source image. The SIFT descriptor has the excellent performance of local image illumination invariance and encoding local image structure; In the step of calculating local gastrointestinal motility: IFT-FLOW is expressed as an optimization problem based on a corresponding search of the cost function: ， ； Among them, ε(p)=(u x (p),u y (p)) is the vector of pixel p=(x,y), s i-1 (p) and s i (p) are the SIFT descriptors extracted from the neighborhood image p in the neighborhood image i−1 and the source image i, respectively. N is the spatial neighborhood of the pixel. The source image is the transformed image using capsule endoscopy motion estimation. R(p,q) constrains the vector with neighboring pixels and regards the vector ε(p) as the local non-rigid motion component.

2. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 1, characterized in that: In the steps of calculating the motion of the capsule endoscope: the honey bee algorithm is used to search for the optimal solution of the motion parameters of the capsule endoscope.

3. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 2, characterized in that: The motion estimation between adjacent frame images is described as: Among them, the matrix M is the transformation matrix of the capsule endoscope, ε is a local flow vector that can find the best local match between two consecutive images, and Z i−1 Yes i−1 The approximate value of the corresponding neighborhood image point.

4. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 3, characterized in that: In the step of calculating the invalid area: if the DMC exceeds 5 pixels, the capsule endoscope is controlled to increase the frame rate.

5. The method for adaptive frame rate adjustment of capsule endoscope based on image motion estimation according to claim 3, characterized in that: In the steps of calculating the invalid area: The first step is backward estimation: according to Z i−1 Estimate the scene deformation from image i to image i−1, and find the points of image i that are not in image i−1 according to the scene deformation. These points are recorded as a set: ； The second step is forward estimation: according to Z i−1 Estimate a set between image i and image i+1: ； Compute the collection: An invalid region of the current frame is a collection: 。

Images