Method and device for measuring travel distance of capsule camera in gastrointestinal tract

Abstract

The present invention relates to a method and apparatus for measuring the travel distance of a capsule camera in a gastrointestinal tract. Specifically, the present invention discloses a method and system for determining the travel distance of a capsule camera. A current global motion vector of a current image in an image sequence is determined, wherein the current global motion vector corresponds to the motion made by the capsule camera between the current image and a reference image associated with the image sequence. Based on the current global motion vector and a previous global motion vector derived for a previous image between an initial image and the current image, the travel distance of the capsule camera in the gastrointestinal tract is determined, wherein the travel distance is measured from an initial position associated with the initial image to a current position associated with the current image. The present invention also discloses a method and system for displaying a sequence of images captured by a capsule camera. The travel distance associated with the image sequence is displayed on a display.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 2019101194736, application date February 18, 2019, and invention name “Method and device for measuring the travel distance of a capsule camera in the gastrointestinal tract”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] The present invention is related to U.S. Patent No. 9,936,151, published on April 3, 2018, and U.S. Patent Application No. 15 / 933,375, filed on March 23, 2018. The entirety of the U.S. Patent and U.S. Patent Application is incorporated herein by reference. Technical Field

[0004] The present invention relates to estimating a travel distance of a capsule camera in a gastrointestinal tract based on images captured by the capsule camera when the capsule camera passes through the gastrointestinal tract. Background Art

[0005] Devices for imaging body cavities or passages in vivo are known in the prior art and include endoscopes and autonomous encapsulated cameras. An endoscope is a flexible or rigid tube that enters the body through a body orifice or surgical opening, typically into the esophagus via the mouth or into the colon via the rectum. An image is formed at the distal end using a lens and transmitted to the proximal end outside the body by a lens relay system or by a coherent fiber optic bundle. Conceptually similar instruments may, for example, record images electronically at the distal end using a CCD or CMOS array and transmit the image data as an electrical signal to the proximal end via a cable. Endoscopes allow the physician to control the field of view and are widely accepted diagnostic tools. However, they do have several limitations, pose risks to the patient, are invasive and uncomfortable for the patient, and their cost limits their use as routine health screening tools.

[0006] Due to the difficulty in traversing the convoluted passages, endoscopes cannot easily reach much of the small intestine, and require specific techniques and precautions that increase costs to reach the entire colon. Endoscopic risks include possible perforation of the body organs traversed and complications caused by anesthesia. Furthermore, a compromise must be made between patient pain and health risks during the procedure and post-procedural downtime associated with anesthesia.

[0007] An alternative in vivo image sensor that solves many of these problems is the capsule endoscope. A camera is housed in a swallowable capsule along with a radio transmitter for transmitting data, primarily including images recorded by the digital camera, to a base station receiver or transceiver and data recorder outside the body. The capsule may also include a radio receiver for receiving instructions or other data from the base station transmitter. Instead of radio frequency transmission, lower frequency electromagnetic signals may be used. Power may be supplied inductively from an external inductor to an internal inductor within the capsule or by a battery within the capsule.

[0008] An autonomous capsule camera system with on-board data storage is disclosed in U.S. Patent No. 7,983,458, entitled "In Vivo Autonomous Camera with On-Board Data Storage or Digital Wireless Transmission in Regulatory Approved Band," which was granted on July 19, 2011. This patent describes a capsule system that stores captured images using on-board storage such as semiconductor non-volatile file memory. After the capsule passes through the body, it is retrieved. The capsule shell is opened and the stored images are transmitted to a computer workstation for storage and analysis. For capsule images received via wireless transmission or retrieved from on-board storage, the images will have to be displayed and reviewed by a diagnostician to identify potential abnormalities.

[0009] Figure 1 An example capsule system with onboard storage is shown. Capsule system 110 includes an illumination system 12 and a camera including an optical system 14 and an image sensor 16. Semiconductor non-volatile archival memory 20 may be provided to enable images to be stored and subsequently retrieved at a docking station outside the body after the capsule is retrieved. System 110 includes a battery power supply 24 and an output port 26. Capsule system 110 may be propelled through the gastrointestinal tract by peristalsis.

[0010] The lighting system 12 may be implemented by LEDs. Figure 1 In the embodiment, the LED is disposed adjacent to the aperture of the camera, but other configurations are possible. For example, the light source may also be disposed behind the aperture. Other light sources, such as laser diodes, may also be used. Alternatively, a white light source or a combination of two or more narrow wavelength band sources may also be used. White LEDs are available, which may include blue or violet LEDs, and phosphorescent materials that are excited by the LED light to emit light of a longer wavelength. The portion of the capsule housing 10 that allows light to pass through may be made of biocompatible glass or polymer.

[0011] The optical system 14 (which may include a plurality of refractive, diffractive or reflective lens elements) provides an image of the lumen wall on the image sensor 16. The image sensor 16 may be provided by a charged-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) type device that converts the received light intensity into a corresponding electrical signal. The image sensor 16 may have a monochromatic response, or include a color filter array, so that a color image can be captured (for example, by using RGB or CYM representation). Preferably, the analog signal from the image sensor 16 is converted to digital form to allow processing in digital form. Such conversion may be implemented by an analog-to-digital (A / D) converter, which may be provided inside the sensor (as is the case here), or in another part inside the capsule housing 10. The A / D unit may be provided between the image sensor 16 and the rest of the system. The LEDs in the illumination system 12 are synchronized with the operation of the image sensor 16. The processing module 22 can be used to provide the processing required by the system, such as image processing and video compression. The processing module can also provide the required system control, such as controlling the LEDs during image acquisition operations. The processing module can also be responsible for other functions, such as managing image acquisition and coordinating image retrieval.

[0012] After the capsule camera passes through the gastrointestinal tract and exits the body, the capsule camera is retrieved and the images stored in the archive memory are read out through the output port. The received images are usually transmitted to a base station for processing and review by a diagnosing physician. The accuracy and efficiency of the diagnosis are of utmost importance. It is expected that the diagnosing physician reviews all images and correctly identifies all abnormalities. Moreover, it is desirable to collect location information of the abnormality, which is useful for possible manipulation or treatment of the abnormality. Although various position detection devices can be embedded or attached to the capsule device, it is desirable to develop a method for determining the travel distance based on the captured images. Summary of the invention

[0013] The present invention discloses a method and system for determining a travel distance of a capsule camera. According to the method, a sequence of images is received, wherein the capsule camera captures the sequence of images as the capsule camera moves through a gastrointestinal (GI) tract. Distance information is received that is associated with an object distance between the capsule camera and a plurality of points in a current image in the sequence of images. The current image is normalized according to the object distance between the capsule camera and the plurality of points in the current image to generate a standardized current image. A current global motion vector of the standardized current image in the sequence of images is determined, wherein the current global motion vector corresponds to a motion made by the capsule camera between the standardized current image and a standardized reference image associated with the sequence of images. A travel distance of the capsule camera in the GI tract is determined according to the current global motion vector and a previous global motion vector derived for a previous image between the standardized initial image and the standardized current image, wherein the travel distance is measured from an initial position associated with the standardized initial image to a current position associated with the standardized current image.

[0014] The travel distance may be estimated by accumulating capsule motion along the longitudinal direction of the gastrointestinal tract based on the current global motion vector and the previous global motion vector. Moreover, capsule motion associated with a target global motion vector along the longitudinal direction of the gastrointestinal tract may be determined by projecting the target global motion vector into the longitudinal direction. The images of the image sequence include panoramic images, wherein each panoramic image corresponds to a scene in a field of view covering 360 degrees of a section of the gastrointestinal tract wall around the capsule camera.

[0015] In one embodiment, for a target image, a global motion vector is derived by dividing the target image into blocks to derive individual motion vectors for the blocks, and the global motion vector is derived from the individual motion vectors. In another embodiment, for a target image, the global motion vector is derived by applying an affine motion model between the target image and a target reference image.

[0016] The method may further comprise providing information associating the travel distance with an image of the sequence of images. For example, the information associating the travel distance with the image of the sequence of images comprises the current travel distance and the identification of the corresponding image.

[0017] The travel distance can be measured in a forward direction or a reverse direction. When measured in a forward direction, the current image is after the initial image in time. When measured in a reverse direction, the current image is before the initial image in time.

[0018] The present invention also discloses a method and system for displaying a sequence of images captured by a capsule camera. According to the method, a sequence of images is received, wherein the capsule camera captures the sequence of images as the capsule camera moves through a gastrointestinal (GI) tract. Travel distances associated with the sequence of images are presented on a display, wherein each travel distance corresponds to an estimated distance measured from an initial position associated with an initial image in the sequence of images to a current position associated with a current image in the sequence of images. One or more graphical representations representing the travel distances are displayed on the display according to a position on one or more graphical representations.

[0019] The method may further include displaying a marker on at least one of the one or more graphical representations to indicate a corresponding image associated with the target location indicated by the marker. In addition, the method may include displaying the corresponding image associated with the target location indicated by the marker on the display. The method may further include displaying the corresponding image associated with the target location indicated by the marker on the display.

[0020] In the above method, the current travel distance may be accumulated from a global motion vector associated with the initial image and the current image in the image sequence. The global motion vector is derived from normalized images of the image sequence, and each normalized image is generated by normalizing the corresponding image according to the distance between the capsule camera and a plurality of points in the corresponding image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An example capsule system with onboard storage is shown, wherein the capsule system includes an illumination system and a camera including an optical system and an image sensor.

[0022] Figure 2 A simplified example of camera geometry is shown, where the image sensor is placed at the focal plane behind the lens.

[0023] Figure 3 A simplified example of distance traveled estimation based on global motion vectors and camera optical parameters is shown.

[0024] Figure 4 A simplified cross-sectional view of a four-lens subsystem is shown, with the optical axes spaced 90° apart in target space.

[0025] Figure 5A and Figure 5B Two examples of camera-object distances in a gastrointestinal environment are shown, where Figure 5B Compared with the example in Figure 5A The middle camera is farther away from the object (polyp) on the gastrointestinal wall.

[0026] Fig. 6AA cross section of the gastrointestinal tract is shown for the case where the capsule camera is located in the center of the gastrointestinal tract wall, wherein the gastrointestinal tract wall is modeled as a perfect round tube.

[0027] Figure 6B Another situation is shown where the capsule is closer to the bottom side of the gastrointestinal wall.

[0028] Fig. 7A Shown in Fig. 6A Uniform motion vector field in an image captured by a panoramic camera under the condition of

[0029] Figure 7B Shown in Figure 6B The changing motion vector field in the image captured by the panoramic camera under the condition of

[0030] Figure 8 Examples of normalized motion vectors depending on the distance between the respective camera and the wall of the gastrointestinal tract are shown.

[0031] Fig. 9 The situation showing the varying distance between the capsule camera and the GI tract wall due to the non-smooth GI tract wall and the non-central position of the capsule camera.

[0032] Fig.10 Shows the Fig. 9 Example of a motion vector field for the situation in .

[0033] Fig.11 It is shown that the varying object-camera distance can be handled by considering local image normalization, which leads to Fig.10 Example of normalized motion vectors for motion vectors in .

[0034] Fig.12 An example of a section of the gastrointestinal tract is shown with the longitudinal direction at the center of the gastrointestinal tract indicated by the dashed line.

[0035] FIG. 13A to FIG. 13E Various embodiments are shown for displaying distance traveled information on a display.

[0036] Fig.14 An example of displaying travel distance information together with image information is shown, wherein a graphical representation of the travel distance and the time code of an image indicated by a marker is displayed in an area on the display.

[0037] Fig.15 Displays the distance traveled and Fig.14 Another example of similar image information, however, the image indicated by the mark is also displayed in the area on the display instead of the time code.

[0038] Fig.16A further example of displaying travel distance information together with image information is shown, wherein anatomical parts of the gastrointestinal tract are also marked on the graphical representation of the travel distance.

[0039] Fig.17 An example based on a laptop computer implementation is shown, wherein a CPU in the laptop computer will perform the required processing and may display the travel distance information and / or corresponding image information on a display (ie, the laptop computer screen).

[0040] Fig.18 An example flow chart for determining the travel distance of a capsule camera according to an embodiment of the present invention is shown.

[0041] Fig.19 An example flow chart for displaying a sequence of images captured by a capsule camera according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] It will be readily appreciated that the components of the present invention as generally described and shown in the accompanying drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention shown in the accompanying drawings is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. Reference to "one embodiment", "embodiment" or similar language in this specification refers to a particular feature, structure or characteristic described in association with the embodiment that may be included in at least one embodiment of the present invention. Therefore, the appearance of "in one embodiment" or "in an embodiment" in different places in this specification does not necessarily refer to the same embodiment.

[0043] Moreover, the features, structures or characteristics described may be combined in any suitable manner in one or more embodiments. However, those skilled in the relevant art will appreciate that the present invention may be implemented without one or more specific details or by other methods, components, etc. In other examples, known structures or operations are not shown or described in detail to avoid obscuring the aspects of the present invention. Example embodiments of the present invention will be better understood by reference to the accompanying drawings, in which like reference numerals represent like parts. The following description is merely an example, briefly describing certain selected embodiments of the apparatus and method consistent with the invention claimed herein.

[0044] An endoscope is typically inserted into the human body through a natural opening such as the mouth or anus. Therefore, the endoscope is preferably small in size to minimize invasiveness. As previously described, an endoscope can be used for diagnosis of the gastrointestinal (GI) tract of the human body. A sequence of captured images can be viewed to identify any possible abnormalities. If an abnormality is found, it is of interest to identify the nature of the abnormality as well as its location. Accordingly, the present invention discloses an endoscope that includes a means for estimating the travel distance of a camera within the GI tract based on captured images of the GI tract.

[0045] In the past, certain types of positioning components such as accelerometers, gyrators, etc. were used to track ingestible devices in the gastrointestinal tract as the ingestible devices containing these components passed through the gastrointestinal tract. However, after swallowing the ingestible device, the subject usually does not remain still. When the subject moves, these components cannot reliably distinguish between the movement of the torso or the device in the gastrointestinal tract. What the components record is a combination of both movements. In order to develop a reliable estimate of the traveled distance, the present invention discloses an image-based method. In particular, the present invention discloses an estimate of the traveled distance based on a global motion vector derived from a sequence of images captured by a capsule camera as it passes through the gastrointestinal tract. By using the global motion estimation method, tracking in the longitudinal direction is tracking of the device movement relative to the gastrointestinal tract. Therefore, it more reliably records the longitudinal curve of the gastrointestinal tract. Although the intestines in the body are not completely fixed in the torso, the uncertainty is much smaller than the entire torso movement.

[0046] Motion estimation is a technique widely used in video compression to estimate the motion between images to compensate for the motion and reduce the difference between the images. Through this reduced difference, the bit rate required to encode the video sequence is greatly reduced. The motion in the scene is usually described by a motion field or motion vector. The motion can correspond to local motion associated with a moving object within the scene or global motion associated with a large area or the entire frame. Global motion is often caused by camera panning or camera motion. For example, in a video sequence captured by a camera on a moving vehicle facing in a direction perpendicular to the direction of motion, the motion vectors in the images will be significantly consistent, and the main motion vector corresponds to the global motion vector. If the camera optical parameters are known, the global motion vector (that is, the motion between the two images) can be simulated by the motion of the camera.

[0047] Figure 2 Simplified example showing camera geometry. In the camera system, the image sensor is placed at the focal plane 220 behind the lens 210. The camera can capture the scene within the field of view extending over an angle α. The focal length f is the distance between the lens and the image sensor. The focal length is often fixed for endoscopic applications and is known by design. If the distance D between the camera and the object is known, the size of the object can be determined from the captured image by measuring the size of the object image in the image. For example, if an object 230 with a height H is at a distance D from the camera, the object image height H can be derived from the object image height h in the image according to the following formula.

[0048]

[0049] In the above equation, h is obtained from image measurement, focal length f is known by design, and distance D is determined by the selected distance measurement method, such as structured light technology. Accordingly, if the distance can be determined, the object size can be derived. The size of the object in the image can be measured in physical size. However, it may be more convenient to capture the image digitally and perform the size measurement in pixel count. Because the physical size and optical footprint of the image sensor surface are known. In addition, the number of pixels is known (for example, 320x240). Therefore, the image size of the object in the image can be measured in pixel count and converted to the physical object image size in the image.

[0050] As shown above, the image size of an object in an image depends on the actual object size and its distance from the camera. A smaller object that is closer may appear to be the same size as a larger object in an image that is farther away.

[0051] Figure 3 A simplified example of distance traveled estimation based on global motion vectors and camera optical parameters is shown. In this example, object 310 is stationary and the camera is moving. At time t1, the camera is at position 320 and object 310 is in front of the camera. Object 310 is projected to center position 330 of capture frame f1. At time t2, the camera moves to position 340. Object 310 is projected to image position 330 of capture frame f2. In capture frame f2, an object position (330') corresponding to the object position (330) in capture frame f1 is indicated. The motion vector between object position 330 and object position 330' can be derived from capture frame f1 and capture frame f2 by using motion estimation techniques. In particular, a corresponding global motion vector (MV g ), where the motion vector is measured in number of pixels.

[0052] Since the image sensor size and resolution are also known, the actual length (l) of the motion vector can be determined according to l = |MV| × pl, where |MV| corresponds to the magnitude of the motion vector and pl corresponds to the distance between pixels. The actual travel distance L can be measured according to the following formula:

[0053]

[0054] The concept of motion estimation can be applied to images captured by a capsule camera as it passes through the gastrointestinal tract of a human subject. However, the motion vectors derived from the images of the gastrointestinal tract captured by the capsule camera are different from those described above. Figure 3The capsule camera may correspond to a forward-looking camera located at one end of the elongated capsule device. A global motion vector representing the motion of the camera may not be easily derived. The capsule camera may also correspond to a panoramic camera, such as the capsule camera system described in U.S. Patent No. 7,940,973, issued on May 10, 2011. Figure 4 A simplified cross-sectional view of four lens subsystems 401 to 404 is shown, where the optical axes are 90° apart in the target space. Lens subsystem 401 covers field of view 411, lens subsystem 402 covers field of view 412, lens subsystem 403 covers field of view 413, and lens subsystem 404 covers field of view 414. The sub-images captured by the multiple cameras can be combined together to form a 360° wide-angle image.

[0055] When the capsule camera passes through the gastrointestinal tract, the capsule camera may not be located in the center of the gastrointestinal tract. Therefore, the distance from the camera to the gastrointestinal wall may be different. Figure 5A and 5B Two examples showing camera-object distances in a gastrointestinal environment. Figure 5A In, with Figure 5B Compared with the example in FIG. 5 , the camera 511 is farther from the object 513 (polyp) on the gastrointestinal wall 512. Therefore, the corresponding Figure 5A The objects in the image will appear smaller than the corresponding Figure 5B of the objects in the image.

[0056] Fig. 6A The cross section shows the capsule camera 611 being located at the center of the gastrointestinal tract wall 612. Fig. 6A In the simulation, the gastrointestinal wall is simulated as a perfect round tube. Each panoramic camera covers a field of view (FOV) corresponding to θ. Fig. 6A As shown in , the FOVs of adjacent cameras overlap slightly. The overlapping image areas will be processed to form a seamless image. Since the capsule is located in the center of the gastrointestinal tract, objects with the same size located on the gastrointestinal wall should appear to have the same size in front of the corresponding cameras. Figure 6B Another situation is shown where the capsule is closer to the bottom side of the gastrointestinal wall. In this case, for objects of the same size located on the gastrointestinal wall, the objects in image 4 (i.e., the image captured by the downward-looking camera) will appear larger than the objects in image 2 (i.e., the image captured by the upward-looking camera). On the other hand, the objects in images 1 and 3 will appear smaller than the object in image 4, but larger than the object in image 2.

[0057] exist Fig. 6A In the case of, if the capsule device moves along the longitudinal direction of the gastrointestinal tract, the image 710 captured by the panoramic camera will show Fig. 7AThe uniform motion vector field shown in . Fig. 7A In FIG. 7 , the panoramic image 710 is composed of four numbered sub-images with seamless sub-image boundaries indicated by dashed lines. Since the panoramic camera in this example covers a 360° field of view, the right side (A′) of the panoramic image 710 is wrapped around the left side (A). Figure 6B In the case of, if the capsule device moves along the longitudinal direction of the gastrointestinal tract, the image 720 captured by the panoramic camera will show Figure 7B . For sub-image 4, the camera is very close to the GI tract wall and the objects will appear larger compared to the image captured by the camera on the opposite side (i.e. associated sub-image 2). Accordingly, the motion vectors in sub-image 4 will be larger compared to the motion vectors in sub-image 2. According to conventional motion estimation techniques, no global motion will be identified.

[0058] exist Figure 7B In the case shown in , the varying motion vectors are caused by the different distances between the corresponding cameras and the gastrointestinal wall. The optical parameters of the cameras are known by design. Therefore, if the distance between the corresponding cameras and the gastrointestinal wall is also known, the motion vectors can be used to estimate the travel distance of the camera according to equation (2), as Figure 3 The distance traveled by the capsule device can be derived as an average of the individual travel distances estimated based on the individual motion vectors. In another example, the motion vectors can be normalized based on the distance between the corresponding cameras and the gastrointestinal wall. For example, the distance between the cameras and the gastrointestinal wall can be normalized relative to a nominal distance, such as the average distance between the cameras and the gastrointestinal wall (i.e., Fig. 6A The distance between each camera and the gastrointestinal wall in the normalized motion vector. Figure 7B In , although the magnitude of the motion vectors varies in the horizontal direction, these motion vectors should be mapped to the same travel distance of the capsule camera. In other words, for two motion vectors MV1 and MV2 of two different image blocks with distances D1 and D2 between the corresponding camera and the gastrointestinal wall, respectively, the following equation holds:

[0059]

[0060] Therefore, the two motion vectors are related according to D1·|MV1|=D2·|MV2|. Normalize all motion vectors, then The normalized motion vectors MV1' and MV2' (i.e., with distance The corresponding motion vector of ) is derived as:

[0061] as well as

[0062]

[0063] Figure 8 An example of normalized motion vectors based on the distance between the corresponding camera and the wall of the gastrointestinal tract is shown. The motion vectors derived from the panoramic image 810 may not be accurate due to noise in the image. Accordingly, the normalized motion vectors may also be inaccurate. However, the average normalized motion vector (820) will provide a more reliable estimate of the true global motion. Figure 8 In the example, the normalized motion vector is normalized relative to the average distance between the camera and the gastrointestinal wall. Although the average of the normalized motion vectors can be used to estimate the global motion vector, other techniques can also be used to estimate the global motion vector. For example, the median or dominant motion vector can be used to estimate the global motion vector.

[0064] In the above discussion, a simplified GI tract wall model (i.e., a cylindrical tube) is used to illustrate that the changing motion vectors may be caused by different distances between the GI tract wall and the capsule camera. In real life, the situation may be more complicated. For example, the GI tract is very different from the simplified cylindrical tube model and the distance between the capsule camera and the GI tract will vary from location to location. Moreover, the longitudinal axis of the capsule camera may not be aligned with the longitudinal direction of the section of the GI tract where the capsule camera is located. In addition, the capsule camera may experience three-dimensional motions, such as tilt and rotation, which further complicates the motion estimation process.

[0065] The longitudinal direction can be derived from the motion vector. For example, the motion vector can be smoothed and represents the shape of the trace of the gastrointestinal tract inside the torso. Information about the longitudinal distance traveled by the capsule camera can provide the doctor with good position information to operate on the lesion found in the image.

[0066] Fig. 9 The changing distance between the capsule camera and the gastrointestinal wall due to the uneven gastrointestinal wall 912 and the non-central position of the capsule camera 910 is shown. Distance lines 921, 922, 923, 931, 932, and 933 show the changing distance due to the uneven gastrointestinal wall 912 and the non-central position and tilt of the capsule camera 910. Although the gastrointestinal wall is uneven, each section of the gastrointestinal wall can be simulated as follows: Fig. 9 The uneven surface of the gastrointestinal tract will locally distort the image captured by the capsule camera. For example, the point with distance line 922 will appear larger in the image than the point with distance line 921 or 923 because the point with distance line 922 is closer to the camera than the other two points.

[0067] The uneven surface of the gastrointestinal tract wall will cause inaccuracies in the motion estimation process. Various motion estimation methods are known in the field of video compression. Among them, block matching is a very popular search algorithm, in which the block matching process searches for the best matching block in a reference picture (that is, a previously processed picture) for a given block in the current picture. The motion between the two corresponding blocks is specified using a motion vector. The motion may correspond to a simple translational motion, which represents the motion of the block by offsetting the block horizontally and / or vertically by a certain offset. The motion may correspond to more complex motions including two-dimensional block rotations and two-dimensional deformations. The affine model can describe two-dimensional block rotations and two-dimensional deformations to transform a square (or rectangle) into a parallelogram. This model can be described as follows:

[0068] x' = a0 + a1 * x + a2 * y, and

[0069] y'=b0+b1*x+b2*y

[0070] In the above equation, (x, y) corresponds to the pixel coordinates of the current image, and (x', y') corresponds to the pixel coordinates of the reference image. For each pixel A(x, y) in the current image, the motion vector between this pixel and its corresponding reference pixel A'(x', y') is (a0+a1*x+a2*y, b0+b1*x+b2*y). Therefore, the motion vector of each pixel is also related to the position. Affine motion estimation is also known in the prior art and the details will not be repeated in this article.

[0071] As described above, the non-smooth surface of the gastrointestinal tract wall will result in different distances between the object position and the camera for pixels in the corresponding image area. This local distance variation will have a negative impact on the accuracy of the motion estimation result. Therefore, according to one embodiment of the present invention, before performing the block matching process, the object-camera distance of the pixels in the search area for motion estimation and the local variation of the object-camera distance of the pixels in the current block are normalized. For example, for the search area indicated by the arc 930 in the cross section, the object-camera distance of the pixels in the search area is normalized based on the object-camera distance of the individual pixels. Object-camera distance normalization will enlarge the small blocks at a longer distance and reduce the small blocks at a closer distance. Image normalization based on object-camera distance can be performed on a sub-block basis by dividing the area into sub-blocks (e.g., 4x4 pixels), and each sub-block is associated with an individual object-camera distance. If the distance of a given sub-block is less than the nominal distance, the sub-block is proportionally reduced according to the nominal distance-individual distance ratio. If the distance of a given sub-block is greater than the nominal distance, the sub-block is proportionally enlarged according to the nominal distance-individual distance ratio. As is known in image processing, pixel interpolation will be required during image enlargement or reduction (i.e., scaling). After image normalization based on the object-camera distance, the motion estimation process should result in a more accurate motion estimation result. Image normalization focuses on the distance changes caused by the uneven surface of the gastrointestinal wall. This normalization is used in the stages before the motion estimation process to improve the accuracy of the motion estimation. Accordingly, the image normalization used above to compensate for the uneven surface of the gastrointestinal wall is called local image normalization for object-camera distance changes. In one embodiment, the effect of the distance change caused by the eccentric position of the capsule camera will be processed after the motion vector is derived. Fig.10 Shows the approach to handling varying object-camera distances by considering local image normalization. Fig. 9 Without local image normalization to compensate for the effects of the uneven GI tract wall, the derived motion vectors will be less accurate.

[0072] exist Fig.10 In the figure, the motion vector mainly points to the upper right, which represents the case where the capsule camera is also rotating around its longitudinal axis. Moreover, since the capsule camera is offset from the center of the gastrointestinal wall, the motion vector in sub-image 2 is usually smaller than the motion vector in sub-image 4. Therefore, the motion vector normalization shown in FIG. 7 and equations (4) and (5) can be applied to normalize Fig.10 In particular, the motion vectors may be normalized relative to a target distance, such as an average distance between a camera and the wall of the gastrointestinal tract. Fig.11 Display for Fig.10 An example of a normalized motion vector of a motion vector in FIG. The average value (1120) of the normalized vectors may be used as a global motion vector for the capsule camera.

[0073] Alternatively, the distance measured within the FOV may be Fig. 9 The camera in the gastrointestinal wall is normalized to unit radius, similar to Fig. 6A For example, the unit radius can be set to 1 cm or 1.25 cm.

[0074] We intend to use the global motion vector to estimate the travel distance of the capsule device in the gastrointestinal tract. The gastrointestinal tract is folded in the human body. From the perspective of the travel distance of the capsule device, the capsule camera motion along the longitudinal direction of the gastrointestinal tract is of interest. Therefore, the motion along the longitudinal direction of the gastrointestinal tract should be measured. Fig.12 , a section of the gastrointestinal tract 1210 is shown and indicated by a dotted line 1220 in the longitudinal direction at the center of the gastrointestinal tract. An image region 1221 associated with the surface of a section of the gastrointestinal tract is indicated by a thick line on the inner surface of the wall of the gastrointestinal tract. A global motion vector 1222 can be derived based on the image region 1221. The image region 1221 may correspond to a 360° image of the target segment. However, the image region 1221 may correspond to a partial image of the target segment. The global motion vector may be determined by dividing the image region 1221 into blocks and first deriving a separate motion vector. Then, the global motion vector is derived from the motion vector. However, the global motion vector may also be derived directly from the image region. For example, by using the image region without dividing it into blocks, parameters associated with the global motion vector (e.g., parameters for an affine motion model) may be derived.

[0075] Since the capsule device moves along the gastrointestinal tract under the action of peristalsis, it may not move precisely in the longitudinal direction. For example, the global motion vectors derived from different segments of the gastrointestinal tract are represented by arrows. In order to measure the travel distance of the capsule device, the motion vector component along the longitudinal direction must be determined. The motion vector component along the longitudinal direction can be determined by projecting the motion vector to the longitudinal direction. For example, the motion vector 1222 is projected to the longitudinal direction to obtain the travel distance 1223 associated with the motion vector 1222. The individual travel distances derived from the image sequence of the gastrointestinal tract can be accumulated to obtain the total travel distance.

[0076] As is known in the art, gastrointestinal peristalsis may cause the capsule device to occasionally move backward (ie, "retrograde") or oscillate. Thus, the motion vector may occasionally point backward (eg, Fig.12 When the capsule device moves backward, the relevant motion vector contributes a negative value to the cumulative travel distance.

[0077] The result of estimating the travel distance may be presented to a user (e.g., a doctor) to evaluate medical images collected while the capsule device passes through the gastrointestinal tract. For example, the travel distance in the longitudinal direction may be displayed as a horizontal line, where the reference point may be set as the origin. The horizontal distance corresponds to the travel distance from the origin. For example, Fig.13A The line 1310 in corresponds to an example of displaying the cumulative travel distance, wherein the origin 1311 corresponds to a first reference position and the end point corresponds to a second reference position. The first reference position and the second reference position may correspond to specific anatomical portions of the gastrointestinal tract. For example, the first reference position may correspond to the duodenum and the second reference position may correspond to the anus. The image being viewed may be displayed in another display window on the display device. The position of the current image on the longitudinal axis of the gastrointestinal tract may be indicated on the horizontal longitudinal line 1310 by a mark, such as a downward arrow 1320. Specific anatomical portions may be determined from the captured images based on the characteristics of these specific anatomical portions. The horizontal axis may be marked based on the measured travel distance. For example, the total length from the origin corresponding to the duodenum to the end point corresponding to the anus may be 7.5 meters. The horizontal axis may be marked every 10 centimeters. Finer or coarser markings may also be used. In another embodiment, the global motion vector associated with the current image may be displayed as Fig. 13B , wherein the arrow 1330 may point forward or backward (i.e., retrograde motion). Furthermore, the position of the retrograde motion may be marked on the travel distance line 1310. For example, a line segment 1340 corresponding to the distance of the retrograde motion may be superimposed on the travel distance line 1310, such as Fig. 13C Alternatively, retrograde motion can be performed as shown in Fig.13D The backward migration 1352 from the original travel distance line 1350a to another original travel distance line 1350b shown in FIG. Fig.13E , showing different display modes, where Fig.13D Migration 1352 is replaced by a line segment 1354 corresponding to the distance traveled associated with the retrograde motion. Although one or more horizontal lines are used to represent the distance traveled, other graphical representations may also be used. For example, one or more horizontal bars may be used. In another example, a numerical value may be displayed to indicate the distance traveled and an up / down arrow may be used to allow the user to select the desired distance traveled. The user may use an input device such as a computer mouse or button to control the selection.

[0078] The method may also display an image or related information on a display along with the travel distance information. Fig.14 An example of displaying distance traveled information along with image information is shown, wherein distance traveled 1420 is displayed in area 1430 on display 1410 along with a graphical representation of the time code of the image indicated by marker 1424 . Fig.15Another example of displaying travel distance information along with image information is shown, wherein an image indicated by marker 1424 is also included in area 1510 on display 1410 . Fig.16 Another example of displaying distance traveled information along with image information is shown, where an anatomical portion of the gastrointestinal tract is labeled 1610 on a graphical representation of distance traveled. Fig.15 Similar to the situation in FIG. 1 , the user can select a point on the travel distance line (e.g., by using a cursor displayed above the distance line) and the image corresponding to that position will be displayed on the Fig.16 In area 1510 on display 1410 in.

[0079] The above method can be implemented by using various programmable devices such as microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs), digital signal processing (DSPs), or any programmable processors. A display can be used to present visual information related to the travel distance and / or image information. In addition, the above method can be implemented using a mobile device such as a tablet or smart phone, because the mobile device usually has a display and sufficient computing power to handle the required processing. A laptop or computer can also serve as a system supporting the above method. For example, Fig.17 An example of a laptop based implementation is shown, wherein a CPU in the laptop 1710 will perform the required processing and may display travel distance information 1730 and / or corresponding image information 1740 on a display 1720 (ie, the laptop screen).

[0080] Fig.18 An example flow chart for determining a travel distance of a capsule camera according to an embodiment of the present invention is shown. According to the method, an image sequence is received in step 1810, wherein the capsule camera captures the image sequence as the capsule camera moves through the GI (gastrointestinal) tract. A current global motion vector is determined in step 1820 for a current image in the image sequence, wherein the current global motion vector corresponds to a motion made by the capsule camera between the current image and a reference image associated with the image sequence. In step 1830, a travel distance of the capsule camera in the GI tract is determined based on the current global motion vector and a previous global motion vector derived for a previous image between an initial image and the current image, wherein the travel distance is measured from an initial position associated with the initial image to a current position associated with the current image.

[0081] Fig.19An example flow chart for displaying a sequence of images captured by a capsule camera according to an embodiment of the present invention is shown. According to the method, in step 1910, a sequence of images is received, wherein the capsule camera captures the sequence of images as the capsule camera moves through the GI (gastrointestinal) tract. In step 1920, travel distances associated with the sequence of images are presented on a display, wherein each travel distance corresponds to an estimated distance measured from an initial position associated with an initial image in the sequence of images to a current position associated with a current image in the sequence of images. In step 1930, one or more graphical representations representing the travel distances are displayed on the display according to the position on the one or more graphical representations.

[0082] The above description is used to enable those of ordinary skill in the art to implement the present invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those of skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but rather has the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are exemplified to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention can be implemented.

[0083] The present invention may be implemented in other specific forms without departing from its spirit or essential characteristics. The above examples should be regarded as illustrative rather than restrictive in all respects. Therefore, the scope of the present invention is represented by the claims rather than the above description. All changes made within the equivalent meaning and scope of the claims will be included within the scope of the claims.

Claims

1. A method for displaying a sequence of images captured by an endoscope, the method comprising: receiving a sequence of images, wherein the sequence of images of a gastrointestinal (GI) tract stored in an endoscopic camera of the endoscope is retrieved; presenting on a display a distance traveled by the endoscope in the GI tract, wherein the travel distance is determined based on a current normalized global motion vector and a previous normalized global motion vector, wherein the current distance traveled corresponds to an estimated distance measured from an initial position of the endoscope associated with an initial image in the sequence of images to a current position of the endoscope associated with a current image in the sequence of images, wherein the current normalized global motion vector and the previous normalized global motion vector are determined for the current image and the previous image, respectively, and wherein each normalized global motion vector is derived by normalizing a corresponding global motion vector associated with an image relative to a nominal object distance between an endoscope camera and the GI tract; and Depending on the position on the one or more graphical representations, the one or more graphical representations representing the distance traveled by the endoscopic camera are displayed on the display.

2. The method of claim 1, further comprising displaying a marker on at least one of the one or more graphical representations to indicate a corresponding image associated with the target location indicated by the marker.

3. The method of claim 2, further comprising displaying on the display the corresponding image associated with the target location indicated by the marker.

4. The method of claim 1, further comprising marking an anatomical portion of the gastrointestinal tract at a target location on at least one of the one or more graphical representations, wherein A corresponding image at the target location is associated with the anatomical portion of the gastrointestinal tract.

5. A system for displaying a sequence of images captured by a capsule camera, the system comprising: a display capable of displaying graphical information; as well as One or more electronic circuits or processors configured to: receiving a sequence of images, wherein an endoscopic camera of the endoscope captures the sequence of images as the endoscope moves through the gastrointestinal (GI) tract; presenting on the display a distance traveled by the endoscope in the gastrointestinal (GI) tract, wherein the travel distance is determined based on a current normalized global motion vector and a previous normalized global motion vector, wherein the current distance traveled corresponds to an estimated distance measured from an initial position of the endoscope associated with an initial image in the sequence of images to a current position of the endoscope associated with a current image in the sequence of images, wherein the current normalized global motion vector and the previous normalized global motion vector are determined for the current image and the previous image, respectively, and wherein each normalized global motion vector is derived by normalizing a corresponding global motion vector associated with an image relative to a nominal object distance between an endoscope camera and the gastrointestinal (GI) tract; and Depending on the position on the one or more graphical representations, one or more graphical representations are displayed representing the distance traveled by the endoscopic camera.

Images