Method for calculating the essential matrix of a capsule endoscope in a tissue cavity

Abstract

The disclosure describes a method for calculating an essential matrix of a capsule endoscope in a tissue cavity, including: collecting a first image of a reconstruction region in the tissue cavity along a first direction in the tissue cavity having a wrinkled inner wall, and causing the capsule endoscope to collect a second image along a second direction forming an included angle with the first direction in the tissue cavity; calculating an optical flow of a first pixel region of the reconstruction region in the first image and a second pixel region corresponding to the first pixel region of the reconstruction region in the second image by a neural network based on a convolution operation, wherein the neural network based on the convolution operation extracts a first feature point set including a plurality of feature points at the first pixel region, and obtains a second feature point set corresponding to the first feature point set at the second pixel region based on the first feature point set and the optical flow; and calculating the essential matrix of the capsule endoscope based on the first feature point set and the second feature point set. Thus, the essential matrix of the capsule endoscope in the tissue cavity can be obtained.

Description

[0001] This application is a divisional application of the patent application with application number 2020111205814, application date October 19, 2020, and the title of the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow. TECHNICAL FIELD

[0002] The present disclosure relates to a method for calculating the essential matrix of a capsule endoscope in a tissue cavity. BACKGROUND

[0003] At present, in the process of pathological diagnosis of a cavity, pathological information is often obtained by observing the internal structure of the cavity. However, for a closed cavity such as the stomach, it is difficult for people to observe from the outside of the cavity, so information is generally collected by introducing a shooting device, and three-dimensional reconstruction is performed by a feature point method.

[0004] In the existing feature point three-dimensional reconstruction method, first, the feature points of two images are extracted respectively, and then the feature points of the two images are paired one by one. The relative pose between the two images can be calculated through the paired feature points, so as to calculate the position and attitude of the shooting device and further calculate the three-dimensional image of the inner wall of the cavity according to the position and attitude of the shooting device.

[0005] However, the existing feature point three-dimensional reconstruction method does not have satisfactory results in scenes with lack of texture and high similarity. For a cavity such as the stomach with less texture and high similarity between textures, the obtained image is easily affected by the light in the cavity, and it is difficult to obtain enough feature points and accurate matching relationship. Therefore, the three-dimensional reconstruction using the traditional feature point method is not suitable for cavities such as the stomach with less texture and high similarity between textures. SUMMARY

[0006] The present disclosure is proposed in view of the above-mentioned prior art, and aims to provide a processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow, which can perform three-dimensional reconstruction on the image of a tissue cavity (such as a stomach cavity) with less texture and high similarity between textures and improve the feature point recognition accuracy of the image.

[0007] To this end, the disclosure provides a processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow, which comprises the following steps: using a capsule endoscope to collect a first image in a first direction for a reconstruction region in a tissue cavity with a wrinkled inner wall, and making the capsule endoscope collect a second image in a second direction for the reconstruction region in the tissue cavity, the first direction and the second direction forming an included angle; calculating an optical flow of a first pixel region of the reconstruction region in the first image and a second pixel region corresponding to the first pixel region in the second image; extracting a first feature point set comprising a plurality of feature points in the first pixel region, and obtaining a second feature point set corresponding to the first feature point set in the second pixel region based on the first feature point set and the optical flow; calculating an essential matrix of the capsule endoscope based on the first feature point set and the second feature point set; and calculating the spatial coordinates of each pixel point of the first pixel region and the second pixel region based on the essential matrix, obtaining a point cloud and performing three-dimensional image reconstruction using the point cloud.

[0008] In this case, the optical flow between the first image and the second image can be accurately obtained, and the corresponding feature points in the first image and the second image can be extracted according to the optical flow, so that sufficient feature points and accurate matching relationship can be extracted in the image of the inner wall of the tissue cavity with less texture, high similarity between textures and serious influence of light, and thus the three-dimensional reconstruction of the inner wall of the tissue cavity can be completed using the accurately matched feature points.

[0009] In addition, in the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the first image and the second image are obtained by the capsule endoscope continuously, the capsule endoscope is a monocular capsule endoscope, and the optical flow is generated by the movement of the capsule endoscope. In this case, the optical flow can be obtained by the movement of the monocular capsule endoscope.

[0010] In addition, in the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the first image and the second image are two images of adjacent frames, the first image is an image of a current frame, and the second image is an image of a next frame. In this case, the influence of gastric cavity peristalsis on the optical flow can be reduced.

[0011] In addition, in the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the plurality of feature points comprise at least 8 feature points. In this way, the accuracy of the essential matrix can be improved.

[0012] In addition, in the method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the optical flow can be calculated by a neural network based on deep learning. In this case, the motion information of each pixel in the tissue cavity inner wall image can be accurately obtained by the neural network based on deep learning.

[0013] In addition, in the method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the training of the neural network based on deep learning can include: preparing a plurality of sets of tissue cavity inner wall image pairs, the tissue cavity inner wall image pairs being two images of the same region in a tissue cavity with a wrinkled inner wall collected in different directions, and being a first training image and a second training image, respectively; inputting the first training image and the second training image of each set of tissue cavity inner wall image pairs into the neural network in sequence to obtain a forward optical flow of a first pixel in the first training image; exchanging the input order of the first training image and the second training image to obtain a backward optical flow of a second pixel corresponding to the first pixel in the second training image; calculating a loss function based on the forward optical flow and the backward optical flow; and evaluating and optimizing the neural network based on the loss function. In this case, the neural network can be trained in an unsupervised manner, and the training can be performed using unlabeled tissue cavity inner wall images.

[0014] In addition, in the method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the loss function can include a structural similarity loss term, a geometric consistency loss term, and an optical flow smoothness loss term. In this case, the neural network can be trained by the loss function.

[0015] In addition, in the method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the structural similarity loss term can include a first structural similarity loss term of a first predicted image and the first training image, and a second structural similarity loss term of a second predicted image and the second training image, the first predicted image being generated by the first training image being acted on by the forward optical flow, and the second predicted image being generated by the first training image being acted on by the backward optical flow. In this case, the accuracy of the calculated forward optical flow and backward optical flow can be evaluated, so that the accuracy of the optical flow calculated by the neural network can be improved by training the neural network.

[0016] In addition, in the method for obtaining a three-dimensional image of a tissue cavity based on optical flow, the optical flow smoothness loss term can include a total variation loss of the forward optical flow and a total variation loss of the backward optical flow. In this case, the smoothness of the optical flow can be improved, so that the smoothness of the optical flow image can be maintained and the influence of noise on the optical flow image can be reduced.

[0017] In addition, in the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow according to an example of the present disclosure, optionally, the geometric consistency loss term is a difference between the forward optical flow of a first pixel point in the first training image and the backward optical flow of a second pixel point corresponding to the first pixel point in the second training image. In this case, the forward optical flow and the backward optical flow calculated by the trained neural network can maintain geometric consistency.

[0018] According to the present disclosure, a processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow can be provided, which can perform three-dimensional reconstruction on images of a tissue cavity (for example, a stomach cavity) with less texture and high similarity between textures, and improve the feature point recognition accuracy of the images. BRIEF DESCRIPTION OF DRAWINGS

[0019] Embodiments of the present disclosure will now be explained in further detail by way of example with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of an implementation process of a processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow according to an example of the present disclosure is shown.

[0021] Figure 2 A schematic diagram of a scene for obtaining a first image and a second image according to an example of the present disclosure is shown.

[0022] Figure 3 A schematic diagram of a neural network based on deep learning according to an example of the present disclosure is shown.

[0023] Figure 4 A schematic diagram of a first preprocessing module in a neural network based on deep learning according to an example of the present disclosure is shown.

[0024] Figure 5 A schematic diagram of a first neural network in a neural network based on deep learning according to an example of the present disclosure is shown.

[0025] Figure 6 A schematic diagram of a variant of a neural network based on deep learning according to an example of the present disclosure is shown.

[0026] Figure 7 A schematic diagram of a training step of a neural network based on deep learning according to an example of the present disclosure is shown.

[0027] Figure 8 A schematic diagram of feature point matching by a neural network based on deep learning according to an example of the present disclosure is shown.

[0028] REFERENCE SIGNS:

[0029] 10 … neural network based on deep learning; 11 … first preprocessing module; 12 … first neural network; 13 … feature combination module; 14 … third neural network; 21 … second preprocessing module; 22 … second neural network; 111 … adjustment unit; 112 … normalization unit; 20 … capsule endoscope; 31 … first image; 32 … second image; 311 … first feature point; 321 … second feature point. DETAILED DESCRIPTION

[0030] All references cited herein are incorporated by reference in their entirety as if fully set forth. Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. One skilled in the art will appreciate that many methods and materials similar or equivalent to those described herein can be used in the practice of the present disclosure. Indeed, the present disclosure is in no way limited to the methods and materials described.

[0031] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same components are designated by the same reference numerals, and repetitive description will be omitted. In addition, the drawings are merely schematic and the ratio of the sizes of the components to each other or the shape of the components, etc. can be different from the actual ones.

[0032] It should be noted that the terms "comprising" and "having" and any variations thereof, such as a process, method, system, product, or apparatus including or having a series of steps or units, are not necessarily limited to those steps or units clearly listed, but can include or have other steps or units that are not clearly listed or inherent to such processes, methods, products, or apparatuses.

[0033] In addition, the subheadings and the like involved in the following description of the present disclosure are not intended to limit the content or scope of the present disclosure, but merely serve as a reading aid. Such subheadings cannot be understood as dividing the content of the article, and the content under the subheadings should not be limited only within the scope of the subheadings.

[0034] The present disclosure relates to a processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow. The processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow can be referred to as a processing method. In the processing method related to the present disclosure, the motion information of each pixel in the three-dimensional image of the tissue cavity can be calculated by the optical flow method. The tissue cavity can be, but is not limited to, a cavity such as a stomach, an abdominal cavity, a thoracic cavity, a large intestine cavity, etc. As an example, the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow related to the present disclosure can be a processing of images of the inner wall of a stomach cavity taken by a capsule endoscope 20 when the capsule endoscope 20 moves in the stomach. In some examples, when the capsule endoscope 20 moves in the stomach, the capsule endoscope 20 with a photographing device can take images of the stomach wall to obtain a three-dimensional image of the stomach, and then construct a complete or partial stomach model based on the three-dimensional image of the stomach.

[0035] Figure 1 A schematic diagram of the implementation process of the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow related to an example of the present disclosure is shown. Figure 2 A schematic diagram of a scene for obtaining a first image 31 and a second image 32 related to an example of the present disclosure is shown.

[0036] In some examples, as shown in Figure 1 The processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow related to the present disclosure can include the following steps: obtaining a first image 31 and a second image 32 of a tissue cavity with a wrinkled inner wall for a reconstruction area (described later) by a capsule endoscope (step S100); calculating the optical flow of a first pixel area of the reconstruction area in the first image 31 and a second pixel area corresponding to the first pixel area of the reconstruction area in the second image 32 (step S200); extracting a first feature point set including a plurality of feature points in the first pixel area, and obtaining a second feature point set corresponding to the first feature point set in the second pixel area based on the first feature point set and the optical flow (step S300); calculating an essential matrix of the capsule endoscope 20 based on the first feature point set and the second feature point set (step S400), and calculating the spatial coordinates of each pixel point of the first pixel area and the second pixel area based on the essential matrix, obtaining a point cloud and performing three-dimensional image reconstruction using the point cloud (step S500).

[0037] In this case, the optical flow between the first image 31 and the second image 32 can be accurately obtained, and the corresponding feature points in the first image 31 and the second image 32 can be extracted according to the optical flow, so that sufficient feature points and accurate matching relationship can be extracted in the inner wall image of the tissue cavity with less texture, high similarity between textures and serious light influence, and thus the three-dimensional reconstruction of the inner wall of the tissue cavity can be completed using the accurately matched feature points.

[0038] In some examples, as shown in Figure 1 and Figure 2As shown, in step S100 of the processing method for obtaining three-dimensional images of tissue cavities based on optical flow, a capsule endoscope can be used to acquire a first image 31 and a second image 32 for the reconstructed area (described later) in a tissue cavity with a folded inner wall. Specifically, the capsule endoscope 20 can be used to acquire the first image 31 for the reconstructed area in the tissue cavity along a first direction in the tissue cavity with a folded inner wall, and the capsule endoscope 20 can be used to acquire the second image 32 for the reconstructed area in the tissue cavity along a second direction in the tissue cavity, and the first direction and the second direction form an angle.

[0039] In some examples, such as Figure 2 As shown, a first image 31 can be acquired along a first direction L1 within a tissue cavity with a folded inner wall using a capsule endoscope 20, targeting the reconstructed region within the tissue cavity. A second image 32 is then acquired along a second direction L2 within the tissue cavity, targeting the reconstructed region, using the capsule endoscope 20. The first and second directions form an angle. In this configuration, first images 31 and second images 32 taken at different angles targeting the reconstructed region can be acquired, thereby allowing the acquisition of the optical flow between the first and second images 31 and 32.

[0040] In some examples, the reconstruction area can be a portion or the entire folded inner wall of a tissue cavity that requires 3D reconstruction to obtain a 3D image.

[0041] In some examples, Figure 2 The folded inner walls of the tissue cavities shown can be, but are not limited to, the folded inner walls of cavities such as the stomach, abdominal cavity, thoracic cavity, and large intestine.

[0042] In some examples, the capsule endoscope 20 in step S100 may have an acquisition device. This acquisition device can acquire pathological information about the gastric cavity. In some examples, the pathological information may include images of the gastric cavity wall. In this case, the capsule endoscope 20 is able to acquire images of the gastric cavity wall, thereby performing three-dimensional reconstruction of the gastric cavity wall.

[0043] In some examples, the first image 31 and the second image 32 can be captured by the same capsule endoscope 20. In this case, the first image 31 is captured in a sequential order and the second image 32 is captured in a sequential order.

[0044] In other examples, the first image 31 and the second image 32 can be captured by different capsule endoscopes 20. In this case, the first image 31 and the second image 32 can be captured simultaneously or sequentially. For example, the first image 31 and the second image 32 can be two images captured simultaneously by two capsule endoscopes 20. In this case, the influence of gastric peristalsis on the optical flow can be reduced.

[0045] In some examples, the capsule endoscope 20 can continuously take pictures inside the stomach cavity. In this case, multiple images of the inner wall of the stomach cavity can be obtained.

[0046] In some examples, the capsule endoscope 20 can be a monocular capsule endoscope, a binocular capsule endoscope, or a multiocular capsule endoscope. In this case, the corresponding capsule endoscope 20 can be selected for taking pictures based on different requirements and scenarios.

[0047] In some examples, the capturing device of the capsule endoscope 20 can be a monocular capturing device. In this case, the image of the inner wall of the stomach cavity can be obtained by a single camera.

[0048] In some examples, the capturing device of the capsule endoscope 20 can be a multiocular capturing device (e.g., a binocular capturing device, a trinocular capturing device, a quadriocular capturing device, etc.). In this case, multiple images of the inner wall of the stomach cavity can be obtained by multiple cameras at the same time, so that multiple images of the inner wall of the stomach cavity at the same position can be obtained at the same light intensity.

[0049] In some examples, the first image 31 and the second image 32 can be images taken by two lenses of a binocular capturing device, respectively. In this case, the first image 31 and the second image 32 can have the same taking time.

[0050] In some examples, the first image 31 can be taken before the second image 32. However, examples of the present disclosure are not limited thereto, and the first image 31 can be taken after the second image 32, or the first image 31 can be taken at the same time as the second image 32.

[0051] In some examples, the first image 31 and the second image 32 can be two images of adjacent frames. The first image 31 can be a current frame image. The second image 32 can be a next frame image adjacent to the current frame. In this case, the taking area of the first image 31 and the taking area of the second image 32 can have more overlapping parts, and the influence of stomach cavity peristalsis on the optical flow can be reduced.

[0052] In other examples, the first image 31 and the second image 32 can not be two images of adjacent frames. Specifically, the first image 31 and the second image 32 can differ by multiple frames. For example, the first image 31 and the second image 32 can differ by 2 frames, 3 frames, 4 frames, etc. In this case, the number of frames between the first image 31 and the second image 32 can be set according to actual requirements.

[0053] In some examples, the capsule endoscope 20 can take images along a predetermined path under the influence of a magnetic field. In this case, the capsule endoscope 20 can scan and image the lining of the stomach along the predetermined path. In some examples, the capsule endoscope 20 can take multiple images so that the imaged area covers the entire stomach cavity, in which case the complete lining of the stomach cavity can be reconstructed. In other examples, the imaged area of ​​the capsule endoscope 20 can be limited to a target area, in which case the target area within the stomach cavity can be reconstructed.

[0054] like Figure 1 As shown, in step S200, the optical flow of the first pixel region of the reconstructed region in the first image 31 and the second pixel region of the reconstructed region in the second image 32 corresponding to the first pixel region can be calculated. For ease of description, the first image 31 can be set to be captured before the second image 32.

[0055] In some examples, the reconstruction area can be the inner wall of a tissue cavity that needs to be reconstructed in three dimensions.

[0056] In some examples, the method of obtaining motion information of an object by calculating the optical flow of the first pixel region of the reconstructed region in the first image 31 and the second pixel region of the reconstructed region in the second image 32 corresponding to the first pixel region can be called the optical flow method.

[0057] In this disclosure, optical flow refers to the method of finding the correspondence between the first image 31 and the second image 32 by utilizing the temporal changes and correlations of pixels in the first image 31 and the second image 32, thereby calculating the motion information of the object between the first image 31 and the second image 32. In this case, the motion information of the object can be described by optical flow.

[0058] In some examples, optical flow can be based on two fundamental assumptions: constant light intensity, meaning that the light intensity of the same target does not change drastically when it moves; and continuous time or "small motion," meaning that changes in time do not cause drastic changes in the target's position. In this case, optical flow can be used to calculate the optical flow of the object between the first image 31 and the second image 32, and the motion information of the object can be obtained based on the optical flow.

[0059] In the embodiments disclosed herein, the optical flow method can be used to find the correspondence between the first image 31 and the second image 32 by utilizing the changes and correlations of pixels in the first image 31 and the second image 32 in the time domain, and calculate the motion information of the capsule endoscope 20.

[0060] In some examples, the optical flow can be caused by the movement of the capsule endoscope 20. In this case, the optical flow can be associated with the movement of the capsule endoscope 20, i.e. the optical flow can be obtained by using the movement information of the capsule endoscope 20. In some examples, the optical flow can refer to the instantaneous velocity of the spatial object in the pixel movement of the observation imaging plane. In some examples, the instantaneous change rate of the light intensity value at the image plane specific coordinate point can be defined as the optical flow. In this case, the light intensity value at the image plane specific coordinate point can be combined with the optical flow to obtain the movement information of the capsule endoscope 20.

[0061] In some examples, the optical flow of a plurality of pixel points can constitute an optical flow field. In this case, the optical flow of each pixel point in the image can be described by the optical flow field.

[0062] In some examples, the optical flow field can be represented by an optical flow image. Specifically, the optical flow is a vector, and the color of each pixel point in the optical flow image can correspond to an optical flow, and the color in the HSV color space can one-to-one correspond to the optical flow. In this case, the optical flow can be represented by color, and the optical flow field can be intuitively represented by the optical flow image.

[0063] In some examples, the light intensity of a pixel point in the first image 31 or the second image 32 can be assumed as I(x, y, t). Wherein x, y respectively represent the position of the pixel point in the width direction and the length direction of the image, and t represents the time dimension thereof. For a preset point on the inner wall of the tissue cavity, the moving distance of the pixel point position of the preset point in the second image 32 and the pixel point position of the preset point in the first image 31 can be represented by (dx, dy), and the moving time can be represented by dt. According to the basic assumption of the optical flow method, the light intensity of the pixel point before and after the movement can be expressed by formula (1):

[0064] I(x, y, t) = I(x + dx, y + dy, t + dt) (1)

[0065] The right end of formula (1) is Taylor expanded as:

[0066]

[0067] ε represents a second-order infinitesimal, which can be ignored, and formula (2) is brought into formula (1) to simultaneously derive the time on both sides as:

[0068]

[0069] Let u, v be the components of the optical flow along the width direction and the length direction of the image respectively, then:

[0070]

[0071] In addition, The partial derivatives of the light intensity of the pixel points in the image with respect to x, y, and t can be represented respectively, and the formula (3) can be converted into formula (5):

[0072] I x u+I y v+I t = 0 (5)

[0073] where I x , I y , and I t can be obtained from the first image 31 or the second image 32. According to the above-mentioned calculation method of the optical flow, there is only one constraint equation, while the optical flow has two components. In some examples, another constraint condition can be introduced to calculate the optical flow, and in this case, the optical flow can be solved by using the additional constraint condition.

[0074] In some examples, according to the introduced constraint condition, the optical flow method can be classified as a gradient-based method, a matching-based method, an energy-based method, a phase-based method, or a neural dynamics method. In this case, the optical flow can be calculated by various calculation methods, so that a more accurate optical flow calculation method can be selected. In other examples, the above-mentioned various optical flow methods can be used in combination, and in this case, a suitable method can be selected for use in combination according to the actual application scenario.

[0075] In some examples, the gradient-based optical flow method can include a Lucas-Kanade algorithm or a Horn-Schunck algorithm. In this case, the optical flow can be solved according to actual requirements.

[0076] In some examples, the Lucas-Kanade algorithm (LK algorithm) can be applied to calculate the optical flow. Specifically, the LK optical flow method adds a "spatial consistency" assumption to the two basic assumptions of the original optical flow method, that is, all adjacent pixel points in the target region have similar motion. That is, each pixel point in the m x m region around the target pixel has the same optical flow. Preferably, m can be set to 2, and in this case, the optical flow can be solved by a simple constraint condition.

[0077] In the m x m region, the LK optical flow method can estimate the optical flow by minimizing the weighted sum of squares of the following formula:

[0078]

[0079] In formula (6), W 2(x) is a window weight function, which makes the weight of the center of the region larger than the surrounding. I(x, y, t) represents the light intensity function of the pixel point, I a represents the partial derivative of I with respect to a (a can be x, y, t), x and y are the positions of the corresponding pixel points, and t is time. Ω represents the range of the region, and for i pixel points X1, X2…X i , set

[0080]

[0081]

[0082]

[0083]

[0084] Therefore, the above solution can be obtained by the least square method:

[0085] A T W 2 AV=A T W 2 b(11),

[0086] Finally:

[0087] V=(A T W 2 A) -1 A T W 2 b(12),

[0088] Thus, the optical flow component u and the optical flow component v along the length and width directions of the image can be solved.

[0089] In some examples, the backward optical flow components u i and v i of the i-th pixel point in the first image 31 to the second image 32 (i=1, 2, …, n) can be solved by the above method.

[0090] In some examples, the Horn-Schunck algorithm (HS algorithm) can be used to calculate the optical flow. Specifically, a global smoothing assumption can be added on the basis of the basic constraint equation of the optical flow, assuming that the change of the optical flow is smooth on the entire image, that is, the image motion vector is smooth or only changes slowly. In this case, the calculated optical flow can be more smooth in the image, and more consistent with the actual optical flow distribution, thereby improving the matching accuracy.

[0091] In some examples, the constraint condition added by the Horn-Schunck algorithm can be expressed as an energy function ζ 2The minimum is taken, where the energy function can be expressed as equation (13):

[0092] ζ 2 =∫∫Ldxdy (13),

[0093] wherein,

[0094]

[0095] wherein, u, v are the optical flow components along the length and width direction of the image respectively, x, y are the positions of the corresponding pixel points, L is the optical flow smoothing term, and a is the smoothing weight coefficient, which represents the weight of the optical flow smoothing term.

[0096] The Euler-Lagrange equation is used and simplified to obtain the iteration formula:

[0097]

[0098]

[0099] wherein, k represents the kth iteration, u k , v k represents the optical flow components after the kth iteration, u k+1 , v k+1 represents the optical flow components after the k+1th iteration, respectively represent the mean values of the optical flow components u and v after the kth iteration. In some examples, the mean values of u and v can be calculated by the nine-point difference format.

[0100] In some examples, the optical flow can be iterated for multiple times until the number of iterations exceeds the allowed number of iterations or the difference of the optical flow between two iterations is less than a given error threshold. In this case, the converged optical flow can be obtained through iteration.

[0101] In other examples, the Horn-Schunck algorithm can be improved. Specifically, a conditional smoothing constraint can be introduced, i.e., different smoothing processing of the gradient is performed through a weighting matrix, or a segmented smoothing method can be introduced. In this case, the improved way can be selected according to the actual situation.

[0102] In other examples, the optical flow can be calculated using a dense optical flow method or a sparse optical flow method. In some examples, the optical flow can be calculated based on the dense optical flow method. Specifically, the dense optical flow method can perform point-by-point matching for a certain region in the first image 31 or the second image 32 and calculate the optical flow of all points on the first image 31 or the second image 32. In this case, a dense optical flow can be calculated, and thus pixel-level feature point matching can be performed through the dense optical flow. In some examples, the optical flow can be calculated based on the sparse optical flow method. Specifically, the dense optical flow method sets the optical flow around a certain point to be the same, and does not need to calculate the optical flow of all points on the first image 31 or the second image 32. In this case, a sparse optical flow can be calculated, and thus the amount of calculation can be reduced and the calculation speed can be improved.

[0103] In some examples, in step S200, the optical flow can be calculated by the neural network 10 based on deep learning. Specifically, the neural network can be trained by inputting pairs of training images (including a first training image and a second training image, described later) to the neural network, and the optical flow between the first image 31 and the second image 32 can be obtained using the trained neural network. In this case, the motion information of each pixel point in the first image 31 or the second image 32 can be accurately obtained by the neural network 10 based on deep learning.

[0104] In some examples, the neural network 10 based on deep learning can also be referred to as a “neural network” in the present disclosure.

[0105] In some examples, the pairs of training images are two images of adjacent frames. In some examples, the pairs of training images are a current frame image and a next frame image. In other examples, the pairs of training images are image pairs that differ by a fixed number of frames, where the number of frames can be 1 frame, 2 frames, 3 frames, 4 frames, or 5 frames, etc. In this case, the number of frames between the input pairs of training images can be adjusted according to actual conditions.

[0106] In some examples, the neural network 10 based on deep learning can be a neural network system based on convolution operation (which can also be referred to as a “convolutional neural network” in the present disclosure), a self-encoding neural network based on multiple layers of neurons, or a neural network pre-trained in the manner of a self-encoding neural network with multiple layers. In this case, the neural network 10 based on deep learning can be constructed according to actual conditions by selecting an appropriate method.

[0107] Hereinafter, the neural network 10 based on deep learning will be described by taking a neural network system based on convolution operation as an example to introduce how to calculate the optical flow, but the method of calculating the optical flow is not limited thereto, and each part of the convolutional neural network described below can also be applied to other neural networks individually or in combination to calculate the optical flow.

[0108] In some examples, when an arbitrary image is taken as input, it can be converted into an RGB color matrix. In this case, it is possible to convert any image into a 3-channel matrix that a computer can recognize.

[0109] Figure 3 A schematic diagram of a deep learning-based neural network involved in the examples of this disclosure is shown.

[0110] In some examples, such as Figure 3 As shown, a neural network system based on convolution operations may include a first neural network 12, a second neural network 22, a feature combination module 13, and a third neural network 14.

[0111] In some examples, the neural network system based on convolution operations may also include a preprocessing module. The preprocessing module may be positioned before the first neural network 12 and the second neural network 22. In this case, the input image of the inner wall of the tissue cavity can be preprocessed before entering the first neural network 12 or the second neural network 22.

[0112] In some examples, the preprocessing module may include a first preprocessing module 11 and a second preprocessing module 21. Specifically, as shown below... Figure 3 As shown, the first preprocessing module 11 and the second preprocessing module 21 can be respectively positioned before the first neural network 12 and the second neural network 22. The output of the first preprocessing module 11 can be connected to the input of the first neural network 12, and the output of the second preprocessing module 21 can be connected to the input of the second neural network 22. However, this embodiment is not limited to this. In other examples, the output of the first preprocessing module 11 can be connected to the input of the second neural network 22, and the output of the second preprocessing module 21 can be connected to the input of the first neural network 12.

[0113] In some examples, the first preprocessing module 11 can receive and process an image from a first input. In some examples, the second preprocessing module 21 can receive and process an image from a second input. In some examples, the first input image can be a first image 31 (or a first training image), and the second input image can be a second image 32 (or a second training image). In some examples, the first input image can be a second image 32 (or a second training image), and the second input image can be a first image 31 (or a first training image).

[0114] In some examples, the preprocessing modules (the first preprocessing module 11 and the second preprocessing module 21) can be used to respectively preprocess the first image 31 and the second image 32 obtained by capturing the same region in the tissue cavity with a wrinkled inner wall in different directions. That is, the first preprocessing module 11 can preprocess the first image 31, and the second preprocessing module 21 can preprocess the second image 32. In some examples, the first preprocessing module 11 and the second preprocessing module 21 can be formed in the same module or independently formed as modules. Hereinafter, the structure and processing process of the convolutional neural network model are described by taking the first preprocessing module 11 preprocessing the first image 31 and the second preprocessing module 21 preprocessing the second image 32 as examples, but the present application is not limited thereto.

[0115] Figure 4 A schematic diagram of the first preprocessing module in the deep learning-based neural network involved in the example of the present disclosure is shown.

[0116] In some examples, the preprocessing module can include an input end. In this case, the neural network can receive the tissue cavity inner wall image for training or calculating the optical flow.

[0117] As described above, the preprocessing modules (the first preprocessing module 11 and the second preprocessing module 21) can be used to respectively preprocess the first image 31 and the second image 32 obtained by capturing the same region in the tissue cavity with a wrinkled inner wall in different directions. Specifically, the first image 31 and the second image 32 can be input in sequence and can enter the first preprocessing module 11 and the second preprocessing module 21, respectively. The first preprocessing module 11 and the second preprocessing module 21 can perform preprocessing such as size adjustment and normalization on the input images. That is, the first preprocessing module 11 can perform preprocessing such as size adjustment and normalization on the first image 31; the second preprocessing module 21 can perform preprocessing such as size adjustment and normalization on the second image 32. In some examples, the input order of the first image 31 and the second image 32 can be exchanged, in which case the first preprocessing module 11 can perform preprocessing such as size adjustment and normalization on the second image 32; the second preprocessing module 21 can perform preprocessing such as size adjustment and normalization on the first image 31.

[0118] In some examples, the first preprocessing module 11 and the second preprocessing module 21 can be set as the same module, and therefore, the first preprocessing module 11 is described in detail below, and the structure of the second preprocessing module 21 can be exactly the same as that of the first preprocessing module 11.

[0119] As Figure 4As shown, the first preprocessing module 11 of the neural network mainly includes a region adjustment unit 111 and a normalization unit 112.

[0120] In some examples, the adjustment unit 111 can be configured to resize the tissue cavity inner wall images (e.g., the first image 31, the second image 32, the first training image, and the second training image). Due to the difference in size of the tissue cavity or the difference in the used lens, the obtained tissue cavity inner wall images can have differences in resolution, inner wall region size, etc. In this case, the obtained tissue cavity inner wall images can be adjusted by the adjustment unit 111.

[0121] In some examples, by the adjustment unit 111, the size of the tissue cavity inner wall images can be adjusted to a specified size (e.g., pixel size) such as 256x256, 512x512, 1024x1024, etc. However, the present embodiment is not limited thereto, and according to specific needs, the size of the tissue cavity inner wall images can also be any other specification size (pixel size) such as 128x128, 768x768, 2048x2048, etc. In this case, the size of the images can be adjusted to a suitable size according to actual needs.

[0122] In some examples, the normalization unit 112 in the first preprocessing module 11 can be configured to normalize the tissue cavity inner wall images (e.g., the first image 31 or the first training image). In this case, the tissue cavity inner wall images can be normalized, so that the differences caused by the differences in different types of tissue cavity inner walls, different image acquisition devices, or different acquisition conditions can be avoided.

[0123] In some examples, the normalization manner of the normalization unit 112 is not particularly limited, and for example, zero mean, unit standard deviation, etc. can be used. In addition, in some examples, normalization in the range of [0, 1] can also be used. In this case, the differences in different types of tissue cavity inner wall images can be overcome, so that the performance of the artificial neural network can be improved.

[0124] In some examples, the first preprocessing module 11 can also have an augmentation unit (not shown). The augmentation unit can be arranged before the adjustment unit 111, but the present embodiment is not limited thereto.

[0125] In some examples, the augmentation unit can be used to perform data augmentation on the tissue cavity inner wall images in the training phase of the neural network. Through the augmentation unit, the obtained tissue cavity inner wall images can be data augmented to expand the sample size of the tissue cavity inner wall images, thereby helping to overcome the overfitting problem and improve the performance of the artificial neural network. In addition, it should be noted that the augmentation unit is generally limited to augmenting data samples in the training phase of the neural network described later, and the augmentation unit can not be used in the test phase of the neural network.

[0126] In addition, the augmentation manner adopted by the augmentation unit is not particularly limited, and in some examples, sample augmentation can be performed by performing various image transformations on the tissue cavity inner wall images. In some examples, the image transformation manner can include symmetry transformation, inversion transformation, rotation transformation, pixel translation, etc., and can also include adjusting the contrast, brightness, color, sharpness, etc. of the image.

[0127] The above has described the constitution and function of the first preprocessing module 11, and similarly, the second preprocessing module 21 can also have the same constitution and function as the first preprocessing module 11. In this case, the tissue cavity inner wall images (such as the second image 32 or the second training image) can pass through the second preprocessing module 21 as a second input to obtain effective preprocessing to meet the subsequent processing of the tissue cavity inner wall images by the artificial neural network (the second neural network 22 and the third neural network 14). In this case, by the first preprocessing module 11 and the second preprocessing module 21, the tissue cavity inner wall images can be effectively preprocessed respectively, thereby helping the subsequent each neural network to further process in the tissue cavity inner wall images.

[0128] In some examples, the first image 31 or the second image 32 can pass through the preprocessing module to obtain a target first image 31 or a target second image 32. The target first image 31 or the target second image 32 can be used as the input of the first neural network 12 and the second neural network 22.

[0129] In the present embodiment, the first neural network 12 can be used to generate a first feature set from an image (e.g., the target first image 31) that has been preprocessed by the first preprocessing module 11. Similarly, the second neural network 22 can be used to generate a second feature set from an image (e.g., the target second image 32) that has been preprocessed by the second preprocessing module 21. The first neural network 12 and the second neural network 22 can achieve an abstract description of the target first image 31 and the target second image 32 by, for example, combining multiple layers of low-level features (pixel-level features).

[0130] Figure 5 A schematic diagram of a first neural network in a deep learning-based neural network involved in an example of the present disclosure is shown.

[0131] In some examples, the first neural network 12 and the second neural network 22 can have the same network structure. The first neural network 12 is described in detail below, and the structure of the second neural network 22 can be identical to that of the first preprocessing module 11.

[0132] In some examples, the first neural network 12 can perform convolutional processing on an input image, in which case the features of the image can be extracted to obtain a feature image of the input image. In some examples, the convolutional processing can be a convolution operation on the target image using a convolution kernel.

[0133] In some examples, two input images (e.g., the target first image 31 and the target second image 32) can be input to the convolutional neural network as inputs, in which case the convolutional neural network can be used to extract features and form a feature set (e.g., motion information), so that an optical flow image can be obtained from the feature set.

[0134] In some examples, the first neural network 12 can include a pooling layer (downsampling layer) and a convolutional layer. In this case, the size of the feature image can be reduced in space by pooling.

[0135] In some examples, the first neural network 12 may not include pooling layers, in which case it is able to preserve information from the input images (e.g., the target first image 31 and the target second image 32).

[0136] In some examples, pooling methods can include max-pooling, mean-pooling, and stochastic-pooling. Pooling operations can reduce feature dimensionality and improve computational efficiency. Furthermore, they can enable neural networks to extract more abstract, high-level features, thereby improving the accuracy of interpreting images of the inner walls of tissue cavities.

[0137] In some examples, the number of convolutional and pooling layers can be increased as needed. In this case, the neural network can extract more abstract high-level features, thereby further improving the accuracy of judging images of the inner wall of tissue cavities.

[0138] In some examples, such as Figure 5 As shown, convolutional layers and pooling layers can be combined alternately. In some examples, the processing flow can sequentially pass through a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer. In this case, convolutional and pooling processes can be performed alternately on the input image (e.g., target first image 31 and target second image 32).

[0139] In some examples, the first neural network 12 and the second neural network 22 can be completely identical. Specifically, the network structure of the first neural network 12 can be exactly the same as that of the second neural network 22. In this case, the number of parameters in the artificial neural network can be reduced, which helps to suppress overfitting of the neural network.

[0140] Furthermore, the neural network structures used in the first neural network 12 and the second neural network 22 are not limited to this; other convolutional neural network structures can also be used, as long as they can ensure the extraction of high-level features from the images of the inner wall of the tissue cavity (e.g., the first image 31, the second image 32, the first training image, or the second training image). In addition, it should be noted that the first neural network 12 and the second neural network 22 involved in this embodiment are mainly used for feature extraction and are not directly output optical flow.

[0141] In this embodiment, such as Figure 3As shown, the feature combination module 13 can be configured to fuse the first feature set generated by the first neural network 12 and the second feature set generated by the second neural network 22 to form a feature combination set. Here, the "feature set" of the present embodiment can refer to "feature sequence", "feature vector", "set of feature values", "feature image", etc., and the meaning thereof should be understood in the broadest way.

[0142] In some examples, the feature combination module 13 can combine the first feature set and the second feature set into a one-dimensional feature vector (feature combination set). In other examples, the feature combination module 13 can also calculate the difference between the first feature set and the second feature set to obtain the feature combination set. In other examples, the feature combination module 13 can also calculate the mean of the first feature set and the second feature set to obtain the feature combination set. In other examples, the feature combination module 13 can perform linear or nonlinear transformation on the first feature set and the second feature set to obtain the feature combination set, etc. In addition, in other examples, the feature combination module 13 can calculate the convolution between the first feature set and the second feature set.

[0143] In some examples, the feature combination module 13 can include convolution layers and pooling layers, and in particular, after the first feature set generated by the first neural network 12 and the second feature set generated by the second neural network 22 are fused to form the feature combination set, the feature combination set can be subjected to one or more convolution processing and pooling processing again. The third convolution layer can be entered and convolution processing can be performed again. In this case, higher-level features of the feature image can be extracted.

[0144] In some examples, through the feature combination module 13, the features generated from the first neural network 12 and the features generated from the second neural network 22 can be fused to facilitate the processing of the subsequent third neural network 14.

[0145] In the present embodiment, the third neural network 14 can be configured to obtain the optical flow according to the result of feature fusion (feature combination set). As shown, the third neural network 14 can be configured to calculate the optical flow according to the input image pair of the inner wall of the tissue cavity based on the result obtained by the feature combination module 13. That is, the third neural network 14 obtains the optical flow according to the feature combination set. Figure 3

[0146] In some examples, the output of the third neural network 14 can be the optical flow of each pixel point.

[0147] ​Further, the network structure of the third neural network 14 is not particularly limited. In some examples, the third neural network 14 can be implemented using various combinations of deconvolution layers, de-pooling layers (up-sampling layers), fully connected layers, and other auxiliary layers (e.g., batch normalization, pooling, convolution, etc.). For example, in some cases, the output layer of the third neural network 14 can use four deconvolution layers, four de-pooling layers, two fully connected layers, and an output layer (softmax layer). Further, in other cases, the output layer of the third neural network 14 can also use two deconvolution layers, two de-pooling layers, three fully connected layers, and an output layer (e.g., softmax layer). In such cases, an optical flow image of the same or similar size as the tissue lumen inner wall image (e.g., the first image 31, the second image 32) can be obtained.

[0148] In some examples, the deep learning-based neural network 10 can include one or more de-pooling layers. In such cases, the size of the optical flow image can be enlarged in space by de-pooling.

[0149] In some examples, the features can be displayed in the form of an image, which can be referred to as a feature image. In such cases, the features can be displayed in an intuitive image. In some examples, the feature image can be the optical flow image described above. In such cases, the features of the image can be displayed in an intuitive optical flow image.

[0150] As described above, in the present embodiment, since the first image 31 and the second image 32 (or the first training image and the second training image) are independently used as input information, respectively, the first neural network 12 can be facilitated to extract features from the first image 31 (or the first training image), and the second neural network 22 can be facilitated to extract features from the second image 32 (or the second training image). Further, by combining the features obtained from the first neural network 12 and the second neural network 22, respectively, and then continuing to obtain the optical flow by the third neural network 14, the accuracy of the optical flow can be significantly improved.

[0151] Figure 6 A schematic view of a variation of the deep learning-based neural network according to the example of the present disclosure is shown.

[0152] In other examples, as Figure 6As shown, the preprocessing module of the neural network system based on convolution operation can include the first preprocessing module 11a, the first neural network 12a, and the third neural network 14a. In this case, the input image of the preprocessing module 11a can be a superimposed image of two tissue cavity inner wall images. Specifically, after the two tissue cavity inner wall images are input into the neural network, they can be respectively converted into a 3-channel matrix, the two 3-channel matrices are superimposed to form a 6-channel matrix, and sequentially pass through the first preprocessing module 11a, the first neural network 12a, and the third neural network 14a. The functions of the first preprocessing module 11a, the first neural network 12a, and the third neural network 14a can refer to the related descriptions of the first preprocessing module 11, the first neural network 12, and the third neural network 14 described above. In this case, the two images can be preprocessed at the same time, and the complexity of the neural network can be reduced.

[0153] In some examples, as shown in FIG. 1, the neural network 10 based on deep learning can include a preprocessing module 11, a first neural network 12, a second neural network 13, and a third neural network 14. Figure 7 As shown, the training step of the neural network 10 based on deep learning can include: preparing a plurality of groups of tissue cavity inner wall image pairs, the tissue cavity inner wall image pair can be two images obtained by collecting the same region in the tissue cavity with a wrinkled inner wall in different directions, and respectively as a first training image and a second training image (step S210); inputting the first training image and the second training image of each group of tissue cavity inner wall image pairs into the neural network in sequence to obtain the forward optical flow of the first pixel point in the first training image (step S220); exchanging the input order of the first training image and the second training image to obtain the backward optical flow of the second pixel point corresponding to the first pixel point in the second training image (step S230); calculating the loss function based on the forward optical flow and the backward optical flow (step S240); and evaluating and optimizing the neural network based on the loss function (step S250). In this case, the neural network can be trained in an unsupervised manner, and the training can be performed using unannotated tissue cavity inner wall images.

[0154] In some examples, the first training image and the second training can be used as training images to train the neural network.

[0155] In some examples, the forward optical flow can refer to the optical flow from the first training image to the second training image. In some examples, the backward optical flow can refer to the optical flow from the second training image to the first training image.

[0156] In some examples, in step S210, a plurality of groups of tissue cavity inner wall image pairs are prepared, the tissue cavity inner wall image pair can be two images obtained by collecting the same region in the tissue cavity with a wrinkled inner wall in different directions, and respectively as a first training image and a second training image.

[0157] In some examples, the pair of tissue cavity inner wall images can be a pair of stomach cavity inner wall images without ground truth. The first training image and the second training image can be images of adjacent frames. In some examples, the first training image and the second training image can be acquired in the same manner as the first image 31 and the second image 32 described above.

[0158] In some examples, in step S220, the first training image and the second training image of each group of pairs of tissue cavity inner wall images can be sequentially input into the neural network to obtain a forward optical flow. In this case, a first predicted image can be generated by the second training image acting on the forward optical flow.

[0159] In some examples, in step S230, the input order of the first training image and the second training image can be exchanged to obtain a backward optical flow. In this case, a second predicted image can be generated by the first training image acting on the backward optical flow.

[0160] In some examples, in step S240, a loss function can be calculated based on the forward optical flow and the backward optical flow. In some examples, the loss function can include a structural similarity loss term, a geometric consistency loss term, and an optical flow smoothness loss term. In this case, the neural network can be trained by the multiple loss functions.

[0161] In some examples, the loss function of the neural network can include a structural similarity loss term, which can represent the similarity between images. Specifically, for an image F and an image S, the calculation formula of the structural similarity SSIM is:

[0162]

[0163] wherein c1=(k1L) 2 , c2=(k2L) 2 , k1=0.01, k2=0.03, SSIM represents the structural similarity between the image F and the image S, μ F , μ S , σ F , σ S are the average value and the variance of the pixel points in the image F and the image S, respectively, σ FS is the covariance between the pixel points in the image F and the image S, and L is the dynamic range of the pixel value.

[0164] In some examples, the structural similarity loss term can include a first structural similarity loss term of the first predicted image and the first training image and a second structural similarity loss term of the second predicted image and the second training image. The first predicted image can be generated by the forward optical flow acting on the second training image, and the second predicted image can be generated by the backward optical flow acting on the first training image. In this case, the accuracy of the calculated forward optical flow and the backward optical flow can be evaluated, so that the accuracy of the optical flow calculated by the neural network is improved through training of the neural network.

[0165] In some examples, the loss function can include an optical flow smoothness loss term. The optical flow smoothness loss term is a sum of a total variation loss of the forward optical flow and a total variation loss of the backward optical flow, in which case the smoothness of the optical flow can be improved, so that the smoothness of the image can be maintained and the noise effect can be reduced.

[0166] In some examples, the total variation loss of the image is:

[0167]

[0168] where x, y respectively represent the position of the pixel point in the width direction and the length direction of the image, and TV represents the total variation loss of the image.

[0169] In some examples, the optical flow smoothness loss term can include a total variation loss of the forward optical flow and a total variation loss of the backward optical flow. In this case, the forward optical flow and the backward optical flow calculated by the trained neural network can be kept smooth in space.

[0170] In some examples, the geometric forward-back consistency can be a difference between the forward optical flow and the backward optical flow. Specifically, by inputting the first training image and the second training image into the neural network system based on convolution operation and exchanging the input order of the first training image and the second training image, the forward optical flow of a first pixel point in the first training image and the backward optical flow of a second pixel point corresponding to the first pixel point in the second training image can be obtained, so that the forward optical flow and the backward optical flow between the corresponding (i.e. matching) pixel points in the first training image and the second training image can be calculated, and the difference between the forward optical flow and the backward optical flow of each pixel point can be calculated, and the sum of the differences between the forward optical flow and the backward optical flow corresponding to each pixel point is taken as the geometric forward-back consistency loss term. In this case, the forward optical flow and the backward optical flow calculated by the trained neural network system can be kept forward-back consistent.

[0171] In some examples, the neural network can be evaluated and optimized based on the loss function in step S250. In some examples, the first neural network 12, the second neural network 22 and the third neural network 14 can be trained together to obtain an optimal neural network structure. In addition, the first neural network 12, the second neural network 22 and the third neural network 14 involved in the present embodiment can be trained together at the same time, and in other examples, the first neural network 12 and the second neural network 22 can be trained first by training an auto-encoder network, and then trained together with the third neural network 14. In this way, the neural network 10 based on deep learning can be trained using training images.

[0172] In some examples, after the neural network 10 based on deep learning is trained using training images, the trained neural network can be tested using test images. The test images can be a plurality of pairs of images of the inner wall of the tissue cavity, which can be two images of the same region in the tissue cavity with a wrinkled inner wall collected in different directions. When the loss function tested is less than a set value, it can be considered that the neural network has been trained.

[0173] In some examples, the test images can include first test images and second test images. In some examples, the first test images and the second test images can be acquired in the same way as the first image 31 and the second image 32 described above.

[0174] As shown in Figure 1 The processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow involved in the present disclosure can extract a first feature point set including a plurality of feature points in a first pixel region and obtain a second feature point set corresponding to the first feature point set in a second pixel region based on the first feature point set and the optical flow in step S300.

[0175] Figure 8 An example of the present disclosure is shown, which shows a schematic diagram of feature point matching based on a neural network based on deep learning.

[0176] In some examples, as shown in Figure 8 The first feature point set can include a plurality of first feature points 311, and the second feature point set can include a plurality of second feature points 321. In some examples, the first feature points 311 and the second feature points 321 can have the same number. In some examples, a single first feature point 311 and a second feature point 321 matched therewith can form a group of feature point pairs.

[0177] In some examples, the optical flow component u iand optical flow component v i Several corner points x can be randomly extracted from the first image 31. j (j=1,2,…,n) is taken as the first feature point 311. Then, the second feature point 321x corresponding to the first feature point 311 in the second image 32 is found by the optical flow at the location of the first feature point 311. j This means that feature matching is completed. In this case, a more accurate feature point matching relationship can be obtained between the first image 31 and the second image 32.

[0178] In some examples, corner points may have the following characteristics: they are generally intersections between contours; they usually have stable properties even when the viewpoint changes for the same scene; the pixels in the vicinity of the point have large variations in both gradient direction and gradient magnitude; in this case, extracting corner points as feature points can improve the accuracy of feature point matching.

[0179] In some examples, to calculate the positional change of the capsule endoscope during the capture of the first image 31 and the second image 32, it is necessary to extract more than 8 pairs of feature points. Specifically, a first feature point set with at least 8 feature points can be extracted from the first image 31, and a second feature point set matching the first feature point set in the second image 32 can be obtained through the optical flow corresponding to the feature points in the first feature point set. In some examples, 8, 10, 12, 14, or 16 pairs of matching feature points can be obtained using the above method. This improves the accuracy of the essential matrix (described later).

[0180] In some examples, camera motion can be solved using epipolar geometry constraints. In such cases, the camera motion and specific position can be accurately solved using epipolar geometry.

[0181] like Figure 1 As shown, in step S400 of the processing method for obtaining three-dimensional images of tissue cavities based on optical flow, the essential matrix of the capsule endoscope 20 can be calculated based on the first feature point set and the second feature point set.

[0182] In some examples, the motion of the capsule endoscope 20 can be accurately solved using epipolar geometry to obtain the essential matrix of the capsule endoscope 20. Specifically, the essential matrix can be obtained by calculating the pixel positions of multiple pairs of feature points, and then decomposing the essential matrix to obtain the rotation matrix and translation matrix. The rotation matrix and translation matrix correspond to the rotation and translation variables of the capsule endoscope 20 during the acquisition of the first image 31 and the second image 32, respectively.

[0183] like Figure 1As shown, the processing method for obtaining a three-dimensional image of a tissue cavity based on optical flow according to the present disclosure can calculate the spatial coordinates of each pixel point in the first pixel region and the second pixel region based on the essential matrix in step S500, obtain a point cloud, and perform three-dimensional image reconstruction using the point cloud.

[0184] In some examples, the spatial coordinates of each pixel point can be calculated by triangulation, a three-dimensional point can be generated, and the three-dimensional point can be stored in a point cloud structure, i.e., the generation of the point cloud (i.e., discrete points) is completed.

[0185] In some examples, the discrete points obtained above can be collected by a point cloud collection device, and noise present in the discrete points can be filtered out by point cloud filtering. Further, a model of a tissue cavity (e.g., a stomach cavity) composed of the discrete points can be obtained by three-dimensional reconstruction technology.

[0186] In some examples, the point cloud filtering can include a pass-through filter, a voxel filter, a statistical filter, a conditional filter, or a radius filter. Thus, one or more filters can be selected according to actual conditions to optimize the noise removal effect.

[0187] In some examples, the three-dimensional reconstruction can be Poisson reconstruction, real-time reconstruction, surface reconstruction, or the like. Thus, the point cloud can be converted into an intuitive three-dimensional image, thereby improving the visual effect.

[0188] Although the present disclosure is specifically described above in combination with the drawings and embodiments, it should be understood that the above description does not limit the present disclosure in any form. Those skilled in the art can make modifications and changes to the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.

Claims

1. A method for calculating the essential matrix of a capsule endoscope within a tissue cavity, characterized in that, The method comprises the following steps: acquiring a first image of a reconstructed region in a tissue cavity with a capsule endoscope along a first direction in the tissue cavity with a wrinkled inner wall, and acquiring a second image of the reconstructed region along a second direction forming an angle with the first direction in the tissue cavity; calculating, by a neural network based on convolution operation, an optical flow of a first pixel region of the reconstructed region in the first image and a second pixel region corresponding to the first pixel region in the second image, extracting a first feature point set comprising a plurality of feature points in the first pixel region, and obtaining a second feature point set corresponding to the first feature point set in the second pixel region based on the first feature point set and the optical flow; and calculating an essential matrix of the capsule endoscope based on the first feature point set and the second feature point set.

2. The method according to claim 1, wherein the capsule endoscope is a monocular capsule endoscope or a multiocular capsule endoscope, and the optical flow is generated by the movement of the capsule endoscope.

3. The method according to claim 1, wherein the first image and the second image are two images of adjacent frames, the first image is an image of a current frame, and the second image is an image of a next frame.

4. The method according to claim 1, wherein the neural network comprises a first neural network, a second neural network, a feature combination module, and a third neural network.

5. The method according to claim 4, wherein the neural network further comprises a preprocessing module, the preprocessing module comprises a first preprocessing module and a second preprocessing module, and the preprocessing module is configured to pre-process the first image and the second image, respectively, wherein the first preprocessing module is configured to pre-process the first image, and the second preprocessing module is configured to pre-process the second image.

6. The method according to claim 4, wherein the feature combination module fuses a first feature set generated by the first neural network and a second feature set generated by the second neural network to form a feature combination set.

7. The method according to claim 1, wherein the training step of the neural network based on convolution operation comprises: preparing a plurality of groups of tissue cavity inner wall image pairs, the tissue cavity inner wall image pair being two images of the same region in a tissue cavity with a wrinkled inner wall acquired in different directions, and being a first training image and a second training image, respectively; inputting the first training image and the second training image of each group of tissue cavity inner wall image pairs into the neural network in sequence to obtain a forward optical flow of a first pixel point in the first training image; exchanging the input order of the first training image and the second training image to obtain a backward optical flow of a second pixel point corresponding to the first pixel point in the second training image; calculating a loss function based on the forward optical flow and the backward optical flow; and evaluating and optimizing the neural network based on the loss function.

8. The computing method of claim 7, wherein the loss function comprises a structural similarity loss term, a geometric consistency loss term, and an optical flow smoothness loss term.

9. The computing method of claim 8, wherein the structural similarity loss term comprises a first structural similarity loss term of a first predicted image and the first training image, and a second structural similarity loss term of a second predicted image and the second training image, the first predicted image being generated by the forward optical flow acting on the second training image, and the second predicted image being generated by the backward optical flow acting on the first training image.

10. The computing method of claim 8, wherein the optical flow smoothness loss term comprises a total variation loss of the forward optical flow and a total variation loss of the backward optical flow, and the geometric consistency loss term is a difference between the forward optical flow of a first pixel in the first training image and the backward optical flow of a second pixel corresponding to the first pixel in the second training image. ​ ​ ​

Images