Intestinal tract three-dimensional reconstruction method and device based on optical flow method, equipment and medium

Abstract

The invention discloses an intestinal tract three-dimensional reconstruction method and device based on an optical flow method, equipment and a medium, and relates to the technical field of intestinal tract three-dimensional reconstruction, and the method comprises the steps: carrying out the depth estimation of each monocular intestinal tract image through employing a deep learning method, and obtaining the depth data of each pixel point; performing intestinal tract contour extraction on the monocular intestinal tract image to obtain two-dimensional intestinal tract contour data; analyzing pixel motion information in the monocular intestinal tract image by adopting an optical flow method, calculating an optical flow field, and calculating the global pose of the endoscope according to the optical flow field; and along with the slow advancing of the endoscope, the reconstruction of the three-dimensional model of the intestinal tract is finally realized according to the corresponding depth data, the two-dimensional outline data of the intestinal tract and the global pose of the endoscope. The method can be stably applied in a dynamic endoscope motion environment, is not limited by illumination conditions and significant feature points, and ensures that effective intestinal tract three-dimensional reconstruction can be performed in a low-texture region.

Description

Technical Field

[0001] The present application relates to the technical field of intestinal three-dimensional reconstruction, and particularly to a method, device, equipment and medium for intestinal three-dimensional reconstruction based on the optical flow method. Background Art

[0002] Intestinal three-dimensional reconstruction technology can help doctors more accurately locate the lesion site. Although traditional two-dimensional images can provide certain information, in a complex intestinal environment, it is difficult to accurately determine the location of the lesion. The three-dimensional reconstruction technology can generate a three-dimensional and real lesion model, enabling doctors to clearly see the specific location of the lesion, which is of great significance for early detection of lesions and formulation of treatment plans. In addition, through the three-dimensional reconstruction method, doctors can more intuitively observe the internal structure of the intestine and better understand the overall structure of the intestine, thereby improving the accuracy of diagnosis.

[0003] Existing intestinal three-dimensional reconstruction technologies, such as reconstruction based on feature point matching and Shape from Shading (SFS) and Shape from Focus (SFF), face many challenges. First, as a typical one-dimensional tubular environment, the intestinal tract often has no obvious feature points, which makes the method based on feature point matching difficult to apply. Second, the SFS and SFF methods rely on specific illumination patterns and focus information to infer the shape of an object, and are limited in a dynamic scene of endoscopic movement, because the movement may cause illumination changes or focus blur, resulting in excessive reconstruction errors or even reconstruction failures.

[0004] Therefore, how to develop a more generally applicable reconstruction method to achieve three-dimensional reconstruction of the intestine in a dynamic scene of endoscopic movement is an important problem that needs to be solved in the current technological development. Summary of the Invention

[0005] The purpose of the present application is to provide a method, device, equipment and medium for intestinal three-dimensional reconstruction based on the optical flow method, which can be stably applied in a dynamic endoscopic movement environment, is not limited by illumination conditions and significant feature points, and ensures effective three-dimensional reconstruction of the intestine in low-texture areas.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] In the first aspect, the present application provides a method for intestinal three-dimensional reconstruction based on the optical flow method, including:

[0008] Real-time acquisition of monocular intestinal images, where the monocular intestinal images are images captured by an endoscope during the forward movement in the intestine;

[0009] Use a deep learning algorithm to perform depth estimation on each of the monocular intestinal images to obtain depth data for each pixel point in each of the monocular intestinal images;

[0010] Extract the intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data;

[0011] Based on the depth data and the corresponding two-dimensional intestinal contour data, obtain local intestinal three-dimensional point cloud data;

[0012] Use the optical flow method to analyze the pixel motion information in the monocular intestinal image, calculate the optical flow field, and calculate the global pose of the endoscope based on the optical flow field;

[0013] Utilize the global pose of the endoscope to convert the local intestinal three-dimensional point cloud data into global intestinal three-dimensional point cloud data;

[0014] Based on the global intestinal three-dimensional point cloud data of all frames, reconstruct the three-dimensional model of the entire intestine.

[0015] In a second aspect, the present application provides an intestinal three-dimensional reconstruction device based on the optical flow method, including:

[0016] An acquisition module for real-time acquisition of monocular intestinal images, where the monocular intestinal images are images captured by an endoscope during the advancement in the intestine;

[0017] A depth estimation module for performing depth estimation on each of the monocular intestinal images using a deep learning algorithm to obtain depth data for each pixel point in each of the monocular intestinal images;

[0018] Extract the intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data;

[0019] A local intestinal three-dimensional point cloud data calculation module for obtaining local intestinal three-dimensional point cloud data based on the depth data and the corresponding two-dimensional intestinal contour data;

[0020] An endoscope global pose calculation module for using the optical flow method to analyze the pixel motion information in the monocular intestinal image, calculating the optical flow field, and calculating the global pose of the endoscope based on the optical flow field;

[0021] A global intestinal three-dimensional point cloud data calculation module for using the global pose of the endoscope to convert the local intestinal three-dimensional point cloud data into global intestinal three-dimensional point cloud data;

[0022] A three-dimensional model reconstruction module for reconstructing the three-dimensional model of the entire intestine based on the global intestinal three-dimensional point cloud data of all frames.

[0023] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method for three-dimensional reconstruction of the intestine based on the optical flow method described in the first aspect above.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for three-dimensional reconstruction of the intestine based on the optical flow method described in the first aspect above.

[0025] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0026] The present application provides a method, device, equipment, and medium for three-dimensional reconstruction of the intestine based on the optical flow method. The method obtains depth data of each pixel point by using a deep learning method to estimate the depth of each monocular intestine image; extracts the intestinal contour of the monocular intestine image to obtain two-dimensional intestinal contour data; analyzes the pixel motion information in the monocular intestine image by using the optical flow method, calculates the optical flow field, and calculates the global pose of the endoscope according to the optical flow field; as the endoscope slowly advances, according to the corresponding depth data, two-dimensional intestinal contour data, and the global pose of the endoscope, the three-dimensional model of the intestine is finally reconstructed. This method can be stably applied in a dynamic endoscope movement environment, is not limited by lighting conditions and significant feature points, and ensures that effective three-dimensional reconstruction of the intestine can also be performed in low-texture areas. Description of the Drawings

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0028] Figure 1 It is an application environment diagram of a method for three-dimensional reconstruction of the intestine based on the optical flow method in Embodiment 1 of the present application;

[0029] Figure 2 It is a flowchart of a method for three-dimensional reconstruction of the intestine based on the optical flow method provided in Embodiment 1 of the present application;

[0030] Figure 3 It is a conceptual diagram of a method for three-dimensional reconstruction of the intestine based on the optical flow method provided in Embodiment 1 of the present application;

[0031] Figure 4 It is a schematic diagram of motion tracking of the optical flow method in an image sequence in Embodiment 1 of the present application;

[0032] Figure 5 Schematic diagram of the mapping process from world coordinates to an image in Embodiment 1 of this application;

[0033] Figure 6 Schematic diagram of the structure of a computer device provided in Embodiment 3 of this application. Detailed implementation manners

[0034] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0035] To make the above objects, features, and advantages of this application more obvious and understandable, the following further describes this application in detail in conjunction with the accompanying drawings and specific implementation manners.

[0036] The intestinal three-dimensional reconstruction method based on the optical flow method provided in the embodiments of this application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set separately, integrated on the server 104, placed in the cloud, or on other servers. The terminal 102 can send the to-be-processed monocular intestinal image to the server 104. After receiving the to-be-processed monocular intestinal image, the server 104 will use a deep learning method to perform depth estimation on each of the monocular intestinal images to obtain the depth data of each pixel point in each of the monocular intestinal images; extract the intestinal contour of each of the monocular intestinal images to obtain two-dimensional intestinal contour data; obtain intestinal local three-dimensional point cloud data according to the depth data and the corresponding two-dimensional intestinal contour data; analyze the pixel motion information in the monocular intestinal image by using the optical flow method, calculate the optical flow field, and calculate the global pose of the endoscope according to the optical flow field; use the global pose of the endoscope to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data; reconstruct the three-dimensional model of the entire intestine according to the intestinal global three-dimensional point cloud data of all frames. The server 104 can feedback the obtained three-dimensional model of the entire intestine to the terminal 102. In addition, in some embodiments, the intestinal three-dimensional reconstruction method based on the optical flow method can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly process the to-be-processed monocular intestinal image by using the intestinal three-dimensional reconstruction method based on the optical flow method, or the server 104 can obtain the to-be-processed monocular intestinal image from the data storage system and process it by using the intestinal three-dimensional reconstruction method based on the optical flow method.

[0037] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0038] In an exemplary embodiment, as Figure 2 shown, a three-dimensional intestinal reconstruction method based on the optical flow method is provided. This method is executed by a computer device, and specifically can be executed alone by a computer device such as a terminal or a server, or can be jointly executed by a terminal and a server. In the embodiment of the present application, taking this method applied to Figure 1 the server 104 therein as an example for illustration, it includes the following steps 201 to step 207. Among them:

[0039] Step 201, obtain monocular intestinal images in real time, where the monocular intestinal images are images captured by an endoscope during the forward movement in the intestine.

[0040] Step 202, use a deep learning algorithm to perform depth estimation on each of the monocular intestinal images to obtain depth data of each pixel point in each of the monocular intestinal images.

[0041] Step 202-1, take each of the monocular intestinal images as input, and use a pre-trained depth estimation model to obtain depth data of each pixel point in each of the monocular intestinal images, where the depth estimation model is a convolutional neural network model.

[0042] Step 203, extract the intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data;

[0043] Step 203-1, perform denoising processing on each of the monocular intestinal images to obtain a denoised monocular intestinal image.

[0044] Step 203-1, perform binarization processing on the denoised monocular intestinal image to obtain a binarized image.

[0045] Step 203-1, perform edge detection on the binarized image, and take the points on the largest contour as two-dimensional intestinal contour data.

[0046] Step 204, according to the depth data and the corresponding two-dimensional intestinal contour data, obtain intestinal local three-dimensional point cloud data.

[0047] Step 205: Use the optical flow method to analyze the pixel motion information in the monocular intestinal image, calculate the optical flow field, and calculate the global pose of the endoscope based on the optical flow field.

[0048] Step 205-1, Taylor expansion is performed on the brightness consistency formula to obtain the Taylor expansion formula, wherein the brightness consistency formula is used to characterize that the brightness value of the pixel point at the same position of two consecutive frames of the monocular intestinal image remains unchanged.

[0049] Step 205-2, combining the brightness consistency formula and the Taylor expansion formula to solve the optical flow vector of each pixel on the monocular intestinal image.

[0050] Step 205-3, using the least squares method to solve the optical flow equation group, to obtain the endoscope motion information of each frame of the monocular intestinal image, wherein the optical flow equation group includes a plurality of optical flow equations, each of which is used to characterize the relationship between the optical flow vector of each pixel point and the endoscope motion information, and the endoscope motion information includes translation and rotation.

[0051] Step 205-4, using the endoscope motion information of each frame of monocular intestinal image as the endoscope motion information of the current frame, and calculating the global pose of the endoscope according to the endoscope motion information of the current frame and the endoscope motion information of the previous frame of monocular intestinal image.

[0052] Step 206, using the global posture of the endoscope, converting the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data;

[0053] Step 207, reconstructing a three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine in all frames.

[0054] It should be noted that the execution order of the above steps 201-207 is only a specific example of the execution order of the intestinal three-dimensional reconstruction method based on the optical flow method. This embodiment does not specifically limit the execution order of these steps. It is sufficient to obtain the corresponding depth data, two-dimensional intestinal contour data, local intestinal three-dimensional point cloud data and endoscope global pose before reconstructing the three-dimensional model of the intestine.

[0055] In order to make those skilled in the art more aware of the technical concept of the intestinal 3D reconstruction method based on the optical flow method in this embodiment, the following combination Figure 3 Provide specific explanation.

[0056] like Figure 3 The conceptual diagram of the intestinal 3D reconstruction method based on the optical flow method shown in the figure includes the following steps:

[0057] Step S1: Perform depth estimation on the monocular image (i.e., monocular intestinal image) captured by the endoscope through a deep learning method. In this embodiment, a convolutional neural network (CNN) model is used to predict the depth of each pixel in the monocular image, so as to be able to predict depth information from a single-frame image and generate a depth map.

[0058] Specifically, in this embodiment, the depth estimation model DepthFM is used for depth prediction. DepthFM is a fast and accurate monocular depth estimation method, which realizes the direct mapping from the input image to the depth map through the flow matching technology. This method not only improves the efficiency but also ensures the quality of the generated depth map. In practical applications, the monocular image captured by the endoscope is input into the DepthFM model, and through the forward inference process, the corresponding depth map is obtained.

[0059] Step S2: Analyze the pixel motion between two consecutive frames of images captured by the endoscope using the optical flow method. Subsequently, according to the calculated optical flow field, the endoscope motion information is solved, and the endoscope motion information includes translation and rotation.

[0060] Figure 4 is a schematic diagram of motion tracking by the optical flow method in an image sequence, showing the process of the optical flow method tracking the motion of a specific point in the image at multiple moments. The optical flow method calculates the pixel motion vector between consecutive frames as follows:

[0061] Assume that the time of the previous frame is t and the time of the next frame is t + δt. Then, the pixel point I(x, y, t) of the previous frame I is at the position I(x + δx, y + δy, z + δz) in the next frame. According to the assumption that the gray value is constant for the two frames, it can be obtained that:

[0062] I(x, y, t) = I(x + δx, y + δy, t + δt).

[0063] Expand the right side above using the Taylor series to get:

[0064]

[0065] where are the partial derivatives of I(x, y, t) with respect to x, y, z; H.O.T is the high-order term of the Taylor series expansion and can be ignored.

[0066] Combine the above two equations, take the derivative of both sides of the equation with respect to t and divide by dt to get the simplified equation:

[0067]

[0068] where, and are the velocity vectors of the optical flow along the x-axis and y-axis respectively, and respectively represent the partial derivatives of the gray level of the pixel points in the image along the X-axis and Y-axis, which can be obtained from the monocular intestinal image captured by the endoscope, (v x , v y ) that is, the desired optical flow vector.

[0069] Analyze the vector field (i.e., the optical flow vector) calculated by the optical flow method to deduce the endoscopic motion information. Assume that the endoscope has a translation T calibrate = [T x T y T z T and a rotation R calibrate = [ω x ω y ω z T .

[0070] The optical flow on the image can be described by the motion of the endoscope (translation and rotation matrices), satisfying the following relationship (i.e., the expression of the optical flow equation):

[0071]

[0072] where, v x (x, y) and v y (x, y) respectively represent the components of the optical flow vector (v x , v y ) of the pixel point (x, y); f x represents the component of the camera's focal length in the x-axis direction; f y represents the component of the camera's focal length in the y-axis direction; ω x represents the rotational angular velocity of the endoscope around the x-axis; ω y represents the rotational angular velocity of the endoscope around the y-axis; ω z represents the rotational angular velocity of the endoscope around the z-axis; T x represents the displacement of the endoscope along the x-axis; T y represents the displacement of the endoscope along the y-axis; T z represents the displacement of the endoscope along the z-axis; z(x, y) is the depth value of the pixel point.

[0073] For each pixel point, the above optical flow equation can be used to further construct an optical flow equation system, representing the relationship between the motion information of the pixel point and the endoscopic posture (i.e., the endoscopic motion information):

[0074] A·p = b;

[0075] where, A is the coefficient matrix related to the optical flow equation, p is the endoscopic motion information to be solved, and b is the calculated optical flow vector.​​

[0076] The motion information (translation T calibrate and rotation R calibrate ) between two frames of the endoscope is obtained by least squares solution. The pose of the endoscope (i.e., the motion information of the endoscope) in the first frame of the monocular intestinal image is known (initial R 0 and T 0 ). By frame-by-frame update, the motion information of the endoscope in the global space (i.e., the global pose of the endoscope) can be obtained:

[0077] R global = R previous ·R calibrate ;

[0078] T global = T previous + R previbus ·T calibrate ;

[0079] where T previous and R previous are the pose matrices of the previous frame of the endoscope (i.e., the motion information of the endoscope in the previous frame of the monocular intestinal image).

[0080] Step S3: Perform a series of processing operations on the monocular intestinal image captured by the endoscope, including using Gaussian blur to reduce noise, dividing the image into foreground and background through threshold segmentation, then applying a contour detection algorithm to extract the intestinal contour, and finally storing the detected contour in an array to obtain two-dimensional intestinal contour data. Specifically, it includes the following sub-steps:

[0081] S31: First, perform Gaussian blur processing on the monocular intestinal image to reduce image noise and details, which helps with subsequent edge detection and contour extraction. Gaussian blur uses a Gaussian function as a filter to perform neighborhood weighted averaging on each pixel value in the image, thereby achieving a smoothing effect.

[0082] S32: Perform binarization processing on the Gaussian-blurred image. Determine a threshold through experiments and use this threshold for threshold segmentation to divide the image into foreground and background parts: the area above the threshold is recognized as the foreground and displayed in white; while the area below the threshold is regarded as the background and displayed in black. Then, perform color inversion processing on the binarized image to more clearly identify the area of interest.

[0083] S33: Perform contour detection in the binarized image. After identifying all the contours, find the largest contour, which usually represents the main area of the intestine. Save the coordinates of all points of this largest contour as a two-dimensional array for subsequent processing and analysis.

[0084] Step S4: Combine each two-dimensional pixel point on the contour (i.e., two-dimensional intestinal contour data) with its corresponding depth value, and convert these points into three-dimensional spatial coordinates through the internal parameter matrix of the camera to generate local three-dimensional point cloud data of the intestine.

[0085] Specifically, the two-dimensional intestinal contour data obtained in step S3, combined with the depth data of the corresponding points obtained in step S1, can generate a point cloud in three-dimensional space (i.e., local three-dimensional point cloud data of the intestine). Figure 5 is a schematic diagram of the mapping process from the world coordinate to the image. Assume a point P = [x w , y w , z w T in the world coordinate system. Through the camera model, it can be projected onto the image plane of the camera, and the corresponding point in the camera coordinate system is P c = [x c , y c , z c T . According to the projection relationship, for the two-dimensional coordinates (u, v) of each point on the contour to the point cloud X point_cloud = (x point_cloud , y point_clud , z point_cloud ) T there is the following transformation formula:

[0086]

[0087] where z c is the depth value of this point, and f x , f y , u 0 , v 0 can be obtained through camera calibration.

[0088] Step S5: Use the motion information (translation and rotation matrix) of the endoscope to convert the point cloud data of the current frame into the global coordinate system. As the endoscope slowly advances, continuously collect new data, and gradually integrate and update the newly generated point cloud with the existing three-dimensional model. Finally, through frame-by-frame accumulation, a complete three-dimensional model is formed.

[0089] Specifically, use the global pose R global and T global of the endoscope to transform the local point cloud into the global coordinate system. After transforming the local point cloud into the global coordinate system, each point cloud (x point_cloud , y point_cloud , z point_cloud ) T is transformed through the following formula:

[0090]

[0091] Among them, (x global , y global , z global ) is the three-dimensional coordinate of the global three-dimensional point cloud data of the intestine; X point_cloud = (x point_cloud , y point_cloud , z point_cloud ) is the three-dimensional coordinate of the local three-dimensional point cloud data of the intestine; R global is the rotation data in the global pose of the endoscope; T global is the translation data in the global pose of the endoscope.

[0092] As the endoscope continuously advances slowly, new data is continuously collected, and the newly generated point cloud is gradually integrated and updated with the existing three-dimensional model, ultimately realizing the three-dimensional reconstruction of the entire intestine.

[0093] The deep learning method of this embodiment effectively extracts the depth information in the image through training the model and is applicable to the scenario of endoscope movement. The optical flow method infers the dynamic changes of the endoscope by analyzing the pixel movement between consecutive frames, which makes it unaffected by specific illumination patterns and focus information, thus still being able to maintain stable performance in the scenario of endoscope movement.

[0094] Compared with the prior art, the method for three-dimensional reconstruction of the intestine based on the optical flow method provided in this embodiment can be stably applied in a dynamic endoscope movement environment, is not restricted by illumination conditions and significant feature points, and ensures that effective three-dimensional reconstruction of the intestine can also be carried out in low-texture areas. This method extracts the depth information and the intestinal contour from the images captured by the endoscope and combines the optical flow method technology to achieve the three-dimensional reconstruction of the intestine, which is more generally applicable than other methods.

[0095] This application also provides an application scenario that applies the above-mentioned method for three-dimensional reconstruction of the intestine based on the optical flow method. Specifically: The method for three-dimensional reconstruction of the intestine based on the optical flow method provided in this embodiment can be applied in the scenario of medical image diagnosis. This scenario includes the intestinal data acquisition link, the image processing and reconstruction link, and the clinical analysis and evaluation link. The method for three-dimensional reconstruction of the intestine based on the optical flow method provided in this embodiment belongs to the key technology in the image processing and reconstruction link. This method can efficiently and accurately reconstruct the three-dimensional structure of the intestine, helping doctors accurately evaluate the lesion location and degree of the intestine, thus providing important support in the diagnosis, surgical planning, and postoperative evaluation of intestinal diseases.

[0096] Embodiment 2

[0097] This embodiment also provides a device for three-dimensional reconstruction of the intestine based on the optical flow method, including:

[0098] An acquisition module, configured to acquire monocular intestinal images in real time, where the monocular intestinal images are images captured by an endoscope during its advancement in the intestine.

[0099] A depth estimation module, configured to perform depth estimation on each of the monocular intestinal images by using a deep learning algorithm to obtain depth data of each pixel point in each of the monocular intestinal images.

[0100] Extract the intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data.

[0101] An intestinal local three-dimensional point cloud data calculation module, configured to obtain intestinal local three-dimensional point cloud data according to the depth data and the corresponding two-dimensional intestinal contour data.

[0102] An endoscope global pose calculation module, configured to analyze the pixel motion information in the monocular intestinal images by using an optical flow method, calculate an optical flow field, and calculate the endoscope global pose according to the optical flow field.

[0103] An intestinal global three-dimensional point cloud data calculation module, configured to use the endoscope global pose to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data.

[0104] A three-dimensional model reconstruction module, configured to reconstruct a three-dimensional model of the entire intestine according to the intestinal global three-dimensional point cloud data of all frames.

[0105] The solution for solving problems provided by the intestinal three-dimensional reconstruction device based on the optical flow method according to the embodiments of the present application is similar to the solution described in the above method. The specific limitations of the device can be referred to the limitations of the intestinal three-dimensional reconstruction method based on the optical flow method in Embodiment 1, and will not be elaborated here.

[0106] This embodiment provides an intestinal three-dimensional reconstruction device based on the optical flow method. The device performs depth estimation on monocular intestinal images by using a deep learning method. Analyze the pixel motion between two consecutive frames of endoscope images (i.e., monocular intestinal images) by using the optical flow method, calculate the optical flow field and deduce the endoscope motion information, including translation and rotation matrices. Subsequently, extract the intestinal contour from the endoscope images and convert the two-dimensional intestinal contour data into intestinal local three-dimensional point cloud data. Finally, use the endoscope motion information to convert the intestinal local three-dimensional point cloud data of the current frame to the global coordinate system. As the endoscope advances slowly, a three-dimensional model of the intestine is reconstructed by the method of frame-by-frame accumulation. The device can be stably applied in a dynamic endoscope motion environment, is not limited by lighting conditions and significant feature points, and ensures effective intestinal three-dimensional reconstruction in low-texture areas.

[0107] Embodiment 3

[0108] This embodiment provides a computer device, which can be a server or a terminal, and its internal structure diagram can be as shown in Figure 6 . The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the three-dimensional intestinal reconstruction method based on the optical flow method. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes the three-dimensional intestinal reconstruction method based on the optical flow method in Embodiment 1.

[0109] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor, and a computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are realized.

[0110] Embodiment 4

[0111] This embodiment provides a computer-readable storage medium storing a computer program, which realizes the three-dimensional intestinal reconstruction method based on the optical flow method in Embodiment 1 when executed by a processor.

[0112] Embodiment 5

[0113] This embodiment provides a computer program product including a computer program, which realizes the three-dimensional intestinal reconstruction method based on the optical flow method in Embodiment 1 when executed by a processor.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0115] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0116] The databases involved in the various embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0117] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0118] In this text, specific examples are used to illustrate the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for three-dimensional reconstruction of intestine based on optical flow method, characterized in that: The intestinal 3D reconstruction method based on the optical flow method comprises: Acquire a monocular intestinal image in real time, wherein the monocular intestinal image is an image captured by the endoscope during the process of advancing the intestinal tract; Using a deep learning algorithm to perform depth estimation on each of the monocular intestinal images to obtain depth data of each pixel in each of the monocular intestinal images; Extracting the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data; Obtaining local three-dimensional point cloud data of the intestine according to the depth data and the corresponding two-dimensional intestinal contour data; An optical flow method is used to analyze pixel motion information in the monocular intestinal image, an optical flow field is calculated, and a global pose of the endoscope is calculated based on the optical flow field; Using the global posture of the endoscope, converting the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data; A three-dimensional model of the entire intestine is reconstructed based on the global three-dimensional point cloud data of the intestine in all frames.

2. The method for intestinal 3D reconstruction based on optical flow method according to claim 1, characterized in that: A deep learning algorithm is used to perform depth estimation on each monocular intestinal image to obtain depth data of each pixel in each monocular intestinal image, specifically including: Each of the monocular intestinal images is used as input, and the depth data of each pixel in each of the monocular intestinal images is obtained using a pre-trained depth estimation model, wherein the depth estimation model is a convolutional neural network model.

3. The method for intestinal 3D reconstruction based on optical flow method according to claim 2, characterized in that: The optical flow method is used to analyze the pixel motion information in the monocular intestinal image, calculate the optical flow field, and calculate the global position and posture of the endoscope according to the optical flow field, specifically including: Taylor expansion is performed on the brightness consistency formula to obtain a Taylor expansion formula, wherein the brightness consistency formula is used to characterize that the brightness value of a pixel point at the same position of two consecutive frames of the monocular intestinal images remains unchanged; Combining the brightness consistency formula and the Taylor expansion formula, solving the optical flow vector of each pixel on the monocular intestinal image; Solving the optical flow equations by the least square method to obtain the endoscope motion information of each frame of the monocular intestinal image, wherein the optical flow equations include a plurality of optical flow equations, each of which is used to characterize the relationship between the optical flow vector of each pixel point and the endoscope motion information, and the endoscope motion information includes translation and rotation; The endoscope motion information of each frame of monocular intestinal image is used as the endoscope motion information of the current frame, and the global position and posture of the endoscope is calculated according to the endoscope motion information of the current frame and the endoscope motion information of the previous frame of monocular intestinal image.

4. The method for intestinal 3D reconstruction based on optical flow method according to claim 3, characterized in that: The expression of the optical flow equation is: Among them, v x (x, y) and v y (x, y) represent the optical flow vector (v) of the pixel point (x, y) respectively. x , v y )'s weight; f x Indicates the component of the camera's focal length in the x-axis direction; f y Represents the component of the camera's focal length in the y-axis direction; ω x represents the angular velocity of the endoscope around the x-axis; ω y represents the angular velocity of the endoscope around the y-axis; ω z represents the angular velocity of the endoscope around the z-axis; T x represents the displacement of the endoscope along the x-axis; T y represents the displacement of the endoscope along the y-axis; T z Represents the displacement of the endoscope along the z-axis; z(x, y) is the depth value of the pixel point (x, y).

5. The method for intestinal 3D reconstruction based on optical flow method according to claim 1, characterized in that: Extracting the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data specifically includes: De-noising each monocular intestinal image to obtain a denoised monocular intestinal image; Binarizing the denoised monocular intestinal image to obtain a binary image; Edge detection is performed on the binary image, and points on the maximum contour are used as two-dimensional intestinal contour data.

6. The method for intestinal 3D reconstruction based on optical flow method according to claim 1, characterized in that: The calculation formula for the intestinal local 3D point cloud data is: Among them, X point_cloud =(x point_cloud ,y point_cloud , z point_cloud ) is the three-dimensional coordinate of the intestinal local three-dimensional point cloud data; z c is the depth data of the pixel point; (u, v) is the two-dimensional coordinate of the corresponding two-dimensional intestinal contour data; f x ,f y ,u0,v0 are the camera calibration data.

7. The method for intestinal 3D reconstruction based on optical flow method according to claim 1, characterized in that: The calculation formula of the intestinal global three-dimensional point cloud data is: Among them, (x global ,y global ,z global ) is the three-dimensional coordinate of the global three-dimensional point cloud data of the intestine; (x point_cloud ,y point_cloud , z point_cloud ) is the three-dimensional coordinate of the intestinal local three-dimensional point cloud data; R global is the rotation data in the global position of the endoscope; T global is the translation data in the global pose of the endoscope.

8. A three-dimensional intestinal reconstruction device based on optical flow method, characterized in that: The intestinal 3D reconstruction device based on the optical flow method comprises: An acquisition module, used for acquiring a monocular intestinal image in real time, wherein the monocular intestinal image is an image captured by the endoscope during the process of advancing the intestinal tract; A depth estimation module, used to perform depth estimation on each monocular intestinal image using a deep learning algorithm to obtain depth data of each pixel in each monocular intestinal image; Extracting the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data; A local three-dimensional point cloud data calculation module for the intestine, used to obtain local three-dimensional point cloud data of the intestine according to the depth data and the corresponding two-dimensional intestinal contour data; An endoscope global posture calculation module, used for analyzing pixel motion information in the monocular intestinal image by using an optical flow method, calculating an optical flow field, and calculating the endoscope global posture according to the optical flow field; An intestinal global three-dimensional point cloud data calculation module is used to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data by using the global posture of the endoscope; The three-dimensional model reconstruction module is used to reconstruct the three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine in all frames.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the optical flow-based intestinal three-dimensional reconstruction method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intestinal three-dimensional reconstruction method based on optical flow method described in any one of claims 1 to 7 is implemented.

Images