Intestinal three-dimensional reconstruction method and device based on optical flow method, equipment and medium

Abstract

The application discloses an intestinal three-dimensional reconstruction method and device based on an optical flow method, equipment and a medium, relates to the technical field of intestinal three-dimensional reconstruction, and the method comprises the following steps: performing depth estimation on each monocular intestinal image by using a deep learning method to obtain depth data of each pixel point; intestinal contour extraction is performed on the monocular intestinal image to obtain two-dimensional intestinal contour data; pixel motion information in the monocular intestinal image is analyzed by using an optical flow method, an optical flow field is calculated, and the global pose of the endoscope is calculated according to the optical flow field; and finally, the reconstruction of the intestinal three-dimensional model is realized according to the corresponding depth data, the two-dimensional intestinal contour data and the global pose of the endoscope as the endoscope slowly advances. The method can be stably applied in a dynamic endoscope motion environment, is not limited by illumination conditions and significant feature points, and ensures that effective intestinal three-dimensional reconstruction can be performed in a low-texture area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intestinal three-dimensional reconstruction, in particular to an intestinal three-dimensional reconstruction method and device based on an optical flow method, equipment and a medium. BACKGROUND

[0002] Intestinal three-dimensional reconstruction technology can help doctors more accurately locate the lesion site. Although traditional two-dimensional images can provide some information, it is difficult to accurately determine the location of the lesion in a complex intestinal environment. The three-dimensional reconstruction technology can generate a three-dimensional and realistic lesion model, allowing doctors to clearly see the specific location of the lesion, which is of great significance for early detection of lesions and development of treatment plans. In addition, through the three-dimensional reconstruction method, doctors can more intuitively observe the internal structure of the intestine and better understand the overall structure of the intestine, thereby improving the accuracy of diagnosis.

[0003] Existing intestinal three-dimensional reconstruction technologies, such as reconstruction based on feature point matching, shape from shading (SFS) and shape from focus (SFF), face many challenges. First, as a typical one-dimensional tubular environment, the feature points of the intestine are often not obvious, which makes it difficult for methods based on feature point matching to be applicable. Second, SFS and SFF methods rely on specific lighting patterns and focal point information to infer the shape of the object, which is limited in dynamic scenes of endoscope movement, because movement can cause changes in lighting or blurring of the focal point, resulting in excessive reconstruction errors or even reconstruction failure.

[0004] Therefore, how to develop a more universally applicable reconstruction method to achieve three-dimensional reconstruction of the intestine in dynamic scenes of endoscope movement is an important problem to be solved in current technical development. SUMMARY

[0005] The purpose of the present application is to provide an intestinal three-dimensional reconstruction method and device based on an optical flow method, which can be stably applied in dynamic endoscope movement environments, is not limited by lighting conditions and significant feature points, and ensures effective three-dimensional reconstruction of the intestine in low-texture areas.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0007] In a first aspect, the present application provides an intestinal three-dimensional reconstruction method based on an optical flow method, comprising:

[0008] real-time acquisition of a monocular intestinal image, wherein the monocular intestinal image is an image captured by an endoscope during the advancement of the intestine;

[0009] perform depth estimation on each of the monocular intestinal images by using a deep learning algorithm to obtain depth data of each pixel point in each of the monocular intestinal images;

[0010] extract an intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data;

[0011] obtain intestinal local three-dimensional point cloud data according to the depth data and corresponding two-dimensional intestinal contour data;

[0012] analyze pixel motion information in the monocular intestinal images by using an optical flow method, calculate an optical flow field, and calculate an endoscope global pose according to the optical flow field;

[0013] convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data by using the endoscope global pose;

[0014] reconstruct a three-dimensional model of the entire intestine according to the intestinal global three-dimensional point cloud data of all frames.

[0015] In a second aspect, the present application provides an intestinal three-dimensional reconstruction device based on an optical flow method, comprising:

[0016] an acquisition module configured to acquire monocular intestinal images in real time, wherein the monocular intestinal images are images captured by an endoscope during intestinal advancement;

[0017] a depth estimation module configured to perform depth estimation on each of the monocular intestinal images by using a deep learning algorithm to obtain depth data of each pixel point in each of the monocular intestinal images;

[0018] extract an intestinal contour from each of the monocular intestinal images to obtain two-dimensional intestinal contour data;

[0019] an intestinal local three-dimensional point cloud data calculation module configured to obtain intestinal local three-dimensional point cloud data according to the depth data and corresponding two-dimensional intestinal contour data;

[0020] an endoscope global pose calculation module configured to analyze pixel motion information in the monocular intestinal images by using an optical flow method, calculate an optical flow field, and calculate an endoscope global pose according to the optical flow field;

[0021] an intestinal global three-dimensional point cloud data calculation module configured to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data by using the endoscope global pose;

[0022] a three-dimensional model reconstruction module configured to reconstruct a three-dimensional model of the entire intestine according to the intestinal global three-dimensional point cloud data of all frames.

[0023] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the first aspect.

[0024] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the first aspect.

[0025] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0026] The present application provides a three-dimensional reconstruction method of the intestinal tract based on the optical flow method, device, equipment and medium, which obtains the depth data of each pixel point by using a deep learning method to estimate the depth of each monocular intestinal tract image; obtains two-dimensional intestinal tract contour data by extracting the intestinal tract contour of the monocular intestinal tract image; calculates the optical flow field by analyzing the pixel motion information in the monocular intestinal tract image using the optical flow method, and calculates the global pose of the endoscope according to the optical flow field; and finally realizes the reconstruction of the three-dimensional model of the intestinal tract according to the corresponding depth data, two-dimensional intestinal tract contour data and global pose of the endoscope as the endoscope slowly advances. The method can be stably applied in a dynamic endoscope motion environment, is not limited by lighting conditions and significant feature points, and ensures that the three-dimensional reconstruction of the intestinal tract can be effectively performed in a low-texture area. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0028] Figure 1 The application environment diagram of the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the embodiment 1 of the present application;

[0029] Figure 2 The flowchart of the three-dimensional reconstruction method of the intestinal tract based on the optical flow method provided in the embodiment 1 of the present application;

[0030] Figure 3 The conceptual diagram of the three-dimensional reconstruction method of the intestinal tract based on the optical flow method provided in the embodiment 1 of the present application;

[0031] Figure 4 The motion tracking diagram of the optical flow method in the image sequence in the embodiment 1 of the present application;

[0032] Figure 5 Schematic diagram of the mapping process from world coordinates to images in Example 1 of the present application;

[0033] Figure 6 A schematic diagram of the structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] The intestinal 3D reconstruction method based on optical flow provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the monocular intestinal image to be processed to the server 104. After the server 104 receives the monocular intestinal image to be processed, the server 104 will use a deep learning method to perform depth estimation on each of the monocular intestinal images to obtain the depth data of each pixel in each of the monocular intestinal images; extract the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data; obtain local three-dimensional point cloud data of the intestine based on the depth data and the corresponding two-dimensional intestinal contour data; use the optical flow method to analyze the pixel motion information in the monocular intestinal image, calculate the optical flow field, and calculate the global pose of the endoscope based on the optical flow field; use the global pose of the endoscope to convert the local three-dimensional point cloud data of the intestine into global three-dimensional point cloud data of the intestine; reconstruct the three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine of all frames. The server 104 can feed back the obtained 3D model of the entire intestine to the terminal 102. Furthermore, in some embodiments, the optical flow-based 3D intestinal reconstruction method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the monocular intestinal image to be processed using the optical flow-based 3D intestinal reconstruction method. Alternatively, the server 104 can obtain the monocular intestinal image to be processed from a data storage system and process it using the optical flow-based 3D intestinal reconstruction method.

[0037] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by a single server or a server cluster composed of multiple servers, and can also be a cloud server.

[0038] In an exemplary embodiment, as shown in Figure 2 A three-dimensional reconstruction method of an intestinal tract based on an optical flow method is provided, which is executed by a computer device, specifically, can be executed by a terminal or a server, or both. In the embodiments of the present application, the method is applied to the server 104 in the system 100, and includes the following steps 201 to 207. In the embodiments of the present application, the method is applied to the server 104 in the system 100, and includes the following steps 201 to 207. Figure 1

[0039] Step 201, real-time acquisition of monocular intestinal tract images, wherein the monocular intestinal tract images are images captured by an endoscope during the advancement of the endoscope in the intestinal tract.

[0040] Step 202, depth estimation of each monocular intestinal tract image by using a deep learning algorithm to obtain depth data of each pixel point in each monocular intestinal tract image.

[0041] Step 202-1, input of each monocular intestinal tract image into a pre-trained depth estimation model to obtain depth data of each pixel point in each monocular intestinal tract image, wherein the depth estimation model is a convolutional neural network model.

[0042] Step 203, intestinal tract contour extraction of each monocular intestinal tract image to obtain two-dimensional intestinal tract contour data.

[0043] Step 203-1, denoising of each monocular intestinal tract image to obtain a denoised monocular intestinal tract image.

[0044] Step 203-1, binarization of the denoised monocular intestinal tract image to obtain a binarized image.

[0045] Step 203-1, edge detection of the binarized image, and taking points on the largest contour as two-dimensional intestinal tract contour data.

[0046] Step 204, intestinal tract local three-dimensional point cloud data obtained according to the depth data and the corresponding two-dimensional intestinal tract contour data.

[0047] ​Step 205 : Analyze the pixel motion information in the monocular intestinal image using an optical flow method, calculate the optical flow field, and calculate the global pose of the endoscope based on the optical flow field.

[0048] Step 205 - 1 , performing Taylor expansion on the brightness consistency formula to obtain a Taylor expansion formula, wherein the brightness consistency formula is used to characterize that the brightness value of a pixel at the same position in two consecutive frames of the monocular intestinal image remains unchanged.

[0049] Step 205 - 2 : Combine the brightness consistency formula and the Taylor expansion formula to solve the optical flow vector of each pixel on the monocular intestinal image.

[0050] Step 205-3, using the least squares method to solve the optical flow equations to obtain the endoscopic motion information of each frame of the monocular intestinal image, wherein the optical flow equations include several optical flow equations, each of which is used to characterize the relationship between the optical flow vector of each pixel point and the endoscopic motion information, and the endoscopic motion information includes translation and rotation.

[0051] Step 205 - 4 , using the endoscope motion information of each frame of monocular intestinal image as the endoscope motion information of the current frame, and calculating the global pose of the endoscope based on the endoscope motion information of the current frame and the endoscope motion information of the previous frame of monocular intestinal image.

[0052] Step 206 , using the global posture of the endoscope, converting the intestinal local 3D point cloud data into intestinal global 3D point cloud data;

[0053] Step 207 : reconstructing a three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine in all frames.

[0054] It should be noted that the execution order of the above steps 201-207 is only the execution order of a specific example of the intestinal three-dimensional reconstruction method based on the optical flow method. This embodiment does not specifically limit the execution order of these steps. It is sufficient as long as the corresponding depth data, two-dimensional intestinal contour data, local intestinal three-dimensional point cloud data and endoscope global pose are obtained before reconstructing the three-dimensional model of the intestine.

[0055] In order to make those skilled in the art more clear about the technical concept of the intestinal 3D reconstruction method based on the optical flow method in this embodiment, the following Figure 3 Provide specific explanation.

[0056] like Figure 3 The conceptual diagram of the intestinal 3D reconstruction method based on the optical flow method shown in FIG. 1 , the method includes the following steps:

[0057] Step S1, depth estimation of monocular images (i.e. monocular intestinal images) captured by an endoscope is performed by a deep learning method. In this embodiment, a convolutional neural network (CNN) model is used to predict the depth of each pixel in the monocular image, so that the depth information can be predicted from a single frame of image to generate a depth map.

[0058] Specifically, in this embodiment, depth prediction is performed based on a depth estimation model DepthFM. DepthFM is a fast and accurate monocular depth estimation method that realizes direct mapping from input images to depth maps through flow matching technology. This method not only improves efficiency, but also ensures the quality of the generated depth map. In practical applications, the monocular images captured by the endoscope are input into the DepthFM model, and the corresponding depth map is obtained through the forward inference process.

[0059] Step S2, the pixel motion between the two consecutive frames of images captured by the endoscope is analyzed by the optical flow method. Then, according to the calculated optical flow field, the endoscope motion information is calculated, including translation and rotation.

[0060] Figure 4 The motion tracking diagram of the optical flow method in the image sequence shows the motion process of a certain point in the image at multiple times. The optical flow method calculates the pixel motion vector between consecutive frames, as follows:

[0061] Assuming that the previous frame time is t and the next frame time is t+δt, then the pixel point I(x, y, t) of the previous frame I is located at I(x+δx, y+δy, z+δz) in the next frame. According to the assumption that the two frames satisfy the constant gray value, it can be obtained that:

[0062] I(x, y, t) = I(x+δx, y+δy, t+δt).

[0063] The right side of the above equation is expanded by Taylor series to obtain:

[0064]

[0065] where is the partial derivative of I(x, y, t) with respect to x, y, and z; H.O.T is the high-order term of the Taylor series expansion, which can be ignored.

[0066] Solving the above two equations, the simplified equation is obtained by taking the derivative of both sides with respect to t and dividing by dt:

[0067]

[0068] where, and are the velocity vectors of the optical flow along the x-axis and y-axis, respectively, and Represent the partial derivatives of the grayscale of the pixel points in the image along the X-axis and Y-axis respectively, It can be obtained from the monocular intestinal image captured by the endoscope, (v x , v y ) is the desired optical flow vector.

[0069] The vector field (i.e., optical flow vector) obtained by optical flow method is analyzed to deduce the endoscope motion information. Assume that the endoscope has a translation T calibrate =[T x T y T z ] T and rotate R calibrate =[ω x ω y ω z ] T .

[0070] The optical flow on the image can be described by the motion of the endoscope (translation and rotation matrices), satisfying the following relationship (i.e., the expression of the optical flow equation):

[0071]

[0072] Among them, v x (x, y) and v y (x, y) represents the optical flow vector (v x , v y ) of the component; f x Indicates the component of the camera's focal length in the x-axis direction; f y Represents the component of the camera's focal length in the y-axis direction; ω x represents the angular velocity of the endoscope around the x-axis; ω y represents the angular velocity of the endoscope around the y-axis; ω z represents the angular velocity of the endoscope around the z axis; T x represents the displacement of the endoscope along the x-axis; T y represents the displacement of the endoscope along the y-axis; T z represents the displacement of the endoscope along the z-axis; z(x, y) is the depth value of the pixel.

[0073] For each pixel, the above optical flow equation can be used to construct an optical flow equation group to express the relationship between the motion information of the pixel and the endoscope posture (i.e., the endoscope motion information):

[0074] A·p=b;

[0075] Among them, A is the coefficient matrix related to the optical flow equation, p is the endoscope motion information to be solved, and b is the calculated optical flow vector.

[0076] The motion information (translation T calibrate and rotation R calibrate ) between two frames of endoscope is solved by least square method. The endoscope pose of the first monocular enteroscopy image (i.e., the endoscope motion information) is known (initialized R0 and T0), and through frame-by-frame updating, the motion information of the endoscope in the global space (i.e., the global pose of the endoscope) can be obtained:

[0077] R global = R previous · R calibrate ;

[0078] T global = T previous + R previbus · T calibrate ;

[0079] wherein T previous and R previous are the pose matrix of the previous frame of endoscope (i.e., the endoscope motion information of the previous frame of monocular enteroscopy image).

[0080] Step S3, a series of processing operations are performed on the monocular enteroscopy image taken by the endoscope, including reducing noise using Gaussian blur, dividing the image into foreground and background by threshold segmentation, then applying a contour detection algorithm to extract the intestinal contour, and finally storing the detected contour in an array to obtain two-dimensional intestinal contour data. Specifically, the following steps are included:

[0081] S31, first, Gaussian blur processing is performed on the monocular enteroscopy image to reduce image noise and details, which helps subsequent edge detection and contour extraction. Gaussian blur achieves a smoothing effect by using a Gaussian function as a filter to perform a neighborhood weighted average on each pixel value in the image.

[0082] S32, the Gaussian blurred image is binarized, a threshold value is determined through experiments, and threshold segmentation is performed using this threshold value to divide the image into foreground and background two parts: the area above the threshold value is identified as foreground and displayed as white; while the area below the threshold value is considered as background and displayed as black. Then, color inversion processing is performed on the binarized image to more obviously identify the region of interest.

[0083] S33, contour detection is performed in the binarized image, and after identifying all contours, the largest contour is found, which usually represents the main area of the intestine. All point coordinates of this largest contour are saved as a two-dimensional array for subsequent processing and analysis.

[0084] Step S4, combine each two-dimensional pixel point on the contour (i.e., two-dimensional intestinal contour data) with its corresponding depth value, convert these points to three-dimensional space coordinates through the intrinsic matrix of the camera, and generate intestinal local three-dimensional point cloud data.

[0085] Specifically, the two-dimensional intestinal contour data obtained in step S3, combined with the depth data of the corresponding points obtained in step S1, can generate point cloud in three-dimensional space (i.e., intestinal local three-dimensional point cloud data). Figure 5 The mapping process from the world coordinate to the image is shown in the schematic diagram. Assuming that a point P = [x w , y w , z w ] T in the world coordinate system, through the camera model, it can be projected onto the image plane of the camera, and the corresponding point in the camera coordinate system is P c = [x c , y c , z c ] T . According to the projection relationship, for the two-dimensional coordinates (u, v) of each point on the contour to the point cloud X point_cloud = (x point_cloud , y point_clud , z point_cloud ) T , there is the following transformation formula:

[0086]

[0087] Where z c is the depth value of the point, f x , f y , u0, v0 can be obtained by camera calibration.

[0088] Step S5, use the motion information (translation and rotation matrix) of the endoscope to convert the point cloud data of the current frame to the global coordinate system. With the slow advancement of the endoscope, new data is continuously collected, and the newly generated point cloud is gradually integrated and updated with the existing three-dimensional model. Finally, through frame-by-frame accumulation, a complete three-dimensional model is formed.

[0089] Specifically, the global pose R global and T global of the endoscope are used to convert the local point cloud to the global coordinate system. The local point cloud is converted to the global coordinate system, and each point cloud (x point_cloud , y point_cloud , z point_cloud ) T is transformed by the following formula:

[0090]

[0091] Where (xglobal , y global , z global ) are three-dimensional coordinates of the global three-dimensional point cloud data of the intestine; X point_cloud = (x point_cloud , y point_cloud , z point_cloud ) are three-dimensional coordinates of the local three-dimensional point cloud data of the intestine; R global is rotation data in the global pose of the endoscope; T global is translation data in the global pose of the endoscope.

[0092] As the endoscope continues to advance slowly, new data is continuously collected, and the newly generated point cloud is gradually integrated and updated with the existing three-dimensional model, ultimately realizing the three-dimensional reconstruction of the entire intestine.

[0093] The deep learning method of the embodiment effectively extracts depth information in the image by training the model, and is suitable for the scene of endoscope movement. The optical flow method infers the dynamic changes of the endoscope by analyzing the pixel motion between consecutive frames, which makes it not affected by specific illumination patterns and focal point information, so it can still maintain stable performance in the scene of endoscope movement.

[0094] Compared with the prior art, the three-dimensional reconstruction method of the intestine based on the optical flow method provided in the embodiment can be stably applied in a dynamic endoscope movement environment, is not limited by illumination conditions and significant feature points, and ensures that effective three-dimensional reconstruction of the intestine can also be performed in low-texture areas. The method extracts depth information and intestinal contours from images captured by the endoscope, combines optical flow technology, realizes three-dimensional reconstruction of the intestine, and has more universal applicability than other methods.

[0095] The application also provides an application scenario applying the three-dimensional reconstruction method of the intestine based on the optical flow method. Specifically, the three-dimensional reconstruction method of the intestine based on the optical flow method provided in the embodiment can be applied in the scene of medical image diagnosis. The scene includes an intestinal data acquisition link, an image processing and reconstruction link, and a clinical analysis and evaluation link. The three-dimensional reconstruction method of the intestine based on the optical flow method provided in the embodiment is a key technology in the image processing and reconstruction link. The method can efficiently and accurately reconstruct the three-dimensional structure of the intestine, help doctors accurately evaluate the lesion site and degree of the intestine, and thus provide important support in the diagnosis, surgical planning and postoperative evaluation of intestinal diseases.

[0096] Embodiment 2

[0097] The embodiment also provides a three-dimensional reconstruction device of the intestine based on the optical flow method, comprising:

[0098] An acquisition module is configured to acquire monocular intestinal images in real time, wherein the monocular intestinal images are images captured by an endoscope during intestinal advancement.

[0099] A depth estimation module is configured to perform depth estimation on each monocular intestinal image by using a deep learning algorithm to obtain depth data of each pixel point in each monocular intestinal image.

[0100] An intestinal contour extraction is performed on each monocular intestinal image to obtain two-dimensional intestinal contour data.

[0101] An intestinal local three-dimensional point cloud data calculation module is configured to obtain intestinal local three-dimensional point cloud data according to the depth data and corresponding two-dimensional intestinal contour data.

[0102] An endoscope global pose calculation module is configured to analyze pixel motion information in the monocular intestinal images by using an optical flow method, calculate an optical flow field, and calculate an endoscope global pose according to the optical flow field.

[0103] An intestinal global three-dimensional point cloud data calculation module is configured to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data by using the endoscope global pose.

[0104] A three-dimensional model reconstruction module is configured to reconstruct a three-dimensional model of the entire intestine according to the intestinal global three-dimensional point cloud data of all frames.

[0105] The implementation scheme for solving the problem provided by the intestinal three-dimensional reconstruction device based on the optical flow method provided in the embodiments is similar to the implementation scheme described in the above method. The specific limitations of the device can be referred to the limitations of the intestinal three-dimensional reconstruction method based on the optical flow method in Embodiment 1, which will not be described here.

[0106] The embodiment provides an intestinal three-dimensional reconstruction device based on an optical flow method. The device performs depth estimation on monocular intestinal images by using a deep learning method. The optical flow method is used to analyze the pixel motion between two consecutive endoscopic images (i.e., monocular intestinal images), calculate an optical flow field, and derive endoscope motion information, including a translation matrix and a rotation matrix. Then, intestinal contour extraction is performed on the endoscopic images, and two-dimensional intestinal contour data is converted into intestinal local three-dimensional point cloud data. Finally, the intestinal local three-dimensional point cloud data of the current frame is converted to a global coordinate system by using the endoscope motion information. By using the frame-by-frame accumulation method, a three-dimensional model of the intestine is reconstructed as the endoscope slowly advances. The device can be stably applied in a dynamic endoscope motion environment, is not limited by lighting conditions and significant feature points, and ensures that effective intestinal three-dimensional reconstruction can be performed in low-texture areas.

[0107] Embodiment 3

[0108] The embodiment provides a computer device which can be a server or a terminal, and an internal structure diagram of the computer device can be as shown in the figure. Figure 6 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the three-dimensional reconstruction method of an intestinal tract based on an optical flow method. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the embodiment 1.

[0109] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the method embodiments described above.

[0110] Embodiment 4

[0111] The embodiment provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the embodiment 1.

[0112] Embodiment 5

[0113] The embodiment provides a computer program product including a computer program, and the computer program is executed by a processor to implement the three-dimensional reconstruction method of the intestinal tract based on the optical flow method in the embodiment 1.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with relevant regulations.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0116] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0117] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered as the scope of the present application.

[0118] The principles and implementation manners of the present application are described herein by using specific examples, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for 3D reconstruction of the intestine based on optical flow method, characterized in that: The intestinal 3D reconstruction method based on the optical flow method includes: Acquire a monocular intestinal image in real time, wherein the monocular intestinal image is an image captured by the endoscope during the process of advancing through the intestine; Using a deep learning algorithm to perform depth estimation on each of the monocular intestinal images to obtain depth data of each pixel in each of the monocular intestinal images; Extracting the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data; Obtaining local three-dimensional point cloud data of the intestine according to the depth data and the corresponding two-dimensional intestinal contour data; An optical flow method is used to analyze pixel motion information in the monocular intestinal image, calculate an optical flow field, and calculate the global pose of the endoscope based on the optical flow field; Using the global posture of the endoscope, converting the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data; reconstructing a three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine in all frames; The optical flow method is used to analyze the pixel motion information in the monocular intestinal image, calculate the optical flow field, and calculate the global pose of the endoscope based on the optical flow field, specifically including: Taylor expansion is performed on the brightness consistency formula to obtain a Taylor expansion formula, wherein the brightness consistency formula is used to characterize that the brightness value of a pixel point at the same position in two consecutive frames of the monocular intestinal image remains unchanged; Combining the brightness consistency formula and the Taylor expansion formula to solve the optical flow vector of each pixel on the monocular intestinal image; Solving the optical flow equations using the least squares method to obtain endoscope motion information for each frame of the monocular intestinal image, wherein the optical flow equations include a plurality of optical flow equations, each of which is used to characterize the relationship between the optical flow vector of each pixel point and the endoscope motion information, wherein the endoscope motion information includes translation and rotation; The expression of the optical flow equation is: Among them, v x (x,y) and v y (x,y) represents the optical flow vector (v x ,v y ) of the component; f x Indicates the component of the camera's focal length in the x-axis direction; f y Represents the component of the camera's focal length in the y-axis direction; ω x represents the angular velocity of the endoscope around the x-axis; ω y represents the angular velocity of the endoscope around the y-axis; ω z represents the angular velocity of the endoscope around the z axis; T x represents the displacement of the endoscope along the x-axis; T y represents the displacement of the endoscope along the y-axis; T z represents the displacement of the endoscope along the z-axis; z(x, y) is the depth value of the pixel point (x, y); The endoscope motion information of each frame of monocular intestinal image is used as the endoscope motion information of the current frame, and the global pose of the endoscope is calculated based on the endoscope motion information of the current frame and the endoscope motion information of the previous frame of monocular intestinal image.

2. The method for 3D reconstruction of the intestine based on optical flow method according to claim 1, characterized in that: A deep learning algorithm is used to perform depth estimation on each monocular intestinal image to obtain depth data of each pixel in each monocular intestinal image, specifically including: Each monocular intestinal image is used as input, and a pre-trained depth estimation model is used to obtain the depth data of each pixel in each monocular intestinal image, wherein the depth estimation model is a convolutional neural network model.

3. The method for intestinal 3D reconstruction based on optical flow method according to claim 1, characterized in that: Performing intestinal contour extraction on each monocular intestinal image to obtain two-dimensional intestinal contour data specifically includes: De-noising each monocular intestinal image to obtain a denoised monocular intestinal image; performing binarization processing on the denoised monocular intestinal image to obtain a binarized image; Edge detection is performed on the binary image, and points on the maximum contour are used as two-dimensional intestinal contour data.

4. The method for 3D reconstruction of the intestine based on optical flow method according to claim 1, characterized in that: The calculation formula for the intestinal local 3D point cloud data is: Among them, X point_cloud =(x point_cloud ,y point_cloud , z point_cloud ) is the three-dimensional coordinate of the intestinal local three-dimensional point cloud data; z c is the depth data of the pixel point; (u, v) is the two-dimensional coordinate of the corresponding two-dimensional intestinal contour data; f x ,f y ,u0,v0 are camera calibration data.

5. The method for 3D reconstruction of the intestine based on optical flow method according to claim 1, characterized in that: The calculation formula of the intestinal global three-dimensional point cloud data is: Among them, (x global ,y global ,z global ) is the three-dimensional coordinate of the global three-dimensional point cloud data of the intestine; (x point_cloud ,y point_cloud , z point_cloud ) is the three-dimensional coordinate of the intestinal local three-dimensional point cloud data; R global is the rotation data in the global pose of the endoscope; T global is the translation data in the global pose of the endoscope.

6. A three-dimensional intestinal reconstruction device based on optical flow method, used to implement the three-dimensional intestinal reconstruction method based on optical flow method according to any one of claims 1 to 5, characterized in that: The intestinal 3D reconstruction device based on the optical flow method includes: an acquisition module, configured to acquire a monocular intestinal image in real time, wherein the monocular intestinal image is an image captured by the endoscope during the process of advancing into the intestine; a depth estimation module, configured to perform depth estimation on each monocular intestinal image using a deep learning algorithm to obtain depth data of each pixel in each monocular intestinal image; Extracting the intestinal contour of each monocular intestinal image to obtain two-dimensional intestinal contour data; A three-dimensional point cloud data calculation module for the intestine, configured to obtain three-dimensional point cloud data of the intestine based on the depth data and the corresponding two-dimensional intestinal contour data; an endoscope global pose calculation module, configured to analyze pixel motion information in the monocular intestinal image using an optical flow method, calculate an optical flow field, and calculate the endoscope global pose based on the optical flow field; an intestinal global three-dimensional point cloud data calculation module, configured to convert the intestinal local three-dimensional point cloud data into intestinal global three-dimensional point cloud data using the endoscope global posture; The three-dimensional model reconstruction module is used to reconstruct a three-dimensional model of the entire intestine based on the global three-dimensional point cloud data of the intestine in all frames.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intestinal three-dimensional reconstruction method based on the optical flow method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intestinal three-dimensional reconstruction method based on the optical flow method described in any one of claims 1 to 5 is implemented.

Images