Cavity branch point detection method and device, electronic equipment and readable storage medium

Abstract

The present application provides a kind of cavity branch point detection method, device, electronic equipment and readable storage medium, the method comprises: by camera, obtain the two-dimensional virtual endoscope image of containing cavity branch point of pre-set quantity of pictures;In two-dimensional virtual endoscope image, select optimal two-dimensional virtual endoscope image;The pose of the camera corresponding to optimal two-dimensional virtual endoscope image is obtained;Two-dimensional depth image corresponding to pose is obtained;Determine target pixel on two-dimensional depth image;According to pose and target pixel, the coordinate of target cavity branch point is determined.Thereby, the coordinate of target cavity branch point can be accurately determined, can provide feature road sign in the registration based on anatomical features in virtual navigation and real-time navigation, to obtain suitable path, to guide instrument from cavity entrance to target (lesion) area through cavity, to assist medical diagnosis or the performance of operation.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of medical technology, and in particular to a method and device for detecting a branch point of a cavity, an electronic device, and a readable storage medium. BACKGROUND

[0002] Vascular reticular branch cavities and tracheal tree branch cavities are typical topological structures of human cavities. Due to the large number of branches and the complex bypass, a suitable path needs to be found to guide the instrument to reach the target (lesion) area during clinical diagnosis and treatment.

[0003] The existing scheme is usually as follows: first, three-dimensional CT (Computed Tomography) or MRI (Magnetic resonance imaging) scanning is performed to obtain three-dimensional anatomical image data of a target part; second, each tissue component (for example, by processing chest three-dimensional CT data from the outside to the inside, the skin, bones, lung area, blood vessels, trachea, etc. can be sequentially segmented) in the three-dimensional anatomical image data is extracted by image segmentation to obtain a vascular reticular branch cavity or a tracheal tree branch cavity; then, a skeleton network or a skeleton tree of the cavity is obtained by a tubular structure thinning and centerline extraction method; finally, the skeleton is analyzed by a graph theory analysis method to automatically extract branch (bifurcation) points on the skeleton line. The position of the branch point can provide a feature landmark for registration based on anatomical features in virtual navigation and real-time navigation, so as to obtain a suitable path to guide the instrument to reach the target (lesion) area from the cavity entrance through the cavity to assist medical diagnosis or surgery.

[0004] However, in the above scheme, since the extracted tubular cavity structure skeleton line is a topological representation, the determined branch point is not an actual anatomical structure, and there is an uncertain distance between the branch point and the actual anatomical structure inner surface (observed by an endoscope), and the two branch points cannot coincide. Therefore, the above scheme has the following disadvantages: the determined branch point is not accurate, thereby affecting the subsequent path determination, and is not conducive to medical diagnosis or surgery. SUMMARY

[0005] Embodiments of the present application provide a method and device for detecting a branch point of a cavity, an electronic device, and a readable storage medium to solve the technical problem that the determined branch point is not accurate in the related art, thereby affecting the subsequent path determination and being not conducive to medical diagnosis or surgery.

[0006] To solve the above technical problem, the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide a method for detecting a branch point of a cavity, and the method comprises the following steps:

[0008] acquiring a preset number of two-dimensional virtual endoscope images containing the cavity branch point through the camera;

[0009] selecting an optimal two-dimensional virtual endoscope image in the two-dimensional virtual endoscope images; wherein the difference between the largest branch opening area and the second largest branch opening area in the optimal two-dimensional virtual endoscope image is the smallest;

[0010] acquiring the pose of the camera corresponding to the optimal two-dimensional virtual endoscope image;

[0011] acquiring a two-dimensional depth image corresponding to the pose;

[0012] determining a target pixel point on the two-dimensional depth image; wherein the target pixel point is the point with the smallest pixel gray value on the mapped line after mapping the line connecting the largest branch opening and the second largest branch opening to the two-dimensional depth image;

[0013] determining the coordinates of the target cavity branch point according to the pose and the target pixel point.

[0014] Preferably, acquiring a preset number of two-dimensional virtual endoscope images containing the cavity branch point through the camera comprises:

[0015] determining the position of the cavity topology skeleton branch point;

[0016] acquiring a preset number of two-dimensional virtual endoscope images containing the cavity branch point through the camera within a preset range of the cavity topology skeleton branch point.

[0017] Preferably, acquiring a preset number of two-dimensional virtual endoscope images containing the cavity branch point through the camera comprises:

[0018] acquiring a preset number of two-dimensional virtual endoscope images containing the cavity branch point according to the user's fine adjustment operation and shooting instruction of the camera;

[0019] wherein the fine adjustment operation is an operation in which the user controls the camera to move within a preset range by a preset amplitude; and after each fine adjustment operation of the user, the camera acquires a two-dimensional virtual endoscope image according to the shooting instruction.

[0020] Preferably, selecting an optimal two-dimensional virtual endoscope image in the two-dimensional virtual endoscope images comprises:

[0021] performing image segmentation processing on each two-dimensional virtual endoscope image to obtain the branch opening of each two-dimensional virtual endoscope image;

[0022] determine a difference between a largest branch opening area and a second largest branch opening area of each of the two-dimensional virtual endoscope images respectively;

[0023] compare the differences;

[0024] determine the two-dimensional virtual endoscope image corresponding to the minimum value in the differences as an optimal two-dimensional virtual endoscope image.

[0025] Preferably, determining the target pixel point on the two-dimensional depth image comprises:

[0026] determine a closed region of the largest branch opening and a closed region of the second largest branch opening in the optimal two-dimensional virtual endoscope image respectively;

[0027] determine a centroid of the closed region of the largest branch opening and a centroid of the closed region of the second largest branch opening respectively;

[0028] connect the centroid of the closed region of the largest branch opening and the centroid of the closed region of the second largest branch opening to form a first connecting line;

[0029] map the first connecting line to a two-dimensional depth image corresponding to the optimal two-dimensional virtual endoscope image to determine a second connecting line;

[0030] determine a point with a minimum pixel gray value on the second connecting line as the target pixel point.

[0031] Preferably, determining the coordinates of the target lumen branch point according to the pose and the target pixel point comprises:

[0032] determine a pixel gray value of the target pixel point;

[0033] determine the coordinates of the target lumen branch point by the following formula:

[0034]

[0035] wherein, the (p', d') is the coordinates of the target lumen branch point, the (p, d) is the pose; p is a position, d is a direction, the pose is a pose corresponding to the optimal two-dimensional virtual endoscope image and the two-dimensional depth image; the Depth is the pixel gray value.

[0036] Preferably, determining the position of the lumen topology skeleton branch point comprises:

[0037] scan a to-be-processed region by using an image scanning technology to obtain a three-dimensional anatomical image of the to-be-processed region;

[0038] perform image segmentation processing on the three-dimensional anatomical image to obtain a target lumen.

[0039] extracting a skeleton centerline of the target lumen tract;

[0040] detecting branch points of the skeleton centerline based on a graph analysis method to determine positions of lumen tract topology skeleton branch points.

[0041] Preferably, the camera is a virtual camera.

[0042] In a second aspect, an embodiment of the present application provides a device for detecting lumen branch points, the device comprising:

[0043] a first obtaining module configured to obtain a preset number of two-dimensional virtual endoscope images containing lumen branch points by using a camera;

[0044] a selecting module configured to select an optimal two-dimensional virtual endoscope image from the two-dimensional virtual endoscope images; wherein a difference between a largest branch opening area and a second largest branch opening area in the optimal two-dimensional virtual endoscope image is the smallest;

[0045] a second obtaining module configured to obtain a pose of the camera corresponding to the optimal two-dimensional virtual endoscope image;

[0046] a third obtaining module configured to obtain a two-dimensional depth image corresponding to the pose;

[0047] a first determining module configured to determine a target pixel point on the two-dimensional depth image; wherein the target pixel point is a point with the smallest pixel gray value on a mapped line after mapping a line connecting the largest branch opening and the second largest branch opening to the two-dimensional depth image;

[0048] a second determining module configured to determine a coordinate of a target lumen branch point according to the pose and the target pixel point.

[0049] Preferably, the first obtaining module is further configured to determine a position of a lumen tract topology skeleton branch point, and obtain a preset number of two-dimensional virtual endoscope images containing lumen branch points by using a camera within a preset range of the lumen tract topology skeleton branch point;

[0050] Preferably, the first obtaining module is further configured to obtain a preset number of two-dimensional virtual endoscope images containing lumen branch points according to a fine-tuning operation and a shooting instruction of a user on the camera;

[0051] wherein the fine-tuning operation is an operation of the user controlling the camera to move within a preset range by a preset amplitude, and the camera obtains a two-dimensional virtual endoscope image according to the shooting instruction after each fine-tuning operation of the user.

[0052] Preferably, the selecting module is further configured to perform image segmentation processing on each two-dimensional virtual endoscope image to obtain a branch opening of each two-dimensional virtual endoscope image;

[0053] determining a difference between the maximum branch opening area and the second largest branch opening area of each two-dimensional virtual endoscope image, respectively;

[0054] comparing the difference values;

[0055] determining the two-dimensional virtual endoscope image corresponding to the minimum value in the difference values as an optimal two-dimensional virtual endoscope image.

[0056] Preferably, the first determining module is further configured to determine a closed region of a branch opening with the largest area and a closed region of a branch opening with the second largest area in the optimal two-dimensional virtual endoscope image, respectively;

[0057] determining a centroid of the closed region of the branch opening with the largest area and a centroid of the closed region of the branch opening with the second largest area, respectively;

[0058] connecting the centroid of the closed region of the branch opening with the largest area and the centroid of the closed region of the branch opening with the second largest area to form a first connecting line;

[0059] mapping the first connecting line to a two-dimensional depth image corresponding to the optimal two-dimensional virtual endoscope image to determine a second connecting line;

[0060] determining a point with the smallest pixel gray value on the second connecting line as a target pixel point.

[0061] Preferably, the second determining module is further configured to determine the pixel gray value of the target pixel point; and determine a coordinate of a target lumen branch point by the following formula:

[0062]

[0063] wherein, the (p', d') is the coordinate of the target lumen branch point, the (p, d) is the pose; p is a position, d is a direction, the pose is a pose corresponding to the optimal two-dimensional virtual endoscope image and the two-dimensional depth image; and the Depth is the pixel gray value.

[0064] Preferably, the first obtaining module is further configured to scan a to-be-processed region by using an image scanning technology to obtain a three-dimensional anatomical image of the to-be-processed region;

[0065] performing image segmentation processing on the three-dimensional anatomical image to obtain a target lumen;

[0066] extracting a skeleton center line of the target lumen;

[0067] The skeleton centerline is subjected to branch point detection based on a graph analysis method to determine the positions of the cavity channel topological skeleton branch points.

[0068] Preferably, the camera is a virtual camera.

[0069] In a third aspect, an electronic device is provided, which comprises a processor, a memory, and a program stored in the memory and executable on the processor, and the program, when executed by the processor, implements the steps of the method for detecting a cavity channel branch point according to the first aspect.

[0070] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the method for detecting a cavity channel branch point according to the first aspect.

[0071] In the embodiments of the present application, the coordinates of the target cavity channel branch point are calculated by using the two-dimensional virtual endoscope image and the two-dimensional depth image and the pose of the camera corresponding thereto, and the image is more accurate in restoring and presenting the natural cavity of the human body than the topological structure. Therefore, compared with the prior art, the coordinates of the target cavity channel branch point (the coordinates of the branch points of the human tree network cavity anatomical structure) can be accurately determined, and by automatically extracting and marking the positions of the branches of the human tree network cavity anatomical structure, feature road signs can be provided for registration based on anatomical features in virtual navigation and real-time navigation, so as to obtain a suitable path to guide the instrument from the cavity entrance to the target (lesion) area through the cavity, so as to assist the medical diagnosis or the performance of the surgery. BRIEF DESCRIPTION OF DRAWINGS

[0072] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Moreover, the same reference numerals are used throughout the same figures. In the drawings:

[0073] Figure 1 A flowchart of a method for detecting a cavity channel branch point according to an embodiment of the present application is provided;

[0074] Figure 2 A schematic diagram of a tracheal topological skeleton branch point and a tracheal anatomical structure branch point according to an embodiment of the present application is provided;

[0075] Figure 3 A flowchart of a method for detecting a cavity channel branch point according to an embodiment of the present application is provided;

[0076] Figure 4A flow chart of a cavity branch point detection method provided by an embodiment of the present application is shown in the figure;

[0077] Figure 5 A structural block diagram of a cavity branch point detection device provided by an embodiment of the present application is shown in the figure;

[0078] Figure 6 A structural block diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0080] Figure 1 A flow chart of a cavity branch point detection method according to an embodiment of the present application is shown in the figure, Figure 1 As shown in the figure, the method comprises:

[0081] Step S101: obtaining a preset number of two-dimensional virtual endoscope images containing cavity branch points through a camera;

[0082] Step S102: selecting an optimal two-dimensional virtual endoscope image in the two-dimensional virtual endoscope images;

[0083] In the optimal two-dimensional virtual endoscope image, the difference between the maximum branch opening area and the second maximum branch opening area is the smallest.

[0084] Step S103: obtaining the pose of the camera corresponding to the optimal two-dimensional virtual endoscope image;

[0085] Step S104: obtaining a two-dimensional depth image corresponding to the pose;

[0086] Step S105: determining a target pixel point on the two-dimensional depth image;

[0087] In the optimal two-dimensional virtual endoscope image, the difference between the maximum branch opening area and the second maximum branch opening area is the smallest.

[0088] Step S106: determining the coordinates of the target cavity branch point according to the pose and the target pixel point.

[0089] It should be noted that the embodiments of the present application can be applied to the scene of human natural cavity three-dimensional image processing and surgical navigation, and the human natural cavity can be a blood vessel cavity or a bronchial cavity. Preferably, the camera is a virtual camera.

[0090] It should be noted that in step S101, the preset number of two-dimensional virtual endoscope images containing the branch points of the cavity can be directly obtained by the camera. Preferably, the preset number of two-dimensional virtual endoscope images containing the branch points of the cavity can be obtained according to the fine adjustment operation of the user on the camera and the shooting instruction; wherein the fine adjustment operation is an operation in which the user controls the camera to move by a preset amplitude within a preset range; and after each fine adjustment operation of the user, the camera obtains a two-dimensional virtual endoscope image according to the shooting instruction. Wherein the branch point of the cavity is also the target branch point of the cavity, that is, the branch point of the cavity anatomical structure.

[0091] In one possible implementation, step S101, obtaining, by the camera, the preset number of two-dimensional virtual endoscope images containing the branch points of the cavity, can further include: determining the position of the branch point of the cavity topology skeleton, and obtaining, by the camera, the preset number of two-dimensional virtual endoscope images containing the branch points of the cavity within a preset range of the branch point of the cavity topology skeleton.

[0092] That is, first, based on the prior art scheme, the position of the branch point of the cavity topology skeleton is determined, which has a certain distance from the branch point of the actual anatomical structure inner surface (observed by the endoscope), as shown in Figure 2 As shown in Figure 2 A schematic diagram of a trachea topology skeleton branch point A and a trachea anatomical structure branch point A' according to an embodiment of the present application), and the branch point of the actual anatomical structure inner surface is the accurate branch point, therefore, the idea of the above implementation is how to determine the coordinates of the target cavity branch point based on the position of the branch point of the cavity topology skeleton. Further, a combination of graph theory and endoscopic image processing technology is used to automatically obtain the branch features of the human tree network cavity, and to extract the actual three-dimensional anatomical branch point, which can provide feature landmarks for registration based on anatomical features in virtual navigation and real-time navigation.

[0093] Based on the above invention idea, determining the position of the branch point of the cavity topology skeleton includes: scanning the to-be-processed region by using image scanning technology to obtain a three-dimensional anatomical image of the to-be-processed region; performing image segmentation processing on the three-dimensional anatomical image to obtain a target tube cavity; extracting a skeleton center line of the target tube cavity; and performing branch point detection on the skeleton center line based on a graph theory analysis method to determine the position of the branch point of the cavity topology skeleton.

[0094] First, the to-be-processed region is scanned by using image scanning technology to obtain a three-dimensional anatomical image of the to-be-processed region; and the three-dimensional anatomical image is subjected to image segmentation processing to obtain a target tube cavity.

[0095] The image scanning technology can be CT or MRI. The specific segmentation method can be: a method based on seed point three-dimensional region growing is used, the target lumen is semi-automatically extracted according to the gray scale or morphological features of the target lumen, a method based on a deep learning neural network can be used, or a commonly used 2D-based segmentation network such as Unet, DeepLab, PVT (Pyramid Vision Transformer) is used to segment and superimpose the anatomical image layer by layer, or a 3D-based segmentation network such as 3DUnet, UNetr, SwinUNetr and the like is directly used to extract the target lumen.

[0096] Secondly, the lumen skeleton center line is extracted.

[0097] The three-dimensional topological refinement method is to continuously remove the boundaries of the object through a morphological erosion operation until only the skeleton thereof is left. Since the method is an iterative process and uses simple morphological operations, it has the advantages of being simple, reliable and parallelizable.

[0098] Finally, the branch points of the skeleton center line are detected.

[0099] Specifically, the following steps can be included: the skeleton points are taken as nodes to construct a connected matrix, wherein N voxels on the three-dimensional skeleton are taken as nodes, two nodes are connected if the distance between the voxels is within the eight-neighborhood, and an N*N connected matrix can be constructed; the number of connected nodes of each node is calculated, specifically, N-1 times of traversal of the N nodes are required, and the number of connected nodes of the node is determined according to the eight-neighborhood; the type of each node is determined according to the number of connected nodes D, specifically, when D=1, the node is an end point of the skeleton; when D=2, the node is a branch point of the skeleton; and when D>2, the node is a branch point of the skeleton. Thus, the position of the lumen topological skeleton branch point can be determined. Then, the target lumen branch point can be determined based on the two-dimensional virtual endoscope image and the corresponding depth image.

[0100] In a possible implementation, the step S101 of acquiring a preset number of two-dimensional virtual endoscope images containing the lumen branch point through the camera includes: acquiring a preset number of two-dimensional virtual endoscope images containing the lumen branch point according to the user's fine adjustment operation and shooting instruction of the camera; wherein the fine adjustment operation is an operation of the user controlling the camera to move within a preset range of the lumen topological skeleton branch point at a preset amplitude; and after each fine adjustment operation of the user, the camera acquires one two-dimensional virtual endoscope image according to the shooting instruction. It can be understood that the image captured by moving the virtual camera near the lumen topological skeleton branch point includes the target lumen branch point.

[0101] It should be noted that by rendering the segmented lumen channel data in three dimensions, a two-dimensional virtual endoscopic image of the three-dimensional lumen channel at different viewing angles can be obtained by changing the camera (viewpoint) pose.

[0102] In a specific application scenario, the virtual camera can be moved to the vicinity of the branch point of the lumen topology skeleton, the virtual camera pose (p, d) is fine-tuned and recorded, where p is the position and d is the direction, and a plurality of two-dimensional endoscopic images of the lumen interior at different poses are obtained. The fine-tuning method can be: moving the virtual camera in the vicinity of the branch point of the lumen topology skeleton with a pitch of ±1° (Pitch, Roll, Yaw), ±20°, a position step of ±0.5mm (x, y, z), ±2mm (or equal to the radius of the current trachea / vascular branch, the radius of the trachea branch is usually 1mm to 10mm), and recording the two-dimensional virtual endoscopic image and the depth image. Wherein, Pitch is the yaw angle, Roll is the pitch angle, and Yaw is the roll angle.

[0103] It should be noted that the two-dimensional virtual endoscopic image corresponding to each pose and the depth image corresponding to each pose can be recorded respectively. The two-dimensional virtual endoscopic image corresponding to each pose can also be recorded, and the optimal two-dimensional virtual endoscopic image and the depth image corresponding to the pose thereof can be recorded.

[0104] Wherein, the depth image (Depth Images), also known as range image (Range Images), refers to an image in which the distance (depth) value of each point in the scene collected by the image collector is taken as the pixel value, which directly reflects the geometric shape of the visible surface of the scene, and many problems in 3D target description can be easily solved using it. Based on this, in a preferred implementation, a ray casting method can be used to intersect the parallel light rays emitted by the camera (viewpoint) at the current pose with the three-dimensional lumen channel data, and the distance between the intersection point and the camera pose, i.e. the two-dimensional depth image corresponding to the two-dimensional virtual endoscopic image at the current camera pose can be constructed.

[0105] The following possible implementation is based on recording the two-dimensional virtual endoscopic image corresponding to each pose and recording the optimal two-dimensional virtual endoscopic image and the depth image corresponding to the pose thereof, in order to achieve the technical effects of improving efficiency and saving resources.

[0106] In a possible implementation, as shown in Figure 3 Step S102, selecting an optimal two-dimensional virtual endoscopic image in the two-dimensional virtual endoscopic image includes:

[0107] Step S301, performing image segmentation processing on each two-dimensional virtual endoscopic image to obtain a branch opening of each two-dimensional virtual endoscopic image;

[0108] Step S302, respectively determine the difference between the maximum branch opening area and the second largest branch opening area of each two-dimensional virtual endoscope image;

[0109] Step S303, compare the difference values;

[0110] Step S304, determine the two-dimensional virtual endoscope image corresponding to the minimum value in the difference value as the optimal two-dimensional virtual endoscope image.

[0111] After obtaining a plurality of two-dimensional endoscopic images inside the lumen, branch opening detection can be performed on the plurality of two-dimensional virtual endoscopic images. Specifically, the two-dimensional virtual endoscopic images obtained at each pose can be calculated, and the two-dimensional virtual endoscopic images can be processed based on a semantic segmentation network with a coding and decoding structure such as FCN, SegNet, Unet, DeepLab, etc.

[0112] In a preferred implementation, a semantic segmentation model based on the ViT (Vision Transformer) architecture, such as the PvT, Swin Transformer, Segmenter, etc. can be used to segment the two-dimensional virtual endoscopic images to obtain the branch openings of the two-dimensional virtual endoscopic images. Then, the difference between the maximum branch opening area and the second largest branch opening area of each two-dimensional virtual endoscopic image can be determined respectively, and the difference values can be compared. The two-dimensional virtual endoscopic image corresponding to the minimum value in the difference value is determined as the optimal two-dimensional virtual endoscopic image. That is, all two-dimensional virtual endoscopic images are traversed, and the optimal two-dimensional virtual endoscopic image with the largest branch opening area and the closest branch opening area is searched.

[0113] It should be noted that after the optimal two-dimensional virtual endoscopic image is determined, the coordinates of the target lumen branch point can be determined according to the virtual camera pose and the target pixel point. Specifically, in a possible implementation, as shown in Figure 4 Step S105, determining the target pixel point on the two-dimensional depth image includes:

[0114] Step S401, respectively determine the closed region of the branch opening with the largest area and the closed region of the branch opening with the second largest area in the optimal two-dimensional virtual endoscopic image;

[0115] Step S402, respectively determine the centroid of the closed region of the branch opening with the largest area and the centroid of the closed region of the branch opening with the second largest area;

[0116] Step S403, connect the centroid of the closed region of the branch opening with the largest area and the centroid of the closed region of the branch opening with the second largest area to form a first connecting line;

[0117] Step S404, mapping the first line to the two-dimensional depth image corresponding to the optimal two-dimensional virtual endoscope image to determine a second line;

[0118] Step S405, determining the point with the smallest pixel gray value on the second line as the target pixel point.

[0119] In a possible implementation, step S106, determining the coordinate of the target cavity branch point according to the pose and the target pixel point comprises: determining the pixel gray value of the target pixel point; determining the coordinate of the target cavity branch point by the following formula:

[0120]

[0121] wherein (p', d') is the coordinate of the target cavity branch point, (p, d) is the pose of the camera (the camera can be a virtual camera); p is a position, d is a direction, the pose is the pose corresponding to the optimal two-dimensional virtual endoscope image and the two-dimensional depth image; and Depth is the pixel gray value.

[0122] That is, the pose corresponding to the optimal two-dimensional virtual endoscope image is recorded, the closed areas of the two branch openings with the largest and most similar areas are determined in the optimal two-dimensional virtual endoscope image, the centers of the two closed areas are connected to form a line connecting the two branch openings, then the line is corresponded to the depth image corresponding to the two-dimensional virtual endoscope image, and the point with the smallest pixel depth value on the line corresponding to the depth image is determined as the target pixel point, and the depth value (gray value) thereof is recorded as Depth. The coordinate of the target cavity branch point is determined by the following formula.

[0123]

[0124] It should be noted that in a possible implementation, the depth image corresponding to the pose of the opening area of the two-dimensional virtual endoscope image can also be a two-dimensional feature point in a two-dimensional image and a depth image under multiple poses of the camera, and the coordinate of the target cavity branch point determined according to the pose and the target pixel point can also be a three-dimensional feature point cloud constructed and processed to extract the branch point (the target cavity branch point) of the three-dimensional lumen.

[0125] Compared with the branch points of the cavity topology skeleton, the embodiment of the present application is more accurate in restoring and presenting the natural cavity of the human body, and therefore, compared with the prior art, the coordinates of the branch points of the target cavity (the coordinates of the branch points of the tree network cavity anatomical structure of the human body) can be accurately determined, and the position of the branch of the tree network cavity anatomical structure of the human body can be automatically extracted and marked, thereby providing a feature landmark for registration based on anatomical features in virtual navigation and real-time navigation, so as to obtain a suitable path to guide the instrument to reach the target (lesion) area from the cavity entrance through the cavity, thereby assisting the medical diagnosis or surgery.

[0126] In the prior art, when the segmented lumen is not smooth, the detected skeleton line often has burrs or false branches, and the shorter branch is easy to be missed, thereby affecting the accuracy of detection. The method shown in the embodiment of the present application can avoid this.

[0127] Figure 5 A structural block diagram of a cavity branch point detection device 50 according to an embodiment of the present application is shown, as shown in the figure, the device 50 comprises: Figure 5

[0128] The first acquisition module 501 is configured to acquire a preset number of two-dimensional virtual endoscope images containing cavity branch points through a camera;

[0129] The selection module 502 is configured to select an optimal two-dimensional virtual endoscope image in the two-dimensional virtual endoscope images;

[0130] The difference between the maximum branch opening area and the second maximum branch opening area in the optimal two-dimensional virtual endoscope image is the smallest;

[0131] The second acquisition module 503 is configured to acquire the pose of the camera corresponding to the optimal two-dimensional virtual endoscope image;

[0132] The third acquisition module 504 is configured to acquire a two-dimensional depth image corresponding to the pose;

[0133] The first determination module 505 is configured to determine a target pixel point on the two-dimensional depth image; wherein the target pixel point is the point with the smallest pixel gray value on the mapped line after mapping the line connecting the maximum branch opening and the second maximum branch opening to the two-dimensional depth image;

[0134] The second determination module 506 is configured to determine the coordinates of the target cavity branch point according to the pose and the target pixel point.

[0135] ​In a possible implementation, the first acquisition module 501 is further configured to determine the positions of the cavity topology skeleton branch points, and acquire a preset number of two-dimensional virtual endoscope images containing the cavity branch points within a preset range of the cavity topology skeleton branch points by using the camera.

[0136] In a possible implementation, the first acquisition module 501 is further configured to acquire the preset number of two-dimensional virtual endoscope images containing the cavity branch points according to a fine adjustment operation of a user on the camera and a shooting instruction.

[0137] The fine adjustment operation is an operation in which the user controls the camera to move by a preset amplitude within the preset range, and the camera acquires a two-dimensional virtual endoscope image according to the shooting instruction after each fine adjustment operation of the user.

[0138] In a possible implementation, the selection module 502 is further configured to perform image segmentation processing on each two-dimensional virtual endoscope image to acquire a branch opening of each two-dimensional virtual endoscope image.

[0139] The selection module 502 is further configured to determine a difference between a maximum branch opening area and a second largest branch opening area of each two-dimensional virtual endoscope image.

[0140] The selection module 502 is further configured to compare the difference.

[0141] The selection module 502 is further configured to determine a two-dimensional virtual endoscope image corresponding to a minimum value in the difference as an optimal two-dimensional virtual endoscope image.

[0142] In a possible implementation, the first determination module 505 is further configured to determine a closed region of a branch opening with a maximum area and a closed region of a branch opening with a second largest area in the optimal two-dimensional virtual endoscope image.

[0143] The first determination module 505 is further configured to determine a centroid of the closed region of the branch opening with the maximum area and a centroid of the closed region of the branch opening with the second largest area.

[0144] The first determination module 505 is further configured to connect the centroid of the closed region of the branch opening with the maximum area and the centroid of the closed region of the branch opening with the second largest area to form a first connecting line.

[0145] The first determination module 505 is further configured to map the first connecting line to a two-dimensional depth image corresponding to the optimal two-dimensional virtual endoscope image to determine a second connecting line.

[0146] The first determination module 505 is further configured to determine a point with a minimum pixel gray value on the second connecting line as a target pixel point.

[0147] In a possible implementation, the second determination module 506 is further configured to determine a pixel gray value of the target pixel point, and determine a coordinate of a target cavity branch point by using the following formula:

[0148]

[0149] Wherein, (p', d') is the coordinate of the target cavity branch point, (p, d) is the pose of the camera (the camera can be a virtual camera); p is the position, d is the direction, the pose is the pose corresponding to the optimal two-dimensional virtual endoscope image and the two-dimensional depth image; Depth is the pixel gray value.

[0150] In a possible implementation, the first acquisition module 501 is further configured to scan the to-be-processed region by using an image scanning technology to acquire a three-dimensional anatomical image of the to-be-processed region.

[0151] The three-dimensional anatomical image is subjected to image segmentation processing to acquire a target tubular cavity.

[0152] The skeleton center line of the target tubular cavity is extracted.

[0153] The skeleton center line is subjected to branch point detection based on a graph theory analysis method to determine the position of the cavity topological skeleton branch point.

[0154] Preferably, the camera is a virtual camera.

[0155] Therefore, the embodiment of the present application can accurately acquire the branch characteristics (the coordinate of the target cavity branch point, i.e., the coordinate of the real three-dimensional anatomical structure branch point, i.e., the coordinate of the A' point shown in the figure) of the three-dimensional tree network natural cavity of the human body, can provide an actual anatomical reference landmark for the accurate positioning of the instrument in virtual navigation and real-time navigation, and can further obtain a suitable path to guide the instrument to reach the target (lesion) region from the cavity entrance through the cavity, to assist the medical diagnosis or the performance of the surgery. Figure 2

[0156] The embodiment of the present application further provides an electronic device 60, as shown in the figure, the electronic device 60 includes a processor 601, a memory 602, and a program stored in the memory 602 and executable on the processor 601, and the program is executed by the processor 601 to implement the steps of the cavity branch point detection method shown in the above embodiment. Figure 6

[0157] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the cavity branch point detection method shown in the above embodiment. The computer readable storage medium includes a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0158] ​​It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0159] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a plurality of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) execute the method described in various embodiments of the present application.

[0160] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which all belong to the protection of the present application.

Claims

1. A method for detecting branch points in a cavity, characterized in that, The method includes: The camera captures a preset number of two-dimensional virtual endoscopic images containing cavity branch points; Among the two-dimensional virtual endoscope images, the optimal two-dimensional virtual endoscope image is selected; wherein, the difference between the largest branch opening area and the second largest branch opening area in the optimal two-dimensional virtual endoscope image is the smallest; Obtain the pose of the camera corresponding to the optimal two-dimensional virtual endoscope image; Obtain a two-dimensional depth image corresponding to the pose; A target pixel is determined on the two-dimensional depth image; wherein, the target pixel is the point with the smallest pixel gray value on the line connecting the largest branch opening and the second largest branch opening after mapping it onto the two-dimensional depth image. The coordinates of the target cavity branch point are determined based on the pose and the target pixel.

2. The method according to claim 1, characterized in that, The camera captures a preset number of two-dimensional virtual endoscopic images containing cavity branch points, including: Determine the location of the branch points of the cavity topology framework; Within a preset range of the branch points of the cavity topology skeleton, a preset number of two-dimensional virtual endoscopic images containing the branch points of the cavity are acquired by a camera.

3. The method according to claim 1, characterized in that, The camera captures a preset number of two-dimensional virtual endoscopic images containing cavity branch points, including: Based on the user's fine-tuning operation and shooting instructions for the camera, a preset number of two-dimensional virtual endoscope images containing the branch points of the cavity are acquired; The fine-tuning operation is the operation by which the user controls the camera to move within a preset range by a preset amplitude; and after each fine-tuning operation by the user, the camera acquires a two-dimensional virtual endoscope image according to the shooting instruction.

4. The method according to claim 1, characterized in that, Selecting the optimal two-dimensional virtual endoscope image from the given two-dimensional virtual endoscope images includes: Each two-dimensional virtual endoscope image is segmented to obtain the branch openings of each two-dimensional virtual endoscope image; Determine the difference between the largest branch opening area and the second largest branch opening area for each two-dimensional virtual endoscope image; Compare the differences; The two-dimensional virtual endoscope image corresponding to the minimum value among the differences is determined as the optimal two-dimensional virtual endoscope image.

5. The method according to claim 4, characterized in that, Determining the target pixel on the two-dimensional depth image includes: In the optimal two-dimensional virtual endoscope image, the closed regions of the branch openings with the largest area and the second largest area are determined respectively. Determine the centroids of the closed regions of the branch opening with the largest area and the closed regions of the branch opening with the second largest area, respectively. Connect the centroids of the closed regions of the branch openings with the largest area and the closed regions of the branch openings with the second largest area to form the first line. The first connection is mapped onto the two-dimensional depth image corresponding to the optimal two-dimensional virtual endoscope image to determine the second connection; On the second connection line, the point with the smallest pixel gray value is determined as the target pixel.

6. The method according to claim 1, characterized in that, Determining the coordinates of the target cavity branch point based on the pose and the target pixel includes: Determine the pixel grayscale value of the target pixel; The coordinates of the target cavity branch point are determined using the following formula: Wherein, (p',d') are the coordinates of the target cavity branch point, and (p,d) are the pose; p is the position, d is the direction, and the pose is the pose corresponding to the optimal two-dimensional virtual endoscope image and the two-dimensional depth image; and Depth is the pixel grayscale value.

7. The method according to claim 2, characterized in that, Determining the location of branch points in the cavity topology framework includes: The area to be processed is scanned using image scanning technology to obtain a three-dimensional anatomical image of the area to be processed. The three-dimensional anatomical image is segmented to obtain the target lumen. Extract the centerline of the skeleton of the target lumen; The branch point detection of the skeleton centerline is performed based on graph theory analysis to determine the location of the branch points of the cavity topology skeleton.

8. The method according to any one of claims 1-7, characterized in that, The camera in question is a virtual camera.

9. A device for detecting the branch point of a cavity, characterized in that, The device includes: The first acquisition module is used to acquire a preset number of two-dimensional virtual endoscope images containing cavity branch points through a camera; The selection module is used to select the optimal two-dimensional virtual endoscope image from the two-dimensional virtual endoscope images; wherein the difference between the largest branch opening area and the second largest branch opening area in the optimal two-dimensional virtual endoscope image is the smallest. The second acquisition module is used to acquire the pose of the camera corresponding to the optimal two-dimensional virtual endoscope image; The third acquisition module is used to acquire a two-dimensional depth image corresponding to the pose; The first determining module is used to determine a target pixel on the two-dimensional depth image; wherein, the target pixel is the point with the smallest pixel gray value on the line connecting the largest branch opening and the second largest branch opening after mapping it onto the two-dimensional depth image; The second determining module is used to determine the coordinates of the target cavity branch point based on the pose and the target pixel.

10. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the method for detecting cavity branch points as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for detecting cavity branch points as described in any one of claims 1-8.

Images