Endoscope propulsion guiding method and device, storage medium and equipment

Abstract

The invention relates to an endoscope propelling guide method and device, a storage medium and equipment. The method comprises the steps that the propelling position of an endoscope in the propelling process and a collected endoscope image are obtained; according to the center line network corresponding to the region of interest, correcting the accumulated error of the propulsion position to obtain a corrected propulsion position; inputting the endoscope image into a pre-trained bifurcation prediction neural network model to obtain an output bifurcation prediction result; according to the center line network and the bifurcation prediction result, predicting a residual propulsion distance and residual propulsion time from the corrected propulsion position to a pre-selected propulsion end point and a target propulsion branch at a bifurcation; and marking a target propulsion branch and a non-target branch for guiding the bifurcation, and performing propulsion guidance according to the residual propulsion distance, the residual propulsion time and the bifurcation with the marked branches. And the endoscope is guided to advance in real time, so that the endoscope is prevented from being pushed to a wrong branch. And the pushing accuracy of the endoscope is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to an endoscope advancement and guidance method, device, storage medium and equipment. Background Art

[0002] Human tissue is not rigid and will exhibit deformations such as stretching, contraction, and displacement at different times, which may cause the endoscope to encounter obstacles during the advancement process. Therefore, it is not only necessary to guide the advancement of the endoscope, but also to determine the position of the endoscope in the human body in real time and avoid advancing to the wrong tissue or organ. Summary of the invention

[0003] In view of the above technical problems, the purpose of the present disclosure is to provide an endoscope advancement and guidance method, device, storage medium and equipment.

[0004] In order to achieve the above-mentioned purpose, a first aspect of the embodiment of the present disclosure provides an endoscope advancement and guidance method, comprising:

[0005] Acquiring the advancement position of the imaging unit of the endoscope during the advancement process and the endoscopic image collected by the imaging unit;

[0006] Correcting the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position;

[0007] Inputting the endoscopic image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model;

[0008] According to the centerline network and the bifurcation prediction result, predict the remaining propulsion distance, remaining propulsion time and target propulsion branch at the bifurcation from the corrected propulsion position to the pre-selected propulsion end point;

[0009] The target propulsion branch and the non-target branch at the fork are marked, and propulsion guidance is performed according to the remaining propulsion distance, the remaining propulsion time and the fork after the branch is marked.

[0010] Optionally, the step of correcting the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position includes:

[0011] Determining a target key node corresponding to the advancement position from the key nodes of the centerline network corresponding to the region of interest;

[0012] According to the node position corresponding to the target key node, the accumulated error of the propulsion position is corrected to obtain the corrected propulsion position.

[0013] Optionally, predicting the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to a pre-selected propulsion end point according to the centerline network and the bifurcation prediction result includes:

[0014] In the case where the bifurcation prediction result indicates that there is a bifurcation ahead of the advancement of the endoscope, predicting the remaining advancement distance from the corrected advancement position to a pre-selected advancement endpoint along the centerline network;

[0015] Predicting the remaining advancement time from the corrected advancement position to a pre-selected advancement endpoint according to the remaining advancement distance and the average speed of the endoscope during the current advancement process;

[0016] From the branches of the bifurcation represented by the bifurcation prediction result, a target propulsion branch from the corrected propulsion position to a pre-selected propulsion endpoint is determined.

[0017] Optionally, the centerline network corresponding to the region of interest is established in the following manner:

[0018] Constructing a three-dimensional model of the region of interest based on the original medical image of the region of interest to be examined;

[0019] Performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type;

[0020] The centerline of the pipeline network corresponding to each pipeline type is extracted to obtain the centerline network corresponding to each pipeline type, wherein the line width of the centerline in the centerline network is one volume element, and there is a unique path between any two volume elements on the centerline.

[0021] Optionally, performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type includes:

[0022] Delimiting a target area including the area of ​​interest in the three-dimensional model;

[0023] Performing volume element scanning on the three-dimensional model within the target area according to the input exposure value range to obtain volume elements corresponding to the target area;

[0024] Taking the input seed point as the starting point, searching for volume elements with a connected relationship in the three-dimensional model according to the exposure value range, and adding display labels to the volume elements with a connected relationship;

[0025] The volume elements without display labels in the three-dimensional model are hidden to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

[0026] Optionally, marking the target propulsion branch and the non-target branch of the guide bifurcation includes:

[0027] Marking the target propulsion branch of the guide bifurcation by presetting a first color and a first preset marking style;

[0028] The non-target propulsion branch of the guide fork is marked by presetting the second color and the second preset marking style.

[0029] A second aspect of the present disclosure provides an endoscope advancement guide device, comprising:

[0030] an acquisition module, configured to acquire a propulsion position of the imaging unit of the endoscope during the propulsion process and an endoscopic image acquired by the imaging unit;

[0031] A correction module is configured to correct the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position;

[0032] An input module is configured to input the endoscopic image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model;

[0033] A prediction module, configured to predict the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to a pre-selected propulsion end point based on the centerline network and the bifurcation prediction result;

[0034] The display module is configured to mark the target propulsion branch and the non-target branch at the fork, and to provide propulsion guidance according to the remaining propulsion distance, the remaining propulsion time and the fork after the branch marking.

[0035] Optionally, the correction module is configured to:

[0036] Determining a target key node corresponding to the advancement position from the key nodes of the centerline network corresponding to the region of interest;

[0037] According to the node position corresponding to the target key node, the accumulated error of the propulsion position is corrected to obtain the corrected propulsion position.

[0038] Optionally, the prediction module is configured to:

[0039] In the case where the bifurcation prediction result indicates that there is a bifurcation ahead of the advancement of the endoscope, predicting the remaining advancement distance from the corrected advancement position to a pre-selected advancement endpoint along the centerline network;

[0040] Predicting the remaining advancement time from the corrected advancement position to a pre-selected advancement endpoint according to the remaining advancement distance and the average speed of the endoscope during the current advancement process;

[0041] From the branches of the bifurcation represented by the bifurcation prediction result, a target propulsion branch from the corrected propulsion position to a pre-selected propulsion endpoint is determined.

[0042] Optionally, the correction module is further configured to establish a centerline network corresponding to the region of interest in the following manner:

[0043] Constructing a three-dimensional model of the region of interest based on the original medical image of the region of interest to be examined;

[0044] Performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type;

[0045] The centerline of the pipeline network corresponding to each pipeline type is extracted to obtain the centerline network corresponding to each pipeline type, wherein the line width of the centerline in the centerline network is one volume element, and there is a unique path between any two volume elements on the centerline.

[0046] Optionally, the correction module is further configured to:

[0047] Delimiting a target area including the area of ​​interest in the three-dimensional model;

[0048] Performing volume element scanning on the three-dimensional model within the target area according to the input exposure value range to obtain volume elements corresponding to the target area;

[0049] Taking the input seed point as the starting point, searching for volume elements with a connected relationship in the three-dimensional model according to the exposure value range, and adding display labels to the volume elements with a connected relationship;

[0050] The volume elements without display labels in the three-dimensional model are hidden to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

[0051] Optionally, the display module is configured as follows:

[0052] Marking the target propulsion branch of the guide bifurcation by presetting a first color and a first preset marking style;

[0053] The non-target propulsion branch of the guide fork is marked by presetting the second color and the second preset marking style.

[0054] According to a third aspect of the embodiments of the present disclosure, there is provided a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscope advancement and guidance method described in any one of the first aspects.

[0055] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0056] a first memory having a computer program stored thereon;

[0057] The second processor is used to execute the computer program in the first memory to implement the steps of the endoscope advancement guidance method according to any one of the first aspects.

[0058] Through the above technical solution, at least the following beneficial effects can be achieved:

[0059] The advancement position of the image part of the endoscope during the advancement process and the endoscope image collected by the image part are obtained; the accumulated error of the advancement position is corrected according to the centerline network corresponding to the region of interest to obtain the corrected advancement position, which can avoid the continuous accumulation of the error of the advancement position, resulting in the inability to accurately determine the exact position of the endoscope; then the endoscope image is input into the pre-trained bifurcation prediction neural network model to obtain the bifurcation prediction result output by the bifurcation prediction neural network model; according to the centerline network and the bifurcation prediction result, the remaining advancement distance, remaining advancement time and target advancement branch at the bifurcation are predicted from the corrected advancement position to the pre-selected advancement end point; the target advancement branch and non-target branch at the bifurcation are marked, and the advancement guidance is performed according to the remaining advancement distance, remaining advancement time and the bifurcation after the branch is marked. The advancement of the endoscope can be guided in real time to avoid advancing the endoscope to the wrong branch. The accuracy of the endoscope advancing to the pre-selected advancement end point is improved.

[0060] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0062] Figure 1 The figure is a flow chart of an endoscope advancement and guidance method according to an embodiment of the present disclosure.

[0063] Figure 2This is an implementation shown according to an embodiment of the present disclosure. Figure 1 Flow chart of the method of step S12 in FIG.

[0064] Figure 3 This is an implementation shown according to an embodiment of the present disclosure. Figure 1 Flow chart of the method of step S14 in FIG.

[0065] Figure 4 The present invention is a flowchart of a method for establishing a centerline network corresponding to a region of interest according to an embodiment of the present invention.

[0066] Figure 5 This is an implementation shown according to an embodiment of the present disclosure. Figure 4 Flow chart of the method of step S402 in FIG.

[0067] Figure 6 It is a block diagram of an endoscope propulsion and guidance device according to an embodiment of the present disclosure.

[0068] Figure 7 It is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0070] Before introducing the endoscope advancement guidance method, device, storage medium and equipment provided by the present disclosure, the application scenario of the present disclosure is first introduced. During the advancement process, the endoscope may encounter a fork, resulting in the inability to determine the advancement direction.

[0071] To this end, the embodiment of the present disclosure provides an endoscope advancement guidance method, which aims to guide the advancement of the endoscope in real time, mark and guide it when encountering a bifurcation, and avoid advancing the endoscope to the wrong branch. The accuracy of advancing the endoscope to the pre-selected advancement endpoint is improved, see Figure 1 As shown, the method includes the following steps.

[0072] In step S11 , the advancement position of the imaging unit of the endoscope during the advancement process and the endoscopic image captured by the imaging unit are acquired.

[0073] It can be explained that in the disclosed embodiment, the imaging unit of the endoscope can be advanced according to a pre-planned advancement path, and during the advancement process, the imaging unit of the endoscope collects images of tissues and pipes in real time to obtain an endoscopic image. The advancement position of the imaging unit of the endoscope during the advancement process is obtained by an inertial sensor configured in the imaging unit of the endoscope.

[0074] Among them, the doctor is the decision maker and executor during the endoscope advancement process. The method provided by the present disclosure can only determine whether the image part of the endoscope is advanced according to the pre-planned advancement path based on the advancement position of the image part of the endoscope. If the endoscope is not advanced according to the pre-planned advancement path during the advancement process, a prompt can be given.

[0075] In step S12, the accumulated error of the advancement position is corrected according to the centerline network corresponding to the region of interest to obtain a corrected advancement position.

[0076] It can be explained that because tissue is not rigid, it may show deformations such as stretching, contraction, and displacement in different states, which may cause the advancement position of the endoscope to deviate from the saved historical data. In some extreme cases, the saved historical data may also differ from the actual surgery or examination. Therefore, it is necessary to analyze the actual acquired advancement position. If the error deviates significantly from the reasonable range and there is a major abnormal risk, not only does it need to correct the advancement position, but also an alarm prompt can be given.

[0077] In the disclosed embodiment, the target position of the endoscope can be determined from the centerline network corresponding to the region of interest, and then the propulsion position of the endoscope can be corrected according to the target position. For example, when the Euclidean distance between the target position and the propulsion position of the endoscope exceeds a preset distance threshold, the propulsion position of the endoscope is replaced with the target position in the centerline network, that is, the target position in the centerline network is used as the corrected retreat position. When the Euclidean distance between the target position and the propulsion position of the endoscope does not exceed the preset distance threshold, the propulsion position of the endoscope is directly used as the corrected propulsion position. In this way, it can be avoided that the propulsion position of the endoscope continuously accumulates errors during the propulsion process, resulting in a continuous increase in the error between the propulsion position of the endoscope and the actual position.

[0078] In step S13, the endoscopic image is input into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model.

[0079] In the disclosed embodiment, the bifurcation prediction neural network model can be, for example, a Yolo model, wherein the original Yolo model can be trained by annotating endoscopic images with bifurcations to obtain a set of model parameters. During surgery or inspection, the endoscopic image collected in real time by the image part of the endoscope is input into the trained Yolo model, and the Yolo model can give a prediction result of whether there is a bifurcation in the endoscopic image. Moreover, in the case of a bifurcation, the prediction result can also mark the direction and size of each branch of the bifurcation. Among them, the Yolo model can quickly and accurately predict the bifurcation.

[0080] In step S14, based on the centerline network and the bifurcation prediction results, the remaining propulsion distance, remaining propulsion time and target propulsion branch at the bifurcation are predicted from the corrected propulsion position to the pre-selected propulsion end point.

[0081] In the disclosed embodiment, the remaining advancement distance to the pre-selected advancement end point can be calculated along the center line network according to the corrected advancement position, and the error prediction of the remaining advancement distance can be performed due to the deformation of the tissue such as stretching, contraction, displacement, etc. Then, the remaining advancement time from the corrected advancement position to the pre-selected advancement end point can be predicted according to the remaining advancement distance and the pre-set advancement direction and advancement speed.

[0082] In the disclosed embodiment, if the bifurcation prediction result indicates that there is no bifurcation ahead, then only one target propulsion branch can be directly displayed. If the bifurcation prediction result indicates that there is a bifurcation ahead, the target propulsion branch from the corrected propulsion position to the pre-selected propulsion end point at the bifurcation is predicted. It is easy to understand that the target propulsion branch can be several optimal propulsion branches, so several target propulsion branches can be displayed, or each target propulsion branch can be selected and one of the optimal target propulsion branches can be displayed, and the other branches are all regarded as non-target propulsion branches.

[0083] In step S15, the target propulsion branch and the non-target branch at the fork are marked, and propulsion guidance is performed according to the remaining propulsion distance, the remaining propulsion time and the marked branch.

[0084] In the disclosed embodiment, the remaining propulsion distance and remaining propulsion time can be announced and guided by voice, for example, and then displayed and guided by display on the user interface. Similarly, the marked target propulsion branch and non-target branch can also be guided by voice at the fork after the branch is marked, and the target propulsion branch can be displayed and guided by displaying a striking color or mark on the user interface, and the non-target branch can be displayed and guided by displaying a non-eye-catching color or mark on the user interface.

[0085] In the disclosed embodiment, the doctor can temporarily change the propulsion direction and propulsion speed of the endoscope based on the displayed endoscopic image, for example, if a new lesion is found, the propulsion path can be temporarily changed. Then, according to the current propulsion position of the image portion of the endoscope and based on the pre-set constraints, several optimal path options from the propulsion position to the pre-selected propulsion end point are planned in real time, and several optimal path options are displayed.

[0086] The above technical solution obtains the advancement position of the image part of the endoscope during the advancement process and the endoscopic image collected by the image part; according to the centerline network corresponding to the area of ​​interest, the cumulative error of the advancement position is corrected to obtain the corrected advancement position, which can avoid the continuous accumulation of the error of the advancement position, resulting in the inability to accurately determine the exact position of the endoscope; then the endoscopic image is input into the pre-trained bifurcation prediction neural network model to obtain the bifurcation prediction result output by the bifurcation prediction neural network model; according to the centerline network and the bifurcation prediction result, the remaining advancement distance, remaining advancement time and target advancement branch at the bifurcation are predicted from the corrected advancement position to the pre-selected advancement end point; the target advancement branch and non-target branch of the guide bifurcation are marked, and the advancement guidance is performed according to the remaining advancement distance, remaining advancement time and the bifurcation after the branch is marked. The advancement of the endoscope can be guided in real time to avoid advancing the endoscope to the wrong branch. The accuracy of the endoscope advancing to the pre-selected advancement end point is improved.

[0087] In the embodiments of the present disclosure, see Figure 2 As shown, in step S12, the accumulated error of the advancement position is corrected according to the centerline network corresponding to the region of interest to obtain the corrected advancement position, including:

[0088] In step S121, a target key node corresponding to the advancement position is determined from the key nodes of the centerline network corresponding to the area of ​​interest.

[0089] In the disclosed embodiment, the key nodes of the centerline network corresponding to the region of interest may be determined based on the type of target tissue, the outline of the tissue, the characteristic size of the tissue, the normal or abnormal index of the tissue, the bifurcation, etc. The key nodes have relatively obvious features and can be clearly seen from the endoscopic image.

[0090] Furthermore, the target key node may be a key node of a previous path of the advancement position, or may be a key node displayed in the current endoscopic image.

[0091] In step S122, the accumulated error of the propulsion position is corrected according to the node position corresponding to the target key node to obtain a corrected propulsion position.

[0092] In the disclosed embodiment, the distance between the node position corresponding to the target key node and the advancement position can be calculated. When the distance exceeds a preset threshold, the advancement position is replaced by the node position to obtain a corrected advancement position. In this way, it is possible to avoid using the node position as the starting point of the next advancement each time during the advancement process, and to avoid the influence of the error of the previous advancement position on the subsequent advancement position.

[0093] In the embodiments of the present disclosure, see Figure 3As shown, in step S14, the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to the pre-selected propulsion end point are predicted based on the centerline network and the bifurcation prediction result, including:

[0094] In step S141, when the bifurcation prediction result indicates that there is a bifurcation ahead of the endoscope, the remaining advancement distance from the corrected advancement position to the pre-selected advancement end point is predicted along the centerline network.

[0095] In the disclosed embodiment, when the bifurcation prediction result indicates that there is a bifurcation in front of the endoscope, multiple remaining propulsion distances from the corrected propulsion position to the pre-selected propulsion end point are predicted along the centerline network. Each remaining propulsion distance corresponds to a bifurcation branch, which facilitates the selection of a suitable target propulsion branch from different remaining propulsion distances. For example, in the case of a bifurcation, if you want to bypass other possible diseased tissues, you can refer to the cost of the remaining propulsion distance of each target propulsion branch to avoid excessive detour costs, which may lead to unreasonable surgery or examination.

[0096] In step S142, the remaining advancement time from the corrected advancement position to the pre-selected advancement end point is predicted based on the remaining advancement distance and the average speed of the endoscope during the current advancement process.

[0097] Among them, due to the differences in tissues of different human bodies, such as the pore size of human ducts, if the remaining advancement time is calculated based on the preset advancement direction and advancement speed, it may lead to large errors in the calculated remaining advancement time. Therefore, the remaining advancement time from the corrected advancement position to the pre-selected advancement end point can be predicted based on the average speed of the endoscope during this advancement process, thereby improving the accuracy of calculating the remaining advancement time.

[0098] It is worth noting that the average speed of the endoscope during this advancement process is calculated based on the cumulative time and cumulative distance when the endoscope is advancing forward. If the endoscope stops at any position for observation, the observation time will not be included in the cumulative time.

[0099] In step S143, a target propulsion branch from the corrected propulsion position to a pre-selected propulsion endpoint is determined from each branch of the fork represented by the fork prediction result.

[0100] There may be one or more target propulsion branches from the corrected propulsion position to the pre-selected propulsion endpoint.

[0101] In the embodiments of the present disclosure, see Figure 4 As shown in Figure 1, the centerline network corresponding to the region of interest is established in the following way:

[0102] In step S401, a three-dimensional model of the region of interest is constructed based on the original medical image of the region of interest to be examined.

[0103] In the embodiment of the present disclosure, the original medical images taken from different angles for the region of interest to be examined may be used to construct a three-dimensional model for the region of interest. Or other existing technologies may be used to construct a three-dimensional model, which will not be described in detail here.

[0104] In step S402, image segmentation is performed on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

[0105] In the disclosed embodiments, a traditional image segmentation algorithm based on morphological operations can be used, or a neural network image segmentation algorithm based on a deep learning network, such as a UNet network, can be used to perform image segmentation on the three-dimensional model to obtain a pipeline network corresponding to each pipeline type in the human pipeline system.

[0106] In step S403, centerline extraction is performed on the pipeline network corresponding to each pipeline type to obtain the centerline network corresponding to the pipeline network of each pipeline type.

[0107] The channels may include: channels that exist naturally in the human body, various sinuses formed by pathological factors, and surgical pathways formed by the dissection of human tissues to reach the surgical area.

[0108] The line width of the center line in the center line network is one volume element, and there is a unique path between any two volume elements on the center line.

[0109] In the disclosed embodiment, for the aforementioned pipeline network, the centerline of the pipeline network can be extracted by a centerline extraction algorithm, and the centerline network finally obtained is equivalent to the pipeline network in terms of topological structure and positioning function. Thus, the network structure is simplified. Among them, the centerline network has the following characteristics: 1) The line width of the centerline is one volume element; 2) There is a unique and non-repetitive path between any two voxels on the centerline network.

[0110] The above technical solution can perform image segmentation on the constructed three-dimensional model, extract the pipeline network, and then determine the centerline network corresponding to the pipeline network, so as to reduce the influence between different types of pipelines in the image.

[0111] In the embodiments of the present disclosure, see Figure 5 As shown, in step S402, the three-dimensional model is segmented to obtain a pipeline network corresponding to the pipeline system of each pipeline type, including:

[0112] In step S4021, a target area including the area of ​​interest is delineated in the three-dimensional model.

[0113] In the disclosed embodiment, a three-dimensional area is delineated from the spatial range of the three-dimensional stereo model as a target area, and the target area includes the patient's lesion area, wherein the shape of the target area can be a cube, an oblique cube, a cylinder, an oblique cylinder, or other irregular three-dimensional bodies.

[0114] Furthermore, in the three-dimensional model, hidden labels are added to other volume elements outside the target area, and when the three-dimensional model is presented, the volume elements marked with the hidden labels are hidden, that is, the volume elements marked with the hidden labels are not displayed.

[0115] In step S4022, according to the input exposure value range, a volume element scan is performed on the three-dimensional model in the target area to obtain a volume element corresponding to the target area.

[0116] It can be understood that the exposure value range includes a lower threshold and an upper threshold, and the volume element scanning is performed on the three-dimensional stereo model in the target area, and hidden labels are added to the volume elements in the target area whose exposure values ​​fall outside the exposure value range, thereby hiding the volume elements in the target area marked with hidden labels.

[0117] In step S4023, taking the input seed point as the starting point, volume elements with connectivity in the three-dimensional model are searched according to the exposure value range, and display labels are added to the volume elements with connectivity.

[0118] In the disclosed embodiment, the input seed point is input by the user in real time according to the surgical requirements. In specific implementation, display labels can be added to volume elements with connectivity, and hidden labels can be added to volume elements without connectivity.

[0119] In step S4024, the volume elements in the three-dimensional model that are not added with display labels are hidden to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

[0120] In the disclosed embodiment, the volume elements marked with hidden labels can be hidden, and the volume elements marked with display labels can be displayed to obtain the pipeline network corresponding to the pipeline system of each pipeline type in the human body pipeline system.

[0121] Optionally, in step S15, marking the target propulsion branch and the non-target branch of the guide bifurcation includes:

[0122] The target propulsion branch of the guide fork is marked by presetting a first color and a first preset marking style.

[0123] The non-target propulsion branch of the guide fork is marked by presetting the second color and the second preset marking style.

[0124] In the embodiment of the present disclosure, the first preset marking style can be to add an arrow guide to the branch, and then the voice prompt can be used to advance according to the arrow of the preset first color. The second preset marking style can be the same as the first preset marking style, and only the target advancement branch and the non-target advancement branch are distinguished by color, or it can be different from the first preset marking style, and then the target advancement branch and the non-target advancement branch are distinguished by different colors and marking styles.

[0125] Furthermore, in the case where there are several target propulsion branches, the target propulsion branches may be marked by different colors, wherein the conspicuousness of the preset first color is higher than the conspicuousness of the preset second color.

[0126] The present disclosure also provides an endoscope advancement guide device, see Figure 6 As shown, the endoscope advancement guide device 600 includes:

[0127] An acquisition module 610 is configured to acquire a propulsion position of the imaging unit of the endoscope during the propulsion process and an endoscopic image acquired by the imaging unit;

[0128] A correction module 620 is configured to correct the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position;

[0129] An input module 630 is configured to input the endoscopic image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model;

[0130] A prediction module 640 is configured to predict the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to the pre-selected propulsion end point based on the centerline network and the bifurcation prediction result;

[0131] The display module 650 is configured to mark the target propulsion branch and the non-target branch at the fork, and provide propulsion guidance according to the remaining propulsion distance, the remaining propulsion time and the marked branch.

[0132] Optionally, the correction module 620 is configured to:

[0133] Determining a target key node corresponding to the advancement position from the key nodes of the centerline network corresponding to the region of interest;

[0134] According to the node position corresponding to the target key node, the accumulated error of the propulsion position is corrected to obtain the corrected propulsion position.

[0135] Optionally, the prediction module 640 is configured to:

[0136] In the case where the bifurcation prediction result indicates that there is a bifurcation ahead of the advancement of the endoscope, predicting the remaining advancement distance from the corrected advancement position to a pre-selected advancement endpoint along the centerline network;

[0137] Predicting the remaining advancement time from the corrected advancement position to a pre-selected advancement endpoint according to the remaining advancement distance and the average speed of the endoscope during the current advancement process;

[0138] From the branches of the bifurcation represented by the bifurcation prediction result, a target propulsion branch from the corrected propulsion position to a pre-selected propulsion endpoint is determined.

[0139] Optionally, the correction module 620 is further configured to establish a centerline network corresponding to the region of interest in the following manner:

[0140] Constructing a three-dimensional model of the region of interest based on the original medical image of the region of interest to be examined;

[0141] Performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type;

[0142] The centerline of the pipeline network corresponding to each pipeline type is extracted to obtain the centerline network corresponding to each pipeline type, wherein the line width of the centerline in the centerline network is one volume element, and there is a unique path between any two volume elements on the centerline.

[0143] Optionally, the correction module 620 is further configured to:

[0144] Delimiting a target area including the area of ​​interest in the three-dimensional model;

[0145] Performing volume element scanning on the three-dimensional model within the target area according to the input exposure value range to obtain volume elements corresponding to the target area;

[0146] Taking the input seed point as the starting point, searching for volume elements with a connected relationship in the three-dimensional model according to the exposure value range, and adding display labels to the volume elements with a connected relationship;

[0147] The volume elements without display labels in the three-dimensional model are hidden to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

[0148] Optionally, the display module 650 is configured to:

[0149] Marking the target propulsion branch of the guide bifurcation by presetting a first color and a first preset marking style;

[0150] The non-target propulsion branch of the guide fork is marked by presetting the second color and the second preset marking style.

[0151] Regarding the endoscope propulsion and guiding device in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0152] Those skilled in the art should understand that the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules can be combined or integrated into one module. In addition, the modules described as separate components may or may not be physically separated. Moreover, each module may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. When implemented using hardware, it may be implemented in whole or in part in the form of an integrated circuit or chip.

[0153] The disclosed embodiment also provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscope advancement and guidance method described in any one of the aforementioned embodiments.

[0154] The present disclosure also provides an electronic device, including:

[0155] a first memory having a computer program stored thereon;

[0156] The second processor is used to execute the computer program in the first memory to implement the steps of the endoscope advancement guidance method in any one of the aforementioned embodiments.

[0157] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. The electronic device 700 can be configured as an endoscope advancement guide device, such as Figure 7 As shown, the electronic device 700 may include: a third processor 701 and a third memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0158] The third processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned endoscope advancement and guidance method. The third memory 702 is used to store various types of data to support the operation of the electronic device 700, and these data may include, for example, instructions for any application or method used to operate on the electronic device 700, and application-related data, such as pictures, etc. The third memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the third memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the third processor 701 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons for operating the endoscope advancement guide. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 705 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0159] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned endoscope advancement guidance method.

[0160] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned endoscope advancement and guidance method are implemented. For example, the computer-readable storage medium can be the above-mentioned third memory 702 including program instructions, and the above-mentioned program instructions can be executed by the third processor 701 of the electronic device 700 to complete the above-mentioned endoscope advancement and guidance method.

[0161] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0162] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0163] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. An endoscope advancement and guidance method, characterized in that: include: Acquiring the advancement position of the imaging unit of the endoscope during the advancement process and the endoscopic image collected by the imaging unit; Correcting the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position; Inputting the endoscopic image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model; According to the centerline network and the bifurcation prediction result, predict the remaining propulsion distance, remaining propulsion time and target propulsion branch at the bifurcation from the corrected propulsion position to the pre-selected propulsion end point; The target propulsion branch and the non-target branch at the fork are marked, and propulsion guidance is performed according to the remaining propulsion distance, the remaining propulsion time and the fork after the branch is marked.

2. The endoscope advancement and guiding method according to claim 1, characterized in that: The step of correcting the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain the corrected advancement position includes: Determining a target key node corresponding to the advancement position from the key nodes of the centerline network corresponding to the region of interest; According to the node position corresponding to the target key node, the accumulated error of the propulsion position is corrected to obtain the corrected propulsion position.

3. The endoscope advancement and guiding method according to claim 1, characterized in that: The method of predicting the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to the pre-selected propulsion end point according to the centerline network and the bifurcation prediction result includes: In the case where the bifurcation prediction result indicates that there is a bifurcation ahead of the advancement of the endoscope, predicting the remaining advancement distance from the corrected advancement position to a pre-selected advancement endpoint along the centerline network; Predicting the remaining advancement time from the corrected advancement position to a pre-selected advancement endpoint according to the remaining advancement distance and the average speed of the endoscope during the current advancement process; From the branches of the bifurcation represented by the bifurcation prediction result, a target propulsion branch from the corrected propulsion position to a pre-selected propulsion endpoint is determined.

4. The endoscope advancement and guiding method according to claim 1, characterized in that: The centerline network corresponding to the region of interest is established in the following way: Constructing a three-dimensional model of the region of interest based on the original medical image of the region of interest to be examined; Performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type; The centerline of the pipeline network corresponding to each pipeline type is extracted to obtain the centerline network corresponding to each pipeline type, wherein the line width of the centerline in the centerline network is one volume element, and there is a unique path between any two volume elements on the centerline.

5. The endoscope advancement and guiding method according to claim 4, characterized in that: The performing image segmentation on the three-dimensional model to obtain a pipeline network corresponding to the pipeline system of each pipeline type includes: Delimiting a target area including the area of ​​interest in the three-dimensional model; Performing volume element scanning on the three-dimensional model within the target area according to the input exposure value range to obtain volume elements corresponding to the target area; Taking the input seed point as the starting point, searching for volume elements with a connected relationship in the three-dimensional model according to the exposure value range, and adding display labels to the volume elements with a connected relationship; The volume elements without display labels in the three-dimensional model are hidden to obtain a pipeline network corresponding to the pipeline system of each pipeline type.

6. The endoscope advancement and guiding method according to any one of claims 1 to 5, characterized in that: The marking of the target propulsion branch and the non-target branch of the guide bifurcation includes: Marking the target propulsion branch of the guide bifurcation by presetting a first color and a first preset marking style; The non-target propulsion branch of the guide fork is marked by presetting the second color and the second preset marking style.

7. An endoscope propulsion guide device, characterized in that: include: an acquisition module, configured to acquire a propulsion position of the imaging unit of the endoscope during the propulsion process and an endoscopic image acquired by the imaging unit; A correction module is configured to correct the accumulated error of the advancement position according to the centerline network corresponding to the region of interest to obtain a corrected advancement position; An input module is configured to input the endoscopic image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model; A prediction module, configured to predict the remaining propulsion distance, remaining propulsion time and target propulsion branch at the fork from the corrected propulsion position to a pre-selected propulsion end point based on the centerline network and the bifurcation prediction result; The display module is configured to mark the target propulsion branch and the non-target branch at the fork, and to provide propulsion guidance according to the remaining propulsion distance, the remaining propulsion time and the fork after the branch marking.

8. The endoscope advancement guide device according to claim 7, characterized in that: The correction module is configured as follows: Determining a target key node corresponding to the advancement position from the key nodes of the centerline network corresponding to the region of interest; According to the node position corresponding to the target key node, the accumulated error of the propulsion position is corrected to obtain the corrected propulsion position.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the first processor, the steps of the endoscope advancement guidance method described in any one of claims 1 to 6 are implemented.

10. An electronic device, characterized in that: include: a first memory having a computer program stored thereon; A second processor is used to execute the computer program in the first memory to implement the steps of the endoscope advancement guidance method according to any one of claims 1 to 6.

Images