Endoscope advancement guidance methods, apparatuses, storage media, and devices
By acquiring the image position and images of the endoscope, and using a centerline network and bifurcation prediction model, the advancement error is corrected and the bifurcation branches are marked, thus solving the problem of inaccurate positioning of the endoscope in human tissue and achieving precise endoscopic advancement guidance.
Patent Information
- Application Number
- CN202311201443.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-15
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2043-09-15
AI Technical Summary
During the advancement of human tissues, endoscopes are prone to encountering obstacles and positional errors, making it impossible to accurately determine the direction of advancement. In particular, at bifurcation points, it is difficult to avoid advancing into the wrong tissues or organs.
By acquiring the image position and image of the endoscope, the centerline network of the region of interest is used to correct the advancement position error, and the data is input into a pre-trained bifurcation prediction neural network model to predict the remaining advancement distance and time, mark the target and non-target branches at the bifurcation point, and guide the advancement of the endoscope in real time.
This improves the accuracy of the endoscope in advancing to the pre-selected endpoint, avoids advancing into the wrong branch, and ensures the safety and effectiveness of the surgery or examination.
Smart Images
Figure CN119632476B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of image processing, in particular, to an endoscope pushing guiding method and device, a storage medium and equipment. BACKGROUND
[0002] Human tissues are not rigid, and at different times, they will exhibit deformation such as stretching, shrinking, displacement, etc., thereby causing the endoscope to encounter obstacles during the pushing process. Therefore, not only is it necessary to guide the pushing during the pushing process of the endoscope, but it is also necessary to determine the position of the endoscope in the human body in real time and avoid pushing to the wrong tissues or organs. SUMMARY
[0003] In view of the above technical problems, the purpose of the present disclosure is to provide an endoscope pushing guiding method, device, storage medium and equipment.
[0004] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present disclosure provides an endoscope pushing guiding method, comprising:
[0005] obtaining a pushing position of an image part of the endoscope in a pushing process and an endoscope image collected by the image part;
[0006] correcting a cumulative error of the pushing position according to a center line network corresponding to the interest region to obtain a corrected pushing position;
[0007] inputting the endoscope image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model;
[0008] predicting a remaining pushing distance, a remaining pushing time and a target pushing branch at a bifurcation from the corrected pushing position to a pre-selected pushing end point according to the center line network and the bifurcation prediction result;
[0009] labeling the target pushing branch and a non-target branch of the bifurcation, and guiding the pushing according to the remaining pushing distance, the remaining pushing time and the bifurcation after branch labeling.
[0010] Optionally, the correcting the cumulative error of the pushing position according to the center line network corresponding to the interest region to obtain the corrected pushing position comprises:
[0011] determining a target key node corresponding to the pushing position from key nodes of the center line network corresponding to the interest region;
[0012] correcting the cumulative error of the pushing position according to a node position corresponding to the target key node to obtain the corrected pushing position.
[0013] Optionally, the predicting, according to the centerline network and the bifurcation prediction result, of a remaining advancing distance, a remaining advancing time and a target advancing branch at a bifurcation from the corrected advancing position to a pre-selected advancing end point comprises:
[0014] in a case that the bifurcation prediction result represents that there is a bifurcation ahead of the advancing direction of the endoscope, predicting a remaining advancing distance from the corrected advancing position to a pre-selected advancing end point along the centerline network;
[0015] predicting a remaining advancing time from the corrected advancing position to a pre-selected advancing end point according to the remaining advancing distance and an average speed of the endoscope in the current advancing process;
[0016] determining a target advancing branch from the corrected advancing position to a pre-selected advancing end point from each branch of the bifurcation represented by the bifurcation prediction result.
[0017] Optionally, the centerline network corresponding to the region of interest is established by:
[0018] constructing a three-dimensional model for the region of interest according to an original medical image of the region of interest to be examined;
[0019] performing image segmentation on the three-dimensional model to obtain a pipe network corresponding to each pipe type of the pipe system;
[0020] extracting a centerline from each pipe network corresponding to each pipe type to obtain a centerline network corresponding to the pipe network of each pipe type, wherein a line width of the centerline in the centerline network is one volume element and there is a unique path between any two volume elements on the centerline.
[0021] Optionally, the performing image segmentation on the three-dimensional model to obtain a pipe network corresponding to each pipe type of the pipe system comprises:
[0022] defining a target region including the region of interest in the three-dimensional model;
[0023] performing volume element scanning on the three-dimensional model in the target region according to an input exposure value range to obtain a volume element corresponding to the target region;
[0024] taking an input seed point as a starting point, searching for volume elements having a connection relationship in the three-dimensional model according to the exposure value range, and adding a display label to the volume elements having the connection relationship;
[0025] Hiding the volume elements in the three-dimensional model which are not added with the display label, to obtain a pipe network corresponding to the pipe system of each pipe type.
[0026] Optionally, the marking the target pushing branch and the non-target branch of the guide bifurcation includes:
[0027] The target pushing branch of the guide bifurcation is marked by a preset first color and a first preset marking style.
[0028] The non-target pushing branch of the guide bifurcation is marked by a preset second color and a second preset marking style.
[0029] In a second aspect of the embodiments of the present disclosure, an endoscope pushing guide device is provided, including:
[0030] The acquisition module is configured to acquire a pushing position of an image part of the endoscope in a pushing process and an endoscope image collected by the image part.
[0031] The correction module is configured to correct a cumulative error of the pushing position according to a center line network corresponding to the region of interest, to obtain a corrected pushing position.
[0032] The input module is configured to input the endoscope image into a bifurcation prediction neural network model pre-trained, to obtain a bifurcation prediction result output by the bifurcation prediction neural network model.
[0033] The prediction module is configured to predict, according to the center line network and the bifurcation prediction result, a remaining pushing distance, a remaining pushing time and a target pushing branch at a bifurcation from the corrected pushing position to a preselected pushing end point.
[0034] The display module is configured to mark the target pushing branch and the non-target branch of the bifurcation, and to guide pushing according to the remaining pushing distance, the remaining pushing time and the bifurcation after branch marking.
[0035] Optionally, the correction module is configured to:
[0036] Determine a target key node corresponding to the pushing position from key nodes of the center line network corresponding to the region of interest.
[0037] Correct the cumulative error of the pushing position according to a node position corresponding to the target key node, to obtain the corrected pushing position.
[0038] Optionally, the prediction module is configured to:
[0039] in a case that the bifurcation prediction result indicates that a bifurcation exists in front of the endoscope, predicting a remaining advancing distance from the corrected advancing position to a pre-selected advancing end point along the centerline network;
[0040] predicting a remaining advancing time from the corrected advancing position to the pre-selected advancing end point according to the remaining advancing distance and an average speed of the endoscope in the current advancing process;
[0041] determining a target advancing branch from the corrected advancing position to the pre-selected advancing end point from each branch of the bifurcation indicated by the bifurcation prediction result.
[0042] Optionally, the correction module is further configured to establish the centerline network corresponding to the region of interest by:
[0043] constructing a three-dimensional model for the region of interest according to an original medical image of the region of interest to be examined;
[0044] performing image segmentation on the three-dimensional model to obtain a pipe network corresponding to a pipe system of each pipe type;
[0045] performing centerline extraction on the pipe network corresponding to each pipe type to obtain a centerline network corresponding to the pipe network of each pipe type, wherein a line width of a centerline in the centerline network is one volume element, and there is a unique path between any two volume elements on the centerline.
[0046] Optionally, the correction module is further configured to:
[0047] demarcating a target region including the region of interest in the three-dimensional model;
[0048] performing volume element scanning on the three-dimensional model in the target region according to an input exposure value range to obtain a volume element corresponding to the target region;
[0049] taking an input seed point as a starting point, searching for volume elements having a connected relationship in the three-dimensional model according to the exposure value range, and adding a display label to the volume elements having the connected relationship;
[0050] hiding volume elements in the three-dimensional model that have not been added with a display label to obtain a pipe network corresponding to a pipe system of each pipe type.
[0051] Optionally, the display module is configured to:
[0052] marking the target advancing branch of the bifurcation by a preset first color and a first preset marking style;
[0053] The non-target pushing branch of the guide bifurcation is marked by presetting a second color and a second preset marking pattern.
[0054] In a third aspect, the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscope pushing guide method according to any one of the first aspect.
[0055] In a fourth aspect, the present disclosure provides an electronic device, comprising:
[0056] a first memory having a computer program stored thereon;
[0057] a second processor configured to execute the computer program in the first memory to implement the steps of the endoscope pushing guide method according to any one of the first aspect.
[0058] The above technical solution can achieve at least the following beneficial effects:
[0059] The pushing position of the image portion of the endoscope in the pushing process and the endoscope image collected by the image portion are obtained, the cumulative error of the pushing position is corrected according to the center line network corresponding to the interest region to obtain a corrected pushing position, which can avoid the continuous accumulation of the error of the pushing position, so that the accurate position of the endoscope cannot be accurately determined. Then, the endoscope image is input into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model. The remaining pushing distance, the remaining pushing time and the target pushing branch at the bifurcation from the corrected pushing position to the preselected pushing terminal point are predicted according to the center line network and the bifurcation prediction result. The target pushing branch and the non-target branch of the bifurcation are marked, and the bifurcation after the branch marking is pushed according to the remaining pushing distance, the remaining pushing time and the bifurcation. The endoscope can be guided in real time to avoid pushing the endoscope to the wrong branch. The accuracy of the endoscope pushing to the preselected pushing terminal point is improved.
[0060] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS
[0061] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the following detailed description, serve to explain the present disclosure. The drawings are not intended to limit the present disclosure, but to illustrate the present disclosure. In the drawings:
[0062] Figure 1 is a flowchart of an endoscope pushing guide method according to an embodiment of the present disclosure.
[0063] Figure 2is a method for establishing a center line network corresponding to an interest region according to an embodiment of the present disclosure. Figure 1 is a flowchart of the method in step S12.
[0064] Figure 3 is a method for establishing a center line network corresponding to an interest region according to an embodiment of the present disclosure. Figure 1 is a flowchart of the method in step S14.
[0065] Figure 4 is a flowchart of a method for establishing a center line network corresponding to an interest region according to an embodiment of the present disclosure.
[0066] Figure 5 is a method for establishing a center line network corresponding to an interest region according to an embodiment of the present disclosure. Figure 4 is a flowchart of the method in step S402.
[0067] Figure 6 is a block diagram of an endoscope pushing guide device according to an embodiment of the present disclosure.
[0068] Figure 7 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0069] The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.
[0070] Before introducing the endoscope pushing guide method, device, storage medium and equipment provided by the present disclosure, the scene to which the present disclosure is applied is introduced. In the pushing process of the endoscope, a bifurcation may be encountered, which causes the pushing direction to be unable to be determined.
[0071] To this end, an endoscope pushing guide method is provided in the embodiments of the present disclosure, which aims to guide the endoscope pushing in real time, mark and guide when a bifurcation is encountered, and avoid pushing the endoscope to the wrong branch. The accuracy of pushing the endoscope to the preselected pushing terminal point is improved, as shown in FIG. 1, which comprises the following steps. Figure 1
[0072] In step S11, the pushing position of the image part of the endoscope in the pushing process and the endoscope image collected by the image part are acquired.
[0073] It can be explained that in the embodiments of the present disclosure, the image part of the endoscope can be pushed according to the preplanned pushing path, and then in the pushing process, the images of the passing tissues and pipelines are collected in real time by the image part of the endoscope to obtain the endoscope image. The pushing position of the image part of the endoscope in the pushing process is acquired by the inertial sensor configured on the image part of the endoscope.
[0074] In the endoscope advancing process, the doctor is the decision maker and executor, and the method provided by the present disclosure can only determine whether the image part of the endoscope is advancing according to the pre-planned advancing path according to the advancing position of the image part of the endoscope. If the endoscope does not advance according to the pre-planned advancing path during the advancing process, a prompt can be given.
[0075] In step S12, the cumulative error of the advancing position is corrected according to the center line network corresponding to the region of interest, to obtain a corrected advancing position.
[0076] It can be explained that because the tissue is not rigid, it may exhibit deformation such as stretching, shrinking, displacement, etc. in different states, thereby causing the advancing position of the endoscope to deviate from the saved historical data. In some extreme cases, the saved historical data may also differ from the actual operation or examination. Therefore, the actually collected advancing position must be analyzed. If the error deviates from the reasonable range significantly and there is a major abnormal risk, not only the advancing position needs to be corrected, but also an alarm prompt can be given.
[0077] In the embodiments of the present disclosure, the target position of the endoscope can be determined from the center line network corresponding to the region of interest, and then the advancing position of the endoscope is corrected according to the target position. For example, in the case where the Euclidean distance between the target position and the advancing position of the endoscope exceeds a preset distance threshold, the target position in the center line network is used to replace the advancing position of the endoscope, that is, the target position in the center line network is used as the corrected advancing position. In the case where the Euclidean distance between the target position and the advancing position of the endoscope does not exceed the preset distance threshold, the advancing position of the endoscope is directly used as the corrected advancing position. In this way, it can be avoided that the advancing position of the endoscope accumulates errors during the advancing process, thereby causing the error between the advancing position of the endoscope and the actual position to increase continuously.
[0078] In step S13, the endoscope image is input into the pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model.
[0079] In the embodiments of the present disclosure, the bifurcation prediction neural network model may, for example, be a Yolo model. The original Yolo model can be trained by labeling the endoscope image with bifurcation to obtain a group of model parameters. During the operation or examination process, the endoscope image collected in real time by the image part of the endoscope is input into the trained Yolo model, and the Yolo model can give a prediction result of whether there is a bifurcation in the endoscope image. Moreover, in the case where there is a bifurcation, the prediction result can also label the directions and sizes of each branch of the bifurcation. The Yolo model can quickly and accurately realize bifurcation prediction.
[0080] In step S14, according to the center line network and the bifurcation prediction result, the remaining advancing distance, the remaining advancing time and the target advancing branch at the bifurcation from the corrected advancing position to the pre-selected advancing end point are predicted.
[0081] In the embodiments of the present disclosure, the remaining advancing distance from the corrected advancing position to the pre-selected advancing end point can be calculated along the center line network, and the error of the remaining advancing distance can be predicted due to the deformation such as stretching, shrinking and displacement of the tissue.
[0082] In the embodiments of the present disclosure, if the bifurcation prediction result indicates that there is no bifurcation in front of the advancing direction, only one target advancing branch can be directly displayed, and if the bifurcation prediction result indicates that there is a bifurcation in front of the advancing direction, the target advancing branch from the corrected advancing position to the pre-selected advancing end point at the bifurcation can be predicted. It is easy to understand that the target advancing branch can be several optimal advancing branches, and therefore, several target advancing branches can be displayed, or each target advancing branch can be selected to display one optimal target advancing branch, and the other branches can be regarded as non-target advancing branches.
[0083] In step S15, the target advancing branch and the non-target branch at the bifurcation are marked, and the advancing guidance is performed according to the remaining advancing distance, the remaining advancing time and the bifurcation after the branch marking.
[0084] In the embodiments of the present disclosure, the remaining advancing distance and the remaining advancing time can be announced and guided by voice, and then displayed by the display of the user interface. Similarly, the target advancing branch and the non-target branch after the branch marking can be guided by voice, and the target advancing branch can be displayed by the display of the user interface with a prominent color or mark, and the non-target branch can be displayed by the display of the user interface with a non-prominent color or mark.
[0085] In the embodiments of the present disclosure, the doctor can temporarily change the advancing direction and the advancing speed of the endoscope based on the displayed endoscopic image, such as finding a new lesion to temporarily change the advancing path. Furthermore, according to the current advancing position of the image part of the endoscope, several optimal path options from the advancing position to the pre-selected advancing end point can be planned in real time based on the pre-set constraint condition, and the several optimal path options can be displayed.
[0086] The technical solution obtains the pushing position of the image part of the endoscope in the pushing process and the endoscope image collected by the image part; the cumulative error of the pushing position is corrected according to the center line network corresponding to the interest region, to obtain a corrected pushing position, which can avoid the error of the pushing position from being accumulated continuously, so that the accurate position of the endoscope cannot be determined accurately; then the endoscope image is input into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model; the remaining pushing distance, the remaining pushing time and the target pushing branch at the bifurcation from the corrected pushing position to the pre-selected pushing end point are predicted according to the center line network and the bifurcation prediction result; the target pushing branch and the non-target branch of the bifurcation are marked, and the bifurcation after the branch marking is pushed according to the remaining pushing distance, the remaining pushing time and the bifurcation. The endoscope can be guided in real time during pushing, and the endoscope can be prevented from being pushed to the wrong branch. The accuracy of pushing the endoscope to the pre-selected pushing end point is improved.
[0087] In the embodiments of the present disclosure, referring to FIG. 12, in step S12, the cumulative error of the pushing position is corrected according to the center line network corresponding to the interest region, to obtain a corrected pushing position, which includes: Figure 2
[0088] In step S121, a target key node corresponding to the pushing position is determined from the key nodes of the center line network corresponding to the interest region.
[0089] In the embodiments of the present disclosure, the key nodes of the center line network corresponding to the interest region can be determined according to the type of the target tissue, the shape contour of the tissue, the characteristic size of the tissue, the normal or abnormal index of the tissue, the bifurcation, etc. The key nodes have relatively obvious characteristics and can be clearly seen from the endoscope image.
[0090] Further, the target key node can be a key node passed once at the pushing position, or a key node displayed in the current endoscope image.
[0091] In step S122, the cumulative error of the pushing position is corrected according to the node position corresponding to the target key node, to obtain a corrected pushing position.
[0092] In the embodiments of the present disclosure, the distance between the node position corresponding to the target key node and the pushing position can be calculated, and in the case where the distance exceeds a preset threshold, the node position is used to replace the pushing position to obtain the corrected pushing position. In this way, the node position can be used as the starting point of the next pushing in each pushing process, and the influence of the error of the previous pushing position on the subsequent pushing position can be avoided.
[0093] In the embodiments of the present disclosure, referring to FIG. 13, in step S13, the remaining pushing distance, the remaining pushing time and the target pushing branch at the bifurcation from the corrected pushing position to the pre-selected pushing end point are predicted according to the center line network and the bifurcation prediction result, which includes: Figure 3 As shown in step S14, the remaining advancing distance, the remaining advancing time, and the target advancing branch at the branch point are predicted from the modified advancing position to the pre-selected advancing end point according to the center line network and the branch prediction result, which comprises:
[0094] In step S141, the remaining advancing distance from the modified advancing position to the pre-selected advancing end point is predicted along the center line network in the case that the branch prediction result represents that there is a branch point in front of the advancing direction of the endoscope.
[0095] In the embodiments of the present disclosure, in the case that the branch prediction result represents that there is a branch point in front of the advancing direction of the endoscope, a plurality of remaining advancing distances from the modified advancing position to the pre-selected advancing end point are predicted along the center line network. Each of the remaining advancing distances corresponds to a branch point branch, so that a suitable target advancing branch can be selected from different remaining advancing distances. For example, in the case that there is a branch point, if it is intended to bypass other possible lesion tissues, the cost of the remaining advancing distance of each target advancing branch can be referred to, so as to avoid bypassing with too large cost, thereby causing unreasonable surgery or examination.
[0096] In step S142, the remaining advancing time from the modified advancing position to the pre-selected advancing end point is predicted according to the remaining advancing distance and the average speed of the endoscope in the current advancing process.
[0097] In the embodiments of the present disclosure, since different human bodies have different tissues, for example, the diameters of the human body pipeline, if the remaining advancing time is calculated according to the pre-set advancing direction and advancing speed, a large error may be caused in the calculated remaining advancing time. Therefore, the remaining advancing time from the modified advancing position to the pre-selected advancing end point can be predicted according to the average speed of the endoscope in the current advancing process, so as to improve the accuracy of the calculated remaining advancing time.
[0098] It is worth noting that the average speed of the endoscope in the current advancing process herein refers to the average speed calculated according to the accumulated time and the accumulated distance in the case that the endoscope is advancing forward. If the endoscope stops at any position for observation, the observation time will not be recorded in the accumulated time.
[0099] In step S143, the target advancing branch from the modified advancing position to the pre-selected advancing end point is determined from each branch of the branch point represented by the branch prediction result.
[0100] The target advancing branch from the modified advancing position to the pre-selected advancing end point can be one or more.
[0101] In the embodiments of the present disclosure, referring to Figure 4 As shown, the center line network corresponding to the interest region is established by the following method:
[0102] In step S401, a three-dimensional model of the region of interest is constructed based on the original medical image of the region of interest to be examined.
[0103] In this embodiment, a three-dimensional model of the region of interest can be constructed from original medical images taken from different angles. Alternatively, other existing technologies can be used to construct the three-dimensional model, which will not be elaborated here.
[0104] In step S402, the three-dimensional model is segmented to obtain the pipe network corresponding to each pipe type.
[0105] In this embodiment of the disclosure, a traditional image segmentation algorithm based on morphological operations can be used, or a neural network image segmentation algorithm based on deep learning networks, such as the UNet network, can be used to segment the three-dimensional model to obtain the pipe network corresponding to each pipe type in the human body pipe system.
[0106] In step S403, the centerline of the pipeline network corresponding to each pipeline type is extracted to obtain the centerline network corresponding to each pipeline type.
[0107] The channels can include: channels that exist naturally in the human body, various sinuses formed by pathological factors, and surgical pathways formed by the anatomy of human tissues to reach the surgical area.
[0108] In this network, the line width of the centerline is a volume element, and there is a unique path between any two volume elements on the centerline.
[0109] In this embodiment, for the aforementioned pipeline network, a centerline extraction algorithm can be used to extract the centerline of the pipeline network. The resulting centerline network is equivalent to the pipeline network in terms of topology and positioning function, thus simplifying the network structure. The centerline network has the following characteristics: 1) the linewidth of the centerline is a volume element; 2) there is a unique and non-repeating path between any two voxels on the centerline network.
[0110] The above technical solution can perform image segmentation on the constructed 3D model, extract the pipe network, and then determine the centerline network corresponding to the pipe network, thereby reducing the influence between different types of pipes in the image.
[0111] In this disclosure embodiment, see Figure 5 As shown, in step S402, the image segmentation of the three-dimensional model to obtain the pipe network corresponding to each pipe type includes:
[0112] In step S4021, a target region including a region of interest is demarcated in the three-dimensional model.
[0113] In the embodiments of the present disclosure, a three-dimensional region is demarcated as a target region from a space range in which the three-dimensional model is located, and the target region includes a lesion region of a patient, wherein the shape of the target region can be a cube, an oblique cube, a cylinder, an oblique cylinder, or other irregular three-dimensional bodies.
[0114] Further, in the three-dimensional model, other volume elements located outside the target region are added with a hidden label, and the volume elements marked with the hidden label are hidden when the three-dimensional model is presented, that is, the volume elements marked with the hidden label are not displayed.
[0115] In step S4022, volume elements of the three-dimensional model in the target region are scanned according to an input exposure value range, and volume elements corresponding to the target region are obtained.
[0116] It can be understood that the exposure value range includes a lower threshold and an upper threshold, the volume elements of the three-dimensional model in the target region are scanned, the volume elements in the target region whose exposure values fall outside the exposure value range are added with a hidden label, and then the volume elements in the target region marked with the hidden label are hidden.
[0117] In step S4023, a seed point input by a user is taken as a starting point, volume elements having a connection relationship in the three-dimensional model are searched according to the exposure value range, and the volume elements having the connection relationship are added with a display label.
[0118] In the embodiments of the present disclosure, the seed point input by the user is input in real time according to the surgical requirements, and in specific implementation, the volume elements having the connection relationship are added with the display label, and the volume elements not having the connection relationship are added with the hidden label.
[0119] In step S4024, the volume elements in the three-dimensional model not added with the display label are hidden, and a pipe network corresponding to a pipe system of each pipe type is obtained.
[0120] In the embodiments of the present disclosure, the volume elements marked with the hidden label can be hidden, and the volume elements marked with the display label can be displayed, and a pipe network corresponding to a pipe system of each pipe type in the human pipe system is obtained.
[0121] Optionally, in step S15, the target pushing branch and the non-target branch of the guide bifurcation are marked, including:
[0122] The target pushing branch of the guide bifurcation is marked by a preset first color and a first preset marking style.
[0123] The non-target pushing branch of the guide bifurcation is marked by presetting a second color and a second preset marking style.
[0124] In the embodiments of the present disclosure, the first preset marking style can be adding an arrow guide to the branch, and then the pushing can be performed according to the preset first color arrow through voice prompts. The second preset marking style can be the same as the first preset marking style, and only the target pushing branch and the non-target pushing branch are distinguished and displayed through color, or the second preset marking style can be different from the first preset marking style, and then the target pushing branch and the non-target pushing branch are distinguished and displayed through different colors and marking styles.
[0125] Further, in the case where there are several target pushing branches, the target pushing branches can be marked by different colors, wherein the eye-catching degree of the preset first color is higher than that of the preset second color.
[0126] The embodiments of the present disclosure also provide an endoscope pushing guide device, referring to FIG. 6, Figure 6 As shown in FIG. 6, the endoscope pushing guide device 600 comprises:
[0127] The acquisition module 610 is configured to acquire a pushing position of an image part of the endoscope in a pushing process and an endoscope image collected by the image part;
[0128] The correction module 620 is configured to correct a cumulative error of the pushing position according to a center line network corresponding to the region of interest, to obtain a corrected pushing position;
[0129] The input module 630 is configured to input the endoscope image into a bifurcation prediction neural network model pre-trained, to obtain a bifurcation prediction result output by the bifurcation prediction neural network model;
[0130] The prediction module 640 is configured to predict a remaining pushing distance, a remaining pushing time and a target pushing branch at a bifurcation from the corrected pushing position to a preselected pushing end point according to the center line network and the bifurcation prediction result;
[0131] The display module 650 is configured to mark the target pushing branch and the non-target branch of the bifurcation, and to perform pushing guidance according to the remaining pushing distance, the remaining pushing time and the bifurcation after branch marking.
[0132] Optionally, the correction module 620 is configured to:
[0133] determine a target key node corresponding to the pushing position from key nodes of the center line network corresponding to the region of interest;
[0134] According to the node position corresponding to the target key node, the cumulative error of the pushing position is corrected to obtain a corrected pushing position.
[0135] Optionally, the prediction module 640 is configured to:
[0136] In a case where the bifurcation prediction result represents that there is a bifurcation at the front of the pushing of the endoscope, a remaining pushing distance from the corrected pushing position to a pre-selected pushing end point is predicted along the center line network;
[0137] According to the remaining pushing distance and an average speed of the endoscope in the current pushing process, a remaining pushing time from the corrected pushing position to the pre-selected pushing end point is predicted.
[0138] From each branch of the bifurcation represented by the bifurcation prediction result, a target pushing branch from the corrected pushing position to the pre-selected pushing end point is determined.
[0139] Optionally, the correction module 620 is further configured to establish the center line network corresponding to the region of interest by:
[0140] According to the original medical image of the region of interest to be checked, a three-dimensional model for the region of interest is constructed;
[0141] Image segmentation is performed on the three-dimensional model to obtain a pipe network corresponding to a pipe system of each pipe type;
[0142] Center line extraction is performed on the pipe network corresponding to each pipe type to obtain a center line network corresponding to the pipe network of each pipe type, wherein the line width of the center line in the center line network is one volume element, and there is a unique path between any two volume elements on the center line.
[0143] Optionally, the correction module 620 is further configured to:
[0144] A target region including the region of interest is demarcated in the three-dimensional model;
[0145] According to an input exposure value range, volume element scanning is performed on the three-dimensional model in the target region to obtain a volume element corresponding to the target region;
[0146] Taking an input seed point as a starting point, according to the exposure value range, volume elements having a connection relationship in the three-dimensional model are searched, and display labels are added to the volume elements having the connection relationship;
[0147] Volume elements in the three-dimensional model that have not been added with display labels are hidden to obtain a pipe network corresponding to a pipe system of each pipe type.
[0148] Optionally, the display module 650 is configured to:
[0149] mark the target advancing branch of the guide fork by presetting a first color and a first preset marking style;
[0150] mark the non-target advancing branch of the guide fork by presetting a second color and a second preset marking style.
[0151] As to the endoscope advancing guide device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.
[0152] Those skilled in the art understand that the device embodiments described above are only illustrative, for example, the division of modules is only a logical functional division, and other division manners can be used in actual implementation, for example, multiple modules can be combined or integrated into one module. In addition, the modules described as separate components can be or can not be physically separated. Moreover, each module can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented entirely or partially in the form of a computer program product. When implemented by hardware, it can be implemented entirely or partially in the form of an integrated circuit or a chip.
[0153] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscope advancing guide method in any of the foregoing embodiments.
[0154] The embodiments of the present disclosure further provide an electronic device, comprising:
[0155] a first memory having a computer program stored thereon;
[0156] a second processor configured to execute the computer program in the first memory to implement the steps of the endoscope advancing guide method in any of the foregoing embodiments.
[0157] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. The electronic device 700 can be configured as an endoscope advancing guide device, as shown in Figure 7 The electronic device 700 can include a third processor 701 and a third memory 702. The electronic device 700 can further include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.
[0158] The third processor 701 is configured to control overall operations of the electronic device 700 to complete all or part of the steps of the endoscope advancement guidance method described above. The third memory 702 is configured to store various types of data to support operations of the electronic device 700, which can include, for example, instructions for any application or method operating on the electronic device 700, and application-related data, such as pictures, etc. The third memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The multimedia component 703 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the third memory 702 or transmitted through the communication component 705. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 704 provides an interface between the third processor 701 and other interface modules, which can be a keyboard, a mouse, a button, etc. These buttons can be virtual buttons or physical buttons, and are configured to operate the endoscope advancement guidance. The communication component 705 is configured to perform wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited herein. Therefore, the corresponding communication component 705 can include a Wi-Fi module, a Bluetooth module, an NFC module, etc.
[0159] In an exemplary embodiment, the electronic device 700 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for performing the endoscope pushing guide method described above.
[0160] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the endoscope pushing guide method described above. For example, the computer-readable storage medium can be the third memory 702 described above including program instructions, which can be executed by the third processor 701 of the electronic device 700 to complete the endoscope pushing guide method described above.
[0161] The preferred embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details of the above-described embodiments. Various simple modifications can be made to the technical solutions of the present disclosure within the technical concept of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
[0162] In addition, it should be noted that each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combinations are not described again in the present disclosure.
[0163] Furthermore, any combination of the various different embodiments of the present disclosure can also be made, as long as it does not deviate from the idea of the present disclosure, and it should also be considered as disclosed by the present disclosure.
Claims
1. An endoscope advancement guidance method, characterized by, The method comprises the following steps: acquiring a pushing position of an image part of the endoscope in a pushing process and an endoscope image collected by the image part; correcting a cumulative error of the pushing position according to a center line network corresponding to a region of interest to obtain a corrected pushing position; inputting the endoscope image into a pre-trained bifurcation prediction neural network model to obtain a bifurcation prediction result output by the bifurcation prediction neural network model; predicting, according to the center line network and the bifurcation prediction result, a remaining pushing distance, a remaining pushing time and a target pushing branch at a bifurcation from the corrected pushing position to a preselected pushing end point; labeling the target pushing branch and a non-target branch at the bifurcation and performing pushing guidance according to the remaining pushing distance, the remaining pushing time and the bifurcation after labeling the branches; wherein the correcting the cumulative error of the pushing position according to the center line network corresponding to the region of interest to obtain the corrected pushing position comprises: determining a target key node corresponding to the pushing position from key nodes of the center line network corresponding to the region of interest; correcting the cumulative error of the pushing position according to a node position corresponding to the target key node to obtain the corrected pushing position, comprising: calculating a distance between the node position corresponding to the target key node and the pushing position, and replacing the pushing position with the node position corresponding to the target key node to obtain the corrected pushing position when the distance exceeds a preset threshold.
2. The endoscope advancement guidance method of claim 1, wherein, The predicting, according to the center line network and the bifurcation prediction result, of the remaining pushing distance, the remaining pushing time and the target pushing branch at the bifurcation from the corrected pushing position to the preselected pushing end point comprises: in a case where the bifurcation prediction result represents that there is a bifurcation in front of the endoscope during pushing, predicting a remaining pushing distance from the corrected pushing position to the preselected pushing end point along the center line network; predicting a remaining pushing time from the corrected pushing position to the preselected pushing end point according to the remaining pushing distance and an average speed of the endoscope in the current pushing process; determining a target pushing branch from the corrected pushing position to the preselected pushing end point from each branch of the bifurcation represented by the bifurcation prediction result.
3. The endoscope advancement guidance method of claim 1, wherein, The center line network corresponding to the region of interest is established in the following manner: constructing a three-dimensional model for the region of interest according to an original medical image of the region of interest to be examined; performing image segmentation on the three-dimensional model to obtain a pipe network corresponding to each pipe type of a pipe system; extracting a center line from each pipe network corresponding to each pipe type to obtain a center line network corresponding to the pipe network of each pipe type, wherein the width of the center line in the center line network is one volume element and there is a unique path between any two volume elements on the center line.
4. The endoscope advancement guidance method of claim 3, wherein, The image segmentation on the three-dimensional model to obtain a pipe network corresponding to each pipe type of a pipe system comprises: delimiting a target region including the region of interest in the three-dimensional model; According to the input exposure value range, a volume element scanning is performed on the three-dimensional model in the target region to obtain a volume element corresponding to the target region; Taking the input seed point as a starting point, volume elements having a connection relationship in the three-dimensional model are searched according to the exposure value range, and display tags are added to the volume elements having the connection relationship; Volume elements in the three-dimensional model that have not added display tags are hidden to obtain a pipe network corresponding to a pipe system of each pipe type.
5. The endoscope advancement guidance method of any of claims 1-4, wherein, The marking of the target advancing branch and the non-target branch of the bifurcation includes: The target advancing branch of the bifurcation is marked by a preset first color and a first preset marking style; The non-target advancing branch of the bifurcation is marked by a preset second color and a second preset marking style.
6. An endoscope advancement guide device, comprising: It includes: An acquisition module configured to acquire a pushing position of an image part of the endoscope in a pushing process and an endoscope image collected by the image part; A correction module configured to correct a cumulative error of the pushing position according to a center line network corresponding to a region of interest to obtain a corrected pushing position; An input module configured to input the endoscope image into a bifurcation prediction neural network model pre-trained to obtain a bifurcation prediction result output by the bifurcation prediction neural network model; A prediction module configured to predict, according to the center line network and the bifurcation prediction result, a remaining pushing distance, a remaining pushing time, and a target advancing branch at a bifurcation from the corrected pushing position to a preselected pushing end point; A display module configured to mark the target advancing branch and the non-target branch of the bifurcation and to perform pushing guidance according to the remaining pushing distance, the remaining pushing time, and the bifurcation after branch marking; The correction module is configured to: Determine a target key node corresponding to the pushing position from key nodes of the center line network corresponding to the region of interest; Correct the cumulative error of the pushing position according to a node position corresponding to the target key node to obtain the corrected pushing position, including: calculating a distance between the node position corresponding to the target key node and the pushing position, and replacing the pushing position with the node position corresponding to the target key node to obtain the corrected pushing position if the distance exceeds a preset threshold.
7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the first processor to implement the steps of the endoscope pushing guidance method of any one of claims 1-5.
8. An electronic device, comprising: It includes: A first memory having a computer program stored thereon; A second processor configured to execute the computer program in the first memory to implement the steps of the endoscope pushing guidance method of any one of claims 1-5.
Citation Information
Patent Citations
Endoscope apparatus
JP2009056239A
Motion navigation method, apparatus, and device for bronchoscope
WO2023124981A1