Endoscope navigation control method, computer device and computer storage medium

Abstract

Embodiments of the present application disclose an endoscope navigation control method, a computer device and a computer storage medium. By acquiring multi-dimensional input information of continuous endoscopic images, quantitatively evaluating the consistency of timing semantics according to the preset path stage related constraints, and finally converting the evaluation results into navigation control actions adapted to the current state, since the path stage related semantic constraints are introduced, the stage misjudgment problem can be found in time, and the noise interference and the real path deviation are distinguished by combining the continuous state of the historical semantics, and then the low credibility output is actively restricted by the hierarchical navigation control actions, instead of only giving a prompt, which can effectively distinguish the short-term semantic anomaly and the continuous path deviation, while ensuring the stability of the navigation output, avoiding the misleading of the low credibility navigation output to the operation, and solving the navigation output error problem caused by single frame semantic misjudgment and short-term image quality interference in the existing endoscope navigation.

Description

Technical Field

[0001] This application relates to the field of endoscope navigation control, specifically to an endoscope navigation control method, computer equipment, and computer storage medium. Background Technology

[0002] Bronchoscopy navigation refers to the technology of using imaging data to construct a three-dimensional path model of the bronchial tree, and using this three-dimensional path model to guide the bronchoscope to the target lesion in real time during the operation. It is now widely used in the diagnosis and interventional treatment of peripheral lung diseases.

[0003] However, existing bronchoscopic navigation systems mostly rely on semantic recognition of single-frame images or single visual confidence to determine whether the navigation is reliable. They cannot distinguish between instantaneous semantic fluctuations caused by short-term obstruction, mucus contamination, and motion blur and actual navigation drift or path stage errors. This can lead to navigation prompts, automatic advancement, or bifurcation selection continuing to be output even when confidence is low, misleading the surgeon. In severe cases, it can even cause airway damage and navigation failure, thereby prolonging the operation time and increasing the risk of complications. Summary of the Invention

[0004] This application provides an endoscopic navigation control method, a computer device, and a computer storage medium to improve the accuracy and stability of endoscopic navigation output, avoid misleading operations due to single-frame misjudgment or short-term image interference, and reduce surgical risks.

[0005] A first aspect of this application provides an endoscope navigation control method, the method comprising:

[0006] Acquire the endoscopic semantic results of continuous endoscopic images during the endoscopic examination, the current navigation path stage, the historical semantic state window, the pose estimation results of the endoscope tip, and the current navigation output;

[0007] Based on the current navigation path stage, the corresponding expected semantic sequence and semantic allowed transition table are retrieved. The expected semantic sequence contains the semantic states expected to occur under the current navigation path stage, and the semantic allowed transition table is used to limit the legal transition relationships between adjacent semantic states.

[0008] Based on the under-vision semantic results, the expected semantic sequence, the semantic allowed transition table, the pose estimation results, the current navigation output, and the image quality factors of the under-vision image, the temporal semantic consistency score of the current navigation state is determined; the temporal semantic consistency score is used to characterize the degree of matching and temporal stability between continuous under-vision semantics and navigation path stages;

[0009] The navigation credibility of the current navigation state is determined based on the temporal semantic consistency score, and corresponding navigation control actions are applied to the current navigation output according to the navigation credibility.

[0010] A second aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect described above.

[0011] A third aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0012] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0013] By acquiring multi-dimensional input information from images under continuous imaging, and performing a quantitative evaluation of temporal semantic consistency based on preset path stage-related constraints, the evaluation results are finally transformed into navigation control actions adapted to the current state. Due to the introduction of path stage-related semantic constraints, stage misjudgment problems can be detected in a timely manner. At the same time, by combining the continuous state of historical semantics, noise interference and true path deviation can be distinguished. Furthermore, by actively limiting low-confidence outputs through graded navigation control actions, rather than simply providing prompts, it can effectively distinguish between short-term semantic anomalies and continuous path deviations. While ensuring the stability of navigation output, it avoids low-confidence navigation outputs from misleading operations, thus solving the navigation output error problem caused by single-frame semantic misjudgment and short-term image quality interference in existing endoscopic navigation. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating an endoscope navigation control method in an embodiment of this application.

[0015] Figure 2 This is a schematic diagram illustrating an exemplary application scenario of the endoscope navigation control system in this application embodiment;

[0016] Figure 3 This is a schematic diagram of a computer device in an embodiment of this application. Detailed Implementation

[0017] This application provides an endoscopic navigation control method, a computer device, and a computer storage medium to improve the accuracy and stability of endoscopic navigation output, avoid misleading operations due to single-frame misjudgment or short-term image interference, and reduce surgical risks.

[0018] Bronchoscope images are easily affected by local occlusion, mucus and blood contamination, changes in lighting, motion blur, and instrument occlusion. Therefore, a low semantic confidence score in a single frame does not necessarily indicate navigation failure, while a high semantic confidence score in a single frame may mislead navigation due to path stage errors.

[0019] The relevant technical approach can be summarized as "single-frame semantic recognition + navigation prompt display". This approach typically consists of a bronchoscopy image acquisition module, a semantic recognition module, a navigation and positioning module, and a display interface. The system segments, identifies bifurcations, lumens, or instruments in the current frame image, and overlays the single-frame semantic results onto the navigation interface or as a reference for navigation prompts.

[0020] The data link for this route is as follows: acquire the current frame of the intraocular image → output the single-frame intraocular semantic results and semantic confidence level → display prompts based on the current navigation pose → the surgeon manually determines whether to continue trusting the navigation. The limitation of this route is that it lacks a stable navigation confidence control logic regarding whether the semantic recognition results match the current path stage, whether they are consistent with the semantic transitions of previous and subsequent frames, and whether they are consistent with the pose estimation. As a result, single-frame noise may lead to erroneous degradation, continuous path deviations may not be identified in a timely manner, and the navigation display and automatic control behavior cannot be stably constrained based on the confidence level of the semantic temporal sequence, severely affecting the accuracy of bronchoscopic navigation.

[0021] Among them, false downgrading refers to the situation where, due to occasional factors such as short-term occlusion, mucus contamination, and motion blur, the semantic confidence of a single frame is abnormal. Existing technologies directly determine that the overall credibility of navigation is insufficient, triggering unnecessary navigation downgrading or pausing, disrupting the smoothness of the surgical operation, and increasing the operator's workload.

[0022] The endoscope navigation control method in the embodiments of this application is described below:

[0023] The endoscopic navigation control method provided in this application can be applied to bronchoscopy navigation scenarios, and can also be extended to other interventional navigation scenarios through natural cavities such as gastroscopy and colonoscopy. This embodiment uses bronchoscopy navigation as a typical application scenario for illustration.

[0024] Please see Figure 1 One embodiment of the endoscopic navigation control method in this application includes:

[0025] 101. Obtain the endoscopic semantic results, current navigation path stage, historical semantic state window, pose estimation results of the endoscope end, and current navigation output of continuous endoscopic images during the endoscopic examination.

[0026] Microscopic semantic results specifically refer to the anatomical structure category labels obtained by semantic segmentation or target detection of microscopic images. For example, they may include categories such as tracheal wall, carina, left main bronchus, right main bronchus, bronchial openings of each lobe segment, lumen center, lesion tissue, surgical instruments, mucus, and blood.

[0027] The semantic results obtained in this step can be semantic results obtained by combining semantic confidence correction. That is, the original semantic recognition results output by the semantic recognition network are filtered or weighted by the corresponding semantic confidence. Recognition results with low confidence are not directly used as the final semantic results. For example, recognition results with confidence below a preset threshold will be marked as unknown areas and will not participate in subsequent matching calculations to avoid recognition errors affecting subsequent judgments.

[0028] The acquisition frequency of images under continuous imaging can be consistent with the frame rate of the endoscope video, or frame extraction can be performed according to computing performance to ensure real-time performance while reducing computing load. For example, on embedded platforms with limited computing performance, one frame can be extracted every three frames for processing.

[0029] The current navigation path stage refers to the specific segment of the path in which the endoscope is located when it advances along the preoperative planned path, such as the trachea stage, carina stage, left main bronchus stage, right main bronchus stage, and bronchus segments of each lobe. Different path stages correspond to different anatomical structural features. Stage-by-stage reference matching can effectively narrow the reference range and reduce cross-stage mismatches.

[0030] The historical semantic state window is a time-series sliding window composed of semantic results, confidence scores, and image quality labels from multiple consecutive historical control cycles. The window length can be dynamically adjusted according to the navigation speed. For example, the window length can be appropriately shortened when the navigation speed is fast, and appropriately extended when the navigation speed is slow or when the navigation is stationary, to ensure the stability of the time-series evaluation. For example, the window can contain the most recent 10 frames of data when the navigation is fast, and the window can contain the most recent 30 frames of data when the navigation is stationary.

[0031] The pose estimation result can be obtained by fusing electromagnetic positioning, visual SLAM, fiber optic positioning or multiple positioning methods. It is used to represent the six-degree-of-freedom pose of the endoscope tip in the preoperative three-dimensional bronchial tree model coordinate system, namely the position coordinates and the attitude angles in three directions. During fusion, Kalman filtering can be used to weight and fuse data from different positioning sources to improve pose stability.

[0032] The current navigation output includes all navigation prompts and automatic control commands output by the system in the current cycle, such as path arrows superimposed on the endoscopic image, distance prompts to the lesion, automatic propulsion commands sent to the robot actuator, and steering control commands.

[0033] Therefore, by acquiring multi-source time-series data instead of relying solely on a single frame image and the current pose, a complete data foundation is provided for subsequent time-series consistency assessment, enabling the system to distinguish between short-term noise and continuous anomalies from the time-series dimension, thereby reducing misjudgments caused by single-frame judgments at the source.

[0034] 102. Based on the current navigation path stage, retrieve the corresponding expected semantic sequence and semantic allowed transition table. The expected semantic sequence contains the semantic states expected to occur under the current navigation path stage, and the semantic allowed transition table is used to limit the legal transition relationships between adjacent semantic states.

[0035] The expected semantic sequence retrieved in this step can be predetermined based on the preoperative three-dimensional bronchial tree model and the planned path. That is, it is the sequence of structures that should be observed in anatomical order during the endoscopic advancement at the current path stage. For example, in the trachea stage, it is expected that the tracheal rings, tracheal wall, carina tip, and carina ridge structures can be observed in sequence. In the right main bronchus stage, it is expected that the right main bronchus lumen, right upper lobe opening, and right intermediate bronchus structures can be observed in sequence. This sequence is arranged strictly according to the anatomical order of the bronchial tree and there will be no skipping of anatomical structures.

[0036] The semantic permissible transfer table is a predefined list of legal transfer relationships between adjacent semantic states based on the fixed anatomical topology of the bronchial tree. Specifically, it can be implemented as a two-dimensional Boolean matrix. In this matrix, rows represent the semantics of the current frame, and columns represent the semantics of the next frame. A matrix value of 1 indicates that the transfer is anatomically permissible, while a value of 0 indicates that the transfer is anatomically impossible. For example, transfers from the tracheal wall to the carina, from the carina to the opening of the left main bronchus, and from the carina to the opening of the right main bronchus are all legal transfers, while a transfer from the tracheal wall directly to the opening of a segmental bronchus is anatomically impossible because the segmental bronchus opening cannot be directly observed from the trachea. This table can be automatically generated based on the planned path during the preoperative planning stage, or dynamically updated during surgery based on the real-time path, to accommodate situations where the surgeon temporarily changes the path to enter other bronchi.

[0037] Therefore, by introducing prior knowledge of bronchial tree anatomy as a reference standard, an objective basis that conforms to anatomical rules is provided for the judgment of temporal consistency, rather than relying solely on data-driven similarity calculations. This significantly improves the accuracy of anomaly judgment and is more in line with the actual clinical scenario of bronchoscopy navigation.

[0038] 103. Based on the under-vision semantic results, the expected semantic sequence, the semantic allowed transition table, the pose estimation results, the current navigation output, and the image quality factors of the under-vision image, determine the temporal semantic consistency score of the current navigation state;

[0039] The temporal semantic consistency score can be used to characterize the degree of matching and temporal stability between sequential endoscopic semantics and navigation path stages. Specifically, the degree of matching and temporal stability between sequential endoscopic semantics and navigation path stages refers to the fact that the endoscopic semantics obtained in the current endoscopic control cycle must not only conform to the pre-defined anatomical location occurrence pattern of the current navigation path stage, but also maintain consistency with the semantic results obtained in previous consecutive control cycles, conforming to the path progression logic. It should not conflict with the expected semantic requirements of the current stage, nor should it exhibit abrupt semantic changes that do not conform to the anatomical path progression pattern within a short period of time.

[0040] Therefore, this embodiment determines the temporal semantic consistency score of the current navigation state. The logic is that navigation deviations in endoscopic surgery are often not instantaneous but accumulate gradually as the path progresses. It is difficult to distinguish between short-term interference and true path deviation based solely on the semantic recognition results of a single frame. Integrating the semantic state, path stage requirements, and pose matching relationships of multiple consecutive frames into a unified consistency score can filter out short-term interference such as occlusion and noise, avoiding unnecessary navigation interruptions or relocations. It can also identify continuously accumulating path deviations in advance, reducing navigation risks through hierarchical control before the deviation develops into a positioning error, thus ensuring the safety of the surgical operation. At the same time, this scoring logic is not bound to a specific semantic recognition model or image preprocessing method, and has good adaptability to different types of endoscopic surgeries and endoscopic equipment with different hardware parameters.

[0041] Specifically, by integrating multi-dimensional consistency evaluation information, the judgment errors caused by image quality and short-term occlusion interference in single-frame semantic recognition results can be avoided. Temporal semantic consistency is transformed into a quantifiable navigation state reliability index, providing a decision-making basis for subsequent hierarchical navigation control. This consistency evaluation information refers to multi-dimensional information used to measure whether the current semantic result conforms to the expected patterns of the path stage and whether it maintains temporal continuity and stability. It can include the matching relationship between the current single-frame semantics and the expected semantics of the current stage, the rationality of historical semantic changes, and the spatial consistency between semantic position and pose estimation results. It can also incorporate penalties and corrections due to image quality. The final score comprehensively reflects the overall reliability of the current navigation result, rather than relying solely on a single dimension, effectively improving the robustness of navigation credibility judgment.

[0042] This approach differs from traditional schemes that rely solely on single-frame semantic confidence or fixed-window averaging. It combines prior constraints of the path phase, temporal stability of historical semantics, and spatial matching of pose estimation. It can accurately distinguish between short-term noise interference and actual navigation path deviation, ensuring navigation response sensitivity while avoiding frequent false triggering of relocation operations, thus balancing the autonomy of autonomous navigation with the safety of surgical procedures.

[0043] Secondly, through multi-dimensional temporal consistency evaluation, it can accurately identify continuous navigation drift and path errors, and effectively filter noise interference caused by short-term occlusion, contamination and blurring. This solves the problem of easy misjudgment in single-frame judgment of existing technologies and greatly improves the accuracy of credibility judgment.

[0044] 104. Determine the navigation credibility of the current navigation state based on the temporal semantic consistency score, and apply corresponding navigation control actions to the current navigation output according to the navigation credibility;

[0045] Optionally, when determining navigation confidence in this step, the temporal semantic consistency score can be smoothed to avoid fluctuations in the navigation confidence caused by single-control-cycle score changes. Smoothing methods can include exponential moving average, Kalman filtering, or window value filtering. For example, when using exponential moving average, the navigation confidence of the current control cycle can be updated based on the temporal semantic consistency score Tscore of the current control cycle and the navigation confidence of the previous cycle Nconf(t-1). The calculation method can be as follows:

[0046] Nconf(t) = beta·Nconf(t-1) + (1-beta)·Tscore; where beta is preferably 0.6, and the adjustable range is 0.4 to 0.8.

[0047] For example, the new navigation confidence value could be 0.8 times the navigation confidence value of the previous control cycle plus 0.2 times the current temporal semantic consistency score, achieving a similar effect to low-pass filtering and filtering out high-frequency single-control cycle fluctuations.

[0048] The advancement of the surgical path is an ordered temporal process. Under normal navigation, the semantics will change in an orderly manner along the expected path stages. Single-frame semantic recognition itself has uncertainties. Short-term occlusion, instrument interference, etc., can lead to errors in single recognition, but will not change the overall semantic order trend of the path. Therefore, this embodiment can effectively filter noise interference from single-frame recognition and promptly detect continuous semantic deviations caused by navigation drift by judging the consistency of temporal semantics and path stages. Without interrupting the surgical rhythm, it can hierarchically control the risk of navigation output, ensuring the efficiency of autonomous navigation and avoiding the surgical risks caused by erroneous navigation.

[0049] The navigation reliability is determined based on temporal semantic consistency scores. The logic behind this method is that the semantic recognition results of a single endoscopic image are susceptible to interference from short-term occlusion, motion blur, and mirror contamination, leading to occasional misidentifications. Relying solely on single-frame results to judge the navigation status can easily cause unnecessary navigation degradation or frequent relocation, affecting the smoothness of the surgical procedure. However, the navigation path itself progresses in stages according to the anatomical path, and the changes in semantic state follow predictable patterns. Combining the continuity of path stages and historical semantic states to judge the reliability of semantic results can effectively distinguish between occasional short-term noise and genuine navigation deviations. This allows for timely identification of true navigation anomalies while avoiding interference from ineffective operations, ultimately improving the stability and robustness of endoscopic autonomous navigation.

[0050] This embodiment applies corresponding navigation control actions to the current navigation output based on navigation reliability because during autonomous navigation, the navigation state is not limited to only two extreme cases: "reliable" and "completely unreliable." Different levels of reliability correspond to different levels of risk, and the corresponding navigation control strategies should match these risk levels. By applying appropriate control actions based on navigation reliability, the smoothness of autonomous navigation can be ensured when navigation reliability is sufficiently high, fully leveraging the advantages of autonomous navigation in improving operational efficiency and reducing the surgeon's workload. Conversely, when navigation reliability decreases, the permissions of autonomous navigation can be gradually restricted to promptly reduce the operational risks caused by navigation output errors, while avoiding direct interruption of navigation and disruption to the surgical rhythm, thus achieving a balance between safety and operational smoothness.

[0051] Therefore, in this embodiment, by acquiring multi-dimensional input information from continuous images, a quantitative evaluation of temporal semantic consistency is completed based on preset path stage-related constraints. Finally, the evaluation result is transformed into navigation control actions adapted to the current state. Due to the introduction of path stage-related semantic constraints, stage misjudgment problems can be detected in a timely manner. At the same time, by combining the continuous state of historical semantics, noise interference and true path deviation can be distinguished. Furthermore, by actively limiting low-confidence output through graded navigation control actions, rather than simply providing prompts, it can effectively distinguish between short-term semantic anomalies and continuous path deviations. While ensuring the stability of navigation output, it avoids low-confidence navigation output from misleading the operation, thus solving the navigation output error problem caused by single-frame semantic misjudgment and short-term image quality interference in existing endoscopic navigation.

[0052] based on Figure 1 In one optional implementation of the embodiment shown, when acquiring multi-dimensional input information of continuous microscopic images, the current microscopic image can be acquired in each control cycle, semantic recognition can be performed on the current microscopic image to obtain microscopic semantic results, and image quality detection can be performed on the current microscopic image to obtain image quality labels.

[0053] The control cycle is a fixed computation cycle of the navigation system, typically set to 33 milliseconds (corresponding to a video frame rate of 30 frames per second), or 100 milliseconds to reduce computational load. Each control cycle obtains the current frame image through an endoscope video stream callback function. Semantic recognition can be implemented using a pre-trained UNet semantic segmentation network or a YOLO object detection network. The input is the current endoscopic image, and the output is the semantic category of each pixel and the confidence score for each category. For example, given an input frame of the right main bronchus, the network can output that 80% of the pixels represent the lumen, 15% represent the tracheal wall, and 5% represent the opening of the right upper lobe, with an overall recognition confidence score of 0.92.

[0054] Image quality detection can be achieved using a lightweight classification network. The input is the current image under the lens, and the output is an image quality label such as clear, blurry, overexposed, underexposed, contaminated, or occluded, along with the corresponding quality score. For example, when there is a small amount of mucus on the lens, the image quality label is slightly contaminated, and the quality score is 0.6.

[0055] Furthermore, the microscopic semantic results and image quality labels from multiple consecutive control cycles can be written as historical semantic states to obtain a historical semantic state window. Optionally, the historical semantic state window can adopt a first-in-first-out sliding queue mechanism. When writing, the current microscopic semantic result, confidence level, image quality label, and timestamp are added as a state tuple to the tail of the queue. When the queue length reaches the upper limit, the oldest state data at the head of the queue is automatically popped to keep the window length fixed.

[0056] Furthermore, the current navigation path stage can be determined based on the preoperative path, pose estimation results, luminal bifurcation topology, and current navigation output. A multi-source voting mechanism can be used to determine the current navigation path stage, where multiple information sources—preoperative path, pose estimation results, luminal bifurcation topology, and current navigation output—vote separately. The stage receiving the most votes is designated as the current navigation path stage. This avoids errors in stage determination due to a single source of information. For example, if the pose estimation results indicate the current pose is in the right main bronchus, the next stage in the preoperative path is the right main bronchus, the current navigation output indicates advancement towards the right main bronchus, and historical semantics have identified the carina structure, all these information sources can vote for the right main bronchus stage, thus determining the current navigation path stage as the right main bronchus stage.

[0057] Therefore, by controlling the synchronous acquisition of data over a control period and fusing multi-source information to determine the current navigation path stage, the temporal alignment of the input data and the accuracy of stage judgment are ensured, which can provide reliable data input for subsequent temporal semantic consistency score calculation.

[0058] exist Figure 1In some embodiments of this example, when calculating the temporal semantic consistency score, evaluations can be performed from multiple dimensions and then fused to obtain the final score. For example, one dimension could be the semantic matching score calculated based on the degree of matching between the microscopic semantic results and the labels of the expected semantic sequence. The principle by which this score affects temporal semantic consistency is that the anatomical structures that should be observed in different current navigation path stages are fixed. By determining whether the microscopic semantic results identified in the current control cycle belong to the structures expected to appear in the current stage, it is possible to quickly determine whether the currently observed content matches the path stage and identify obvious path errors from the single-frame label dimension.

[0059] Specifically, the implementation method could be to obtain the microscopic semantic results of the current control cycle, traverse all semantic labels in the expected semantic sequence, and determine whether the microscopic semantic results exist in the expected semantic sequence. If the microscopic semantic results are single labels, the semantic matching score is 1 if they exist, and 0 if they do not exist. If the microscopic semantic results are multiple labels, the proportion of the number of matched labels to the total number of identified labels is calculated as the semantic matching score. For example, if the current navigation path stage is the right main bronchus stage, the expected semantic sequence includes the right main bronchus lumen, the right upper lobe opening, and the right intermediate trunk. If the microscopic semantic results of the current control cycle identify the right main bronchus lumen, which belongs to the label in the expected semantic sequence, then the semantic matching score is 1. If the microscopic semantic results of the current control cycle identify the left lower lobe opening, which is not in the expected semantic sequence, then the semantic matching score is 0.

[0060] In addition, the matching degree can also be calculated based on the position of the microscopic semantic result in the expected semantic sequence. For example, the closer the current microscopic semantic result is to the semantic node corresponding to the expected path position, the higher the matching score, and the farther away it is, the lower the score.

[0061] The second dimension can be to calculate a transfer legitimacy score based on whether the transfer between endoscopic semantic results in adjacent control cycles belongs to the legal transfer in the semantically permissible transfer table. The principle behind this score affecting temporal semantic consistency is that the bronchial tree has a fixed tree-like topology, the endoscope moves a limited distance between adjacent control cycles, and the observed anatomical structures must be anatomically adjacent. It is impossible for there to be a semantic jump across levels. By judging whether the semantic transfer between adjacent control cycles is legal, abnormal jumps caused by semantic recognition errors or navigation drift can be identified.

[0062] Specifically, the implementation method can be to obtain the microscopic semantic results of the previous control cycle and the current control cycle, and query the boolean value corresponding to the semantic pair in the semantic allowed transfer table; if the semantic of the previous control cycle is unknown, the transfer is legal by default and the transfer legality score is 1; if the corresponding value in the semantic allowed transfer table is 1, it means that it is a legal transfer and the transfer legality score is 1; if the corresponding value is 0, it means that it is an illegal transfer and the transfer legality score is 0.

[0063] For example, if the semantic result under endoscopy in the previous control cycle was the carina, and the semantic result under endoscopy in the current control cycle was the opening of the right main bronchus, then according to the semantic allowed transfer table, the transfer from the carina to the opening of the right main bronchus is a legal transfer, so the transfer legality score is 1. If the semantic result under endoscopy in the previous control cycle was the carina, and the semantic result under endoscopy in the current control cycle was the opening of the left lower lobe, then the corresponding value for this transfer in the semantic allowed transfer table is 0, which is an illegal transfer, so the transfer legality score is 0.

[0064] The third dimension can be a historical stability score calculated based on the consistency between the semantic state within the historical semantic state window and the direction of advancement in the current navigation path stage. The principle behind this score's influence on temporal semantic consistency is that while semantic recognition in a single control cycle may be affected by noise and result in errors, the semantic change trend across multiple consecutive control cycles should be consistent with the navigation direction. By judging temporal consistency within the historical semantic state window, the influence of single-frame noise can be filtered out, identifying a continuously stable semantic state and avoiding misjudgments caused by single-frame errors.

[0065] Specifically, the implementation method can be to traverse all the under-vision semantic results in the historical semantic state window, first count the number of control cycles belonging to the expected semantics of the current navigation path stage, divide it by the total length of the historical semantic state window to obtain the base score; then count whether the semantic transfer direction is consistent with the current navigation output's direction of advancement, and deduct the corresponding score according to the number of reverse transfers when a reverse transfer occurs, and finally obtain the historical stability score.

[0066] For example, if the historical semantic state window has a length of 15 control cycles, and 13 control cycles identify the relevant structures of the right main bronchus, while 2 control cycles are identified as unknown due to mucus obscuring them, with no reverse transition, then the historical stability score is 13 / 15≈0.87. If 5 of these control cycles identify the structures of the left main bronchus, which is opposite to the direction of propulsion, the historical stability score can be reduced to below 0.5.

[0067] The four dimensions can be: a pose consistency score can be calculated based on the spatial consistency between the semantic region location of the endoscopic semantic results and the pose estimation results, and the current navigation output. The principle behind this score affecting temporal semantic consistency is that the position of the anatomical structure identified by the endoscopic semantic results in the three-dimensional bronchial tree is fixed. This position should spatially match the endoscope position indicated by the pose estimation results. Spatial consistency judgment can identify navigation errors caused by pose estimation drift, avoiding the problem of correct semantic recognition but incorrect navigation position when pose is incorrect.

[0068] Specifically, the implementation method can be as follows: the anatomical structure region identified by the semantic results under the current control cycle is projected onto the preoperative three-dimensional bronchial tree model through camera intrinsic and extrinsic parameters to obtain the spatial position of the structure in the three-dimensional model; the Euclidean distance between this position and the current position of the endoscope indicated by the pose estimation result is calculated. When the distance is less than a preset distance threshold, the pose consistency score is 1, the larger the distance, the lower the score, and when it exceeds the maximum distance threshold, the pose consistency score is 0; at the same time, combined with the current navigation output propulsion direction, it is determined whether the semantic position is in the propulsion direction. If it is not in the propulsion direction, the score can be deducted appropriately.

[0069] For example, if the pose estimation result shows that the endoscope is at the entrance of the right main bronchus, and the semantic results under the endoscope identify the opening of the right upper lobe, the position of the right upper lobe opening after projection is 2mm away from the pose estimation position, which is less than the preset distance threshold, then the pose consistency score is 0.9; if the pose estimation result shows that the endoscope is in the right main bronchus, but the semantic results under the endoscope identify the right intermediate trunk, the position of the right upper lobe opening after projection is 15mm away from the pose estimation position, which exceeds the maximum distance threshold, then the pose consistency score is 0.2.

[0070] The fifth dimension can be a noise penalty score calculated based on the image quality label of the under-lens image, which represents the image noise. The principle behind this score affecting temporal semantic consistency is that when the under-lens image has quality problems such as blurriness, contamination, or occlusion, the reliability of the semantic recognition result will be greatly reduced. It is necessary to penalize the overall score based on the image quality to avoid poor-quality images from affecting the calculation of the temporal semantic consistency score.

[0071] Specifically, this can be achieved by obtaining the image quality label for the current control period. Different quality labels correspond to different penalty values: a penalty value of 0 for a clear image, 0.3 for slightly blurred or contaminated images, 0.6 for moderately blurred or contaminated images, and 1 for severely blurred, contaminated, or completely occluded images. The noise penalty score is 1 minus the penalty value. For example, if the image is clear and unoccluded in the current control period, the image quality label is "clear," the penalty value is 0, and therefore the noise penalty score is 1; if the lens is completely occluded by a large clump of slime in the current control period, the image quality label is "severely contaminated," the penalty value is 0.9, and therefore the noise penalty score is 0.1.

[0072] Furthermore, the semantic matching score, the transition legitimacy score, the historical stability score, the pose consistency score, and the noise penalty score can be fused to obtain the temporal semantic consistency score. These multiple dimensions evaluate temporal consistency from five perspectives: single-frame matching, adjacent transitions, historical trends, spatial location, and image quality. Weighted fusion can integrate information from each dimension to obtain a more accurate and robust overall consistency score, avoiding the limitations of single-dimensional judgment.

[0073] Specifically, this can be achieved by assigning preset weights to each of the aforementioned scores, multiplying each score by its corresponding weight, and then summing the results. The final temporal semantic consistency score can be narrowed down to the 0-1 range (or other numerical ranges). For example, the calculation formula could be as follows:

[0074] Tscore = c1·Ssem + c2·Stran + c3·Shist + c4·Spose - c5·Snoise;

[0075] Among them, Ssem refers to semantic matching score; Stran refers to transition legitimacy score; Shist refers to historical stability score; Spose refers to pose consistency score; and Snoise refers to noise penalty score.

[0076] The optional weight values ​​are as follows: c1=0.30, c2=0.25, c3=0.20, c4=0.15, c5=0.10, which can limit the Tscore to 0 to 1. Of course, other weight values ​​can also be set. The above weights are preferred embodiments and can be adjusted according to different path stages. For example, increase the Stran weight in the fork entry stage and increase the Snoise weight in the tool operation stage.

[0077] The fusion method can also be calculated using dynamic time warping, hidden Markov models, temporal neural networks, etc. Regardless of the calculation method used, the final temporal semantic consistency score is a value in the range of 0-1 (or other numerical ranges). The higher the score, the better the matching degree between the semantics under the continuous mirror and the current navigation path stage and the better the temporal stability.

[0078] Therefore, by using multi-dimensional temporal semantic consistency score evaluation, we can accurately identify continuous navigation drift and path errors, and effectively filter noise interference caused by short-term occlusion, contamination, and blurring. This solves the problem of easy misjudgment in single-frame judgment of existing technologies and greatly improves the accuracy of navigation credibility judgment.

[0079] exist Figure 1In some embodiments of this example, when applying corresponding navigation control actions to the current navigation output based on navigation credibility, the navigation credibility level can be divided first, and then the corresponding navigation control actions can be applied to the current navigation output based on the navigation credibility level. The navigation control actions can be used to hierarchically restrict the display of navigation output and the control permissions of autonomous navigation.

[0080] Specifically, multiple navigation credibility levels can be divided based on the numerical range of navigation credibility. Different navigation credibility levels correspond to different navigation control actions. The navigation control actions cover two dimensions of hierarchical restrictions: the display method of navigation output and the control authority of autonomous navigation. The display layer and the control layer are restricted independently. When the navigation credibility is low, prompts can be retained but the prompt weight is reduced, while high-risk navigation control actions are restricted.

[0081] For example, one classification method could be as follows: when the navigation confidence level Nconf ≥ 0.80, it is classified as G0 high-confidence navigation; when 0.65 ≤ Nconf < 0.80, it is classified as G1 observational confidence; when 0.45 ≤ Nconf < 0.65, it is classified as G2 degraded navigation; and when Nconf < 0.45 or consecutive illegal semantic transitions exceed Nbad, it is classified as G3 relocation / review. To avoid frequent switching between different navigation confidence levels, an Nconf ≥ 0.70 requirement can be set for recovery from G2 to G1, and an Nconf ≥ 0.85 requirement for recovery from G1 to G0.

[0082] Of course, the threshold range for each level can be adjusted according to the risk requirements of different surgical scenarios. For scenarios with higher navigation safety requirements, the trigger threshold for each level can be appropriately increased to initiate degradation control earlier. Furthermore, the number and naming of navigation confidence levels can also be adjusted according to actual needs to meet the design requirements of navigation systems with different complexities. This embodiment does not impose any limitations on this.

[0083] Optionally, a recovery hysteresis mechanism can be set when classifying navigation confidence levels. That is, the threshold for dropping from a high navigation confidence level to a low navigation confidence level is lower than the threshold for rising from a low navigation confidence level to a high navigation confidence level. For example, the threshold for dropping from a high confidence level to an observed confidence level is 0.8, and the threshold for rising from an observed confidence level to a high confidence level is 0.9. This avoids frequent switching of navigation confidence levels when the navigation confidence level is near the threshold, which may affect the smoothness of the operation. It realizes a hierarchical management logic of full functionality when the confidence level is high, gradually restricting permissions when the confidence level is low, and stopping automatic control when the confidence level is unreliable.

[0084] Therefore, by controlling navigation reliability in a hierarchical manner, navigation risks can be managed in layers. When the reliability is high, navigation smoothness is guaranteed; when the reliability is low, high-risk operations are gradually restricted; and when the reliability is unreliable, automatic control is stopped in time and manual intervention is prompted. This minimizes unnecessary navigation interruptions while ensuring surgical safety, thus balancing navigation safety and operational smoothness.

[0085] After classifying the navigation confidence level, different control strategies correspond to different navigation confidence levels. When the navigation confidence level is high, the navigation path is displayed and the execution permission of autonomous navigation control actions is granted. The system is allowed to output automatic advancement commands and steering assistance commands, and the scope can be automatically advanced according to the planned path. At this time, the operator only needs to monitor the navigation status. For example, if the navigation is stable in the right main bronchus and the temporal semantic consistency score is higher than 0.9 for 20 consecutive control cycles, the system displays a green navigation path arrow and automatically advances the scope at a speed of 5 mm / s (or other speed), while automatically adjusting the steering to stay in the center of the lumen.

[0086] When the navigation confidence level is at the observation confidence level, reduce the display transparency or priority of navigation prompts, and display the temporal semantic consistency score and its explanation on the navigation interface. For example, indicate whether the current situation is a single-frame blur or a slight pose deviation. At the same time, prohibit the output of navigation control actions exceeding the preset gain, that is, prohibit large-angle turning and high-speed advancement, and only allow low-speed, small-range scope adjustments. At this time, the surgeon can pay appropriate attention to the actual situation under the scope, but does not need to interrupt the navigation process. For example, if a small amount of mucus floats in front of the lens and causes a single control cycle recognition error, the temporal semantic consistency score drops to 0.7, the navigation path arrow turns into a semi-transparent yellow, the "poor single-frame image quality" prompt is displayed in the corner of the interface, the automatic advancement speed is reduced to 1 mm per second, and automatic turning exceeding 15 degrees is prohibited. After the mucus floats past and the field of vision is restored, it automatically returns to the high confidence level.

[0087] When the navigation confidence level is downgraded, automatic propulsion control commands for navigation control are prohibited. The navigation interface prompts the user to re-observe the endoscopic field of view or control the endoscope to retract slowly. At the same time, the system can generate multiple repositioning candidate poses for the endoscope tip based on the confidence status over the most recent historical period, allowing the surgeon to select and confirm. In this case, the automatic propulsion function is disabled, but auxiliary functions such as turning prompts are still available. The surgeon can manually control the endoscope to adjust its position. For example, if the endoscope tip touches the tracheal wall, causing the field of view to be obstructed by the wall, and the temporal semantic consistency score is lower than 0.6 for 5 consecutive control cycles, the system immediately stops automatic propulsion, and the interface prompts "Please retract the endoscope and re-observe the field of view." At the same time, multiple possible repositioning candidate poses are generated based on the confidence pose in the last 3 seconds before the obstruction for the surgeon to select.

[0088] When the navigation reliability level is at the repositioning verification level, all automatic navigation control commands are stopped. A prominent message appears on the navigation interface indicating that the current navigation is unreliable and requests manual intervention for confirmation. Navigation control commands can only be resumed after the user completes the pose repositioning operation at the endoscope tip, or after the system completes the re-registration of the endoscopic image with the preoperative model. At this time, all automatic functions are frozen, and navigation can only continue after the surgeon confirms that the position is correct. For example, if the endoscope accidentally enters the left main bronchus but the navigation still thinks it is in the right main bronchus, and the temporal semantic consistency score is lower than 0.4 for 10 consecutive control cycles, the system stops all automatic navigation control commands, and a red message pops up on the interface: "Navigation unreliable, please confirm the current position." The system resumes navigation function only after the doctor manually operates the endoscope to complete the repositioning.

[0089] Therefore, by controlling navigation confidence levels at multiple levels, corresponding control permissions are matched to navigation states with different confidence levels. This ensures smooth operation in high confidence states, gradually restricts high-risk operations in low confidence states, and freezes automatic functions in a timely manner in case of serious abnormalities, so as to avoid misleading the operator and causing airway damage due to erroneous navigation control actions.

[0090] exist Figure 1 In some implementations of this embodiment, the system can detect abnormal input states in real time. When an input abnormality occurs, the navigation confidence level is directly prohibited from entering the high confidence level, avoiding the output of high-risk navigation control actions based on unreliable input data. Specific abnormal input states include various situations such as failure to acquire images under the lens, empty output of image semantic recognition, asynchronous timestamps of the under-lens image and pose timestamps, and unavailable image quality labels for multiple consecutive control cycles.

[0091] Among these issues, failures in acquiring images under the microscope are usually caused by interrupted camera connections, complete lens obstruction, or loss of video signal. For example, a loose video connection will result in a completely black image, and the acquisition interface will return an empty frame. Empty outputs in image semantic recognition are typically due to poor image quality that prevents the identification of any anatomical structures. For instance, if the field of view is completely filled with blood, the semantic network cannot identify any valid structures and will output an empty result.

[0092] The time stamp and pose time stamp of the microscopic image are not synchronized. This is usually due to the delay of the positioning signal or the delay of the image transmission, which causes the image and pose to be data from different times. For example, when electromagnetic positioning is interfered with by the operating room equipment, the pose data will be delayed by more than 200 milliseconds, which is more than 6 control cycles later than the current image.

[0093] An image quality label of "unavailable" for multiple consecutive control cycles is usually due to the lens being completely contaminated with blood or secretions, or the lens being pressed against the airway wall, resulting in a complete loss of field of view. For example, when a large clump of mucus completely covers the lens, no structure can be identified in images for multiple consecutive control cycles.

[0094] Secondly, if the above-mentioned abnormal input states continue for more than the preset number of control cycles, the system can directly adjust the navigation confidence level to a downgraded navigation level and prompt the surgeon to re-observe the field of view under the microscope and investigate the cause of the abnormality. For example, if the image interruption lasts for 3 seconds, i.e. 90 control cycles, the system will automatically downgrade and prompt the surgeon to check the video connection.

[0095] Therefore, by detecting input anomalies, high navigation confidence level permissions can be restricted in a timely manner when the data source is unreliable. This avoids the risk of outputting incorrect navigation control actions based on erroneous data from the input end, thus ensuring the reliability of navigation input.

[0096] exist Figure 1 In some implementations of this embodiment, the system can detect semantic mutations and pose conflict anomalies in real time. When an illegal semantic transfer not recorded in the semantically permissible transfer table occurs in the under-vision semantic results of an adjacent control cycle, the system can determine the anomaly type by combining the noise penalty score. If the illegal transfer occurs only in a single control cycle, and the noise penalty score of that control cycle is higher than the noise threshold, it indicates that the anomaly is likely due to a recognition error caused by single-frame noise such as slime occlusion or motion blur. In this case, the system can mark the current control cycle as a short-term semantic anomaly and temporarily not trigger a downgrade. The anomaly mark will be automatically cleared after the subsequent control cycle returns to normal.

[0097] For example, during the advancement of the right main bronchus, a control cycle image was mistakenly identified as the opening of the left lower lobe due to mucus obscuring the image. This transfer was an illegal transfer, but the image quality label for that control cycle was severely contaminated, and the noise penalty score was higher than the threshold. The system only marked it as a short-term anomaly and did not trigger a downgrade. The anomaly label was automatically cleared after the recognition returned to normal in the next control cycle.

[0098] If illegal transfers continue for more than the preset number of control cycles, it indicates that the anomaly is not single-frame noise, but a real navigation drift or path error. At this time, the system can adjust the navigation confidence level to a downgraded navigation level or a relocation verification level, and prompt the operator to confirm the location.

[0099] For example, if the endoscope actually enters the left main bronchus but the navigation system still identifies it as being in the right main bronchus, and the system identifies the structure as being in the left main bronchus for five consecutive control cycles, all transfers are considered illegal transfers, and the system directly adjusts to the relocation verification level to prompt for relocation.

[0100] When the anatomical location corresponding to the semantic result under endoscopy is inconsistent with the anatomical location indicated by the pose estimation result, and this inconsistency continues for more than a preset number of control cycles, it indicates that the pose estimation may have drifted. At this time, the system can reduce the pose consistency score and generate repositioning candidate poses for the endoscope tip, suggesting that the surgeon may need to perform a repositioning operation. For example, if the pose estimation result is incorrectly displayed in the right upper lobe, but the anatomical features of the right intermediate trunk are identified for 5 consecutive control cycles, with a position difference of one bronchial segment, the system will reduce the pose consistency score from 0.9 to 0.3 and generate multiple repositioning candidate poses such as the right intermediate trunk and the posterior segment of the right upper lobe for the doctor to choose from.

[0101] Therefore, by distinguishing between single-frame semantic noise and continuous semantic anomalies, unnecessary degradation caused by single-frame recognition errors is avoided, and real navigation drift and pose errors can be detected in a timely manner, improving the accuracy of anomaly handling and reducing unnecessary interruptions to the surgical procedure.

[0102] exist Figure 1 In some embodiments of this example, after outputting navigation control actions, the system can also perform an output interlock check. The system can wait for the actuator to return the corresponding navigation control action execution feedback. If the corresponding action execution feedback is not received within a preset response time, or if the action execution feedback indicates that the navigation control action has failed, such as robot joint jamming, command rejection, or communication interruption, the system can immediately prohibit the output of automatic propulsion control commands and control commands for turning beyond a preset angle, that is, prohibit high-risk automatic operations, such as prohibiting large-angle automatic turning exceeding 15 degrees, to prevent the mirror from rotating at a large angle and damaging the airway wall when the state is unknown.

[0103] After implementing the above restrictions, the system maintains the navigation confidence level at a low confidence level and prompts on the navigation interface that manual confirmation is required before navigation operations can resume. This avoids inconsistencies caused by the system believing that navigation control actions have been executed when the actual mirror has not moved, and prevents subsequent navigation control actions from being output based on incorrect states, which could lead to danger. For example, if the system issues a navigation control action to move forward 5 millimeters and the robot joints get stuck, and no action execution feedback is received within 500 milliseconds, the system immediately prohibits automatic propulsion and large-angle turning, prompting "Control command not executed, please check robot status." Only after the doctor investigates the stuck problem and manually confirms it will the system restore normal navigation functions.

[0104] Therefore, by using the output interlock mechanism, closed-loop safety verification of navigation control actions is achieved, which can avoid state errors caused by navigation control actions not being actually executed, and improve the control safety of automatic navigation.

[0105] exist Figure 1In some implementations of this embodiment, when the navigation confidence level is lower than the high confidence level, the system does not directly restore to the high navigation confidence level, but instead restores it by gradually increasing the level. Specifically, the system can only increase the navigation confidence level by one and remove the navigation restrictions corresponding to that level when the preset level restoration conditions are met for multiple consecutive control cycles. It does not skip levels to ensure the stability of the restoration process. For example, it cannot directly restore from the relocation verification level to the high confidence level; it must be increased gradually in the order of relocation verification level to downgraded navigation level, downgraded navigation level to observation confidence level, and observation confidence level to high confidence level. Each level needs to meet the observation requirements individually.

[0106] Removing the navigation restrictions corresponding to each level means that only the control permissions allowed for that level are gradually opened after each level is upgraded. Not all high-risk automatic operations will be opened at once. Full autonomous navigation control permissions are only granted after the navigation state has been verified and confirmed to be stable and reliable at each level. This avoids immediately enabling full-function automatic propagation immediately after repositioning, reducing the possibility of outputting high-risk operations when the pose is not fully stable, further ensuring navigation safety during the recovery process, and reducing the risk of errors recurring after anomaly correction.

[0107] The specific conditions for level recovery may include a temporal semantic consistency score higher than a preset recovery threshold, semantic transitions in adjacent control cycles being legal transitions, convergence of the deviation between the lens semantic results and pose estimation results, and all navigation control actions having corresponding effective action execution feedback. Only when all these conditions are met simultaneously can a continuous cycle be counted. If any one condition is not met, the continuous cycle count is reset to zero.

[0108] Among them, the temporal semantic consistency score is higher than the recovery threshold. For example, the observation confidence level needs to be raised to a high confidence level, and the score needs to be higher than 0.9 continuously. The semantic transitions of adjacent control cycles are all legal transitions, that is, there are no anatomically impossible semantic jumps in multiple consecutive control cycles. The deviation between the microscopic semantic results and the pose estimation results converges, that is, the difference between the semantic position and the pose position is stable within the allowable range and no longer increases. All navigation control actions have corresponding effective action execution feedback, that is, all previously issued navigation control actions have received action execution feedback of successful execution.

[0109] The recovery process is carried out strictly in accordance with the navigation confidence level order, and each level needs to meet the continuous cycle requirements. For example, it takes 10 consecutive control cycles to meet the recovery conditions of the level to upgrade from the downgraded navigation confidence level, and 15 consecutive control cycles to meet the recovery conditions of the level to upgrade from the observation confidence level to the high confidence level.

[0110] Therefore, by restoring step by step and confirming continuously in cycles, the risks caused by immediately resuming high-level automatic navigation as soon as the anomaly improves are avoided, ensuring the stability and safety of the recovery process.

[0111] exist Figure 1 In some implementations of this embodiment, during the observation period of level recovery, if the temporal semantic consistency score fluctuates repeatedly within the score range of the recovery threshold and cannot be stably higher than the recovery threshold, the system can increase the number of continuous control cycles required for level recovery observation, extend the observation time, and at the same time maintain the current navigation confidence level unchanged, and not upgrade the level because the condition is met for an occasional control cycle.

[0112] For example, if the current navigation confidence level is the observation confidence level, and the temporal semantic consistency score fluctuates between 0.88 and 0.92 when the level recovers, sometimes exceeding the threshold of 0.9 and sometimes falling below 0.9, the requirement that originally required 15 consecutive control cycles to meet the condition can now be raised to require 30 consecutive control cycles to advance to the next higher level, while maintaining the observation confidence level without increasing.

[0113] Therefore, by dynamically adjusting the observation period, the risks caused by prematurely upgrading before the status is stable are avoided, the level recovery process is made more consistent with the actual navigation status, the reliability of the recovery phase is further improved, and the possibility of enabling high-risk automatic operations in unstable states is reduced.

[0114] The system can further determine the cause of the fluctuations. If the fluctuations are caused by occlusion of the endoscopic image or contamination of the lens, such as a small amount of mucus adhering to the lens causing occasional fluctuations in the recognition results, the system can prompt the surgeon to clean the lens or wait for the field of vision to recover naturally on the navigation interface. If the fluctuations are caused by a continuous conflict between the endoscopic semantic results and the pose estimation results, it indicates that the pose drift problem has not been truly resolved. In this case, the system can maintain the repositioning request of the endoscope's end pose, continuously prompt the surgeon to complete the repositioning, and not automatically increase the navigation confidence level until the pose conflict is completely resolved.

[0115] For example, there is a persistent deviation between the current pose estimation result and the actual anatomical position under the microscope. Only a few control cycles occasionally match the correct result due to image recognition noise, but this does not fundamentally solve the navigation drift problem. Therefore, we will not rashly increase the navigation reliability level to avoid the surgical risks caused by incorrect automatic navigation recovery.

[0116] Therefore, by differentiating the causes of fluctuations and addressing them specifically during the recovery process, we can handle temporary anomalies that can recover on their own, such as minor contamination, and avoid misjudging unresolved issues such as pose conflicts as recovery, thus further improving the safety of the recovery process.

[0117] This application also provides an endoscopic navigation control system, which includes an endoscopic semantic acquisition module, a path stage reference module, a temporal consistency evaluation module, a navigation reliability control module, a degradation relocation module, a recovery interpretation module, and an execution and display interface. Wherein:

[0118] The microscopic semantic acquisition module is used to acquire continuous microscopic images and output microscopic semantic results, confidence scores, and image quality labels. It can be implemented using semantic segmentation networks and quality classification networks.

[0119] The path stage reference module is used to determine the current navigation path stage and retrieve the corresponding expected semantic sequence and semantically allowed transition table, which can be implemented based on the preoperative path and real-time pose.

[0120] The temporal consistency evaluation module is used to calculate the temporal semantic consistency score, which can be implemented using weighted summation, dynamic time warping, or temporal networks.

[0121] The navigation confidence control module is used to update navigation confidence and classify navigation confidence levels. It can be implemented using a state machine or threshold judgment and supports smoothing and hysteresis mechanisms.

[0122] The degradation and relocation module is used to detect various anomalies and trigger degradation or relocation. It can detect four types of anomalies: input anomalies, semantic mutations, pose conflicts, and output interlocks.

[0123] The recovery interpretation module is used to control the recovery level and handle abnormal fluctuations during the recovery process, enabling step-by-step recovery and cause-by-cause processing.

[0124] The execution and display interface is used to output display prompts and navigation control actions, and read action execution feedback to achieve interlocking. It is responsible for communicating with the display interface and the robot's actuator.

[0125] Each module works collaboratively according to the aforementioned method steps to jointly achieve temporal semantic consistency assessment and navigation reliability classification control, effectively distinguishing between short-term noise and continuous anomalies, and improving the safety and smoothness of bronchoscopy navigation. The specific working process of each module corresponds one-to-one with the steps in the aforementioned method embodiments, and will not be repeated here.

[0126] For example, Figure 2 This figure illustrates an exemplary application scenario of the endoscope navigation control system. The system includes a endoscopic semantic acquisition module 110, a path stage reference module 120, a temporal consistency evaluation module 130, a navigation reliability control module 140, a degradation relocation module 150, a recovery interpretation module 160, an execution and display interface 170, and a pose estimation module 180. These modules are connected via data streams.

[0127] The under-magnification semantic acquisition module 110 outputs the under-magnification semantic results, image quality labels, and semantic confidence to the temporal consistency evaluation module 130. The path stage reference module 120 outputs the expected semantic sequence to the temporal consistency evaluation module 130. The pose estimation module 180 outputs the pose estimation results and the current navigation path stage to the temporal consistency evaluation module 130 and the path stage reference module 120. The current navigation output 190 outputs the path bifurcation and distance information to the temporal consistency evaluation module 130 and the navigation confidence control module 140.

[0128] The temporal consistency evaluation module 130 outputs the temporal semantic consistency score to the navigation credibility control module 140. The navigation credibility control module 140 outputs the navigation credibility level to the downgrade relocation module 150, the recovery interpretation module 160, and the execution and display interface 170, respectively. The downgrade relocation module 150 outputs downgrade, relocation, and relocation candidate pose information to the execution and display interface 170. The recovery interpretation module 160 outputs the recovery observation window and anomaly cause information to the execution and display interface 170. The execution and display interface 170 outputs navigation interface prompts and robot control commands, and simultaneously transmits the execution feedback and image feedback back to the under-lens semantic acquisition module 110, and feeds back the control output to the current navigation output 190.

[0129] Therefore, through the connection and flow of the above data streams, a complete closed-loop processing from the input of the microscopic image to the output of the navigation control can be achieved. The temporal semantic consistency evaluation results are integrated into the dynamic adjustment of navigation credibility. Finally, safe and reliable automatic navigation is achieved through hierarchical control and anomaly handling, effectively reducing unnecessary operation interruptions and timely preventing surgical risks caused by real navigation anomalies.

[0130] The computer device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 3 One embodiment of the computer device in this application includes:

[0131] The computer device 300 may include one or more central processing units (CPUs) 301 and a memory 305, in which one or more applications or data are stored.

[0132] The memory 305 can be volatile or persistent storage. The program stored in the memory 305 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 301 can be configured to communicate with the memory 305 and execute the series of instruction operations stored in the memory 305 on the computer device 300.

[0133] Computer device 300 may also include one or more power supplies 302, one or more wired or wireless network interfaces 303, one or more input / output interfaces 304, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0134] The central processing unit 301 can perform the aforementioned... Figure 1 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments are not described in detail here.

[0135] This application also provides a computer storage medium, one embodiment of which includes: the computer storage medium storing instructions, which, when executed on a computer, cause the computer to perform the aforementioned... Figure 1 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An endoscope navigation control method, characterized in that, The method includes: Acquire the endoscopic semantic results of continuous endoscopic images during the endoscopic examination, the current navigation path stage, the historical semantic state window, the pose estimation results of the endoscope tip, and the current navigation output; Based on the current navigation path stage, the corresponding expected semantic sequence and semantic allowed transition table are retrieved. The expected semantic sequence contains the semantic states expected to occur under the current navigation path stage, and the semantic allowed transition table is used to limit the legal transition relationships between adjacent semantic states. Based on the under-vision semantic results, the expected semantic sequence, the semantic allowed transition table, the pose estimation results, the current navigation output, and the image quality factors of the under-vision image, the temporal semantic consistency score of the current navigation state is determined; the temporal semantic consistency score is used to characterize the degree of matching and temporal stability between continuous under-vision semantics and navigation path stages; The navigation credibility of the current navigation state is determined based on the temporal semantic consistency score, and corresponding navigation control actions are applied to the current navigation output according to the navigation credibility.

2. The method according to claim 1, characterized in that, The acquisition of endoscopic semantic results, current navigation path stage, historical semantic state window, pose estimation results of the endoscope tip, and current navigation output from continuous endoscopic images during the endoscopic examination includes: In each control cycle, the current under-lens image is acquired, semantic recognition is performed on the current under-lens image to obtain the under-lens semantic result, and image quality detection is performed on the current under-lens image to obtain the image quality label; The under-microscopic semantic results and the image quality labels of multiple consecutive control cycles are written as historical semantic states to obtain the historical semantic state window; The current navigation path stage is determined based on the preoperative path, the pose estimation result, the luminal bifurcation topology, and the current navigation output.

3. The method according to claim 1, characterized in that, The step of determining the temporal semantic consistency score of the current navigation state based on the under-vision semantic results, the expected semantic sequence, the semantic permissible transition table, the pose estimation results, the current navigation output, and the image quality factors of the under-vision image includes: A semantic matching score is calculated based on the degree of label matching between the microscopic semantic results and the expected semantic sequence; a transfer legality score is calculated based on whether the transfer between microscopic semantic results of adjacent control periods belongs to the legal transfer in the semantic allowed transfer table. The historical stability score is calculated based on the degree of consistency between the semantic state within the historical semantic state window and the direction of advancement of the current navigation path stage. A pose consistency score is calculated based on the spatial consistency between the semantic region location of the under-vision semantic results and the pose estimation results and the current navigation output; a noise penalty score is calculated based on the image noise characterized by the image quality label of the under-vision image. The semantic matching score, the transition legality score, the historical stability score, the pose consistency score, and the noise penalty score are fused to obtain the temporal semantic consistency score.

4. The method according to claim 3, characterized in that, The step of applying corresponding navigation control actions to the current navigation output based on the navigation confidence level includes: Navigation reliability levels are determined based on the aforementioned navigation reliability. Based on the navigation credibility level, a corresponding navigation control action is applied to the current navigation output. The navigation control action is used to hierarchically restrict the display of the navigation output and the control permissions of autonomous navigation.

5. The method according to claim 4, characterized in that, The step of applying corresponding navigation control actions to the current navigation output based on the navigation confidence level includes: When the navigation confidence level is high, the navigation path is displayed and the execution permission for autonomous navigation control actions is granted; When the navigation confidence level is the observation confidence level, reduce the display transparency or display priority of the navigation prompt information, display the temporal semantic consistency score and its explanation on the navigation interface, and prohibit the output of navigation control actions exceeding the preset gain. When the navigation confidence level is downgraded, the automatic propulsion control command for navigation control is prohibited. The navigation interface prompts the user to re-observe the field of view under the endoscope or to control the endoscope to retract at a low speed, and generates multiple repositioning candidate poses for the endoscope tip. When the navigation reliability level is at the repositioning verification level, all automatic navigation control commands are stopped, the navigation interface prompts that the current navigation is unreliable and requests manual intervention for confirmation, and the output of navigation control commands is only resumed after the user completes the pose repositioning operation at the end of the endoscope or completes the re-registration of the endoscopic image.

6. The method according to claim 5, characterized in that, The method further includes: When an abnormal input state occurs, the navigation confidence level is prohibited from entering the high confidence level; the abnormal input states include: failure to acquire the under-lens image, empty output of image semantic recognition, asynchronous timestamp of under-lens image and pose estimation timestamp, and unavailable image quality labels for multiple consecutive control cycles. If the abnormal input status continues for more than a preset number of cycles, the navigation reliability level will be adjusted to the downgraded navigation level, and the navigation interface will prompt the user to re-observe the field of view under the lens.

7. The method according to claim 5, characterized in that, The method further includes: When an illegal semantic transfer not recorded in the semantic allowed transfer table occurs in the microscopic semantic results of an adjacent control cycle, if the illegal semantic transfer occurs in only a single control cycle and the noise penalty score is higher than the noise threshold, the current control cycle is marked as a short-term semantic anomaly and the degradation is not triggered temporarily. If the illegal semantic transfer continues for more than a preset number of cycles, the navigation credibility level will be adjusted to the downgraded navigation level or the relocation review level. When the anatomical structure position corresponding to the endoscopic semantic result is inconsistent with the anatomical structure position indicated by the pose estimation result and continues for more than the preset number of cycles, the value of the pose consistency score is reduced and a candidate pose for repositioning of the endoscope tip is generated.

8. The method according to claim 5, characterized in that, After outputting the navigation control action, the method further includes: If no action execution feedback corresponding to the navigation control action is received within the preset response time, or if the action execution feedback indicates that the navigation control action has failed, then the output of automatic propulsion control commands and control commands for turning beyond the preset angle is prohibited. Maintain the navigation confidence level at a low confidence level, and resume navigation operation after prompting for manual confirmation on the navigation interface.

9. The method according to claim 5, characterized in that, The method further includes: When the navigation confidence level is lower than the high confidence level, if the preset level recovery condition is met for multiple consecutive control cycles, the navigation confidence level is gradually increased and the navigation restriction corresponding to the navigation confidence level is removed. The recovery conditions include: the temporal semantic consistency score is higher than a preset recovery threshold, the semantic transitions of adjacent control cycles are all legal transitions, the deviation between the under-vision semantic results and the pose estimation results converges, and all navigation control actions have corresponding action execution feedback.

10. The method according to claim 9, characterized in that, The method further includes: During the observation period of grade recovery, if the temporal semantic consistency score fluctuates within the score range of the recovery threshold, the number of consecutive control cycles required for grade recovery observation is increased, and the current navigation confidence level is maintained. If the fluctuation is caused by occlusion of the endoscopic image or contamination of the mirror, the navigation interface will prompt the user to clean the mirror or wait for the field of view to be restored; if the fluctuation is caused by a conflict between the endoscopic semantic result and the pose estimation result, the repositioning request for the endoscope end pose will be maintained.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 10.

12. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 10.

Images