Intraoperative state perception method and system applied to endoscopic surgery navigation

Abstract

The application relates to the technical field of surgical navigation, in particular to an intraoperative state perception method and system applied to endoscopic surgical navigation. The method comprises the following steps: acquiring an intraoperative endoscope video stream in real time; organizing continuous video frames in the endoscope video stream into video clips; performing spatio-temporal joint feature extraction on the video clips to obtain a shared feature vector for representing a spatio-temporal state of an intraoperative scene; inputting the shared feature vector into task processing branches corresponding to multiple intraoperative perception tasks respectively to obtain multiple intraoperative perception results; and outputting the multiple intraoperative perception results. The intraoperative state perception method provided in the application generates a shared feature vector based on the same endoscope video clip, reduces repeated feature extraction and reasoning delay, improves the reliability of the perception result and the adaptability to the navigation system, provides timely and accurate intraoperative navigation information for doctors, reduces the information screening burden, improves the precision and safety of intraoperative operation, and reduces the surgical risk of patients.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a method and system for intraoperative status perception applied to laparoscopic surgical navigation. Background Technology

[0002] During laparoscopic surgery, intraoperative video can reflect the surgical area, instrument movements, tissue exposure, and changes in the surgical process. With the development of technologies such as surgical navigation, intraoperative assisted recognition, and intelligent prompts, there is a need to convert laparoscopic video into intraoperative status information that can be accessed by computing devices. Related technologies typically employ hand-crafted features combined with traditional classifiers, neural networks, or video-based models to analyze surgical videos, and establish separate processing models for perception tasks such as surgical stages, instruments, anatomical structures, intraoperative events, and spatial states. For example, video classification models such as VideoMAE are used for surgical stage recognition, YOLO series object detection models are used for instrument detection, and U-Net segmentation models are used for anatomical structure segmentation. Each task corresponds to an independent model instance, an independent feature extraction process, and an independent inference process.

[0003] In intraoperative scenarios requiring real-time response, extracting video features separately for multiple tasks can easily lead to redundant calculations and accumulated latency. Furthermore, processing video and outputting results independently by each task makes it difficult to ensure that different perceptual results are based on a consistent spatiotemporal state, thus affecting the collaborative use of multiple perceptual results. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of this application provide an intraoperative state perception method and system for laparoscopic surgical navigation. This method generates shared features based on the same laparoscopic video segment and allows multiple intraoperative perception tasks to share these shared features, thereby reducing redundant feature extraction and obtaining multiple intraoperative perception results from consistent sources. Specifically, this application uses a shared video encoder as its core to construct an intraoperative perception architecture of "one-time encoding - multi-task parallel reasoning - process constraint fusion - dual-channel directional output." It also integrates a perception degradation self-assessment and confidence dynamic adjustment mechanism to achieve low-latency, high-reliability intraoperative state perception that adapts to navigation layering requirements.

[0005] To achieve the above objectives, in a first aspect, embodiments of this application provide an intraoperative state perception method for laparoscopic surgical navigation, comprising: acquiring an intraoperative laparoscopic video stream in real time; organizing continuous video frames in the laparoscopic video stream into video segments; extracting spatiotemporal joint features from the video segments to obtain a shared feature vector characterizing the spatiotemporal state of the intraoperative scene; inputting the shared feature vector into the task processing branches corresponding to multiple types of intraoperative perception tasks to obtain multiple types of intraoperative perception results; wherein, the multiple types of intraoperative perception tasks share the shared feature vector; and outputting the multiple types of intraoperative perception results.

[0006] To achieve the above objectives, in a second aspect, embodiments of this application provide a laparoscopic video intraoperative state perception system, comprising: a video acquisition module for real-time acquisition of intraoperative laparoscopic video streams; a segment organization module for organizing continuous video frames in the laparoscopic video stream into video segments; a shared feature extraction module for performing spatiotemporal joint feature extraction on the video segments to obtain a shared feature vector characterizing the spatiotemporal state of the intraoperative scene; a multi-task perception module for inputting the shared feature vector into the task processing branches corresponding to multiple types of intraoperative perception tasks to obtain multiple types of intraoperative perception results; and an output module for outputting the multiple types of intraoperative perception results; wherein the multiple types of intraoperative perception tasks share the shared feature vector.

[0007] To achieve the above objectives, in a third aspect, embodiments of this application provide a computing device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the intraoperative status perception method for laparoscopic surgical navigation provided by any of the optional implementations of the first aspect.

[0008] The intraoperative state perception method for laparoscopic video provided in this application organizes the laparoscopic video stream into video segments with continuous frame relationships, extracts shared features representing spatiotemporal states from the video segments, and uses these shared features as common inputs to multiple task processing branches. Therefore, multiple task processing branches do not need to repeatedly extract underlying video features separately, reducing redundant computation. Each task processing branch outputs perception results based on the shared features corresponding to the same video segment, enabling different categories of intraoperative perception results to have the same spatiotemporal state basis, and providing homogeneous multi-class perception results for subsequent consistency verification, quality assessment, or downstream calls. Attached Figure Description

[0009] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is an overall flowchart of a laparoscopic video intraoperative status perception method provided in some embodiments of this application; Figure 2 This is a schematic diagram of the architecture of a shared video encoder and multiple intraoperative perception task processing branches provided in some embodiments of this application; Figure 3 This is a flowchart of an endoscopic video segment organization method provided in some embodiments of this application; Figure 4 This is a flowchart of a spatiotemporal joint feature extraction method provided in some embodiments of this application; Figure 5This is a flowchart of a shared video encoder overlapping frame multiplexing encoding method provided in some embodiments of this application; Figure 6 This is a flowchart of surgical procedure context consistency verification and state fusion provided in some embodiments of this application; Figure 7 This is a flowchart of a perceptual degradation detection and confidence reduction process provided in some embodiments of this application; Figure 8 This is a schematic diagram of a computing device provided for some embodiments of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0012] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0013] Furthermore, in this application, directional terms such as "upper," "lower," "inner," and "outer" are defined relative to the indicated placement of the components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and can change accordingly depending on the placement of the components in the accompanying drawings.

[0014] In recent years, laparoscopic surgical robots have become an indispensable and highly valuable intelligent tool in minimally invasive surgical practice. Working in synergy with robotic arms and imaging technology, laparoscopic surgical robots provide precise and effective support for complex laparoscopic surgical scenarios such as urinary system tumor resection, gynecological disease treatment, thoracic and abdominal organ surgery, and gastrointestinal interventional procedures, and are widely used in multiple specialties.

[0015] Laparoscopic surgical robots can integrate laparoscopic surgical navigation systems. These systems provide surgeons with real-time anatomical positioning references by spatially registering preoperative medical images with actual intraoperative anatomical structures. They also offer navigation prompts, allowing surgeons to stay informed about the current surgical status and other information. This is a crucial technology for improving the accuracy and safety of laparoscopic surgery.

[0016] The intraoperative state perception method for laparoscopic video provided in this application can be applied to scenarios involving real-time or near-real-time analysis of laparoscopic surgical videos. This method can be executed by a computing device, which can be an intraoperative workstation, an edge computing server, a computing node in a surgical navigation system, or other computing devices with video processing capabilities.

[0017] The intraoperative status awareness method for laparoscopic video provided in this application can be applied to a multi-layer closed-loop navigation system. This architecture includes a semantic map layer, an intraoperative awareness layer, a registration and localization layer, an intelligent decision-making layer, and an enhanced display layer.

[0018] The system comprises three layers: a semantic map layer providing preoperative anatomical structure and spatial risk information; an intraoperative perception layer acquiring information on surgical stages, instruments, anatomical structures, and risk events based on intraoperative video; a registration and positioning layer establishing the correspondence between preoperative information and intraoperative space; an intelligent decision-making layer generating task-level planning results by combining intraoperative status, surgical task map, and knowledge retrieval results; and an enhanced display layer outputting corresponding navigation prompts. Registration deviations can be fed back to the semantic map layer to correct the map. Surgeons can re-enter the perception process through intraoperative video, and task planning can trigger knowledge retrieval, adjusting subsequent planning or retrieval strategies based on surgeon feedback. The layers can exchange data using a standardized message format defined by JSON Schema, forming a closed-loop navigation process combining map correction, perception decision-making, prompt feedback, and knowledge enhancement.

[0019] The method provided in this application is applied to laparoscopic surgical navigation and can provide intraoperative status perception results for the registration and positioning layer and intelligent decision-making layer of the laparoscopic surgical navigation system.

[0020] The intraoperative status perception method for laparoscopic video provided in this application embodiment can be specifically executed by the intraoperative perception layer in a multi-layer closed-loop navigation system.

[0021] Figure 1 This is a flowchart of an intraoperative status perception method for laparoscopic video provided in some embodiments of this application.

[0022] Figure 2 This is a schematic diagram of the architecture of a shared video encoder and multiple intraoperative perception task processing branches provided in some embodiments of this application.

[0023] like Figure 1 and Figure 2 As shown, some embodiments of this application provide a laparoscopic video intraoperative status perception method that may include the following steps: Step S110: Acquire intraoperative laparoscopic video stream in real time.

[0024] For example, the computing device can acquire intraoperative laparoscopic video streams in real time from a laparoscopic camera, video capture card, surgical robot system, intraoperative video server, or video caching module. The laparoscopic video stream can be a real-time video stream.

[0025] In some embodiments, the laparoscopic video stream may include a monocular laparoscopic video stream or a binocular laparoscopic video stream.

[0026] By acquiring laparoscopic video streams, instruments, tissues, surgical areas, image quality, and event changes in the intraoperative field of view can be fed into the computing device as continuous video data.

[0027] Step S120: Organize consecutive video frames in the endoscopic video stream into video segments.

[0028] For example, the computing device can acquire continuous video frames from the endoscopic video stream and divide the continuous video frames into at least one video segment according to a preset video segmentation rule.

[0029] Video clips can serve as processing units with defined time ranges. Compared to frame-by-frame processing, video clips can preserve changes in motion, tissue exposure, field of view, and event duration between adjacent frames. Compared to processing an entire video clip at once, video clips facilitate real-time or near-real-time inference using a sliding window approach. This allows subsequent shared features to correspond to defined frame sequences and temporal positions.

[0030] Step S130: Perform spatiotemporal joint feature extraction on the video clip to obtain a shared feature vector used to characterize the spatiotemporal state of the intraoperative scene.

[0031] For example, shared features can be used to represent spatial content and temporal changes in a video clip. Spatial content may include visible structures, instrument positions, instrument shapes, surgical areas, or image quality. Temporal changes may include instrument movement, changes in tissue exposure, phase transitions, changes in field of view, or changes in intraoperative events.

[0032] Shared feature vectors can include video segment-level spatiotemporal feature vectors, or a set of vectors organized according to spatial location, temporal location, or feature hierarchy. For detection, segmentation, or spatial estimation tasks, the shared video encoder can also retain multi-level spatiotemporal features corresponding to the shared feature vectors.

[0033] After the underlying image information and cross-frame temporal information in the video clip are extracted using shared features, they are collectively encoded into the same spatiotemporal state representation. Since multiple intraoperative perception tasks subsequently use this shared feature, the underlying video encoding, which might have been repeatedly performed by each task, can be centralized into a single shared feature extraction process, thereby reducing redundant computation and providing a basis for the common source of results from multiple tasks.

[0034] Step S140: Input the shared feature vector into the task processing branch corresponding to the multi-class intraoperative perception task to obtain the multi-class intraoperative perception results.

[0035] For example, multiple task processing branches can receive the same shared feature vector and perform different types of intraoperative perception tasks based on the shared feature vector.

[0036] Among them, the various intraoperative perception tasks may include at least two of the following: surgical phase identification task, instrument status identification task, anatomical structure or surgical area identification task, intraoperative event identification task, and spatial state estimation task.

[0037] Because each task processing branch receives the same shared feature vector, results such as stage, instrument, structure, event, and spatial state can correspond to the same video segment, rather than to different frame sequences or different encoding processes. In this way, after the perception results of subsequent intraoperative tasks are output, comparable and fusionable state relationships can be established among multiple results from the same source.

[0038] Step S150: Output multiple types of intraoperative perception results.

[0039] For example, the multiple types of intraoperative sensing results may include at least two of the following: surgical phase sensing results, instrument status sensing results, structural region sensing results, intraoperative event sensing results, and spatial status sensing results.

[0040] By outputting multiple types of intraoperative perception results, the computing device can convert the internal inference results obtained from shared feature vectors and task processing branches into structured state information that can be called by downstream modules, and simultaneously provide input to downstream functions such as spatial registration, localization, process judgment, status display, or prompt information generation. Since these results are derived from shared feature vectors of the same video segment, the output results have a consistent temporal source and spatiotemporal state basis.

[0041] Through the above steps, the computing device organizes the endoscopic video stream into video segments with continuous frame relationships, extracts shared feature vectors representing spatiotemporal states from these video segments, and uses these shared feature vectors as common inputs to multiple task processing branches. This eliminates the need for multiple task processing branches to repeatedly extract underlying video features, reducing redundant computation. Each task processing branch outputs perception results based on the shared feature vector corresponding to the same video segment, ensuring that intraoperative perception results of different categories have the same spatiotemporal state foundation, and providing homogeneous multi-class perception results for subsequent consistency verification, quality assessment, or downstream applications.

[0042] Figure 3 This is a flowchart of a laparoscopic video segment organization provided in some embodiments of this application.

[0043] like Figure 3 As shown, in some embodiments, step S120 can be implemented in the following ways: Step S121: Preprocess the consecutive video frames to obtain preprocessed video frames.

[0044] In some embodiments, the computing device may perform at least one of the following on consecutive video frames in the endoscopic video stream: resizing, pixel value normalization, denoising, brightness correction, and time synchronization. Through preprocessing, video frames from different sources, with different brightness states, or different timestamps can be converted into inputs more suitable for processing by a shared video encoder, thereby reducing the impact of acquisition differences on subsequent shared feature vector extraction.

[0045] In some examples, computing devices can scale consecutive video frames to a preset spatial resolution, such as 448×256 or other spatial resolutions. Resizing enables each frame in subsequent video segments to have a consistent height and width, facilitating shared video encoders to perform convolutions, attention calculations, or spatiotemporal block partitioning at a fixed spatial scale.

[0046] In some examples, pixel value normalization can convert the pixel values ​​of video frames to a range of 0 to 1, or it can be normalized based on a preset mean and standard deviation. Normalization can reduce the differences in pixel distribution caused by different cavity light sources, exposure states, or acquisition devices, enabling the shared video encoder to obtain a more stable input distribution under different video sources.

[0047] In some examples, denoising can employ Gaussian filtering, temporal smoothing, or learned denoising techniques to reduce the impact of acquisition noise or compression noise on image details.

[0048] In some examples, brightness correction can be achieved using histogram equalization, gamma correction, or local brightness compensation to reduce brightness differences caused by variations in the cavity mirror light source, reflections, or local shadows.

[0049] In some examples, time synchronization processing can align frame timestamps in monocular or binocular video, giving frames within the same video segment a defined temporal order.

[0050] Step S122: Use a fixed-length sliding window to determine a continuous frame sequence from the preprocessed video frames.

[0051] In this method, overlapping frames are set between adjacent sliding windows, with the number of overlapping frames being greater than half the window length. This allows for the preservation of sufficient continuous temporal information between adjacent video segments, reducing the fragmentation of stage transitions or changes in machine motion at segment boundaries, thereby ensuring the temporal continuity of stage transition and motion change recognition.

[0052] For example, a fixed-length sliding window can move sequentially along the video frames, each time covering consecutive video frames as a consecutive frame sequence. The window length is used to limit the number of video frames contained in each consecutive frame sequence, and the sliding step size is used to limit the interval between the starting frames of adjacent windows.

[0053] In one example, the window length is T frames, T=16, the sliding step is 4 frames, and adjacent windows overlap by 12 frames. By setting overlapping frames, motion or state changes at the end of the previous window can continue to the beginning of the next window, thus maintaining the continuity of stage transition and motion change recognition. The window length and sliding step can also be adjusted according to the video frame rate, computing device performance, and real-time requirements.

[0054] Step S123: Determine the continuous frame sequence as a video segment.

[0055] For example, the computing device can identify a continuous sequence of frames covered by each sliding window as a video segment and record a segment identifier, start frame number, end frame number, center frame number or timestamp for the video segment, so that subsequent shared feature vectors, multi-class intraoperative perception results, process constraint verification results and dual-channel output results can be traced back to the same video segment.

[0056] In one example, each video segment in the preprocessed video segment sequence has a shape of [T, 3, H, W], where T represents the number of video frames contained in the video segment, 3 represents that each frame has three color channels (red, green, and blue), and H and W represent the height and width of the video frame, respectively. Thus, continuous endoscopic videos are organized into a sequence of video segments with a uniform number of frames, channels, and spatial resolution, so that shared feature vector extraction can be performed on a segment-by-segment basis.

[0057] By using the above video segment organization method, the computing device can perform shared feature vector extraction on a segment-by-segment basis while preserving continuous motion changes. When there are overlapping frames between adjacent video segments, the motion or state change at the end of the previous video segment can continue to the beginning of the next video segment, reducing the probability that stage transitions, changes in machinery movements, or events are truncated by segment boundaries.

[0058] Figure 4 This is a flowchart of spatiotemporal joint feature extraction provided in some embodiments of this application.

[0059] like Figure 4 As shown, in some embodiments, step S130 can be implemented in the following way: Step S131: Input the video clip into the shared video encoder.

[0060] For example, a shared video encoder can be a video base model, a 3D convolutional network, a spatiotemporal Transformer network, a video joint embedding prediction model, an encoding model pre-trained on surgical videos, or a lightweight video encoding model.

[0061] In some examples, shared video encoders can employ a video joint embedding prediction architecture. Taking the SurgMotion video joint embedding prediction architecture as an example, SurgMotion can be built upon a V-JEPA-like native video architecture. This type of architecture does not extract features from each frame individually and then perform simple temporal stitching; instead, it takes video segments as input and learns the spatiotemporal dynamic relationships between consecutive frames in the latent space. In this way, the encoder can more directly capture the temporal changes in instrument movement trajectories, tissue deformation, the effects of energy instruments, phase transition trends, and events such as smoke or bleeding.

[0062] In some examples, SurgMotion may include a context encoder, a target encoder, and a predictor. The context encoder encodes unmasked spatiotemporal blocks of video to obtain a contextual representation; the target encoder forms a target representation for masked or predicted spatiotemporal blocks of video; and the predictor predicts the motion or state features of the masked spatiotemporal blocks in the latent space based on the contextual representation. Through this joint embedding prediction approach, the shared video encoder can learn the spatiotemporal continuity within video segments, rather than relying solely on the texture of a single frame.

[0063] In some embodiments, a shared video encoder can be obtained through the following training process.

[0064] S131a, performs self-supervised pre-training on the shared video encoder.

[0065] In some examples, self-supervised pre-training can be performed on no fewer than 100,000 surgical video clips.

[0066] The pre-training task can be: randomly masking a portion of the spatiotemporal blocks in the input video clip, and then predicting the motion or state features of the masked spatiotemporal blocks in the latent space using an encoder and predictor structure. This stage does not require manual task annotation for each frame, enabling the shared video encoder to learn general temporal dynamic representations in surgical videos.

[0067] S131b performs surgical domain adaptation on a pre-trained shared video encoder.

[0068] In some examples, domain-adaptive fine-tuning can be performed using approximately 500 complete videos of specific laparoscopic surgeries, with video sources covering different surgeons, patient conditions, and acquisition devices. The domain-adaptive phase allows the shared video encoder to adapt to the anatomical structures, surgical phase divisions, instrument operation modes, image quality variations, and event types within a specific laparoscopic surgery. It should be noted that the number and sources of the videos mentioned above are merely examples and are not specific limitations.

[0069] S131c, jointly trains a shared video encoder and task processing branches.

[0070] In some examples, stage annotations, instrument boxes and motion annotations, structure mask annotations, event annotations, and pseudo-depth supervision data can be used in combination to perform end-to-end multi-task fine-tuning of the shared video encoder and individual task heads.

[0071] The training strategy can employ a gradual unfreezing approach. First, the shared video encoder is frozen, and only the task heads are trained until convergence. Then, the shared video encoder and the processing branches of each task are jointly fine-tuned with a small learning rate, for example, 1 / 10 of the pre-training learning rate. The total loss for multi-task training can be a weighted sum of the losses of each task, or an adaptive weighting strategy based on task uncertainty can be used (this strategy dynamically adjusts the loss weights according to the learning uncertainty of each task; the higher the learning uncertainty of a task, the lower the corresponding loss weight, thus achieving balanced training across multiple tasks).

[0072] Step S132: Jointly encode the spatial and temporal information in the video segment using a shared video encoder to obtain the joint encoding result.

[0073] For example, spatial information may include visible structures, instrument shapes, instrument positions, surgical areas, tissue textures, or image quality states in video frames. Temporal information may include instrument movement, changes in tissue exposure, field of view shifts, phase transitions, smoke changes, bleeding changes, or occlusion changes between consecutive frames. The shared video encoder jointly encodes the spatial and temporal information in the video segment, so that the joint encoding result simultaneously includes single-frame spatial content and cross-frame temporal changes.

[0074] In some embodiments, when there are overlapping frames between adjacent video segments, the shared video encoder may use an overlapping frame multiplexing method for joint encoding.

[0075] Figure 5 This is a flowchart of a shared video encoder overlapping frame multiplexing encoding provided in some embodiments of this application.

[0076] like Figure 5 As shown, step S132 above can be implemented in the following way: Step S1321: Determine whether there are overlapping frames between the video segment and the previous video segment.

[0077] For example, a computing device can determine whether a current video segment and a previous video segment contain the same video frame based on the frame number, timestamp, or cache location of the current video segment and the previous video segment.

[0078] In some examples, if the current video segment is frame numbered 104 to 119 and the previous video segment is frame numbered 100 to 115, then frames 104 to 115 can be identified as overlapping frames. By first determining whether overlapping frames exist, the computing device can determine whether to perform incremental coding or full coding subsequently.

[0079] Step S1322: When there are overlapping frames between a video segment and the previous video segment, obtain the intermediate encoding result generated by the shared video encoder when jointly encoding the previous video segment.

[0080] For example, intermediate encoding results may include intermediate layer features corresponding to overlapping frames, attention key features, spatiotemporal block features, frame-level latent vectors, or other cached features that can participate in the encoding of the current video segment. Since these intermediate encoding results have already been generated during the joint encoding process of the previous video segment, the current video segment does not need to repeat the complete encoding of overlapping frames from the original pixels.

[0081] Step S1323: Identify the non-overlapping frames in the video segment, excluding overlapping frames.

[0082] For example, a computing device can determine non-overlapping frames in the current video segment, excluding overlapping frames, based on the frame number, timestamp, or cache location of the current video segment.

[0083] Taking the aforementioned example, if the current video segment has frame numbers 104 to 119 and the previous video segment has frame numbers 100 to 115, then frame numbers 116 to 119 can be identified as non-overlapping frames. Non-overlapping frames carry additional time information about the current video segment compared to the previous video segment.

[0084] Step S1324: Encode the non-overlapping frames and obtain the joint encoding result based on the intermediate encoding result and the encoding result of the non-overlapping frames.

[0085] For example, a shared video encoder can encode non-overlapping frames to obtain the encoding results of the non-overlapping frames, and generate a joint encoding result of the current video segment based on the intermediate encoding result of the previous video segment and the encoding results of the non-overlapping frames. This generation process may include splicing, attention fusion, temporal position rearrangement, weighted fusion, or reprojection. Thus, the joint encoding result of the current video segment contains both the historical continuous information corresponding to the overlapping frames and the newly added state changes corresponding to the non-overlapping frames.

[0086] In the absence of overlapping frames, the computing device can encode all video frames in the current video segment to obtain the joint encoding result of the current video segment. Therefore, when overlapping frames exist between adjacent video segments, the computing device can reduce the repetitive encoding of overlapping frames; when no overlapping frames exist, the computing device can still complete the encoding of regular video segments. This method preserves the temporal continuity provided by sliding segments while reducing the amount of repetitive computation on overlapping video segments.

[0087] Step S133: Generate a shared feature vector based on the joint encoding result.

[0088] For example, the computing device can pool, project, normalize, or fuse features on the joint encoding results to generate a shared feature vector with fixed dimensions or a preset structure.

[0089] In some examples, the shared feature vector can be a video segment-level spatiotemporal feature vector.

[0090] In one example, the shape of the video clip-level spatiotemporal feature vector output by the shared video encoder is [1, d], where d represents the feature dimension and can be 768.

[0091] In other examples, the shared feature vector may include multiple levels of spatiotemporal feature maps for use by different task processing branches such as detection, segmentation, event recognition, or spatial estimation. The shared feature vector can be used to characterize at least one of instrument movement, changes in tissue exposure, phase transitions, changes in image quality, and changes in intraoperative events.

[0092] In some embodiments, each task processing branch may share a shared feature vector output by a shared video encoder and output intraoperative perception results of the corresponding type. The task processing branches may include at least two of the following: surgical stage recognition branch, instrument status recognition branch, anatomical structure or surgical area recognition branch, intraoperative event recognition branch, and spatial state estimation branch.

[0093] In this embodiment, for scenarios that require simultaneous execution of N types of intraoperative perception tasks, the shared video encoder performs low-level spatial and temporal feature extraction only once on the current video segment. Each task processing branch reuses the shared feature vector obtained from this feature extraction and completes the classification, detection, segmentation, event recognition, or spatial state estimation of the corresponding task based on the shared feature vector.

[0094] In the approach of deploying independent video coding models for N types of tasks, N types of tasks require N complete feature extractions, and the accumulated inference delay during the procedure makes it difficult to meet the millisecond-level response requirements of real-time navigation.

[0095] This embodiment eliminates the need to repeat the complete feature extraction process N times for the same video segment, and can merge repeated video encoding processes into a single shared encoding process.

[0096] Since video encoding typically constitutes the main computational overhead in intraoperative perception reasoning, a shared video encoder can reduce the amount of repetitive encoding computation that increases with the number of tasks. Each task processing branch can use a lightweight task head to process shared feature vectors in parallel, enabling the reasoning processes of different tasks to be executed at least partially in parallel. This avoids the accumulation of latency caused by end-to-end serial reasoning of each task, making multi-task intraoperative perception more suitable for real-time or near-real-time navigation scenarios.

[0097] In one test example, the total end-to-end inference latency for the computing device to complete five types of intraoperative perception tasks does not exceed 50 milliseconds; this value is only used to illustrate the performance of an exemplary implementation and does not constitute a limitation on the computing platform, the number of tasks, or the inference latency.

[0098] In some embodiments, the task processing branch can be a lightweight task head, or at least one of a decoding network, a classification network, a detection network, a segmentation network, an event recognition network, or a spatial estimation network.

[0099] In some embodiments, step S140 may include at least two of steps S141 to S145.

[0100] Step S141: Input the shared feature vector into the surgical stage recognition branch to obtain the surgical stage perception result.

[0101] For example, the surgical stage identification branch can output the probability distribution of each candidate surgical stage based on a shared feature vector, and determine the stage whose probability meets a preset condition as the surgical stage perception result. The surgical stage identification branch may include a fully connected layer, a temporal classification layer, or an attention classification layer, used to output the stage probability distribution based on the shared feature vector.

[0102] In some examples, the surgical stage identification branch can employ a two-layer fully connected network and a Softmax layer. This branch can receive a d-dimensional shared feature vector, for example, d=768; the first fully connected network can map the 768-dimensional features to 256-dimensional features and apply ReLU activation; the second fully connected network can map the 256-dimensional features to N_phase-dimensional stage features, where N_phase can be 8 or other stages; the Softmax layer can output the probability distribution of each candidate stage. This structure is used to illustrate a lightweight implementation of the stage identification task head, and does not limit this application to using the same dimensions or the same activation function.

[0103] The surgical phase perception results may include at least one of the following: current phase, phase confidence, probability distribution of each phase, and phase transition state.

[0104] Taking partial nephrectomy as an example, the various surgical stages can be shown in Table 1 below: Table 1

[0105] The surgical phase perception results may include at least one of the following: current phase, phase confidence, probability distribution of each phase, and phase transition state.

[0106] Taking partial nephrectomy as an example, the perception results at each stage of the surgery are shown in Table 2 below: Table 2

[0107] Step S142: Input the shared feature vector into the instrument status recognition branch to obtain the instrument status perception result.

[0108] For example, the device state recognition branch can output at least one of device category, device position, device motion, device action, or device tracking number based on a shared feature vector. The device state recognition branch may include a detection head and a motion recognition head, wherein the detection head is used to output the device category and device position, and the motion recognition head is used to output the device action based on device region features or a shared feature vector.

[0109] In some examples, the device status recognition branch can employ a lightweight DETR detection head. This head can take a shared feature vector as input, set N_query learnable query vectors (e.g., N_query=16), and output the category prediction and bounding box corresponding to each query through a cross-attention mechanism of two Transformer decoders. An action recognition branch can be placed at the end of the detection head to output the action category and action confidence. Device tracking can employ an online association strategy based on feature similarity, maintaining the tracking number through cross-frame matching of detection results.

[0110] The results of device status perception can include device category, device bounding box, device key points, action category, action confidence, detection confidence, or cross-frame tracking number.

[0111] Identifiable instrument categories may include grasping forceps, scissors, suction devices, electrocoagulation devices, ultrasonic scalpels, needle holders, and vascular occlusion forceps, etc.; identifiable action categories may include traction, separation, clamping, cutting, electrocoagulation, suction, suturing, and knotting, etc.

[0112] Taking partial nephrectomy as an example, the status sensing results of each instrument are shown in Table 3 below: Table 3

[0113] Step S143: Input the shared feature vector into the anatomical structure or surgical area recognition branch to obtain the structural region perception result.

[0114] For example, the anatomical structure or surgical region recognition branch can output at least one of the following: visible structure category, structural region, structural mask, surgical region, or exposure state. The anatomical structure or surgical region recognition branch may include a segmentation decoding head, a region classification head, or a multi-scale feature fusion layer for outputting structural region perception results.

[0115] In some examples, the anatomical structure or surgical region identification branch can employ a lightweight FPN semantic segmentation decoder. This branch processes shared feature vectors through three upsampling branches at different scales and outputs a pixel-level classification probability map after fusing multi-scale information. In this way, the structural region identification branch can both distinguish structural categories using high-level semantic information and determine structural boundaries or region masks using high-resolution features.

[0116] Taking partial nephrectomy as an example, the anatomical structures or surgical areas can be categorized as shown in Table 4 below: Table 4

[0117] The results of structure region perception can include structure category, pixel-level mask, region confidence, and exposure or occlusion status.

[0118] Taking partial nephrectomy as an example, the perception results of each structural region are shown in Table 5 below: Table 5

[0119] Step S144: Input the shared feature vector into the intraoperative event recognition branch to obtain the intraoperative event perception result.

[0120] For example, the intraoperative event recognition branch can output event results such as smoke, bleeding, blurred vision, instrument occlusion, overexposure, underexposure, rapid movement of the field of vision, or insufficient visible area. The intraoperative event recognition branch can be a multi-label event classification head, enabling the same video clip to correspond to multiple events such as smoke, bleeding, blur, or occlusion simultaneously.

[0121] In some examples, the intraoperative event recognition branch can employ a two-layer fully connected network and a sigmoid output layer. This branch can receive a d-dimensional shared feature vector, for example, d=768; the first fully connected network can map the 768-dimensional features to 128-dimensional features, and the second fully connected network can map the 128-dimensional features to N_event-dimensional event features, where N_event can be 4 or other event numbers; the sigmoid output layer can output the independent detection probability of each event. Due to the use of a multi-label classification approach, the same video segment can simultaneously correspond to multiple events such as bleeding, smoke, blurred vision, and instrument occlusion.

[0122] Taking partial nephrectomy as an example, the types of intraoperative events are shown in Table 6 below: Table 6

[0123] Intraoperative event perception results can include event type, whether the event was detected, event severity, affected area, and event confidence level.

[0124] Taking partial nephrectomy as an example, the results of perception of various intraoperative events are shown in Table 7 below: Table 7

[0125] Step S145: Input the shared feature vector into the spatial state estimation branch to obtain the spatial state perception result.

[0126] For example, the spatial state estimation branch can output at least one of spatial cues, relative depth, absolute depth, point cloud, spatial relationships between structures, or spatial state confidence. The spatial state perception results can be used to generate spatial state information and provide input for registration and positioning, spatial relationship calculation, or navigation cues reliability assessment.

[0127] The spatial state estimation branch can include a depth estimation head, a spatial relation estimation head, or a point cloud generation unit, which is used to output perception results related to the spatial state.

[0128] In some examples, under monocular laparoscopic conditions, the spatial state estimation branch can output a relative depth map, the distance relationships between structures, the relative positional relationships between instruments and structures, or spatial state confidence. The monocular implementation does not rely on left-right view parallax, and therefore can be applied to devices that only provide a single laparoscopic video stream.

[0129] Under monocular endoscopy, the spatial state estimation branch can employ a lightweight depth estimation decoder based on the MiDaS architecture. It outputs a dense relative depth map spatially aligned with the input frame and calculates the relative spatial relationships between key structures based on the depth map and segmentation mask. For example, spatial relationships could represent "the instrument tip is located above the surface of the renal parenchyma" or "the instrument is far from the renal hilum region."

[0130] In other examples, within robotic surgical systems, if endoscopic pose information can be obtained through the robot's kinematics interface, the computing device can fuse depth estimation results with kinematic data to output more accurate 3D spatial coordinates and support multi-view spatial information accumulation. The computing device can generate depth information based on disparity information between the left and right views, and then estimate the spatial state based on this depth information. The computing device can match features from the left and right views to obtain a disparity map, and calculate depth values ​​based on the camera baseline distance, focal length, and disparity values. Furthermore, the computing device can back-project pixel coordinates into a 3D point cloud in the camera coordinate system based on depth information and camera intrinsic parameters.

[0131] In other examples, under the condition of a robotic surgical system, if the computing device can also obtain the laparoscopic pose information or the endoscopic pose information of the surgical robot system, the computing device can fuse the depth information with the pose information to generate multi-time accumulated spatial state information.

[0132] Taking partial nephrectomy as an example, the results of spatial state perception are shown in Table 8 below: Table 8

[0133] Figure 6 This is a flowchart of surgical procedure context consistency verification and state fusion provided in some embodiments of this application.

[0134] like Figure 6 As shown, in some embodiments, the computing device may also perform process constraint consistency verification and state fusion after obtaining multiple types of intraoperative perception results, which may include steps S410 to S440.

[0135] Step S410: Obtain the constraints of the laparoscopic surgery procedure.

[0136] For example, laparoscopic surgical procedure constraints are used to characterize prior matching relationships between multiple types of intraoperative state elements. These multiple types of intraoperative state elements may include prior associations between at least two of the following: surgical stage, instrument status, anatomical structure, surgical area, and intraoperative events.

[0137] In some embodiments, the constraints of the laparoscopic surgical procedure can be represented by a rule table, association matrix, probability graph, knowledge graph, state transition relation, statistical prior model, or a trained constraint model.

[0138] In some examples, constraints in laparoscopic surgical procedures may include stage-instrument association matrices, stage-anatomical region association matrices, and instrument-action prior distributions. The stage-instrument association matrix represents the strength of association between different surgical stages and different instrument types; the stage-anatomical region association matrix represents the strength of association between different surgical stages and different visible structures or surgical regions; and the instrument-action prior distribution represents the probability or degree of matching of a particular type of instrument performing different actions.

[0139] Taking partial nephrectomy as an example, the candidate stages may include establishing the approach and surgical field, fascia or fat management, vascular identification and control, exposure of the target organ or region, confirmation of target boundaries, target resection, tissue suturing or collection system management, and specimen retrieval, examination, and finishing. It should be noted that the names of these stages can be replaced according to the specific surgical procedure.

[0140] Phase-device association matrix M_phase_instr[i][j]: Represents the association strength between the i-th surgical phase and the j-th device, with values ​​[0, 1], as shown in Table 9 below: Table 9

[0141] Phase-anatomical region association matrix M_phase_region[i][k]: represents the association strength between the i-th surgical phase and the k-th anatomical region / structure.

[0142] The device-action prior distribution P_action|instr(a|j) represents the conditional probability distribution of each action performed by the j-th device.

[0143] Step S420: Based on prior matching relationships, perform surgical procedure context consistency verification on multiple types of intraoperative perception results.

[0144] In some embodiments, step S420 may include steps S421 to S424.

[0145] Step S421: Calculate the instrument consistency score based on the surgical stage perception results and instrument status perception results.

[0146] In some embodiments, if the surgical stage perception result is candidate stage P_i and the observed instrument set is O_instr, then the instrument consistency score Score_instr(P_i) can be calculated according to the following formula: Formula 1; In Formula 1, j represents the observed instrument, M_phase_instr[i][j] represents the correlation strength between candidate phase P_i and instrument j, conf(j) represents the detection confidence of instrument j, and |O_instr| represents the number of observed instruments.

[0147] It should be noted that the above formula is only an exemplary calculation method. The device consistency score can also be determined by weighted average, maximum association strength, normalized probability, or output of a trained model.

[0148] Step S422: Calculate the region consistency score based on the perception results during the surgical phase and the perception results of the structural region.

[0149] In some embodiments, the regional consistency score can represent the degree of matching between the candidate surgical stage and the currently visible structure or surgical area.

[0150] The regional consistency score, Score_region(P_i), can be calculated using the following formula: Formula 2; In Formula 2, k represents the observed anatomical region / structure, M_phase_region[i][k] represents the correlation strength between candidate phase P_i and anatomical region / structure k, conf(k) represents the detection confidence of anatomical region / structure k, and |O_region| represents the number of observed anatomical regions.

[0151] Thus, the stage determination not only depends on the probability of the stage identification branch, but can also be constrained by the actual visible structures or surgical areas in the current image.

[0152] Step S423: Weight and fuse the device consistency score and the regional consistency score to obtain the overall consistency score; In some embodiments, the computing device can calculate a comprehensive consistency score based on the instrument consistency score and the regional consistency score. For example, the comprehensive consistency score Score_consistency(P_i) can be calculated according to the following formula: Formula 3; In Formula 3, Here, Score_region(P_i) represents the weighting coefficient, and Score_region(P_i) represents the region consistency score. The computing device can also adjust the confidence level of the surgical stage perception results based on the overall consistency score. For example, the fused stage confidence level P_fused[i] can be calculated according to the following formula: Formula 4; In Formula 4, P_phase[i] represents the original probability of the candidate stage P_i output by the stage recognition branch. It is a consistency enhancement factor.

[0153] Step S424: Based on the overall consistency score, determine whether the perception results of the surgical stage meet the current process context.

[0154] In some embodiments, the current process context can be determined by the stage results of the previous video segment, the multiple intraoperative perception results of the current video segment, the preset stage transition relationship, or the state sequence within a historical time window. The computing device can perform a weighted fusion of the instrument consistency score and the region consistency score to obtain a comprehensive consistency score, and determine whether the surgical stage perception results match the current process context based on the comprehensive consistency score.

[0155] Step S425: If the perception result of the surgical stage does not meet the current process context, reduce the confidence of the perception result of the surgical stage, or mark the perception result of the surgical stage as pending confirmation, extend the observation window, and perform secondary verification based on the perception result of subsequent video segments.

[0156] In some embodiments, if the corrected result does not match the current process context, or if the maximum confidence stage conflicts with process constraints, the computing device can reduce the confidence of the surgical stage perception result, or mark the surgical stage perception result as pending confirmation. In this way, the process constraints do not replace the multi-task perception result itself, but rather serve as a verification mechanism to restrict clearly inconsistent single results from being directly output as a definitive state.

[0157] The computing device can also extend the observation window, continue to accumulate multiple types of intraoperative perception results in subsequent video segments, and make secondary judgments based on more video frames.

[0158] In some embodiments, process constraint consistency verification may further include at least one of steps S426 and S427.

[0159] Step S426: Perform device-stage joint verification on the detected devices.

[0160] In some embodiments, for each detected device, the computing device can verify whether the device belongs to a highly correlated device in the current verification stage based on the stage-device association matrix. The computing device can compare the association strength between the current verification stage and the device with a preset association strength threshold.

[0161] In one example, the preset association strength threshold θ can be 0.3. If an instrument with an association strength lower than θ is detected, and the detection confidence of that instrument is greater than 0.8, the computing device can increase the sensitivity of the stage transition flag and determine whether a stage transition has been entered or whether there is a deviation in the current stage judgment. The above threshold is only an example and can be adjusted according to the surgical procedure, instrument type, or calibration data.

[0162] Through joint instrument-stage verification, instrument test results with low correlation but high confidence can be used as the basis for verifying stage changes or stage deviations, rather than being directly output as isolated instrument states.

[0163] Step S427: Perform a structural-stage consistency assessment on the identified anatomical structures.

[0164] In some embodiments, the computing device can verify whether the identified anatomical structure matches the expected anatomical structure or expected visible area in the current confirmation stage, based on the current confirmation stage and the stage-anatomical region correlation matrix. If an abnormal structure that should not be exposed in the current stage is identified, the computing device can generate an anatomical abnormality exposure marker and increase the risk level of the current scene.

[0165] In one example, if the current confirmation stage is P2, but the structural region identification branch has already identified an "open region of the collection system," the computing device can mark this structure as an anomalous structure in the current stage and use the dissection anomaly exposure mark as the basis for generating a risk state. Thus, the structural identification result not only describes the visible content but also reverse-verifies the stage state and reflects any obvious conflicts between the stage and the structure in the risk state.

[0166] Step S430: Based on the consistency verification results, correct the confidence level of at least one type of result among the multiple types of intraoperative perception results.

[0167] In some embodiments, the computing device may adjust the confidence level of at least one type of result, such as stage, structure, event, or spatial state, based on instrument consistency score, regional consistency score, overall consistency score, stage transition relationship, or anomalous structure marker. Confidence adjustment may include increasing the confidence level of results consistent with process constraints, decreasing the confidence level of results conflicting with process constraints, or marking the relevant results as pending confirmation.

[0168] Step S440: Generate fused intraoperative status information based on the corrected multi-class intraoperative perception results.

[0169] In some embodiments, the fused intraoperative status information may include at least one of stage status, instrument status, anatomical status, event status, spatial status, perceived quality, and overall confidence. By performing consistency checks among the results of multiple tasks, the fused intraoperative status information is no longer just a simple juxtaposition of multiple task results, but rather reflects the degree of matching between different status elements.

[0170] When multiple intraoperative perception tasks output results independently, each task processing branch makes judgments based solely on its own recognition results, lacking the constraints of the surgical procedure context of partial nephrectomy, and cross-validation between results from different tasks is impossible. Even if a task processing branch outputs a high raw confidence level, the result may still not match the current surgical stage, instrument status, instrument movement, visible anatomical structures, or surgical area. This can easily lead to logical contradictions such as inconsistencies between the surgical stage and instrument status, between the surgical stage and visible anatomical structures, or between instrument movement and the current surgical area. As a result, single-task misjudgments are difficult to detect in a timely manner, reducing the reliability of multi-task intraoperative perception results.

[0171] To improve the reliability of multi-task intraoperative perception results, this embodiment introduces a priori surgical procedures for partial nephrectomy, establishing prior matching relationships between surgical stages and instrument status, surgical stages and anatomical structures or surgical areas, and instrument types and instrument actions. Based on these prior matching relationships, the consistency of the workflow context of multiple types of intraoperative perception results is verified.

[0172] The computing device calculates an instrument consistency score based on the surgical stage perception results and instrument status perception results, and a region consistency score based on the surgical stage perception results and structural region perception results. It then performs a weighted fusion of the instrument consistency score and the region consistency score to obtain a comprehensive consistency score. When a task result does not match the current surgical procedure context, the computing device lowers the confidence level of the relevant result or marks it as pending confirmation. It can also extend the observation window and perform secondary verification using subsequent video clips.

[0173] Therefore, the results of each task are no longer directly output as independent results, but are mutually verified using the surgical procedure context and other task results. This reduces single-task misjudgments and logical contradictions between stages, instruments, anatomical structures and surgical areas, improves the accuracy, stability and interpretability of stage identification results, and enhances the overall reliability of multi-task intraoperative perception results.

[0174] Figure 7This is a flowchart of a perceptual degradation detection and confidence reduction process provided in some embodiments of this application.

[0175] like Figure 7 As shown, in some embodiments, the computing device can also detect perceptual degradation events based on shared feature vectors and perform confidence-weighted processing on the affected intraoperative perceptual results. This process may include steps S610 to S640.

[0176] Step S610: Detect perceived degradation events through the intraoperative event identification task branch based on shared feature vectors.

[0177] For example, a perception degradation event may include at least one of smoke, bleeding, blurred vision, instrument occlusion, overexposure, underexposure, rapid movement of the field of vision, and insufficient visible area.

[0178] The computing device can process the shared feature vector through an intraoperative event recognition branch or a perceived quality branch, outputting the presence of a perceived degradation event and its corresponding probability. Since perceived degradation events share the same feature vector with other intraoperative sensing tasks, degradation detection results can correspond to the same video segment with stage, instrument, structural, and spatial status results.

[0179] Step S620: Determine the event type, severity, and affected area of ​​the perceived degradation event.

[0180] For example, the event type can be smoke, bleeding, blurred vision, equipment obstruction, overexposure, underexposure, rapid movement of the field of view, or insufficient visible area.

[0181] The severity can be determined based on the percentage of degraded areas, the intensity of degradation, the duration, or the confidence level of event detection.

[0182] The affected area can be an image region, a structural region, an instrument region, or a time interval within a video clip.

[0183] In some examples, the severity of smoke obscuration can be determined based on the smoke coverage area and smoke concentration, while the severity of bleeding events can be determined based on the bleeding area, color intensity, and duration (number of frames).

[0184] Step S630: Determine the intraoperative sensory tasks affected by the sensory degradation event based on the event type, severity, and affected area.

[0185] For example, a computing device can determine the affected tasks based on a preset task influence relationship.

[0186] In some examples, smoke or blurred vision may simultaneously affect stage identification, instrument detection, and structural region identification; bleeding may primarily affect structural region identification, event identification, and spatial state estimation; instrument occlusion may primarily affect structural region identification and spatial state estimation within the occluded area; and rapid movement of the field of view may affect stage identification and spatial state estimation, which depend on the stability of consecutive frames.

[0187] By first identifying the affected tasks, computing devices can differentiate between tasks rather than uniformly downweighting all perceived results.

[0188] Step S640: Perform confidence-weighted processing on the intraoperative perception results output by the affected intraoperative perception task.

[0189] In some embodiments, step S640 may include steps S641 to S644.

[0190] Step S641: Determine the confidence decay coefficient based on the event type and severity.

[0191] For example, the confidence decay coefficient can be found in a preset decay table, or it can be calculated based on the event type, severity, area of ​​influence, and task type.

[0192] For example, the confidence decay results of different event types on different perceived outcomes are shown in Table 10 below: Table 10

[0193] By using this weighting method that differentiates between event type and task type, computing devices can first determine which type of perception task is affected by the degradation event, and then perform differentiated confidence correction on the corresponding task results, thereby avoiding the degradation of task results that are not significantly affected.

[0194] Step S642: Correct the confidence of the affected intraoperative perception results using the confidence attenuation coefficient to obtain the corrected confidence.

[0195] For example, the computing device can multiply the original confidence level of the affected task by a confidence decay factor to obtain a corrected confidence level. If a task is not affected by the current perceptual degradation event, the confidence level of that task can remain unchanged or be adjusted according to a smaller decay factor.

[0196] Step S643: Generate perceived quality information based on the corrected confidence level.

[0197] For example, perceived quality information may include a comprehensive perceived quality score, corrected confidence levels for each task, degradation factors, affected areas, degradation duration, or quality level. The comprehensive perceived quality score can be calculated based on the corrected confidence levels for multiple tasks, the severity of degradation events, and the duration of degradation events. Thus, downstream modules can determine which perceived results in the current video segment are more reliable and which should have their weight reduced.

[0198] Step S644: Generate perception quality prompt information for downstream navigation modules based on perception quality information, which is used to instruct the downstream navigation modules to adjust the calling weight of the corresponding intraoperative perception results.

[0199] For example, the prompt message can be used to indicate a decrease in the perceived reliability of the current video segment, to indicate that a certain type of result is in a pending confirmation state, or to prompt downstream modules to reduce the usage weight of the corresponding result. This prompt message is used to reflect the reliability of the perceived result and is not limited to specific medical processing methods.

[0200] Without the ability to self-assess perception quality, the raw confidence level output by the intraoperative perception model cannot accurately reflect the actual reliability of the perception results in the current scenario. Especially in scenarios of perception degradation such as smoke, bleeding, blurred vision, instrument obstruction, overexposure, underexposure, or rapid movement of the field of vision, instruments, anatomical structures, or spatial information in the video may be affected to varying degrees, making it easy for the model to output perception results with high confidence but low reliability. If the corresponding perception quality information is not synchronized to downstream modules, the registration and positioning layer and the intelligent decision-making layer will not be able to know the current reliability of the perception results. Directly calling relevant results may cause registration errors, misjudgments of status, or navigation prompt deviations, resulting in insufficient robustness of the intraoperative perception system in degraded scenarios.

[0201] To enhance the robustness of the intraoperative perception system under the aforementioned degradation scenarios, through steps S610 to S644, the computing device incorporates a built-in perception degradation detection and confidence dynamic weighting mechanism. While outputting perception results such as stage, instrument, structure, event, and spatial state, it also performs a self-evaluation of the perception quality of the current video segment.

[0202] The computing device detects perception degradation events and determines the affected intraoperative perception tasks based on the type, severity, and affected area of ​​the degradation event; for example, smoke or blurred vision may affect both instrument detection and anatomical structure recognition, while instrument occlusion may primarily affect structure recognition and spatial state estimation within the occluded area.

[0203] The computing device dynamically downweights the results of affected tasks based on preset task impact relationships, instead of uniformly downweighting the results of tasks that are not significantly affected. It also generates perception quality information that includes degradation factors, affected tasks, corrected confidence levels, and perception quality levels, and synchronously provides this perception quality information to the registration and positioning layer and the intelligent decision-making layer.

[0204] As a result, downstream modules can perceive the current credibility of various results and reduce the call weight of related results, mark related states as pending confirmation, or adopt a degraded navigation prompt method accordingly. This avoids navigation deviations caused by directly using high-confidence erroneous results in degraded scenarios and enhances the scenario robustness of the intraoperative perception and navigation system.

[0205] In some embodiments, the process constraint consistency verification and state fusion process may further include step S450.

[0206] Step S450: Generate a risk status based on the consistency verification results, perceived degradation events, and perceived quality information.

[0207] For example, a computing device can generate a risk status for the current scenario by combining phase consistency scores, perceived degradation events, abnormal structure markers, and a comprehensive perceived quality score. The risk status may include a risk level and a list of risk sources.

[0208] In some examples, a risk level can be determined as high risk when at least two of the following conditions are met: a phase consistency score of less than 0.3; a bleeding event detected with a severity greater than 0.5; the presence of an anatomical abnormality exposure marker; and a perceived quality of life composite score of less than 0.4.

[0209] When any of the above conditions are met, the risk level can be determined as medium risk.

[0210] When the above conditions are not met and the consistency score is greater than 0.7, the risk level can be determined as low risk. The above combination of thresholds and conditions is only used to illustrate one way of generating a risk status and can be adjusted according to specific procedures, calibration data, or system security policies.

[0211] In some embodiments, the fused scene state information can be as shown in Table 11 below: Table 11

[0212] In some embodiments, outputting multiple types of intraoperative perception results in step S150 may include steps S151 to S153.

[0213] Step S151: Generate spatial state information based on multiple intraoperative perception results.

[0214] For example, spatial state information may include at least one of visible structure information, instrument position information, instrument motion information, spatial cues, depth information, point cloud information, spatial state confidence, and perceived quality information.

[0215] Visible structural information in spatial state information may include structural category, structural region, structural mask, or structural confidence level; instrument location information may include instrument category, bounding box, key points, tracking number, or instrument tip location; spatial cues may include relative depth, distance relationship between structures, positional relationship between instruments and structures, or point cloud data.

[0216] By organizing this information into spatial state information, downstream spatial registration, positioning, or spatial relationship calculation modules can directly call the results related to spatial location without having to re-parse all semantic results.

[0217] In some examples, spatial state information can be represented using structured fields such as timestamp, frame number, list of visible structures, list of instrument locations, spatial cues, and perceived quality.

[0218] For example, the spatial state information can be shown in Table 12 below: Table 12

[0219] Step S152: Generate semantic state information based on the results of multiple intraoperative perceptions.

[0220] For example, semantic state information may include at least one of the following: current surgical stage, instrument movement, anatomical state, event flags, cue status, and overall confidence level.

[0221] The current surgical stage in the semantic state information may include stage identifier, stage name, stage confidence level, or whether it needs to be confirmed; instrument action may include instrument category, action category, and action confidence level; event flag may include event type, whether it is activated, severity, and affected area; prompt status may include perceived quality prompt, status needing confirmation prompt, or process context inconsistency prompt.

[0222] By organizing this information into semantic state information, the downstream state display, process judgment, or prompt information generation modules can obtain results that are more suitable for process semantic processing.

[0223] In some examples, semantic state information can be represented by the following structured fields: current surgical stage, stage transition status, instrument list, anatomical status, event flag, risk status, and overall confidence level.

[0224] For example, the semantic state information is shown in Table 13: Table 13

[0225] Step S153: Output spatial state information and semantic state information respectively.

[0226] Among them, spatial state information is geared towards endoscope navigation registration and positioning scenarios, while semantic state information is geared towards endoscope navigation intelligent decision-making scenarios.

[0227] For example, spatial state information and semantic state information can be output in JSON format, or in message objects, interface data packets, or other structured data formats.

[0228] In some embodiments, step S153 may include steps S1531 to S1533.

[0229] Step S1531: Output spatial state information to the registration and positioning layer of the laparoscopic surgical navigation system through the spatial perception output channel.

[0230] For example, the spatial perception output channel can be used in downstream applications such as spatial registration, positioning, or spatial relationship calculation. The spatial perception output channel can output structured data, message objects, or interface data packets, enabling the registration and positioning layer to read spatially relevant content such as visible structures, instrument positions, depth information, and point cloud information.

[0231] Step S1532: Output semantic state information to the intelligent decision layer of the laparoscopic surgery navigation system through the semantic perception output channel.

[0232] For example, the semantic-aware output channel can be used in downstream scenarios such as intraoperative status display, process judgment, or prompt information generation. The semantic-aware output channel can output structured data, message objects, or interface data packets, enabling the intelligent decision-making layer or status display layer to read semantically relevant content such as the current surgical stage, instrument movements, anatomical status, event markers, risk status, and overall confidence level.

[0233] Step S1533: Make the spatial state information and semantic state information originate from the same set of multi-class intraoperative perception results.

[0234] For example, instead of re-executing two independent sensing processes for the two output channels, the computing device organizes the output content according to different downstream usage requirements based on the same shared feature vector and the same set of multi-class intraoperative sensing results. Thus, information homogeneity can be maintained between the spatial sensing output channel and the semantic sensing output channel, reducing state deviations between different channels.

[0235] When multiple types of intraoperative perception results are output as a unified whole dataset, the registration and localization layer and the intelligent decision-making layer will receive complete perception results containing both spatial and semantic information. Since the registration and localization layer primarily requires spatial data such as visible structure locations, instrument locations, depth information, point cloud information, and spatial relationships, while the intelligent decision-making layer primarily requires semantic data such as surgical stage, instrument movements, anatomical status, event markers, and risk status, both downstream modules must manually filter, parse, and transform the required information from the complete perception results. This not only increases the transmission and repeated parsing of irrelevant data, causing data redundancy, but also increases the workload of downstream interface design, data filtering, and module adaptation, resulting in poor adaptability of the intraoperative perception results output and increasing the integration complexity of the navigation system.

[0236] To adapt to the layered architecture of the laparoscopic surgical navigation system, this embodiment adopts a dual-channel directional output mechanism consisting of a spatial perception output channel and a semantic perception output channel. Based on the calling requirements of different downstream modules, the same set of multiple intraoperative perception results are categorized and organized. Specifically, the spatial perception output channel mainly outputs spatial state information such as the location of visible structures, instrument positions, depth information, point cloud information, spatial relationships, and spatial state confidence, and directs this information to the registration and positioning layer to meet spatial processing requirements such as spatial registration, coordinate transformation, positioning, and spatial relationship calculation. The semantic perception output channel mainly outputs semantic state information such as the current surgical stage, instrument movements, anatomical status, event markers, risk status, and overall confidence level, and directs this information to the intelligent decision-making layer to meet semantic decision-making requirements such as process judgment, status display, and navigation prompt information generation.

[0237] Therefore, the registration and positioning layer does not need to sift through spatial data from all semantic results such as stages and events, and the intelligent decision-making layer does not need to parse complete spatial data such as point clouds and depth maps. This reduces irrelevant data transmission, repeated parsing, and data redundancy, thereby reducing the data filtering burden on downstream modules and the system integration cost, and improving the output adaptability between intraoperative perception results and the hierarchical architecture of the navigation system. At the same time, since the two output channels originate from the same shared feature vector and the same set of multi-class intraoperative perception results, the temporal and source consistency of spatial state information and semantic state information can also be guaranteed.

[0238] In some embodiments, obtaining the laparoscopic video stream in step S110 may include a monocular laparoscopic video stream acquisition method or a binocular laparoscopic video stream acquisition method.

[0239] In some examples, obtaining the endoscope video stream in step S110 can be achieved in the following way: Acquire a monocular endoscope video stream and estimate the spatial state based on the temporal changes between consecutive video frames in the monocular endoscope video stream, instrument motion, and visible structure changes.

[0240] For example, after acquiring the monocular endoscope video stream, the computing device can organize the single video frames into video segments and estimate the spatial state based on the temporal changes between consecutive video frames, the movement of the instrument, and the changes in the visible structure.

[0241] Alternatively, obtaining the endoscope video stream in step S110 can be achieved in the following way: The system acquires a binocular endoscope video stream, generates depth information based on the disparity information between the left and right views in the binocular endoscope video stream, and then estimates the spatial state based on the depth information.

[0242] For example, after acquiring the binocular endoscope video stream, the computing device can obtain the left view video frame and the right view video frame respectively.

[0243] Alternatively, the acquisition of the endoscope video stream in step S110 can be achieved in the following way: The system acquires intraoperative binocular laparoscopic video stream and endoscopic pose data of the surgical robot in real time, and generates depth information based on the parallax information between the left and right views in the binocular laparoscopic video stream. The depth information is then fused with the endoscopic pose data to estimate the spatial state.

[0244] The computing device generates a depth map based on the parallax information between the left and right views, and converts the pixel coordinates in the depth map into three-dimensional coordinates in the endoscope camera coordinate system based on camera intrinsic parameters. Endoscope pose data may include rotation and translation information of the endoscope relative to the robot base coordinate system, patient reference coordinate system, or navigation reference coordinate system. Based on the coordinate transformation relationship corresponding to the endoscope pose data, the computing device transforms the three-dimensional coordinates in the camera coordinate system to the target reference coordinate system, and fuses the three-dimensional coordinates at one or more time points to obtain the spatial state.

[0245] In some embodiments, the laparoscopic video intraoperative state perception system may include a video acquisition module, a segment organization module, a shared feature extraction module, a multi-task perception module, and an output module. The video acquisition module can be used to acquire laparoscopic video streams from a laparoscopic camera, video capture card, surgical robot system, intraoperative video server, or video caching module. The segment organization module can be used to preprocess continuous video frames and organize them into video segments according to a preset video segmentation rule. The shared feature extraction module can be used to input the video segments into a shared video encoder and obtain shared feature vectors representing spatiotemporal states. The multi-task perception module can be used to input the shared feature vectors into the task processing branches corresponding to multiple types of intraoperative perception tasks to obtain multiple types of intraoperative perception results. The output module can be used to output the multiple types of intraoperative perception results.

[0246] In some embodiments, the system may further include at least one of a process constraint module, a perception degradation processing module, and a dual-channel output module. The process constraint module can be used to acquire the constraint relationships of the laparoscopic surgical process and perform consistency verification and state fusion on multiple types of intraoperative perception results based on these constraints. The perception degradation processing module can be used to detect perception degradation events, determine the event type, severity, and affected area, and perform confidence-weighted processing on the affected intraoperative perception results. The dual-channel output module can be used to generate spatial state information and semantic state information, and output them through the spatial perception output channel and the semantic perception output channel respectively.

[0247] The modules described above can be implemented through software, hardware, or a combination of both. Modules can be deployed on the same computing device or across multiple computing nodes, interacting with each other via data bus, network interface, or inter-process communication. The division between modules is for illustrative purposes only and does not limit the actual division of software packages, processes, threads, or hardware units in the product.

[0248] The following example illustrates the method provided in this application embodiment using a continuous operation example during laparoscopic surgery. This example can be executed by the aforementioned laparoscopic video intraoperative status awareness system and is only used to illustrate the method's operation process, not to limit specific surgical sites, equipment models, video parameters, or model structures.

[0249] Example 1: Intraoperative sensing during partial nephrectomy based on monocular endoscopy. This embodiment describes the complete implementation process of the method of this application in robot-assisted laparoscopic partial nephrectomy (RAPN).

[0250] Implementation environment: In this embodiment, the surgical procedure is a robot-assisted laparoscopic partial nephrectomy, and the surgical robot system is the Agibot surgical robot system. A 30-degree 3D high-definition laparoscope is used, and perception is achieved using its left-eye monocular video stream, with a video resolution of 1920×1080 and a frame rate of 30 frames per second. The computing platform is an edge computing server equipped with an NVIDIA RTX 4090 GPU with 24GB of video memory, an Intel Xeon W-2400 CPU, and 64GB of RAM. The operating system is Ubuntu 22.04 LTS, the deep learning framework is PyTorch 2.0, and the inference acceleration tool is TensorRT 8.6. Inter-module communication uses a publish-subscribe message bus based on ZeroMQ. The above hardware and software configuration is used to illustrate an implementable environment and does not constitute a necessary limitation.

[0251] Model preparation, i.e., the offline phase: During the pre-training phase, the SurgMotion encoder underwent self-supervised pre-training on 100,000 surgical video clips. These video clips came from public or private datasets such as Cholec80, M2CAI, and Private Kidney, and covered various surgical types including cholecystectomy, nephrectomy, and hysterectomy. Pre-training used eight A100 GPUs and lasted approximately 72 hours.

[0252] In the domain adaptation phase, fine-tuning was performed using 500 complete videos of partial nephrectomy surgeries, each video ranging from approximately 90 to 180 minutes in length. Training parameters included a learning rate of 1e-5, a batch size of 4, and 10 epochs of training.

[0253] In the multi-task fine-tuning phase, the following labeled data were used for end-to-end fine-tuning: uniform sampling and frame-by-frame labeling of 8 stages from 500 videos, totaling approximately 500,000 frames; bounding box labeling of instrument categories and actions, totaling approximately 5,000 frames, with annotations completed by professional physicians; pixel-level polygon labeling of anatomical structures, totaling approximately 2,000 frames; labeling of bleeding, smoke, blurring, and occlusion events using a multi-label classification method, totaling approximately 3,000 frames; and pseudo-depth maps generated from binocular videos as depth estimation supervision signals, totaling approximately 10,000 frames.

[0254] The total loss during multi-task training can be expressed as: Formula 5; In Formula 5 above, L_phase represents the phase identification loss, L_instrument represents the instrument status identification loss, L_structure represents the structural region identification loss, L_event represents the intraoperative event identification loss, and L_depth represents the spatial or depth estimation loss. The weights are determined by grid search optimization.

[0255] The performance metrics for each task on the independent validation set are as follows: Top-1 accuracy for stage recognition is 91.3%, mAP@0.5 accuracy for instrument detection is 87.6%, mDice accuracy for anatomy segmentation is 0.82, macro F1 accuracy for event detection is 0.85, and δ1 accuracy for depth estimation is 0.79. The above data scale, training parameters, loss weights, and performance metrics are used to illustrate a model preparation process and do not limit this application to using the same parameters or achieving the same performance.

[0256] Intraoperative real-time reasoning, i.e., the online phase: The following uses three key time points of a RAPN surgery as an example to illustrate the real-time operation of the system according to the video segment organization, shared feature vector extraction, multi-task intraoperative perception, process constraint fusion and dual-channel output process of this application.

[0257] Time point A is 5 minutes after the start of surgery, corresponding to stage P1, i.e., the establishment of the approach and field of vision.

[0258] Step 1, corresponding to steps S110 and S120, involves the video acquisition module receiving the endoscope video stream, and the segment organization module organizing the most recent 16 frames into video segments, numbered 100 to 115, and then scaling each video frame to 448×256 before normalization. This step provides video segments with continuous frame relationships and a unified input format for subsequent shared encoding.

[0259] Step 2, corresponding to step S130, involves the SurgMotion encoder performing inference on the video segment, taking approximately 12 milliseconds and outputting a 768-dimensional spatiotemporal feature vector. This spatiotemporal feature vector serves as a shared feature vector for the five task processing branches, enabling subsequent tasks to correspond to the spatiotemporal state of the same video segment.

[0260] Step 3, corresponding to step S140, involves parallel inference across five task processing branches based on a shared feature vector. The phase identification branch outputs P_phase=[P1:0.92, P2:0.05, P3:0.02, ...], current_phase=P1, and confidence=0.92; the instrument status identification branch detects the ultrasonic scalpel and outputs bbox=[320,180,380,240], track_id=1, action=separation, and conf=0.88; the structural region identification branch outputs psoas major muscle surface visible with conf=0.85 and retroperitoneal fat visible with conf=0.90; the intraoperative event identification branch detects no events, and the probability of each event is less than 0.1; the spatial state estimation branch outputs a relative depth map and the spatial relationship between the instrument and the retroperitoneal surface, approximately 2 mm to 5 mm. Thus, the same shared feature vector is converted into five categories of intraoperative perception results: phase, instrument, structure, event, and space.

[0261] Step 4 involves the process of consistency verification of process constraints, state fusion, and risk state generation. In stage P1, the expected set of instruments is {ultrasonic scalpel, electrocoagulation device}, the observed set of instruments is {ultrasonic scalpel}, and the instrument consistency score is calculated. The expected region set for stage P1 is {psoas major muscle surface, retroperitoneal fat}. The observed region matches this set, and the region consistency score is [not specified]. Overall consistency score This indicates high consistency. (Fusing confidence level) The calculation result was truncated to 1.0. Since no sensory degradation event was detected, the risk level was low. Through this step, the stage identification result was confirmed after joint verification of the device and area status.

[0262] Step 5 corresponds to step S150 and the dual-channel output process. The spatial awareness output channel T1 outputs {timestamp=300.0, visible_structures=[{psoas muscle surface, mask, 0.85}], instrument_positions=[{ultrasound scalpel, bbox, track_id=1}], spatial_cues={relative depth map, ...}, perception_quality={0.88}}. The semantic awareness output channel T2 outputs {current_phase={P1, approach and field of view establishment, 1.0}, instruments=[{ultrasound scalpel, separation, track_id=1, 0.88}], anatomical_status=[{psoas muscle surface, partial exposure, 0.85}], event_flags=[], risk_status={low}, overall_confidence=0.90}. T1 provides spatial information for registration and localization, while T2 provides semantic and risk information for status display or intelligent decision-making.

[0263] Time point B is 28 minutes after the start of surgery, corresponding to stage P4, i.e., the exposure of the kidney and tumor area.

[0264] Steps 1 and 2 are executed in the same manner as in time node A. The segment organization module generates the current video segment, the SurgMotion encoder outputs the shared feature vector, and the encoder inference takes about 12 milliseconds.

[0265] In step 3, the phase identification branch outputs P_phase=[P3:0.25, P4:0.68, P5:0.05, ...], current_phase=P4, and confidence=0.68; the instrument status identification branch detects the dissection forceps and outputs bbox=[400,200,460,280], track_id=2, action=free, and conf=0.85; the structural region identification branch outputs that the main renal artery is visible and conf=0.82, the renal vein is visible and conf=0.88, and the renal hilum region is exposed and conf=0.90; the intraoperative event identification branch outputs a small amount of smoke, severity=0.3, and the affected area occupies about 15% of the current image; the spatial state estimation branch outputs that the tip of the dissection forceps is about 3 mm away from the surface of the renal artery.

[0266] In step 4, the expected instrument set for stage P4 is {dissection forceps, ultrasonic scalpel}, the observed instrument set is {dissection forceps}, and the instrument consistency score is... The expected region set for stage P4 is {renal artery, renal vein, and renal hilum}. The observed region matches this set, with a region consistency score of Score_region ≈ 0.87. Based on the aforementioned weights, the overall consistency score is calculated... The results indicate high consistency, thus confirming stage P4. For a small amount of smoke, the perception degradation processing module applies a decay coefficient of 0.85 to the anatomical structure confidence level, adjusting the renal artery confidence level from 0.82 to approximately 0.70. Considering the proximity to a large blood vessel and the mild smoke, the risk level is determined to be moderate. This step first utilizes process constraints to confirm the stage, then corrects the affected structural perception results based on the severity of the smoke.

[0267] In step 5, the spatial perception output channel T1 outputs {spatial_cues={separation clamp distance to renal artery surface approximately 3mm}, perception_quality={0.78, degradation factor=[smoke, severity=0.3]}}; the semantic perception output channel T2 outputs {current_phase={P4, kidney and tumor area exposure, 0.68}, risk_status={medium, source=[vascular manipulation, mild smoke]}}. Thus, the downstream module can simultaneously obtain the intraoperative status and be aware of the perception quality changes caused by smoke.

[0268] Time point C is 45 minutes after the start of surgery, corresponding to stage P6, i.e. tumor resection, at which point active bleeding occurs during surgery.

[0269] Steps 1 and 2 are executed in the same manner as in time node A, generating shared spatiotemporal features for the current video segment.

[0270] In step 3, the phase identification branch outputs P_phase=[P5:0.10, P6:0.82, P7:0.06, ...] and current_phase=P6; the instrument status identification branch detects the electrocoagulation instrument and outputs track_id=3, action=electrocoagulation and conf=0.80, and also detects the aspirator and outputs track_id=4, action=aspiration and conf=0.90; the structural region identification branch outputs that the tumor region is visible and conf=0.75, and the bleeding point conf=0.92; the intraoperative event identification branch outputs the bleeding event severity=0.7 and the smoke event caused by electrocoagulation severity=0.6; the spatial state estimation branch outputs that the tip of the electrocoagulation instrument is located about 2 mm above the bleeding point.

[0271] In step 4, the perception degradation processing module applies attenuation to the confidence level of the anatomical structure based on the bleeding event. The attenuation amount for bleeding is... The attenuation coefficient was 0.72, and the confidence level for the tumor region was adjusted from 0.75 to [missing value]. The process constraint module then performs consistency checks on the revised stages, instruments, structures, and event results. Since the bleeding severity is 0.7, the smoke severity is 0.6, and the stage consistency is moderate, the overall risk level is determined to be high. This step converts the degree of perception degradation into a change in task confidence, and then combines this with the consistency check results to generate a risk status.

[0272] In step 5, the spatial perception output channel T1 outputs {perception_quality={0.45, degradation factors=[bleed, severity=0.7; smoke, severity=0.6]}}; the semantic perception output channel T2 outputs {event_flags=[{bleed, activation, severity=0.7}, {smoke, activation, severity=0.6}], risk_status={high, source=[active bleed, heavy smoke], suggested processing=[severe degradation of perception quality, reduce navigation cue confidence]}}. The suggested processing field expresses the category of the system-generated cue information and can be replaced with perception reliability cue, navigation devaluation cue, or pending confirmation cue.

[0273] Example 2: Enhanced 3D Spatial Perception Using Binocular Parallax. This example, building upon Example 1, uses a binocular endoscope to acquire left and right views to enhance 3D spatial perception. Other processing steps can be referenced from Example 1.

[0274] During video acquisition, the video acquisition module simultaneously receives two video streams: the left view and the right view from the binocular endoscope. Each video stream has a resolution of 1920×1080 and a frame rate of 30 frames per second.

[0275] During video encoding, the SurgMotion encoder extracts spatiotemporal features from the left and right views respectively. The left view features are used for five perceptual tasks: stage recognition, instrument status recognition, structural region recognition, intraoperative event recognition, and spatial state estimation. The right view features are used only for the spatial state estimation task and are used together with the left view features for stereo matching.

[0276] In the spatial estimation process, the spatial state estimation branch includes a disparity calculation branch. This branch performs cross-correlation calculations on the left and right view features at multiple disparity levels to generate a cost volume, aggregates the costs through 3D convolution, and outputs a disparity map through a soft-argmin operation. The computing device combines the baseline distance b and focal length f from the camera calibration parameters, according to... The disparity map is converted into an absolute-scale depth map. Using the depth map and camera intrinsics, the computing device backprojects the pixel coordinates into a 3D point cloud in the camera coordinate system.

[0277] During the registration enhancement process, the spatial_cues field in the spatial perception output channel T1 is upgraded from a relative depth description to a structured point cloud containing absolute-scale 3D coordinates. This structured point cloud can be directly input into the registration and localization layer for ICP registration, eliminating the need for the registration and localization layer to generate a point cloud based on the structure mask and depth map backprojection, thus reducing the data conversion steps before registration.

[0278] In one test example, the absolute error of depth estimation of the renal artery trunk in binocular mode decreased from approximately ±3.2 mm in monocular mode to approximately ±1.1 mm, and the registration accuracy (TRE) of the T1 channel output point cloud improved from approximately 4.5 mm in monocular mode to approximately 2.1 mm. These values ​​are used to illustrate a possible effect of binocular parallax enhancement in 3D spatial perception and point cloud registration, and do not limit this application to achieving these values.

[0279] In some embodiments, the complete operation may also include stage transition processing. When the surgical stage perception results of multiple consecutive video segments change, and the changed candidate stage satisfies the process constraints with the instrument state, structural region, or intraoperative event, the computing device can output a stage transition status. If the candidate stage changes significantly but the instrument consistency score or region consistency score is low, the computing device may temporarily not directly confirm the stage transition, but instead mark the stage result as a pending confirmation state, or continue to accumulate multiple types of intraoperative perception results in subsequent video segments. Thus, the stage transition judgment does not only rely on a single classification probability, but can also be verified by combining multiple state elements of the current video segment.

[0280] In some embodiments, the complete operation may also include quality status feedback. After generating perceived quality information, the computing device can simultaneously write the perceived quality information into spatial state information and semantic state information. For spatial state information, perceived quality information can represent the reliability of depth information, point cloud information, structural regions, or instrument positions; for semantic state information, perceived quality information can represent the reliability of stages, actions, events, or prompt states. Thus, when downstream modules call the spatial perception output channel or the semantic perception output channel, they can obtain quality references related to their own usage content, without having to re-analyze the original endoscope video.

[0281] In summary, this embodiment performs spatiotemporal feature extraction on the same video segment once through a shared video encoder, and reuses the shared feature vector by multiple task processing branches, reducing redundant computation and inference latency accumulation caused by independent deployment of multiple tasks. Cross-validation of results from various tasks, such as stages, instruments, structures, regions, and events, is performed through surgical procedure constraints, reducing logical inconsistencies between different task results. The spatial perception output channel and semantic perception output channel organize the same perception results according to downstream needs, reducing irrelevant data transmission and downstream module filtering. Perception degradation detection, task impact analysis, and confidence weighting processing enable downstream modules to understand the reliability of results in degradation scenarios. These processes form a continuous processing chain from shared perception, result verification, classification output to quality feedback, helping to provide timely, consistent, and quality-informed multi-task perception results for the intraoperative navigation system.

[0282] Figure 8 This is a schematic diagram of a computing device provided for some embodiments of this application.

[0283] like Figure 8 As shown, the computing device 800 includes a processor 801 and a memory 802. Exemplarily, the computing device 800 may also include a communications interface 803 and a communications bus 804.

[0284] The processor 801, memory 802, and communication interface 803 communicate with each other via communication bus 804. The communication interface 803 may include a transmitter and receiver for communicating with other devices or communication networks, and may be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet interface (GE).

[0285] In some embodiments, the processor 801 is used to execute program 805, specifically performing the relevant steps in the above method embodiments. Exemplarily, program 805 may include program code, which includes computer-executable instructions.

[0286] For example, processor 801 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of this application. Computing device 800 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs. The CPU may be a single-core CPU or a multi-core CPU.

[0287] In some embodiments, memory 802 is used to store program 805. Memory 802 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0288] Specifically, program 805 can be invoked by processor 801 and executed by computing device 800 using the above method.

[0289] It should be noted that, in this application, terms such as "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0290] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0291] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0292] In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. The above embodiments are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this application should be included within the scope of protection of this application.

Claims

1. A method for intraoperative status perception applied to laparoscopic surgical navigation, characterized in that, include: Real-time acquisition of intraoperative laparoscopic video stream; Organize consecutive video frames in the endoscopic video stream into video segments; Spatiotemporal joint feature extraction is performed on the video segment to obtain a shared feature vector that characterizes the spatiotemporal state of the intraoperative scene; The shared feature vector is input into the task processing branches corresponding to the multiple types of intraoperative perception tasks to obtain multiple types of intraoperative perception results; wherein, the multiple types of intraoperative perception tasks share the shared feature vector. Output the results of the various types of intraoperative perception.

2. The intraoperative status sensing method for laparoscopic surgical navigation according to claim 1, characterized in that, Spatiotemporal joint feature extraction is performed on the video segment, including: Input the video clip into the shared video encoder; The shared video encoder is used to jointly encode the spatial and temporal information in the video segment to obtain the joint encoding result. The shared feature vector is generated based on the joint encoding result; The shared feature vector is used to characterize at least one of instrument movement, tissue exposure changes, phase transitions, image quality changes, and intraoperative event changes.

3. The intraoperative status sensing method for laparoscopic surgical navigation according to claim 1, characterized in that, The method further includes: Obtain the constraints of the laparoscopic surgical procedure, which are used to characterize the prior matching relationship between multiple types of intraoperative state elements, including at least two of the following: surgical stage, instrument status, anatomical structure, surgical area, and intraoperative events. Based on the prior matching relationship, the surgical procedure context consistency is verified for the multiple types of intraoperative perception results. Based on the consistency verification results, the confidence level of at least one of the multiple types of intraoperative perception results is corrected; The fused intraoperative status information is generated based on the corrected intraoperative perception results of multiple types.

4. The intraoperative status sensing method for laparoscopic surgical navigation according to claim 3, characterized in that, Based on the prior matching relationship, the surgical procedure context consistency is verified for the multiple types of intraoperative perception results, including: Based on the perception results during the surgical phase and the perception results of the instrument status, the instrument consistency score is calculated; Based on the surgical phase perception results and structural region perception results, a region consistency score is calculated; The consistency scores of the instruments and the consistency scores of the regions are weighted and fused to obtain a comprehensive consistency score. Based on the overall consistency score, it is determined whether the surgical stage perception result satisfies the current process context; If the perception result of the surgical stage does not meet the current process context, the confidence level of the perception result of the surgical stage is reduced, or the perception result of the surgical stage is marked as pending confirmation, the observation window is extended, and a second verification is performed based on the perception result of subsequent video segments.

5. The intraoperative status sensing method for laparoscopic surgical navigation according to any one of claims 1-4, characterized in that, The method further includes: Based on the shared feature vector, sensory degradation events are detected through the intraoperative event recognition task branch. The sensory degradation events include at least one of smoke, bleeding, blurred vision, and instrument obstruction. Determine the event type, severity, and affected area of ​​the perceived degradation event; Based on the event type, the severity, and the affected area, determine the intraoperative perception task affected by the perception degradation event; The confidence weighting of the intraoperative perception results output by the affected intraoperative perception task is reduced.

6. The intraoperative status sensing method for laparoscopic surgical navigation according to claim 5, characterized in that, The confidence-weighted processing of the intraoperative perception results output from the affected intraoperative perception task is performed, including: The confidence decay coefficient is determined based on the event type and the severity. The confidence level of the affected intraoperative perception results is corrected based on the confidence level decay coefficient to obtain the corrected confidence level. Perceived quality information is generated based on the corrected confidence level; Based on the perceived quality information, perceived quality prompts are generated for the downstream navigation module, which are used to instruct the downstream navigation module to adjust the calling weight of the corresponding intraoperative perceived results.

7. The intraoperative status sensing method for laparoscopic surgical navigation according to any one of claims 1-4, characterized in that, Output the various types of intraoperative perception results, including: Spatial state information is generated based on the various intraoperative perception results. The spatial state information includes at least one of the following: visible structure information, instrument position information, instrument motion information, spatial cues, depth information, point cloud information, spatial state confidence, and perception quality information. Semantic state information is generated based on the multiple intraoperative perception results. The semantic state information includes at least one of the following: current surgical stage, instrument movement, anatomical state, event markers, prompting state, and overall confidence level. The spatial state information and the semantic state information are output respectively, wherein the spatial state information is for the endoscope navigation registration and positioning scenario, and the semantic state information is for the endoscope navigation intelligent decision-making scenario.

8. The intraoperative status sensing method for laparoscopic surgical navigation according to claim 7, characterized in that, Output the spatial state information and the semantic state information respectively, including: The spatial state information is output to the registration and positioning layer of the laparoscopic surgical navigation system through the spatial perception output channel; the spatial state information is used for spatial registration, positioning, or spatial relationship calculation. The semantic state information is output to the intelligent decision-making layer of the laparoscopic surgical navigation system through the semantic perception output channel; the semantic state information is used for intraoperative status display, process judgment or prompt information generation. The spatial state information and the semantic state information are derived from the same set of intraoperative perception results.

9. The intraoperative status sensing method for laparoscopic surgical navigation according to any one of claims 1-4, characterized in that, Real-time acquisition of intraoperative laparoscopic video streams, including: Real-time acquisition of intraoperative monocular laparoscopic video stream, and estimation of spatial state based on temporal changes, instrument movement and visible structure changes between consecutive video frames in the monocular laparoscopic video stream; Alternatively, the intraoperative binocular laparoscopic video stream can be acquired in real time, and depth information can be generated based on the parallax information between the left and right views in the binocular laparoscopic video stream, and then the spatial state can be estimated based on the depth information. Alternatively, the intraoperative binocular laparoscopic video stream and the endoscopic pose data of the surgical robot can be acquired in real time, and depth information can be generated based on the parallax information between the left and right views in the binocular laparoscopic video stream. The depth information is then fused with the endoscopic pose data to estimate the spatial state.

10. The intraoperative status sensing method for laparoscopic surgical navigation according to any one of claims 1-4, characterized in that, Organizing consecutive video frames in the endoscopic video stream into video segments includes: The continuous video frames are preprocessed to obtain preprocessed video frames; A fixed-length sliding window is used to determine a continuous frame sequence from the preprocessed video frames; wherein, overlapping frames are set between adjacent sliding windows, and the number of overlapping frames is greater than half the window length; The consecutive frame sequence is determined as the video segment.

11. The intraoperative status sensing method for laparoscopic surgical navigation according to any one of claims 1-4, characterized in that, The various intraoperative sensing tasks include at least two of the following: surgical phase identification task, instrument status identification task, anatomical structure or surgical area identification task, intraoperative event identification task, and spatial status estimation task. The multiple types of intraoperative perception results include at least two of the following: surgical stage perception results corresponding to the surgical stage identification task, instrument state perception results corresponding to the instrument state identification task, structural region perception results corresponding to the anatomical structure or surgical region identification task, intraoperative event perception results corresponding to the intraoperative event identification task, and spatial state perception results corresponding to the spatial state estimation task.

12. A laparoscopic video intraoperative status perception system, characterized in that, include: The video acquisition module is used to acquire intraoperative laparoscopic video streams in real time; The segment organization module is used to organize continuous video frames in the endoscopic video stream into video segments; A shared feature extraction module is used to perform spatiotemporal joint feature extraction on the video segment to obtain a shared feature vector that characterizes the spatiotemporal state of the intraoperative scene. The multi-task perception module is used to input the shared feature vector into the task processing branches corresponding to multiple types of intraoperative perception tasks to obtain multiple types of intraoperative perception results. The output module is used to output the various intraoperative perception results; The shared feature vector is used in all of the various intraoperative perception tasks.

13. A computing device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the intraoperative status awareness method for laparoscopic surgical navigation as described in any one of claims 1 to 11.

Images