A surgery video multi-granularity intelligent inference method and system based on knowledge enhancement and a large model

Abstract

The application discloses a kind of surgical video multi-granularity intelligent inference method and system based on knowledge enhancement and large model belongs to surgical video intelligent analysis technical field.For the problem that existing surgical video analysis is difficult to consider multi-granularity recognition, time sequence coherence and medical semantic consistency, surgical video data is obtained and multi-granularity recognition is carried out, and initial multi-granularity recognition result is obtained;Surgical task graph for representing hierarchical relationship and time sequence constraint in surgical procedure is constructed, and time sequence correction is carried out based on surgical task graph, and time sequence correction result is obtained;Surgical knowledge base is constructed, relevant knowledge content is obtained based on search enhancement generation mechanism, initial multi-granularity recognition result, time sequence correction result and knowledge content are input into large model inference correction module for semantic consistency verification, and semantic correction result is generated;Time sequence correction result and semantic correction result are fused and output, improve the time sequence coherence, cross-task consistency and explainability of surgical video analysis result.

Description

Technical Field

[0001] This invention relates to the field of video data recognition technology, and in particular to a multi-granularity intelligent reasoning method and system for surgical videos based on knowledge enhancement and large models. Background Technology

[0002] With the widespread adoption of minimally invasive surgery and robot-assisted surgical systems, surgical videos have become a fundamental data modality for surgical quality control, standardized surgical procedure evaluation, and clinical teaching. Compared to natural scene videos, surgical videos carry a more rigorous structured semantic hierarchy: from macroscopic surgical stages and steps, to mesoscopic surgical instruments, surgical actions, and anatomical tissues, and then to microscopic instrument-tissue spatial relationships and temporal boundaries of actions, forming a multi-level semantic pyramid. Achieving a complete analysis of this semantic pyramid and logically consistent reasoning is the core technical goal of the field of intelligent surgical video analysis.

[0003] Currently, intelligent analysis technology for surgical videos mainly follows two technical routes, but neither has systematically achieved the aforementioned goals.

[0004] The first technical approach is a multimodal question-answering and reasoning method based on surgical videos. This type of method uses video clips as input units, employing multiple feature extractors such as instrument detection networks, anatomical segmentation networks, and temporal recognition networks to acquire multidimensional visual representations. Based on these representations, it automatically generates "video clip-question-answer pairs" training data to supervise and fine-tune a large multimodal model, ultimately achieving answer generation for specific questions. The core technology of this approach is establishing a mapping from a joint "visual-text" space to a discrete answer space.

[0005] However, this approach has the following technical limitations: First, the output representation lacks the ability to structure and organize multi-granular semantics. The output of this method is defined as a set of discrete answer texts associated with a specific question, rather than a unified, structured surgical semantic graph. This means that the system cannot synchronously output a complete set of multi-granular information, such as stages, steps, instruments, actions, tissues, spatial localization, and triplet relationships, corresponding to a video segment within the same inference cycle. The recognition results between different tasks are decoupled, lacking a shared semantic representation layer to uniformly encode and express the holographic surgical scene of "when-what stage-what step-what instrument-what action-what tissue-what spatial relationship".

[0006] Second, the reasoning mechanism lacks the ability to directly verify and correct the basic visual recognition results. Although this type of method introduces a surgical knowledge base to perform visual-text token similarity matching during the feature selection stage to filter compact features related to the surgery, the knowledge base in this architecture only serves as an attention bias for feature selection, rather than an independent logical verification engine. The system's reliance on multimodal large-scale models to generate answers means that when faced with erroneous visual features generated by the visual encoder due to factors such as noise, occlusion, and changes in lighting, it cannot perform truth verification and correction of the basic recognition results (such as instrument category and action type) at the knowledge level. In other words, the correctness of the question-answering answer is based on the implicit assumption of the reliability of visual features, which is not always valid in complex clinical scenarios.

[0007] The second technical approach is a surgical stage or step recognition method that incorporates temporal constraints. This type of method takes a video frame sequence as input, uses a spatial encoder and a temporal encoder to extract appearance features and temporal dynamic features respectively, generates a spatiotemporal joint representation through an adaptive fusion module, and outputs a frame-by-frame stage or step probability distribution via a classification head. Some methods further introduce post-processing modules such as diffusion models or temporal state machines to learn the stage / step transition probability distribution of a standard surgical procedure, smoothing and correcting abnormal jumps that violate transition constraints in the initial classification sequence. The technical essence of this approach is to establish a classification mapping from low-level visual representation to high-level procedure labels in the video spatiotemporal feature space, and apply first-order Markov or higher-order sequence constraints to the output space.

[0008] Although this approach improves the continuity of the stage / step identification sequence by introducing temporal constraints, it has the following technical limitations: First, the task granularity is singular, and the semantic coverage is sparse. The output space of this technical approach is strictly limited to a set of labels for surgical stages or steps. While this coarse-grained process labeling can describe the macroscopic progress of the surgery, it completely loses the mesoscopic and microscopic operational semantics, such as instruments, actions, anatomical tissues, and their spatial relationships. This means that even if the model correctly identifies the current frame as belonging to the "gallbladder triangle dissection" step, it cannot further answer a series of detailed operational questions that are of substantial significance to the assessment of surgical quality, such as "what instruments are used," "what actions are performed," "which tissue is acted upon," and "what is the spatial relationship between instruments and tissues."

[0009] Second, the temporal correction mechanism lacks multi-granularity constraints and cross-task consistency. The temporal constraints imposed by diffusion models or temporal adapters in existing methods only apply to a single semantic level—the sequence of stages or steps. The nodes and edges defined in their graph structure express the predecessor-successor and prohibited jump relationships between stages / steps, without incorporating finer-grained temporal patterns such as surgical actions and instrument operations into the same constraint framework. This leads to two problems: first, even if the step sequence is macroscopically corrected to a legal transition, its internal action sequences or instrument usage sequences may still exhibit jitter and misalignment that does not conform to the operational norms of that step; second, this temporal correction is independent of other tasks such as instrument recognition and action recognition, completely disregarding cross-task semantic constraints such as "step and instrument," "step and action," and "instrument, action, and tissue," thus lacking the ability to detect and correct logical conflicts between multi-task recognition results.

[0010] Examining these two technical approaches from a unified perspective reveals a common structural flaw: the three core functional modules—visual recognition, temporal modeling, and knowledge reasoning—lack a closed-loop feedback and collaborative correction mechanism. The visual recognition module unilaterally transmits noisy recognition results to either the temporal or reasoning module, while the latter lacks the ability to perform truth verification and reverse correction based on medical knowledge and task graph constraints. This directly leads to the following technical problems that cannot be effectively solved within the existing framework: First, there's the issue of robustness in complex surgical scenarios. Under complex imaging and operational conditions such as smoke, bleeding, instrument obstruction, tissue deformation, lighting variations, and differences in surgeons' operating habits, visual models inevitably produce errors such as misidentification of instruments, misclassification of actions, stage jumps, and spatial positioning deviations. Current architectures lack a mechanism to automatically detect and label these anomalies that contradict medical common sense, based on the medical logical relationships of "instrument-action-tissue," the procedural constraints of "stage-step-operation," and the spatial relationships of anatomical structures.

[0011] Second, there is the issue of cross-task semantic conflicts in multi-granularity recognition results. When stage recognition, step recognition, instrument recognition, action recognition, and tissue recognition are run as independent tasks, logical contradictions may arise between their outputs. For example, the step recognizer outputs "suture and hemostasis," the action recognizer outputs "electrocoagulation," and the instrument recognizer outputs "needle holder," constituting a semantic conflict that does not conform to the norms of surgical procedures. The existing architecture lacks a consistent reasoning layer across tasks and cannot arbitrate and jointly optimize conflicting results based on rules in the surgical knowledge graph (such as "the standard instrument for electrocoagulation is an electrocoagulation hook or bipolar electrocoagulation forceps, not a needle holder").

[0012] Third, there are issues with the interpretability of the output and the lack of medical evidence. Existing methods, whether outputting a stage label or a question-and-answer format, do not provide a transparent reasoning process to the user. Users cannot know whether the conclusion is based on a visual pattern learned by the model from a single frame, adherence to a standard surgical procedure, or logical deduction after ruling out other impossible scenarios. In clinical settings, reasoning results lacking a supporting chain of medical evidence have fundamentally limited credibility and acceptability.

[0013] In summary, existing technologies fail to integrate multi-granularity visual recognition, multi-level temporal constraints guided by task graphs, and retrieval-enhanced knowledge reasoning into a unified closed-loop system in their architectural design. This results in systemic deficiencies in key indicators such as multi-granularity semantic coverage, robustness, cross-task consistency, and interpretability. This is precisely the core technical problem that this invention aims to solve. Summary of the Invention

[0014] To address the shortcomings of existing technologies, this invention provides a method and system for multi-granular intelligent reasoning of surgical videos based on knowledge enhancement and large models.

[0015] To achieve the above objectives, the technical solution of the present invention is as follows: A multi-granularity intelligent reasoning method for surgical videos based on knowledge enhancement and large models includes the following steps: Acquire surgical video data and perform multi-granularity recognition on the surgical video data to obtain initial multi-granularity recognition results; A surgical task graph corresponding to the surgical video data is constructed. The surgical task graph is used to represent the hierarchical relationships and temporal constraints in the surgical process. The initial multi-granularity recognition results are temporally corrected based on the surgical task graph to obtain the temporal correction results. Construct a surgical knowledge base, and obtain knowledge content related to the initial multi-granularity recognition result and / or the temporal correction result from the surgical knowledge base based on a retrieval enhancement generation mechanism; The initial multi-granularity recognition result, the temporal correction result, and the knowledge content are input into the large model inference correction module, which performs semantic consistency verification and generates semantic correction result. The temporal correction result is fused with the semantic correction result to obtain the final correction result.

[0016] The specific implementations of video data preprocessing, video segmentation and frame cutting, construction of multi-granularity surgical video datasets, basic visual recognition model structure, construction of surgical task graph G=(V,E,A), forward noise addition and reverse noise reduction process of diffusion-type temporal correction, construction of surgical knowledge base and knowledge vectorization, construction of query generation for retrieval enhancement and vector similarity retrieval, multi-class consistency verification performed by the large model inference correction module, and fusion and output of dual correction results can be referred to the following specific implementation methods.

[0017] Furthermore, prior to the multi-granularity recognition, the procedure includes dividing the surgical video data into multiple video segments. The video segments are divided using a fixed time window method or a dynamic window method based on changes in surgical stage, steps, or actions. The dynamic window determines the start and end times of the segments based on the probability of stage changes, the probability of actions, or the boundary detection results output by the visual model.

[0018] Furthermore, the multi-granularity recognition is based on a pre-constructed multi-granularity surgical video dataset; the multi-granularity surgical video dataset is constructed by performing multi-granularity annotation on surgical video data, and the multi-granularity annotation includes at least surgical stage annotation, surgical step annotation, surgical instrument annotation, and surgical action annotation.

[0019] Furthermore, the multi-granularity recognition is achieved through a basic visual recognition model, which includes a spatial feature extraction network, a temporal feature extraction network, an adaptive feature fusion module, and a classification prediction module.

[0020] Furthermore, the surgical task graph is denoted as G=(V,E,A), where V represents the set of nodes, E represents the set of edges, and A represents the set of attributes of nodes or edges; nodes represent surgical stages, surgical steps, or surgical actions, edges represent allowed transfer relationships, prohibited jump relationships, or hierarchical relationships, and attributes include duration range, common instruments, common actions, or common tissue information.

[0021] Furthermore, the temporal correction is a diffusion-type temporal correction, which includes a forward noise addition process and a reverse noise removal process. The forward noise addition process is used to add random perturbations to the real labeled sequence during the training phase to simulate noise in the identification sequence. The reverse noise removal process is used to learn the mapping from the noisy sequence to the real sequence and to progressively predict and remove sequence noise under the constraints of the surgical task map.

[0022] Furthermore, the step of obtaining relevant knowledge content from the surgical knowledge base based on the retrieval enhancement generation mechanism includes: converting the initial multi-granularity recognition result and the temporal correction result into a retrieval query; calculating the similarity between the retrieval query and the vector of each knowledge entry in the surgical knowledge base; and selecting several knowledge contents with the highest similarity. The semantic consistency verification includes cross-task consistency verification and temporal consistency verification.

[0023] Furthermore, the fusion adopts at least one of sequential complementarity, reverse verification complementarity, or confidence-weighted fusion.

[0024] Furthermore, this invention provides a surgical video multi-granularity intelligent reasoning system based on knowledge enhancement and a large model, comprising: A multi-granularity recognition module is used to acquire surgical video data and perform multi-granularity recognition on the surgical video data to obtain an initial multi-granularity recognition result. The surgical task graph construction module is used to construct a surgical task graph, which is used to represent the hierarchical relationships and temporal constraints in the surgical procedure. The timing correction module is used to perform timing correction on the initial multi-granularity recognition result based on the surgical task map to obtain the timing correction result; The surgical knowledge base module is used to build a surgical knowledge base; The retrieval enhancement large model inference correction module is used to obtain knowledge content related to the initial multi-granularity recognition result and the temporal correction result from the surgical knowledge base based on the retrieval enhancement generation mechanism, and input the initial multi-granularity recognition result, the temporal correction result and the knowledge content into the large model inference correction module for semantic consistency verification, and generate semantic correction result; The fusion output module is used to fuse the temporal correction result with the semantic correction result to obtain the final correction result.

[0025] The beneficial effects of this invention are as follows: By constructing a multi-granularity surgical video dataset, this invention enables the same video segment to simultaneously associate information such as surgical stage, surgical steps, surgical instruments, surgical actions, anatomical tissues or organs, instrument-action-tissue triples, spatial localization, temporal boundaries, scene description, prediction of the next action, and multi-round reasoning question answering. Therefore, compared with existing methods that only target single-stage recognition, single-instrument recognition, or single-video question answering tasks, this invention can form a more complete semantic representation of surgical videos, reduce the problem of data fragmentation between different tasks, and improve the consistency of model training, testing, and comprehensive evaluation.

[0026] This invention further introduces a surgical task graph and a task graph-guided diffusion-based temporal correction mechanism, using permissible transition relationships, prohibited jump relationships, duration ranges, and hierarchical relationships between surgical stages, steps, and actions as temporal constraints. Therefore, when the basic visual recognition model is affected by smoke, bleeding, occlusion, blurring, lighting changes, or surgeon differences, resulting in short-term stage jumps, step misalignments, or action boundary jitter, the system can use task graph constraints and a diffusion-based denoising process to correct the noisy recognition sequence, thereby improving the temporal coherence and process rationality of long-term surgical video analysis results.

[0027] Meanwhile, this invention constructs a surgical knowledge base and combines retrieval-enhanced generation and a large-model inference correction mechanism. This enables the system to further retrieve relevant surgical procedure knowledge, instrument function knowledge, action definition knowledge, anatomical structure knowledge, triplet relationship knowledge, and spatial relationship knowledge after obtaining initial visual recognition results and temporal correction results for stages, steps, instruments, and actions. This information serves as the basis for semantic judgment and result correction by the large model. Therefore, this invention can not only output recognition results but also determine whether there is consistency between stages and steps, instruments and actions, actions and tissues, instrument positions and tissue positions, and temporal boundaries and scene descriptions. This reduces cross-task logical conflicts and improves the medical rationality, spatial consistency, semantic consistency, and interpretability of the output results.

[0028] Furthermore, this invention employs a dual complementary strategy of task graph-guided diffusion-based temporal correction and retrieval-enhanced large-model inference correction. The former primarily addresses sequence-level issues such as process jumps and boundary instability, while the latter mainly resolves knowledge-semantic level problems like instrument-action-organ mismatch, unreasonable spatial relationships, and inconsistent descriptions. Therefore, compared to solutions that solely rely on visual models, rule-based post-processing, or large-model generation, this invention can verify and correct multi-granularity recognition results of surgical videos from both temporal logic and knowledge-semantic perspectives, improving the accuracy, robustness, and traceability of intelligent analysis of complex surgical videos. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0030] Figure 1 This is a flowchart illustrating the overall process of the method provided in this embodiment of the invention. Figure 2 A flowchart illustrating the construction process of the multi-granularity surgical video dataset provided in this embodiment of the invention; Figure 3 A schematic diagram of the system provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings, are intended to cover non-exclusive inclusion.

[0033] For clarification and / or ease of reference, the following terms are defined herein: Multi-granularity recognition results specifically refer to one or more of the following: surgical stage, surgical step, surgical instrument, surgical action, anatomical tissue or organ, instrument-action-tissue triplet, instrument spatial positioning, tissue and organ spatial positioning, temporal action boundary, scene description, next action prediction, and multi-round reasoning results.

[0034] The instrument-action-tissue triple, also used in this article, refers specifically to a structured operative event consisting of surgical instruments, surgical actions, and tissues or organs. "Tissues" includes anatomical tissues and organs.

[0035] A surgical task graph, specifically, refers to a graph structure used to represent the relationships between surgical stages, steps, and actions. This graph includes nodes, edges, and attributes. Nodes represent stages, steps, or actions; edges indicate whether transitions are allowed or prohibited; and attributes include information such as duration, hierarchical relationships, common instruments, common actions, and common tissues.

[0036] The hierarchical relationship refers to the hierarchical relationship between surgical stages, surgical steps, and surgical actions; the temporal constraint refers to the constraint on the time sequence of each stage, step, and action in the surgical procedure, including allowed transfer relationships, prohibited jump relationships, and duration range constraints.

[0037] Task graph-guided diffusion-based temporal correction specifically refers to a mechanism that uses surgical task graph constraints and diffusion-based denoising models to correct the stage sequence, step sequence, or action sequence output by a basic visual recognition model.

[0038] Retrieval-enhanced generation specifically refers to a mechanism that first retrieves relevant knowledge from the surgical knowledge base based on the current recognition results, temporal correction results, contextual results, or reasoning tasks, and then provides the retrieved knowledge to a large model for reasoning, judgment, correction, or interpretation.

[0039] Large-scale model inference correction specifically refers to a mechanism in which a large model makes reasonable judgments and corrections on results such as stage-step, instrument-action, action-organization, triples, spatial relationships, temporal boundaries, and scene descriptions based on initial identification results, temporal correction results, retrieval knowledge, and contextual results.

[0040] The semantic consistency verification refers to a mechanism in which a large model, based on surgical knowledge base and retrieval knowledge, judges and corrects the rationality of multi-granularity recognition results such as stages, steps, instruments, actions, tissues, triples, spatial relationships, and temporal boundaries. This includes stage-step consistency verification, step-instrument consistency verification, instrument-action consistency verification, action-tissue consistency verification, triple combination rationality verification, spatial consistency verification, temporal consistency verification, and temporal boundary verification. The cross-task consistency verification is used to determine whether there are logical conflicts between the results of different recognition tasks, and the temporal consistency verification is used to determine whether the stages, steps, or actions are consistent with the order of the preceding and following processes.

[0041] Dual correction fusion specifically refers to the mechanism of fusing the task graph-guided diffusion-based temporal correction results with the retrieval-enhanced large model inference correction results to obtain the final recognition and inference results.

[0042] This invention addresses the intelligent analysis of surgical videos, proposing a multi-granularity intelligent reasoning method and system for surgical videos based on knowledge enhancement and a large model. This method combines multi-granularity surgical video dataset construction, a basic visual recognition model, a surgical task graph, a task graph-guided diffusion-based temporal correction model, a surgical knowledge base, a retrieval enhancement generation mechanism, and a large-model reasoning correction mechanism. This invention can uniformly identify, organize, temporally correct, verify knowledge, and semantically correct surgical stages, steps, instruments, actions, anatomical tissues, instrument-action-tissue triples, instrument spatial positioning, tissue and organ spatial positioning, temporal action boundaries, surgical scene descriptions, next action prediction, and multi-round reasoning content within surgical videos.

[0043] This invention is not simply a video question-and-answer method, nor is it a single surgical stage identification method. Instead, it combines video visual recognition results, surgical task graphs, surgical knowledge bases, and large-scale model inference results in a closed loop. This allows the system to first obtain initial recognition results such as stages, steps, instruments, and actions from the video. Then, guided by the task graph, a diffusion-style temporal correction module determines whether the sequence of stages, steps, and actions conforms to the standard surgical procedure. Subsequently, it combines multi-task labeled data, question-and-answer data, knowledge base candidate information, or the large-scale model inference process to obtain other multi-granular auxiliary information. Finally, based on the surgical knowledge base and the large-scale model inference mechanism, it determines whether the results conform to the instrument functional logic, instrument-action-organ relationship logic, anatomical spatial relationship logic, and cross-task semantic consistency requirements. It also corrects unreasonable recognition results and finally outputs structured intelligent analysis results of surgical videos.

[0044] Specifically, a first aspect of this invention provides a multi-granularity intelligent reasoning method for surgical videos based on knowledge enhancement and a large model. For example... Figure 1 As shown, the method includes the following main steps: S1, acquire surgical video data and perform preprocessing; S2: Divide the preprocessed surgical video data from step S1 into multiple video segments, and slice each video segment into frames to obtain a frame sequence; S3: Construct a multi-granularity surgical video dataset to form multi-granularity information corresponding to each video segment formed in step S2; S4: Obtain the initial multi-granularity recognition result of the video segment, wherein the initial multi-granularity recognition result includes at least one or more of the following: stage recognition sequence, step recognition sequence, device recognition result, and action recognition sequence; S5: Construct a surgical task diagram, which is used to represent the standard surgical procedure and its constraints; S6: Input the stage recognition sequence, step recognition sequence and action recognition sequence in the initial recognition result into the task graph-guided timing correction module. The task graph-guided timing correction module uses the surgical task graph constructed in step S5 as a constraint condition to correct the stage jumps, step misalignments and action boundary jitters in the initial recognition result sequence that do not conform to the surgical procedure logic, and obtain the timing correction results corresponding to each sequence. S7: Construct a surgical knowledge base and vectorize the knowledge entries in the surgical knowledge base to form a searchable knowledge index; the surgical knowledge base shall include at least one or more of the following: surgical procedure knowledge, surgical step knowledge, instrument function knowledge, action definition knowledge, anatomical structure knowledge, instrument-action-tissue relationship knowledge, anatomical spatial relationship knowledge, and instrument spatial positioning knowledge. S8: Convert the initial identification result, the temporal correction result, and the multi-granularity information into structured reasoning input; S9: Based on the retrieval enhancement generation mechanism, relevant knowledge content is retrieved from the surgical knowledge base as the basis for reasoning according to the current video segment, the current recognition result and the current reasoning task; S10: Input the video segment information, the initial recognition result, the temporal correction result, the knowledge content retrieved in step S9, the auxiliary reasoning information, and the preset reasoning instructions into the large model reasoning correction module; S11: The large model inference correction module performs the verification and correction operations: S12: The task graph-guided diffusion-based temporal correction result is fused with the semantic correction result generated by the large model inference correction module to obtain the final correction result; S13: Output at least one of the following: multi-granularity recognition results, temporal correction results, semantic correction results, reasoning explanation, risk warning, next action prediction, or structured analysis report.

[0045] The method and system of this invention are applied to the analysis of various surgical videos, performing multi-granularity recognition, temporal correction, knowledge retrieval, semantic correction, and structured output.

[0046] In some embodiments of the present invention, the original surgical videos acquired in steps S1 and S2 include, but are not limited to, robot-assisted radical prostatectomy, laparoscopic surgery, endoscopic surgery video analysis, and other surgical videos with clear procedures, instrument operations, anatomical structures, and action sequences, such as robot-assisted surgery, laparoscopic surgery, thoracoscopic surgery, gynecological laparoscopic surgery, urological surgery, gastrointestinal surgery, hepatobiliary surgery, etc.

[0047] To facilitate subsequent model processing, step S1 preprocesses the original video. Specifically, preprocessing may include one or more of the following operations: (1) Unify video format, convert videos from different sources into a unified encoding format; (2) Frame rate standardization, for example, sampling at 1 frame / second, 30 frames / second or other preset frame rates; (3) Resolution standardization: scaling the video frames to the size required for the model input; (4) Remove invalid segments, such as black screens, transitions, non-surgical scenes, and severely out-of-focus scenes; (5) Image quality assessment, evaluating the clarity, brightness, occlusion degree and image stability of video frames; (6) Timestamp alignment: Establish a time index for each video frame or video segment that corresponds to the original video timeline.

[0048] After preprocessing, in step S2, the surgical video is divided into multiple video segments. Specifically, the video segments can be divided using a fixed time window method or a dynamic window method based on changes in surgical stage, steps, or actions. The fixed time window can be set to 5 seconds, 10 seconds, 30 seconds, 60 seconds, or other durations; the dynamic window can determine the start and end times of the segment based on the probability of stage changes, the probability of action changes, or the boundary detection results output by the visual model. The dynamic boundary can be automatically determined based on changes in surgical stage, surgical steps, instrument movement, action boundary detection results, camera angle changes, or image quality changes.

[0049] Besides segmenting by time window, segmentation can also be performed using keyframe clustering, scene change detection, action start and end point detection, model confidence abrupt change detection, or manually labeled boundaries. The video sampling frame rate can also be set to 1 frame / second, 5 frames / second, 10 frames / second, or other frame rates depending on the task requirements. All of the above different segmentation and sampling methods can be used to obtain video segments suitable for subsequent multi-granularity recognition and intelligent inference.

[0050] For each video segment, the system saves the following basic information: (1) Video ID; (2) Segment number; (3) Segment start time; (4) End time of the segment; (5) Fragment frame sequence; (6) Video source information; (7) Image quality information; (8) Corresponding annotation information or Q&A information.

[0051] In some embodiments of the present invention, in step S3, the constructed multi-granularity surgical video dataset does not contain only single-stage labels or single question-answer pairs, but rather establishes labels, questions, answers, and inference chains of multiple granularities for the same video segment, specifically including multi-task labeled data and multi-granularity question-answer data. Specifically, each video segment corresponds to one or more data elements. A data element includes at least video segment information and corresponding multi-granularity information. Preferably, a data element may include the following fields: (1) Video segment identifier; (2) Surgical stage label; (3) Surgical procedure labels; (4) Surgical instrument labels; (5) Surgical action labels; (6) Labels for anatomical tissues or organs; (7) Instrument-Motion-Tissue triplet label; (8) Spatial positioning information of the instrument; (9) Spatial location information of tissues and organs; (10) The time interval in which the action occurs; (11) Action switching time point; (12) Surgical scene description text; (13) Next action prediction label or text; (14) Multi-round reasoning question and answer; (15) Image quality or video quality label; (16) Manual confirmation mark or expert review mark.

[0052] The spatial positioning information of the instrument can be represented using bounding box coordinates, key point coordinates, segmentation masks, or region descriptions. Bounding box coordinates can be represented in normalized coordinate form as follows: [x_min, y_min, x_max, y_max] Where x_min represents the x-coordinate of the top left corner of the bounding box, y_min represents the y-coordinate of the top left corner of the bounding box, x_max represents the x-coordinate of the bottom right corner of the bounding box, and y_max represents the y-coordinate of the bottom right corner of the bounding box. All coordinate values ​​are normalized to between 0 and 1.

[0053] Spatial location information of tissues and organs can be represented in the same or similar way as that of instrument spatial location, or it can be represented in the form of segmentation masks. For tissue or organ regions with irregular boundaries, segmentation masks are preferred; for scenarios with high annotation efficiency requirements, bounding box coordinates or region description text can also be used for representation.

[0054] The instrument-action-tissue triple is used to represent the core operative events in a surgical video, and its form is as follows: Surgical instruments, surgical procedures, and the tissue or organ targeted. For example, <bipolar clamp, traction, prostate>, <monopolar curved scissors, separation, bladder neck>, <needle holder, suturing, dorsal deep vein complex>. Through ternaries, the system can uniformly represent "what instruments are in the image", "what actions the instruments are performing", and "what tissue or organ the actions are acting on" as structured events.

[0055] Multi-granularity question-answering data can include the following types: (1) Surgical stage identification problem, used to determine which surgical stage a video segment belongs to; (2) Surgical step identification problem, used to determine which specific step is being performed in a video segment; (3) Surgical instrument identification problem, used to identify currently active or visible instruments; (4) Surgical action recognition problem, used to identify the actions currently being performed by the instrument; (5) Anatomical tissue identification problem, used to identify the tissues or organs involved in the current operation; (6) Triplet identification problem, used to identify the combination relationship between instruments, actions and tissues; (7) Instrument spatial positioning problem, used to output the position of a specified instrument in the picture; (8) Spatial positioning of tissues and organs, used to output the position of a specified tissue or organ in the picture; (9) Timing action boundary location problem, used to determine the start and end time of an action or the time point of action transition; (10) Surgical video scene description problem, used to generate a complete description of the current video segment; (11) Next action prediction problem, used to predict the subsequent operation that may occur after the current segment ends; (12) Multi-round reasoning questions are used to deduce the stage, steps, equipment, actions, organization and operational significance of the current video segment through continuous question and answer.

[0056] To improve dataset consistency, multiple question templates can be set for each question type. The same video clip can correspond to multiple question templates and multiple answer formats. For example, a video clip can simultaneously correspond to stage selection questions, step selection questions, equipment multiple selection questions, action multiple selection questions, triplet multiple selection questions, spatial positioning open-ended questions, temporal boundary selection questions, and comprehensive description questions.

[0057] In some embodiments of the present invention, the dataset construction process is as follows: Figure 2 As shown, the specific process includes the following: S3.1. Input the segmented video clip; S3.2. Expert annotation or semi-automatic annotation; S3.3. Generation stages, steps, equipment, and actions; S3.4. Label the instrument-motion-tissue ternary set; S3.5. Mark the spatial location of tissues / organs; S3.6. Mark the start and end times of the action and the action switching point; S3.7. Generate question templates and standard answers; S3.8. Expert review and consistency check; S3.9. Create a multi-granularity surgical video dataset; In step S3.2, annotation can be performed by experts (i.e., direct annotation by human experts) or semi-automatic annotation. Semi-automatic annotation involves the basic visual model first generating candidate labels for stages, steps, instruments, and actions, and then experts reviewing, supplementing, or correcting these candidate labels, as well as other multi-task annotation content such as organization, triples, spatial localization, temporal boundaries, scene descriptions, and next-step predictions. Semi-automatic annotation reduces the cost of manual annotation and improves annotation efficiency.

[0058] The annotation format in multi-granularity datasets can also be replaced. For example, spatial localization of instruments and spatial localization of tissues and organs can be represented by bounding boxes, segmentation masks, key points, trajectory lines, heatmaps, region description text, or any combination thereof. Triple annotations can be represented as "instrument-action-tissue", or can be extended to higher-order structures such as "instrument-action-tissue-time interval" or "instrument-action-tissue-spatial location".

[0059] In summary, any dataset that can establish correspondences between multiple granularities of information, such as stages, steps, instruments, actions, tissues, spaces, time sequences, descriptions, predictions, or inferences, around the same surgical video segment can be used as an alternative to the multi-granularity dataset construction method of this invention.

[0060] In step S4, the video clip is input into the basic visual recognition model to obtain initial recognition results such as stages, steps, instruments, and actions. In some embodiments of the present invention, the basic visual recognition model may include one or more sub-models, which may work in parallel or serially.

[0061] In some embodiments of the present invention, the basic visual recognition model is mainly used to perform four initial classifications on video clips, and preferably includes the following sub-modules: (1) Surgical stage identification module; (2) Surgical step recognition module; (3) Surgical instrument identification module; (4) Surgical action recognition module.

[0062] Basic visual recognition models are not limited to a specific network structure. Specifically, they can employ convolutional neural networks, 3D convolutional neural networks, visual Transformers, temporal Transformers, multi-scale visual Transformers, graph neural networks, temporal convolutional networks, recurrent neural networks, long short-term memory networks, large video models, multimodal visual language models, or combinations of the above models.

[0063] In this embodiment, the basic visual recognition model is used to output initial recognition results for surgical stages, surgical steps, surgical instruments, and surgical actions corresponding to surgical video segments. For surgical stage and surgical step recognition, TAPIS, ProstaFormer, SlowFast, TimeSformer, Video Swin Transformer, MViT, or other video understanding models can be used. For surgical instrument recognition, video classification models, object recognition models, or multi-label classification models can be used. For surgical action recognition, temporal action recognition models, temporal segmentation models, sequence labeling models, or boundary detection models can be used.

[0064] In other alternative implementations, the basic visual recognition model can be further extended to a multi-task visual model capable of outputting results such as anatomical tissues or organs, spatial localization of instruments, spatial localization of tissues and organs, temporal action boundaries, scene descriptions, or predictions of the next action. However, in this embodiment, the above-mentioned extended results are not necessarily the direct output of the basic visual recognition model, and can also be provided by multi-task labeled data, question-answering data, surgical knowledge base, retrieval enhancement generation module, large model inference correction module, or auxiliary model.

[0065] As long as the basic visual recognition model can at least output initial recognition results such as stages, steps, instruments and actions corresponding to surgical video segments, or further expand to output other multi-granularity auxiliary results in alternative embodiments, it can be used as an implementation scheme of the basic visual recognition module of the present invention.

[0066] Specifically, for the recognition of surgical stages, surgical steps, surgical instruments, and surgical actions, video models that can simultaneously model spatial and temporal features are preferred.

[0067] In a preferred implementation, the basic visual recognition model includes a spatial feature extraction network, a temporal feature extraction network, an adaptive feature fusion module, and a classification prediction module.

[0068] The spatial feature extraction network extracts spatial visual information from sampled video frames, including instrument morphology, tissue texture, anatomical boundaries, and local manipulation areas. The temporal feature extraction network extracts information on surgical action changes, instrument movement, tissue traction changes, and stage / step transitions from consecutive video frames. The adaptive feature fusion module fuses spatial and temporal features, enabling the model to adaptively select feature addition, feature concatenation, or a weighted combination of both based on the characteristics of different video segments. The classification and prediction module outputs initial recognition results and their confidence levels for surgical stages, surgical steps, surgical instruments, and surgical actions.

[0069] The method employs an adaptive feature fusion approach to combine spatial and temporal features. This fusion approach is not limited to weighted concatenation or weighted addition. Alternative methods include, but are not limited to, direct concatenation, element-wise addition, element-wise multiplication, attention fusion, gated fusion, cross-attention fusion, multi-head attention fusion, feature pyramid fusion, low-rank fusion, bilinear fusion, or confidence-based post-fusion.

[0070] The fusion location can also vary. It can be performed early in the feature extraction stage, mid-stage in the classification and prediction stage, or late in the stage after multiple sub-models have output their results. Any fusion method that can combine spatial visual information and temporal dynamic information for multi-granularity recognition of surgical videos is a viable alternative to this invention.

[0071] In some embodiments of the present invention, adaptive feature fusion is implemented in the following manner: Let the spatial feature be Fs, the temporal feature be Ft, the concatenation fusion result be F_concat, the addition fusion result be F_add, and the adaptive weight be α. Then the fusion feature F can be expressed as: F =α·F_concat + (1 - α)·F_add Here, α is a weight automatically generated by the neural network based on the features of the current video segment, with a value ranging from 0 to 1. When the current task relies more on spatial details, α can be increased to increase the proportion of stitching and fusion; when the current task relies more on temporal continuity, α can be decreased to increase the proportion of additive fusion, or the meaning of the weight can be adjusted according to the specific implementation method.

[0072] The initial results output by the basic visual recognition model can be represented in a structured form. For example, for a video clip, the output of the basic visual recognition model may include: Video clip number; Phase prediction results and confidence levels; Step-by-step prediction results and confidence levels; Instrument prediction results and confidence levels; Action prediction results and confidence levels.

[0073] The initial results can be saved as a structured file, such as a JSON file, or directly passed to the subsequent inference module.

[0074] In step S5, the constructed surgical task graph is used to represent the standard surgical procedure and its constraints, and serves as the basis for subsequent temporal correction. The surgical task graph can be constructed using expert rules, obtained through statistical learning from labeled data, or constructed using a combination of expert rules and data-driven methods. The task graph can be a fixed graph or an updatable graph; it can be a single-procedure task graph or a multi-procedure shared task graph; it can be a directed graph, or a hierarchical graph, probabilistic graph, state transition graph, knowledge graph, or finite state machine.

[0075] In some embodiments of the present invention, the surgical task graph is denoted as G=(V,E,A), where V represents the set of nodes, E represents the set of edges, and A represents the set of attributes of nodes or edges. Nodes can represent surgical stages, surgical steps, surgical actions, surgical events, instrument states, anatomical regions, or risk nodes; edges represent allowed transitions, prohibited transitions, conditional transitions, probabilistic transitions, hierarchical relationships, causal relationships, or spatial relationships; the attribute set A may include information such as node name, node type, allowed predecessor nodes, allowed successor nodes, prohibited jump nodes, shortest duration, longest duration, common instruments, common actions, and common tissues or organs.

[0076] For example, a task graph may include: Stage nodes: S1, S2, S3, ..., Sn; Step nodes: P1, P2, P3, ..., Pm; Action nodes: A1, A2, A3, ..., Ak; Allowed transition edges: S1→S2, S2→S3, P1→P2, P2→P3; Prohibited transfer relationship: S1 cannot directly jump to a distant stage that is not adjacent and has no reasonable clinical basis; Duration constraint: The duration of a certain stage or step should be within a preset time range; Superior-subordinate constraints: A certain step should belong to the corresponding surgical stage, and a certain action should conform to the current step or the function of the current instrument.

[0077] In some embodiments of the present invention, the surgical task diagram is constructed jointly by expert rules, surgical procedure specifications, historical labeled data, and model statistical results. Different task diagrams can be established for different surgical procedures; for different hospitals or surgeons performing the same surgical procedure, optional branches or allowed variations can be set on the basis of the standard task diagram, thereby taking into account both procedural standardization and differences in clinical operation.

[0078] In step S6, to further address the issues of stage jumps, misjudgments of steps, discontinuous action boundaries, and inconsistent process logic that easily occur in long-term surgical videos using the basic visual recognition model, this invention sets up a task graph-guided temporal correction module. This module is used to correct the basic visual recognition results at the sequence level, complementing the subsequent retrieval enhancement large model inference correction module. It utilizes the surgical task graph, process constraints, contextual information, or model confidence to correct abnormal jumps, misalignments, or boundary jitters in the stage, step, or action sequences output by the basic visual model.

[0079] In some embodiments of the present invention, the task graph-guided temporal correction module is specifically a diffusion-type temporal correction, that is, a diffusion model is used for temporal correction. Alternatively, other sequence correction models can be used to achieve similar purposes. For example, Hidden Markov Models, Conditional Random Fields, Viterbi Decoding, Bayesian Filtering, Kalman Filtering, Particle Filtering, Dynamic Programming, Temporal Convolutional Networks, Recurrent Neural Networks, Long Short-Term Memory Networks, Transformer Sequence Models, Graph Neural Networks, Reinforcement Learning Strategies, Rule-Constrained Decoding, or Probabilistic State Machines can be used to perform temporal correction on the initial recognition sequence.

[0080] For example, diffusion-based temporal correction can be replaced by a sequence re-scoring method under task graph constraints. That is, the system first generates multiple candidate phases, steps, or action sequences, and then sorts the candidate sequences according to visual model confidence, task graph transition score, duration score, context consistency score, and knowledge matching score, and selects the candidate sequence with the highest score as the correction result.

[0081] In some embodiments of the present invention, the sequence of stages, steps, or actions output by the basic visual recognition model can be represented as Y0={y1,y2,…,yt,…,yT}, where yt represents the initial prediction result of the t-th video segment, and T represents the total number of video segments. Due to factors such as smoke, bleeding, occlusion, blurring, or surgeon differences, Y0 may contain short-term noise or abnormal jumps, such as S1→S5→S3. The task graph-guided diffusion-based temporal correction module treats this initial sequence as a noisy sequence and recovers the target sequence Y* that better conforms to the constraints of the surgical task graph through a diffusion-based denoising process.

[0082] In one implementation, diffusion-based timing correction includes a forward noise addition process and a reverse noise reduction process.

[0083] The forward noise addition process is used to simulate noise perturbations in the recognition sequence. During the training phase, the system adds random perturbations to the real labeled sequence Y to obtain sequences Yτ with different noise intensities. Perturbation methods can include class substitution, short-term jumps, fragment missing, boundary shifts, or confidence perturbations to simulate errors that the basic visual recognition model may produce in real-world scenarios.

[0084] The inverse denoising process is used to learn the mapping from the noisy sequence Yτ to the true sequence Y. The system inputs the noisy sequence Yτ, basic visual features, model confidence, contextual fragment features, and task graph constraints into a diffusion-based temporal correction network. This network progressively predicts and removes sequence noise, making the output sequence more consistent with the allowed transition relationships, duration constraints, and hierarchical constraints defined in the task graph.

[0085] During the inference phase, the basic visual recognition model first outputs the initial sequence Y0 and its probability distribution P0. The task graph-guided diffusion-based temporal correction module takes Y0, P0, and the task graph G as input and obtains the corrected sequence Yd through several denoising iterations. Each denoising step is constrained by the task graph. If a candidate result violates the prohibited transition relationship or hierarchical constraint in the task graph, the score of the candidate result is reduced; if the candidate result satisfies the allowed transition relationship, duration constraint, and contextual consistency, the score of the candidate result is increased.

[0086] In one alternative implementation, the task graph constraint score can be represented as: GraphScore(yt-1, yt, yt+1) = λ1 · EdgeScore(yt-1, yt) + λ2 ·EdgeScore(yt, yt+1) + λ3 · DurationScore(yt) + λ4 · HierarchyScore(yt) Among them, EdgeScore indicates whether the transition between adjacent nodes conforms to the allowed edges of the task graph, DurationScore indicates whether the duration of the current node conforms to the preset range, HierarchyScore indicates whether the hierarchical relationship between steps and stages, and between actions and steps is consistent, and λ1 to λ4 are weight parameters.

[0087] The diffusion-based temporal correction module considers both the visual model probability and the task graph constraint score when generating candidate sequences. The comprehensive score of candidate nodes can be expressed as: FinalScore(yt) = β1 · VisualScore(yt) + β2 · GraphScore(yt-1, yt,yt+1) + β3 · ContextScore(yt) Wherein, VisualScore represents the prediction confidence of the basic visual recognition model for the current node, GraphScore represents the task graph constraint score, ContextScore represents the consistency score between the current node and the context of the preceding and following video segments, and β1 to β3 are weight parameters.

[0088] In this way, when the basic visual recognition model outputs a short-term abnormal jump, the task graph-guided diffusion-based temporal correction module can make corrections based on prior knowledge of the process. For example, if the allowed stage transition in a routine surgical procedure is S1→S2→S3, and the initial recognition result is S1→S5→S3, and S5 does not meet the constraints of instruments, actions, tissues, and duration of the preceding and following segments, the system will identify S5 as a noise node and correct it to S2 or other reasonable nodes that meet the constraints of the task graph during the denoising process.

[0089] Step S6 primarily addresses errors at the "sequence level," including stage jumps, step jumps, discontinuous action sequences, unstable boundaries, and short-term misidentification. Its output includes the corrected stage sequence, step sequence, action sequence, action boundaries, and temporal correction confidence.

[0090] In step S7, the constructed surgical knowledge base is used to provide external knowledge support for the reasoning and correction of the large model. The knowledge base can include structured knowledge, semi-structured knowledge, and textual knowledge. Specifically, it can be implemented using structured databases, relational databases, document databases, graph databases, vector databases, knowledge graphs, rule bases, ontology bases, or hybrid databases. The content in the knowledge base can come from expert rules, public datasets, surgical guidelines, surgical procedure specifications, teaching materials, historical surgical video annotation results, model output results, and manual review feedback results.

[0091] Preferably, the surgical knowledge base includes the following: (1) Knowledge of surgical stages, including the names, definitions, start conditions, end conditions and typical visual features of different surgical stages; (2) Surgical procedure knowledge, including the specific steps included in each stage, the order of execution of the steps, and the transfer relationship between the steps; (3) Surgical instrument knowledge, including instrument name, instrument function, operable actions, common usage scenarios and appearance characteristics; (4) Surgical procedure knowledge, including procedure name, procedure definition, conditions for procedure occurrence, characteristics of procedure duration, and transition relationships between procedures; (5) Knowledge of anatomical tissues and organs, including tissue or organ names, anatomical locations, visible features, adjacent structures and common operating methods; (6) Instrument-Motion-Tissue Triad Knowledge, including reasonable or common combinations of instruments, motions and tissues in specific surgical scenarios; (7) Surgical procedure knowledge, including standard surgical flowcharts, stage transfer rules, step transfer rules, rules for allowing jumps and rules for prohibiting jumps; (8) Spatial relationship knowledge, including spatial relationships between instruments and target tissues such as contact, proximity, traction, cutting, separation, and suturing; (9) Temporal relationship knowledge, including the range of action duration, action switching order, range of step duration, and range of stage duration; (10) Quality assessment knowledge, including abnormal operation prompts, risk points, operating procedures, quality control rules and expert experience; (11) Question and answer template knowledge, including question templates, answer formats and evaluation criteria corresponding to different task types.

[0092] In some embodiments of the present invention, each piece of knowledge in the knowledge base is organized into knowledge entries. Knowledge entries can be represented in natural language text form, or in the form of tables, triples, JSON, XML, graph nodes, regular expressions, or vectors. The knowledge base can be built offline or updated online; it can be deployed locally or on a server or in the cloud. Each knowledge entry may include fields such as knowledge number, knowledge type, keywords, knowledge text, structured fields, applicable stage, applicable steps, related instruments, related actions, related tissues or organs, source of evidence, and update time.

[0093] For example, a triplet knowledge can be represented as: Knowledge type: Equipment-Motion-Organizational Relationship; Applicable stage: Bladder neck identification and separation; Applicable procedure: Incision between the prostate and the bladder neck; Equipment: Single-pole bending shears; Action: Separation; Tissue or organ: Prostate or bladder neck region; Rule Explanation: In this step, unipolar curved shears are commonly used for tissue dissection along the junction of the prostate and bladder neck.

[0094] A sequence of events can be represented as: Knowledge type: Surgical procedure transfer relationship; Current step: Suturing and ligating the dorsal deep vein complex; Next candidate step: urethral transection or subsequent related steps; Transfer prohibited: Directly redirect to a remote step that does not conform to the current process; Rule Explanation: If the current segment shows the deep dorsal vein complex ligation action, the next prediction should prioritize the process node adjacent to the completion of the ligation.

[0095] After constructing the knowledge entries, the system vectorizes the knowledge base. Specifically, the system converts the text content and structured fields of each knowledge entry into a text representation, then generates a vector representation using a text embedding model, and stores it in a vector database or index structure. The vectorized knowledge base supports similarity retrieval.

[0096] In steps S8 and S9, the system converts the initial recognition results (stages, steps, instruments, actions, etc.) output by the basic visual recognition model, as well as the task graph-guided diffusion-based temporal correction results, into structured inference input. It then uses the retrieval enhancement generation module to retrieve relevant knowledge from the surgical knowledge base. Information such as tissue or organ, instrument-action-tissue triples, spatial localization, temporal boundaries, scene descriptions, and next action predictions can be used as multi-task labeled data, question-answering data, knowledge base candidate information, intermediate inference information from large models, or subsequent output information for retrieval and inference.

[0097] Retrieval enhancement generation refers to retrieving relevant knowledge from an external knowledge base based on the input content before generating or reasoning in a large model. The retrieved results are then used as part of the large model's input to enhance the reasoning basis. Retrieval can employ vector retrieval, keyword retrieval, rule matching, knowledge graph queries, structured database queries, hybrid retrieval, or reordering retrieval. Similarity calculation methods can include cosine similarity, dot product similarity, Euclidean distance, edit distance, BM25 algorithm, neural reordering models, or other semantic matching methods. Retrieval queries can be constructed from initial recognition results, contextual fragments, user questions, scene descriptions, triple results, or conflicting items. Retrieval results can be directly input into the large model or first filtered, deduplicated, compressed, sorted, or rule-filtered before being input. The retrieval enhancement generation module in this invention is not only used to generate answers but also for performing rationality analysis and result correction on multi-granularity recognition results.

[0098] In some embodiments of the present invention, the initial identification results and timing correction results of the current video segment, such as its stage, step, instrument, and action, are first converted into a retrieval query. The retrieval query may include the following: (1) Current stage prediction results; (2) Prediction results of the current step; (3) Current instrument prediction results; (4) Current action prediction result; (5) Recognition results of adjacent segments before and after the current video segment; (6) Task graph-guided diffusion-based timing correction results; (7) The reasoning problem or correction task currently to be solved; (8) Candidate information of tissues or organs, candidate information of device-action-tissue triples, spatial location information of devices, spatial location information of tissues and organs or time interval information of actions provided by multi-task labeled data, question and answer data, knowledge base candidates, manual annotation results, auxiliary models or intermediate inference results of large models.

[0099] Among them, the prediction results of stages, steps, instruments and actions are the direct outputs of the basic visual recognition model; information such as tissues or organs, triples, spatial positioning and action time intervals can be used as auxiliary reasoning information to participate in the retrieval, and are not limited to the direct outputs of the basic visual recognition model.

[0100] The system then converts the search query into a vector and calculates its similarity with the vectors of each knowledge entry in the surgical knowledge base. The system selects the highest similarity entries as the search results. Similarity can be calculated using cosine similarity, dot product similarity, Euclidean distance, or other vector similarity methods.

[0101] In one implementation, the retrieval score can be represented as: Score(q, k_i) = cos(E(q), E(k_i)) Where q represents the current search query, k_i represents the i-th knowledge item in the knowledge base, E represents the text embedding model, and cos represents the cosine similarity. The system selects the top k knowledge items as candidate knowledge items according to their scores from high to low, where k can be 3, 5, 10, or other preset values.

[0102] To improve search accuracy, the system can also employ a hybrid search approach, using both keyword matching and vector retrieval. Keyword matching ensures terminological accuracy, such as instrument names, tissue names, and action names; vector retrieval ensures semantic relevance, such as synonyms, similar steps, and similar operational scenarios.

[0103] In steps S10 and S11, the system inputs the initial recognition results (stages, steps, instruments, actions, etc.) output by the basic visual recognition model, the task graph-guided diffusion-style temporal correction results, the retrieved knowledge content, auxiliary reasoning information, and reasoning instructions into the large-model reasoning correction module. The large-model reasoning correction module then performs rationality analysis, cross-task consistency verification, and semantic correction on the multi-granularity recognition results. In some embodiments of the present invention, the large model can be a locally deployed large language model, a cloud-based large language model, a multimodal large model, a dedicated model fine-tuned from surgical video data, or a multi-model collaborative reasoning architecture. The large model can be a general-purpose language model, a medical domain model, a surgical domain model, or a visual language model. Alternatively, the large-model reasoning correction module can also be implemented using a rule-based reasoning engine, an expert system, a knowledge graph inferencer, a probabilistic graphical model, a classification corrector, a discriminative verification model, a small language model, or a combination of multiple dedicated sub-models. For example, a model can be set up to judge the consistency of stages and steps, a model to judge the consistency of instruments and actions, a model to judge the consistency of actions and organizations, and a model to judge the rationality of spatial relationships. The outputs of these models can be summarized into the final correction result.

[0104] The auxiliary reasoning information may include one or more of the following: multi-task labeled data, question-and-answer data, knowledge base candidates, manually labeled results, auxiliary model results, or intermediate reasoning results of large models.

[0105] Specifically, the inputs to the large model inference correction module may include: (1) Basic information of the current video segment; (2) Initial recognition results of stages, steps, instruments, actions, etc., output by the basic visual recognition model; (3) Task graph-guided diffusion-based timing correction results; (4) Recognition results of stages, steps, equipment, actions, etc. of adjacent video segments; (5) Retrieve relevant knowledge returned by the enhanced generation module; (6) Candidate information of tissues or organs, candidate information of device-action-tissue triples, candidate information of spatial positioning, candidate information of temporal boundary, candidate information of scene description or candidate information of next action provided by multi-task labeled data, question and answer data, knowledge base candidates, manual annotation results, auxiliary models or intermediate inference results of large models; (7) Task instructions, such as "determine whether the initial recognition result is reasonable", "correct unreasonable results", "output explanation", "predict the next action", etc.; (8) Output format constraints, such as requiring the output to be in structured JSON format or table format.

[0106] The main functions of the large model inference correction module include: (1) Stage-step consistency check: Determine whether the current step belongs to the current stage. If not, correct the stage or step according to the stage-step relationship in the knowledge base and the context result. (2) Step-device consistency verification: Determine whether the device appearing in the current step conforms to the common device set of the step; (3) Instrument-action consistency verification: Determine whether the current instrument can perform the current action; (4) Action-tissue consistency check: Determine whether the current action usually acts on the current tissue or organ; (5) Rational verification of the ternary combination: Determine whether the <instrument, action, tissue> ternary combination conforms to the reasonable combination recorded in the surgical knowledge base; (6) Spatial consistency verification: Determine whether the spatial position of the instrument and the spatial position of the tissue or organ support the current action relationship, such as whether the instrument is close to the target tissue or whether it is located within the operation area; (7) Timing consistency check: Determine whether the current stage, step or action is consistent with the flow order of the preceding and following segments, and whether there are any abnormal jumps; (8) Timing boundary verification: Determine whether the start and end times of the action are consistent with the scene description, the movement of the equipment, and the changes in the actions before and after; (9) Next action prediction: Based on the current stage, step, triplet and process knowledge, predict the possible subsequent actions or steps; (10) Result interpretation generation: Explain the reasons why the identification results are retained or corrected.

[0107] Large model inference correction can be performed using the following logic: Input: Initial recognition result R0, task graph diffusion-based temporal correction result Rd, context result C, retrieval knowledge K, inference instruction P; Output: Large model inference correction result Rl, correction reason E, confidence level or credibility level L.

[0108] The processing procedure is as follows: The first step is to read the initial recognition result R0 and the timing correction result Rd; The second step is to construct a retrieval query based on R0, Rd, and the context result C. The third step is to retrieve relevant knowledge K from the knowledge base; The fourth step is to organize R0, Rd, C, K, and P into cue words; The fifth step involves the large model judging whether the initial identification results of stages, steps, instruments, and actions in R0 and the temporal correction results in Rd are reasonable. It also combines auxiliary reasoning information to judge whether the results of tissue or organ, triplet, spatial relationship, temporal boundary, scene description, and prediction of the next action are consistent with surgical knowledge and contextual logic. Step 6: If a result is reasonable, retain that result; Step 7: If a result is unreasonable, select a candidate correction result based on the knowledge base and context. Step 8: Output the large model inference correction result Rl and its interpretation E.

[0109] To avoid format instability caused by the free generation of large models, this invention preferably employs structured output constraints. The output of a large model can adopt the following structure: { "Interim Results": { Initial prediction: xxx, Timing correction result: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Step Results": { Initial prediction: xxx, Timing correction result: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Instrument Results": { Initial prediction: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Action Result": { Initial prediction: xxx, Timing correction result: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Tissue or organ outcome": { Candidate Information: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Triple Result": { Candidate Information: xxx, Semantic correction result: xxx, "Should I correct this?": Yes / No Basis: xxx }, "Spatial positioning results": { "Instrument Coordinates": xxx, "Tissue or organ coordinates": xxx, "Spatial Relationship Judgment": xxx }, "Temporal Boundary Results": { "Action start and end time": xxx "Action Switching Point": xxx, Is it reasonable?: xxx }, "Predicted Next Move": xxx, "Comprehensive Explanation": xxx } Step S12 implements a dual-correction fusion strategy, complementing and fusing the task graph-guided diffusion-type temporal correction module with the retrieval-enhanced large-model inference correction module. The task graph-guided diffusion-type temporal correction module primarily operates from a time-series perspective, focusing on determining the reasonableness of the sequence of surgical stages, steps, and actions on the timeline. It is suitable for correcting issues such as short-term jumps, boundary jitter, stage misalignment, and discontinuous processes. The retrieval-enhanced large-model inference correction module primarily operates from a knowledge semantic perspective, focusing on determining whether stages, steps, instruments, actions, tissues, spatial positioning, triple relationships, and scene descriptions conform to the surgical knowledge base, instrument functions, anatomical structures, and operational rules. It is suitable for correcting issues such as instrument-action mismatch, action-tissue mismatch, step-instrument mismatch, spatial relationships not supporting the current operation, and contradictions between scene descriptions and structured results. In some embodiments of this invention, the complementary strategy includes the following methods: First, sequential complementarity. The system first guides the diffusion-based timing correction module through the task graph to correct the timing of stages, steps, and action sequences. Then, the timing correction results are used as input to the large model inference correction module, enabling the large model to perform semantic reasoning in a more reasonable process context.

[0110] Second, the system employs a reverse verification-based complementary approach. The system first detects semantic conflicts through a large model inference and correction module, such as a mismatch between a certain instrument-action-tissue triple and the current step. Then, it feeds back this conflict information to the task graph-guided diffusion-based temporal correction module, prompting it to reassess whether there are any temporal misjudgments in the current step or action sequence.

[0111] Third, confidence-weighted fusion. The system obtains the diffusion-based temporal correction result Rd and the large-model inference correction result Rl separately, and calculates the final result Rf based on their confidence, consistency score, and knowledge matching score. The final result can be expressed as: Rf = Fusion(Rd, Ld, Rl, Ll, C) Where Rd represents the diffusion-based time-series correction result, Ld represents the diffusion-based time-series correction confidence level, Rl represents the large-model inference correction result, Ll represents the large-model inference correction confidence level, and C represents the cross-task consistency score. Fusion represents the fusion function, which can employ rule-based voting, confidence-weighted averaging, priority selection, or manual review triggering strategies.

[0112] When Rd and Rl are consistent, the system directly adopts this consistent result as the final correction result. When Rd and Rl are inconsistent, the system selects different processing methods according to the conflict type: If the conflict mainly involves the sequence of stages, steps, or actions, the task diagram-guided diffusion-style timing correction results should be used first. If the conflict mainly involves equipment, action, organization, triples, or spatial relationships, the retrieval-enhanced large model reasoning correction results should be used first. If both confidence levels are low or the conflict cannot be resolved automatically, the system will output a suspected anomaly marker and prompt for manual review.

[0113] Through the above dual correction fusion strategy, the present invention can simultaneously utilize the process constraint capability of the task graph and the knowledge reasoning capability of the large model to achieve temporal correction and semantic correction of the multi-granularity recognition results of surgical videos.

[0114] The execution order and fusion method of task graph diffusion-based temporal correction and retrieval enhancement large model can be varied. For example, task graph temporal correction can be performed first, followed by large model inference correction; or large model inference correction can be performed first, followed by task graph temporal correction; or the two can be executed in parallel and then fused together. The fusion method can employ rule priority, confidence weighting, voting mechanisms, cascading correction, conflict type judgment, threshold triggering, manual review triggering, or a learned fusion network. When the outputs of the two correction modules are consistent, the result can be directly adopted; when they are inconsistent, the result can be selected based on the conflict type, for example, temporal conflicts prioritize the task graph temporal correction result, and semantic conflicts prioritize the large model inference correction result; selection can also be based on confidence, knowledge matching degree, or consistency score; and a pending review mark can be output when automatic judgment is not possible.

[0115] In some embodiments of the present invention, to improve the reliability of multi-granularity recognition results, a cross-task consistency verification mechanism is established. This mechanism does not judge whether a single recognition result has the highest confidence level, but rather judges whether multiple recognition results support each other. Specifically, it can be implemented using rule-based judgment, statistical models, machine learning models, graph models, large model judgment, or a combination of rules and models. Verification objects can include stages and steps, steps and instruments, instruments and actions, actions and tissues, instrument locations and tissue locations, temporal boundaries and scene descriptions, and the current step and the next prediction, etc.

[0116] In some embodiments of the present invention, cross-task consistency verification includes at least the following rules: Rule 1: Stage and Step Consistency Rule. Each surgical step should belong to one or more pre-defined surgical stages. If the step identification result does not belong to the step set corresponding to the stage identification result, a stage-step conflict is determined to exist.

[0117] Rule 2, Step-to-Instrument Consistency Rule. Different steps typically have common instrument sets. If the instrument identification result does not match the common instrument set of the current step at all, and the model confidence is low, it is determined that there may be an instrument identification error or a step identification error.

[0118] Rule 3: Instrument-Action Consistency Rule. Different instruments have different functions. For example, a suction device typically corresponds to suction, cleaning, or assisted traction actions; a needle holder typically corresponds to clamping, suturing, knotting, or suture pulling actions; scissor-like instruments typically correspond to actions such as separation, cutting, or electrocoagulation. If the combination of instrument and action does not conform to the functional relationship defined in the knowledge base, an instrument-action conflict is determined to exist.

[0119] Rule 4: Action-Tissue or Organ Consistency Rule. Different surgical actions typically act on specific tissue or organ regions. If the combination of action and tissue or organ does not conform to the operational rules of the current step or stage, an action-tissue conflict is determined.

[0120] Rule 5, Spatial Position Consistency Rule. After the system identifies the spatial positions of the instrument and the target tissue or organ, it determines whether the instrument position and the target tissue or organ position satisfy the current action relationship. For example, for actions that require direct action on the target tissue, such as cutting, separating, suturing, and knotting, the instrument tip or operating area should be in contact, overlap, or be close to the target tissue area. If the spatial distance exceeds a preset threshold, it is determined that the spatial relationship does not support the current triplet.

[0121] Rule Six: Temporal Consistency Rule. The system determines the rationality of the sequence of stages, steps, or actions based on the standard surgical procedure, task graph, or historical context. If the current identification result forms a significantly unreasonable jump with the preceding and following segments, a temporal conflict is determined to exist.

[0122] Rule 7: Consistency between Description and Structured Results. The surgical scene description generated by the system should be consistent with the structured recognition results. For example, if the structured results show that the instrument is a single-pole curved shear, the action is dissection, and the tissue is the prostate, then the description text should not generate instruments, actions, or tissues that contradict these findings.

[0123] In one implementation, the system can set a consistency score for each consistency rule. The consistency score can be a weighted sum, or it can be represented by the maximum conflict item, minimum confidence level, logical constraint satisfaction, knowledge graph path score, classifier output probability, or the confidence level of a large model output. The conditions for triggering correction can be a score falling below a threshold, the occurrence of a specific conflict type, a decrease in confidence, knowledge matching failure, or a manual review rule.

[0124] In some embodiments of the present invention, a weighted summation method is specifically used for scoring: Let the stage-step consistency score be C1, the instrument-action consistency score be C2, the action-organ consistency score be C3, the spatial consistency score be C4, the temporal consistency score be C5, and the descriptive consistency score be C6, then the comprehensive consistency score C_total can be expressed as: C_total = w1C1 + w2C2 + w3C3 + w4C4 + w5C5 + w6C6; Here, w1 to w6 are preset weights or weights learned through the validation set. When C_total is lower than the preset threshold, the system triggers the large model correction process; when C_total is higher than the preset threshold, the system retains the initial recognition results or only generates explanatory descriptions.

[0125] In step S13, the system outputs multi-granularity intelligent analysis results. The output content can be adjusted according to different application scenarios. In some embodiments of the present invention, one or more of the following results can be output: (1) Corrected surgical stage identification results; (2) Corrected surgical procedure identification results; (3) Corrected surgical instrument identification results; (4) Corrected surgical action recognition results; (5) Corrected anatomical tissue or organ identification results; (6) The corrected instrument-motion-tissue ternary set; (7) Instrument spatial positioning results; (8) Spatial localization results of tissues and organs; (9) The time interval of the action and the action switching point; (10) Description of the surgical video scene; (11) Prediction of the next action; (12) Results of multiple rounds of reasoning; (13) Abnormal or risk warnings; (14) Basis for result correction; (15) Structured analysis report.

[0126] The structured analysis report may include basic video information, segment numbers, stage / step results, instrument / action / tissue results, triplet results, spatial localization results, temporal boundary results, next prediction results, risk warnings, and reasoning basis. Outputs can be structured JSON files, tables, text reports, visualization interfaces, video timeline annotations, risk warning lists, teaching feedback reports, or quality control scoring reports. Output content may include one or more of the following: final recognition results, initial prediction results, temporal correction results, semantic correction results, reasons for correction, confidence level, risk warnings, next predictions, and expert review comments.

[0127] It can also output results with different granularities depending on the application scenario. For example, in the training data construction scenario, it outputs annotation suggestions and items to be reviewed; in the postoperative review scenario, it outputs a process analysis report; in the teaching scenario, it outputs key operation segments and improvement suggestions; and in the model evaluation scenario, it outputs the accuracy, consistency score and error type statistics for each task.

[0128] Corresponding to the knowledge-enhanced and large-model-based intelligent reasoning method for surgical videos described in the above embodiments, this invention also provides a knowledge-enhanced and large-model-based intelligent reasoning system for surgical videos. Each module of this system can implement each step of the knowledge-enhanced and large-model-based intelligent reasoning method for surgical videos.

[0129] like Figure 3 As shown, the system includes: The video input module is used to acquire surgical video data; The video preprocessing module is used to preprocess the surgical video data acquired by the video input module, including format conversion, frame rate sampling, resolution unification, image quality assessment, and invalid segment removal. The video segmentation module is used to divide the preprocessed surgical video data into multiple video segments and to cut each video segment into frames to obtain a frame sequence. The multi-granularity dataset construction module is used to generate multi-task labeled data and multi-granularity question-and-answer data corresponding to each video segment divided by the video segmentation module, so as to establish the correspondence between video segments and labels or question-and-answer data such as stages, steps, instruments, actions, organizations, triples, spatial positioning, temporal boundaries, scene descriptions and next step predictions. The basic visual recognition module is used to recognize the frame sequence obtained by the video segmentation module to obtain an initial recognition result, which includes a stage recognition sequence, a step recognition sequence, an instrument recognition result, and an action recognition sequence. The initial result structuring module is used to transform the initial recognition results obtained by the basic visual recognition module into structured reasoning input. Specifically, it converts the initial recognition results such as stages, steps, instruments, and actions into a unified format. The surgical task graph module is used to construct a surgical task graph, which uses surgical stages, surgical steps and surgical actions as nodes, and allows transfer relationships, prohibits jump relationships, duration range constraints and hierarchical relationships as edges. The task graph-guided diffusion-type temporal correction module uses the surgical task graph constructed by the surgical task graph construction module as the constraint condition for the diffusion process. It corrects the stage jumps, step misalignments, and action boundary jitters in the initial recognition sequence that do not conform to the surgical procedure logic through a stepwise denoising method, and obtains the temporal correction results corresponding to each sequence. It realizes the repair of abnormal jumps in the stage, step, and action sequences according to the constraints of the task graph. The surgical knowledge base module is used to construct a surgical knowledge base, which includes at least surgical procedure knowledge, instrument function knowledge, action definition knowledge, anatomical structure knowledge, instrument-action-tissue relationship knowledge, and anatomical spatial relationship knowledge, thereby storing knowledge such as surgical procedure, instrument function, action definition, anatomical structure, triplet relationship, spatial relationship, and temporal relationship. The knowledge vectorization and retrieval module is used to vectorize the knowledge entries in the surgical knowledge base to form a searchable knowledge index, so as to realize similarity retrieval; The retrieval enhancement generation module is used to retrieve relevant knowledge content from the surgical knowledge base as the basis for reasoning based on the retrieval enhancement generation mechanism, the current video segment, the current recognition result, and the current reasoning task. The large model inference correction module is used to perform verification and correction operations on video segment information, the initial recognition results obtained by the basic visual recognition module, the temporal correction results obtained by the diffusion temporal correction module, the knowledge content retrieved by the retrieval enhancement generation module, auxiliary inference information, and preset inference instructions. The auxiliary inference information includes one or more of the following: multi-task labeled data, multi-granularity question and answer data, knowledge base candidate options, manual labeled results, auxiliary model results, or large model intermediate inference results, thereby realizing rationality analysis and result correction based on retrieved knowledge and initial results. The dual-correction fusion module is used to fuse the temporal correction result obtained by the diffusion-type temporal correction module with the semantic correction result generated by the large model inference correction module to obtain the final correction result; The output and report generation module is used to output at least one of the following: multi-granularity recognition results, temporal correction results, semantic correction results, reasoning explanations, risk warnings, next action predictions, or structured analysis reports.

[0130] In some embodiments of the present invention, a cross-task consistency verification module is further provided to determine whether there is a logical conflict between different identification tasks.

[0131] In some embodiments of the present invention, a manual review and feedback iteration module is further provided, which is used to receive the expert review results and feed them back to the dataset, knowledge base, task graph or model training process.

[0132] Manual review and feedback iteration can be completed by a single expert, or by multiple experts independently annotating the data and then arbitrating consistency. Feedback can be used to update multi-granularity datasets, surgical task graphs, knowledge bases, visual recognition models, diffusion-based temporal correction models, or fine-tuning data for large models.

[0133] Human feedback can be reviewed item by item or in batches; it can review the final result or intermediate results, such as task graph correction results, knowledge retrieval results, reasons for large model correction, or consistency conflicts. The feedback strategy can adopt an active learning approach, prioritizing low-confidence, high-conflict, or high-risk segments for expert review.

[0134] Therefore, any solution that can improve the quality of the dataset, task graph, knowledge base, or model performance by utilizing the results of manual review can be used as an alternative to the feedback iteration module of this invention.

[0135] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0136] In some embodiments of the present invention, the methods and systems of the present invention can be applied to the analysis of various surgical videos for multi-granularity recognition, temporal correction, knowledge retrieval, semantic correction, and structured output, including but not limited to robot-assisted radical prostatectomy, laparoscopic surgery, endoscopic surgery video analysis, and other surgical videos with clear procedures, instrument operations, anatomical structures, and action sequences, such as robot-assisted surgery, laparoscopic surgery, thoracoscopic surgery, gynecological laparoscopic surgery, urological surgery, gastrointestinal surgery, hepatobiliary surgery, etc.

[0137] The following is an algorithm flow of a specific embodiment of the present invention.

[0138] Input: A surgical video clip Clip_t and its adjacent clips Clip_t-1 and Clip_t+1.

[0139] Step 1: Clip_t to obtain the frame sequence Frame_t.

[0140] Step two: Input Frame_t into the basic visual recognition model to obtain the initial recognition result R0. R0 includes the prediction results and their confidence levels for stages, steps, instruments, and actions.

[0141] Step 3: Combine R0 with the stage, step, instrument, and action recognition results of the preceding and following segments to form context C.

[0142] Step four: Input the initial sequences of stages, steps, and actions in R0 into the task graph-guided diffusion-based temporal correction module. This module performs denoising correction on short-term jumps, step misalignments, and action boundary jitter based on the surgical task graph G, model confidence, and context C, obtaining the temporal correction result Rd.

[0143] Step 5: Construct the retrieval query Q based on R0, Rd, C, and auxiliary reasoning information A. Auxiliary reasoning information A may include candidate tissue or organ information, device-action-tissue triplet candidate information, spatial positioning candidate information, action temporal boundary candidate information, initial scene description, or next action candidate information provided by multi-task labeled data, question-answering data, knowledge base candidates, manual annotation results, intermediate reasoning results from auxiliary models or large models, spatial positioning candidate information, action temporal boundary candidate information, initial scene description, or next action candidate information. Q may include content such as "current stage + current step + device + action + preceding and following steps + temporal correction result + candidate tissue + candidate triplet".

[0144] Step 6: Input Q into the knowledge vectorization and retrieval module to obtain the relevant knowledge set K.

[0145] Step seven: The system calculates a consistency score based on R0, Rd, C, auxiliary reasoning information A, and retrieved knowledge K, including stage-step consistency, instrument-action consistency, action-organization consistency, spatial consistency, and temporal consistency. Stage, step, instrument, and action are derived from the initial recognition results of the basic visual recognition model, while information such as organization, triples, spatial location, and temporal boundaries can be derived from auxiliary reasoning information A or retrieved knowledge K.

[0146] Step 8: If all consistency scores are higher than the preset threshold, the relevant results are retained as the final results, and an explanation is generated.

[0147] Step 9: If there are items with consistency scores lower than the preset threshold, input R0, Rd, C, auxiliary reasoning information A, K, and conflict items into the large model reasoning correction module.

[0148] Step 10: The large model inference correction module outputs the semantic correction result Rl and explains the reason for the correction.

[0149] Step 11: The dual-calibration fusion module performs fusion based on Rd, Rl, confidence level, and conflict type to obtain the final result Rf.

[0150] Step 12: Output Rf as the final multi-granular intelligent reasoning result of the surgical video based on knowledge enhancement and a large model.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-granularity intelligent reasoning method for surgical videos based on knowledge enhancement and large models, characterized in that, Includes the following steps: Acquire surgical video data and perform multi-granularity recognition on the surgical video data to obtain initial multi-granularity recognition results; A surgical task graph corresponding to the surgical video data is constructed. The surgical task graph is used to represent the hierarchical relationships and temporal constraints in the surgical process. The initial multi-granularity recognition results are temporally corrected based on the surgical task graph to obtain the temporal correction results. Construct a surgical knowledge base, and obtain knowledge content related to the initial multi-granularity recognition result and / or the temporal correction result from the surgical knowledge base based on a retrieval enhancement generation mechanism; The initial multi-granularity recognition result, the temporal correction result, and the knowledge content are input into the large model inference correction module, which performs semantic consistency verification and generates semantic correction result. The temporal correction result is fused with the semantic correction result to obtain the final correction result.

2. The method for multi-granular intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, Before the multi-granularity recognition, the procedure further includes dividing the surgical video data into multiple video segments. The video segments are divided using a fixed time window method or a dynamic window method based on changes in surgical stage, steps, or actions. The dynamic window determines the start and end times of the segments based on the probability of stage changes, the probability of actions, or the boundary detection results output by the visual model.

3. The method for multi-granularity intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The multi-granularity recognition is achieved based on a pre-constructed multi-granularity surgical video dataset; the multi-granularity surgical video dataset is constructed by performing multi-granularity annotation on surgical video data, and the multi-granularity annotation includes at least surgical stage annotation, surgical step annotation, surgical instrument annotation, and surgical action annotation.

4. The method for multi-granularity intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The multi-granularity recognition is achieved through a basic visual recognition model, which includes: a spatial feature extraction network for extracting spatial visual information of the surgical scene, including instrument shape, tissue texture, anatomical structure boundaries, and local operating areas; a temporal feature extraction network for extracting temporal information of surgical action changes, instrument movement, tissue traction changes, and stage or step transfers; an adaptive feature fusion module for fusing spatial and temporal features; and a classification prediction module for outputting the initial recognition results and confidence levels of surgical stages, surgical steps, surgical instruments, and surgical actions.

5. The method for multi-granularity intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The surgical task graph is denoted as G=(V,E,A), where V represents the set of nodes, E represents the set of edges, and A represents the set of attributes of nodes or edges. Nodes represent surgical stages, surgical steps, or surgical actions, and edges represent allowed transfer relationships, prohibited jump relationships, or hierarchical relationships. Attributes include duration range, common instruments, common actions, or common tissue information.

6. The method for multi-granularity intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The timing correction is a diffusion-type timing correction, which includes a forward noise addition process and a reverse noise reduction process; the forward noise addition process is used to add random perturbations to the real labeled sequence during the training phase to simulate noise in the recognition sequence; The reverse denoising process is used to learn the mapping from the noisy sequence to the real sequence, and under the constraints of the surgical task graph, it progressively predicts and removes sequence noise so that the corrected sequence conforms to the constraint relationship defined by the surgical task graph.

7. The method for multi-granularity intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The process of obtaining relevant knowledge content from the surgical knowledge base based on the retrieval enhancement generation mechanism includes: converting the initial multi-granularity recognition results and the temporal correction results into retrieval queries; calculating the similarity between the retrieval queries and the vectors of each knowledge entry in the surgical knowledge base; and selecting several knowledge contents with the highest similarity. The semantic consistency verification includes cross-task consistency verification and temporal consistency verification.

8. The method for multi-granular intelligent reasoning of surgical videos based on knowledge enhancement and large models according to claim 1, characterized in that, The fusion adopts at least one of sequential complementarity, reverse verification complementarity, or confidence-weighted fusion.

9. A multi-granularity intelligent reasoning system for surgical videos based on knowledge enhancement and large models, characterized in that, include: A multi-granularity recognition module is used to acquire surgical video data and perform multi-granularity recognition on the surgical video data to obtain an initial multi-granularity recognition result. The surgical task graph construction module is used to construct a surgical task graph, which is used to represent the hierarchical relationships and temporal constraints in the surgical procedure. The timing correction module is used to perform timing correction on the initial multi-granularity recognition result based on the surgical task map to obtain the timing correction result; The surgical knowledge base module is used to build a surgical knowledge base; The retrieval enhancement large model inference correction module is used to obtain knowledge content related to the initial multi-granularity recognition result and the temporal correction result from the surgical knowledge base based on the retrieval enhancement generation mechanism, and input the initial multi-granularity recognition result, the temporal correction result and the knowledge content into the large model inference correction module for semantic consistency verification, and generate semantic correction result; The fusion output module is used to fuse the temporal correction result with the semantic correction result to obtain the final correction result.

Images