A multi-granularity surgical video visual question answering method based on hierarchical visual representation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2026-04-03
- Publication Date
- 2026-06-09
Smart Images
Figure CN122176610A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent question answering technology, specifically a multi-granularity surgical video visual question answering method based on hierarchical visual representation. Background Technology
[0002] Surgical videos are a vital resource widely used in modern medical education and clinical practice, primarily for recording surgical procedures, assessing surgical skills, and analyzing surgical steps and instrument usage. They play a crucial role in medical student training, surgical quality control, and clinical decision support. Surgical videos contain rich clinical information, including the progress of surgical steps, the location and number of instruments and organs, and details of instrument-tissue interactions, presented in a hierarchical structure of global-local-detail. Furthermore, the questions in surgical video visual question answering are multi-granular, such as surgical stage or step, instrument or organ location, number of instruments, and the actions and states of instrument-tissue interactions. With the rapid development of surgical techniques and the massive accumulation of surgical video data, effectively understanding and analyzing surgical video content has become an important research direction in the field of medical artificial intelligence. In recent years, the rise of visual question answering technology has provided new possibilities for the intelligent analysis of surgical videos. By combining computer vision and natural language processing technologies, surgical video visual question answering systems can automatically understand video content and answer relevant questions, thus providing medical professionals with intelligent auxiliary tools. This technology not only improves the efficiency and accuracy of surgical video analysis but also promotes the standardized dissemination of medical knowledge and the effective accumulation of clinical experience.
[0003] Early research on surgical video visual question answering focused on proposing new surgical video datasets and using visual-language encoders to more accurately understand surgical scenarios. For example, VisualBert ResMLP constructed two surgical video VQA datasets, Cholec80-VQA and EndoVis-18-VQA, and adopted a dual visual-text encoder. It innovatively introduced a visual Bert encoder model with a residual attention mechanism, effectively enhancing the interaction between visual and textual tags. GVLE-LViT designed a gated visual-language module to construct input features for the visual-language transformer and added a detection head for accurate localization. These global-based methods showed good performance but neglected local features. To improve answer quality, recent research has begun to incorporate spatial and motion information from surgical instruments and anatomical structures. SSG-VQA-Net integrates global and local features, improving reasoning ability for solving complex problems. While these methods consider the relationship and fusion between global and local information, they all respond to questions of different granularities with a uniform process, lacking awareness and differentiation of question granularity. Furthermore, the reasoning and contextual learning capabilities of large models have not been fully explored. With large language models such as GPT and LLaMA demonstrating strong generalization capabilities in instruction compliance and contextual reasoning, recent work has begun to explore multimodal pipelines centered on LLMs, aligning visual evidence with language reasoning in surgical scenarios to enhance open-ended responses and multi-turn interactions. One approach constructs visual instruction data or adaptation modules within surgical scenarios to improve the ability to describe common questions and scene entities. For example, SurgVLM, developed based on Qwen2.5-VL, adapts to different surgical tasks through instruction fine-tuning and has constructed the multimodal surgical video understanding dataset SurgVLM-DB. Surgical-LVLM, by combining a pre-trained LLM with a dedicated visual perception LoR module, performs exceptionally well in complex surgical scenarios. Another approach combines graph structures or local referencing capabilities to attempt to incorporate steps, instruments, and tissue relationships into semantic chains. Surgical-MambaLLM combines Mamba2 with large language models for the surgical video domain, fully leveraging Mamba2's ability to perceive information and capture dependencies to improve the large model's understanding of surgical images. These methods improve fine-grained understanding of surgical images and generalization ability to problems of different granularities by incorporating large models. However, surgical videos are presented in a global-local-detail hierarchical structure, and existing methods lack a hierarchical representation consistent with the structure of surgical videos. The inability to achieve cross-hierarchical cue fusion and alignment leads to reduced accuracy and affects the interpretability of the model.
[0004] Existing visual question answering methods for surgical videos suffer from several problems. First, the questions in surgical video visual question answering are multi-granular; however, existing works typically employ a uniform strategy to handle questions of different granularities. This uniform approach introduces unnecessary detailed computations when dealing with questions requiring global context, leading to feature redundancy. Furthermore, in questions requiring local or detailed information, the lack of granularity matching constraints results in missed detections and misjudgments of boundaries, interactions, and subtle states. This impacts the model's accuracy and efficiency. Second, surgical videos exhibit a hierarchical structure of global-local-detail. Existing methods lack a consistent hierarchical question answering mechanism with surgical videos, failing to fuse and align cues between different levels. For example, the global context fails to constrain local regions, and local structures and spatial relationships fail to convey fine-grained information to the detailed level. This leads to deviations in the localization of key areas and the judgment of states and actions, affecting the model's interpretability. This makes it difficult for medical experts to understand and verify the system's reasoning process, limiting its application value in actual clinical decision-making. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to propose a multi-granularity surgical video visual question-answering method based on hierarchical visual representation, comprising: Step 1: Obtain the user's question text for the surgical video and the current image of the surgical video; Step 2: Clean and standardize the problem text to obtain the processed problem text, and then determine the target granularity of the processed problem text; Step 3: Generate an intermediate representation based on the target granularity and the current image of the surgical video; Step 4: Construct constrained hints. Input the intermediate representation, the current image, the processed question text, and the constrained hints into the large language model to obtain the answer text.
[0006] Optionally, step 2, which involves cleaning and standardizing the problem text, specifically includes: The process unifies capitalization and punctuation in the problem text, removes stop words, and, in conjunction with a pre-defined surgical dictionary, corrects spelling and terminology, mapping instruments, organs, stages, and steps to a unified expression, resulting in the processed problem text.
[0007] Optionally, the target granularity of the processed question text in step 2 is determined in the following way: The processed question text is identified to obtain keywords. When the keywords contain stage, step, or process, the processed question text is classified as scene-level and its initial granularity is set to 1. When the keywords contain organ or spatial location and quantity statistics of instruments, the processed question text is classified as region-organ level and its initial granularity is set to 2. When the keywords contain judgment of instrument status, action, or boundary dynamics, the processed question text is classified as pixel-level and its initial granularity is set to 3. To set up structured prompts, the processed question text, the current image of the surgical video, and the structured prompts are input into the large language model to obtain the prediction granularity and confidence score. When the confidence score is less than the preset threshold, or when the prediction granularity is inconsistent with the initial granularity, the initial granularity is incremented by 1 to become the target granularity. When the confidence score is greater than or equal to the preset threshold, and the prediction granularity is consistent with the initial granularity, the initial granularity is directly used as the target granularity.
[0008] Optionally, step 3 specifically includes: When the target granularity is 1, the current image of the surgical video is input into the fine-tuned recognition model MTMS-TCN, which outputs a scene-level posterior distribution. and global semantic summarization, Together with the global semantic summary, it serves as an intermediate representation.
[0009] Optionally, step 3 may also include: When the target granularity is 2, execute the steps for when the target granularity is 1 to generate the scene-level posterior distribution. Based on global semantic summarization, entities are extracted from the processed question text to obtain an entity set E; the current image of the surgical video is segmented using a fine-tuned segmentation model to obtain the entity mask for each entity. Instance count and relative position, Entity mask The intermediate representation is composed of instance count and relative position; Among them, entity mask Regions with a median value of 1 are entities. e Local area, entity mask All local regions constitute an entity e of Clipping Collection ,in, For entities The A local region; a local region is an instance, and the relative position is the spatial relationship between the instance and the organ or the spatial relationship between different instances.
[0010] Optionally, step 3 may also include: When the target granularity is 3, execute the steps for target granularity 2 to generate a scene-level posterior distribution. Global semantic summary, entity mask , Clipping Collection This leads to the generation of entity label distribution after scenario prior fusion. The entity label distribution after scenario prior fusion, the judgment reasons, and the evidence citations constitute the intermediate representation.
[0011] Optionally, the entity label distribution after prior scene fusion is generated in the following way: Define action tag space With state label space Based on the constraints of the device itself, A and S are filtered to obtain a subset of action labels. and status tag subset This leads to the construction of structured cue triples; Each The large language model is input with structured prompt triples to obtain... The corresponding entity label distribution, judgment reasoning, and evidence citation, wherein the entity label distribution is an entity action label distribution. or entity status label distribution ; Quality-weighted aggregation of entity tag distributions for the same entity yields multiple results. Entity action tag weighted aggregation or more Entity state label weighted aggregation Specifically, this is achieved through the following formula: ; ; in, For normalization operations, For the first indivual Quality weights; Will Mapping these to the action space and state space respectively, we obtain the action heuristic distribution of the scene prior mapping. State heuristic distribution mapped to scene priors ; The following formula will be used to... and Lightweight linear fusion is performed to obtain the entity action label distribution after scene prior fusion. Or, by using the following formula and Lightweight linear fusion is performed to obtain the entity state label distribution after scene prior fusion. ,Will or As the distribution of entity labels after scene prior fusion; ; ; in The fusion strength.
[0012] Optionally, the structured prompt triple is <context, visual evidence, criterion template>, and the superscript and subscript include the entityified question. The entity problem, in conjunction with global semantic summarization. It is obtained by instantiating the processed problem text; visual evidence includes the current image and entity mask. The superimposed image also includes Clipping Collection The criterion template includes textual judgment criteria and a subset of action tags. or a subset of status tags .
[0013] The beneficial effects of adopting the above technical solution are as follows: This invention proposes a problem-granularity perception method, using problem granularity as a global control signal to achieve adaptive problem triage from the scene level to the region / organ level and pixel level. Different processing paths are adopted for different granularities to ensure the coherence of reasoning between different behavioral goals and visual granularities. A hierarchical visual representation framework is constructed, using problem granularity labels to drive the triage and alignment of cross-scale evidence at the scene level, region / organ level, and pixel level, and a unified intermediate representation to carry global information, entity-level structured evidence, and pixel-level fine-grained clues. It conforms to the hierarchical structure of global-local-detail in surgical videos, realizing hierarchical visual representation and evidence alignment of global-local-detail.
[0014] Unlike existing methods that rely solely on single-level features, this invention obtains multi-level visual representations by dividing questions into multiple granularities, significantly improving the model's answering performance in surgical video visual question answering while enhancing interpretability and reliability. Specifically, this invention uses a rule- and dictionary-driven template matching mechanism, along with a large model to divide surgical video questions into three granularities: scene-level, region / organ-level, and pixel-level. Global planning is performed for questions of different granularities to select the most suitable processing path. For scene-level questions requiring global features, a stage and step recognition model is used to obtain global information. For region / organ-level questions requiring local features, a surgical video-specific segmentation model generates corresponding masks to capture finer-grained local features. For even more detailed pixel-level questions, the synergy of global and local features captures both global information and preserves detailed regional features, facilitating further analysis and reasoning by the large model. Attached Figure Description
[0015] Figure 1 This is one of the flowcharts illustrating a multi-granularity surgical video visual question answering method based on hierarchical visual representation in an embodiment of the present invention; Figure 2 This is a second flowchart illustrating a multi-granularity surgical video visual question-answering method based on hierarchical visual representation in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the problem granularity awareness in an embodiment of the present invention; Figure 4 This is a flowchart of the hierarchical visual representation mechanism in an embodiment of the present invention; Figure 5 This is a flowchart of hierarchical reasoning and response in an embodiment of the present invention. Detailed Implementation
[0016] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0017] To address the shortcomings of existing surgical video visual question answering models, this invention provides a multi-granularity surgical video visual question answering method based on hierarchical visual representation. This method utilizes an adaptive question granularity perception mechanism and a hierarchical visual representation mechanism to improve the accuracy and interpretability of medical visual question answering. Figure 1 and Figure 2 This may include the following steps: Step 1: Obtain the user's question text for the surgical video and the current image of the surgical video; Step 2: Clean and standardize the problem text to obtain processed problem text, then determine the target granularity of the processed problem text, and combine it with... Figure 3 Specifically, the following steps are included: Step 2.1: Clean and standardize the problem text to obtain the processed problem text; Specifically, the text unifies capitalization and punctuation, removes stop words, and corrects spelling and terminology without altering the semantics by combining a pre-defined surgical dictionary. It also maps instruments, organs, stages, and steps to unified expressions, normalizes synonyms and abbreviations, and resolves ambiguities. The resulting processed text is more stable, parsable, and reusable in form and semantics, thus improving the robustness of subsequent granular judgments from the source.
[0018] Step 2.2: Recognize the processed question text to obtain keywords. When the keywords contain stage, step, or process, classify the processed question text as scene-level and set its initial granularity to 1. When the keywords contain organ or spatial location and quantity statistics of instruments, classify the processed question text as region-organ level and set its initial granularity to 2. When the keywords contain judgment of instrument status, action, or boundary dynamics, classify the processed question text as pixel-level and set its initial granularity to 3. In this step, based on the core intent of the processed question text, key semantic clues and sentence patterns such as "stage / step," "location / positioning," "quantity statistics," and "state / action" are automatically identified, providing clear semantic support for subsequent granular mapping and judgment. Based on reviewable templates and keyword rules, structured semantic clues are mapped to three levels of granularity to achieve rule / dictionary-driven initial granular judgment. Step 2.3: Set up structured prompts. Input the processed question text, the current image of the surgical video, and the structured prompts into the SurgicalGPT large language model to obtain the prediction granularity and confidence level. Further, output the judgment reason. Compare the prediction granularity with the initial rule judgment for consistency. If the confidence level is less than the preset threshold, or if the prediction granularity is inconsistent with the initial granularity, add 1 to the initial granularity as the target granularity to ensure evidence coverage. If the confidence level is greater than or equal to the preset threshold, and the prediction granularity is consistent with the initial granularity, directly use the initial granularity as the target granularity. After completing the problem granularity classification, the hierarchical structural features of the surgical video are fully utilized to construct a hierarchical three-layer progressive information extraction mechanism: for scene-level problems, a global recognition model is used to obtain the overall semantics such as stages and steps; for region / organ-level problems, a segmentation model is combined to extract refined anatomical structure and spatial relationship features; for pixel-level problems, fine-grained reasoning of complex interactive states and minute movements is performed under the joint representation of global and local data, which is specifically achieved through step 3.
[0019] Step 3: Generate an intermediate representation based on the target granularity and the current image of the surgical video, and combine it with... Figure 4 include: When the target granularity is 1, the current image of the surgical video is input into the fine-tuned recognition model MTMS-TCN, which outputs a scene-level posterior distribution. And a global semantic summary (e.g., “phase=dissection, conf=0.82”); this global representation serves as a unified contextual prior that runs through all subsequent scales, which can directly support scene-level decision-making and provide operational intent and process constraints in region / organ-level and pixel-level tasks, thereby improving the consistency and interpretability of cross-scale reasoning.
[0020] Will Together with the global semantic summary, it serves as an intermediate representation, with a scene-level posterior distribution. The definition is as follows: ; in In mathematics, it is a symbol belonging to. This is the mathematical representation of a probability distribution. This refers to the number of stages / steps, i.e., the total number of stages / steps that can be divided into in a surgical video.
[0021] The recognition model MTMS-TCN was fine-tuned using three datasets: Cholec80-VQA, EndoVis-18-VQA, and PSI-AVA-VQA.
[0022] When the target granularity is 2, execute the steps for when the target granularity is 1 to generate the scene-level posterior distribution. The system extracts entities, relation types, and action verbs from the processed question text using global semantic summarization to obtain an entity set E. For example, from the question "Is the grasper used in gallbladder dissection?", it obtains the entities "grasper" and "gallbladder dissection", with the relation "used in", and generates the entity set E. and the mask space domain to which each entity belongs. ,in The image spatial domain is the complete pixel coordinate space of a single frame of the surgical video.
[0023] The current image of the surgical video is segmented using a fine-tuned segmentation model to obtain the entity mask for each entity. Instance count and relative position; The segmentation model used is the SurgicalSAM segmentation model, which is fine-tuned on three datasets: Cholec80-VQA, EndoVis-18-VQA, and PSI-AVA-VQA.
[0024] Among them, entity mask It is a binary matrix with the exact same size as the current image, where each pixel position (i,j) corresponds to the pixel coordinates (x,y) of the image. (Entity mask) Regions with a median value of 1 are entities. e The local region precisely marks all pixel positions of the entity in the image spatial domain Ω, thus establishing a direct mapping relationship between the entity and its coordinates. When constructing the ROI (Region of Interest) of an entity based on the entity mask and coordinates, the process begins with the entity mask... Extract the set of pixel coordinates with a value of 1, and then calculate the minimum bounding box of this coordinate set, or directly use the mask region as the boundary to obtain the local region of interest (ROI) of the entity. If there are multiple instances of the same entity (for example, if there are 3 grasping objects of the same type in the image, there will be 3 grasping object masks, each mask representing an instance), then the entity mask... All local regions constitute an entity e of Clipping Collection ,in, For entities The Each local region is considered an instance, and its relative position represents the spatial relationship between the instance and the organ, or the spatial relationship between different instances (e.g., the position of instance 1 relative to the gallbladder, the relative orientations of multiple instances, etc.). Entity mask Instance counts and relative positions form an intermediate representation; these ROIs and clipping sets together serve as subsequent visual evidence.
[0025] The above process integrates regional evidence into an intermediate representation, serving as a traceable basis for pixel-level judgment and final answer, and enhances the relevance and interpretability of location, counting, and interactive question answering with structured mesoscale evidence.
[0026] When the target granularity is 3, execute the steps for target granularity 2 to generate a scene-level posterior distribution. Global semantic summary, entity mask , Clipping Collection This leads to the generation of entity label distribution after scene prior fusion; Define action tag space With state label space Specifically, it is expressed as follows: ; ; Based on the constraints of the instrument itself (e.g., the main function of a gripper is to pull and it cannot perform shearing), A and S are filtered to obtain a subset of action tags. and status tag subset Construct a structured prompt triple (context, visual evidence, criterion template), wherein the upper and lower parts include the entityified question. The entity problem, in conjunction with global semantic summarization. It is obtained by instantiating the processed problem text; visual evidence includes the current image and entity mask. The overlaid image (with highlighted solid areas) also includes Clipping Collection The criterion template includes textual judgment criteria and a subset of action tags. or a subset of status tags You can select only one of the action subset or state subset based on the actual needs of the problem; Each The large language model is input with structured prompt triples to obtain... The corresponding entity label distribution, judgment reasoning, and evidence citation, wherein the entity label distribution is an entity action label distribution. or entity status label distribution The formula is as follows: ; ; in for The number of action categories included. for The number of action categories included.
[0027] To reduce fluctuations caused by potential field-of-view and segmentation errors in surgical video images, quality-weighted aggregation of entity label distributions for the same entity is performed to obtain multiple... Entity action tag weighted aggregation or more Entity state label weighted aggregation Specifically, this is achieved through the following formula: ; ; in, For normalization operations, For the first indivual The quality weights can be obtained from the mask integrity, sharpness, and segmentation error.
[0028] To maintain consistency with the scene context, Mapping these to the action space and state space respectively, we obtain the action heuristic distribution of the scene prior mapping. State heuristic distribution mapped to scene priors ; Will and Lightweight linear fusion is performed to obtain the entity action label distribution after scene prior fusion. , or and Lightweight linear fusion is performed to obtain the entity state label distribution after scene prior fusion. Specifically, this is achieved through the following formula: ; ; in Fusion strength; Will or As the entity label distribution after scene prior fusion, the entity label distribution after scene prior fusion, the judgment reason and the evidence citation constitute the intermediate representation, realize the traceable alignment between pixel-level conclusions and upper-level context—regional evidence, and thereby drive the closed loop of multi-level reasoning and response.
[0029] Based on this, combined Figure 5 This invention, based on problem-level granularity and hierarchical visual representation, merges scene-level posterior and entity-level details into a unified intermediate representation to ensure cross-scale alignment and traceability. Specifically, we use the scene-level surgical stage / step probability distribution as the core global context, serving as a unified benchmark for integrating all hierarchical information. For region / organ-level information, we fully preserve the structured evidence at this level, including entity masks, anatomical structure instance analysis results, and organ spatial relationship features, ensuring that this level of evidence is not lost and is traceable. For pixel-level information, we first weight and summarize the action / state judgment results (including judgment reasons and evidence reference areas) of each entity in different ROI regions according to ROI quality weights to obtain the overall action / state judgment result of the entity. Then, we map the scene-level global context as a priori reference for the action / state dimension and integrate it into the above overall judgment result in a lightweight linear fusion manner, ensuring that the pixel-level conclusions are consistent with the global context. Ultimately, all evidence entries at the scene, region / organ, and pixel levels are integrated into the same intermediate representation, ensuring that conclusions at each level can be traced back to the corresponding evidence, thus achieving cross-scale information alignment and interpretability.
[0030] For the unified intermediate representation, the system employs constrained hints to enable the LLM to generate answers that are consistent with the question type and have traceable evidence. Constrained hints, based on the unified intermediate representation, define inference rules, evidence, output specifications, and consistency requirements for the large language model, ensuring that the answer matches the question type, the reasoning is traceable, and the logic and evidence are aligned. For scenario-level questions, the posterior a posteriori of each stage / step is included. As a core constraint, the conclusion must be consistent with the overall context; for regional / organ-level questions, in Beyond the provided global context constraints, the model is further required to utilize local information such as masks, instance counts, and spatial relationships (here, spatial relationships correspond to instance-level spatial relationship modeling generated by SurgicalSAM, specifically referring to the relative positional relationships between different instances or between instances and organs, such as the position of the grasping forceps instance relative to the gallbladder, the mutual orientation and contact state between multiple instrument instances, etc.). The constraint answer must generate localization, counting, or interaction-related conclusions along with structured local descriptions. For pixel-level problems, entity-level fusion distribution is used as the core basis, while also referencing corresponding... The key points of the judgment reasoning and evidence citation ensure a strict consistency mapping from conclusion to context to evidence, and are presented in an interpretable manner with standardized answer text, key evidence and citation positions.
[0031] Step 4: Construct constrained hints. Input the intermediate representation, the current image, the processed question text, and the constrained hints into the large language model to obtain the answer text; This invention provides a multi-granularity surgical video visual question answering method based on hierarchical visual representation. For performance verification of this method, this embodiment predicts the answer based on given surgical video frame images and questions. Three different medical surgical visual question answering datasets are used: Cholec80-VQA, EndoVis-18-VQA, and PSI-AVA-VQA. Cholec80-VQA is a laparoscopic surgery visual question answering dataset containing question-answer pairs corresponding to 40 laparoscopic cholecystectomy surgery videos from the Cholec80 dataset, totaling 21.6k frames and 34,000 question-answer pairs. EndoVis-18-VQA is a visual question answering dataset based on 14 robotic nephrectomy surgery videos, with questions covering one organ, four tool localizations, 13 tool-organ interactions, and eight surgical tools. PSI-AVA-VQA is a visual question-answering dataset derived from eight prostatectomy surgeries. This dataset includes four classification tasks: surgical stage recognition, surgical step recognition, tool position, and action state, totaling 2.2k frames. ACC, Recall, and F1-Score were selected as evaluation metrics, and the results were compared with the following methods on three datasets: MFB, MFH, Mutan, VisualBert, VisualBert ResMLP, SSG-VQA-Net, and SurgicalGPT. The method of this invention is named S2P-VQA. The best results are shown in bold in Tables 1, 2, and 3.
[0032] Table 1 Experimental results on the Cholec80-VQA dataset
[0033] Table 2 Experimental results on the EndoVis-18-VQA dataset
[0034] Table 3 Experimental results of the PSI-AVA-VQA dataset
[0035] This invention uses precision, recall, and F1 score as evaluation metrics. Precision measures the proportion of correctly predicted samples, reflecting the overall accuracy of the classification results. Recall focuses on evaluating the model's ability to identify positive samples, reflecting the completeness of its coverage of true positive examples. The F1 score, as the harmonic mean of precision and recall, effectively balances the performance of both classes, providing a more robust comprehensive evaluation, especially in datasets with imbalanced class distributions. Under these evaluation metrics, compared with the other two methods, this invention achieves the best performance on the Cholec80-VQA, EndoVis-18-VQA, and PSI-AVA-VQA datasets. Its performance advantage mainly stems from question granularity awareness, hierarchical visual representation, and the synergy between multi-level reasoning and response, demonstrating good cross-type generalization and robustness.
[0036] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A multi-granularity surgical video visual question answering method based on hierarchical visual representation, characterized in that, include: Step 1: Obtain the user's question text for the surgical video and the current image of the surgical video; Step 2: Clean and standardize the problem text to obtain the processed problem text, and then determine the target granularity of the processed problem text; Step 3: Generate an intermediate representation based on the target granularity and the current image of the surgical video; Step 4: Construct constrained hints. Input the intermediate representation, the current image, the processed question text, and the constrained hints into the large language model to obtain the answer text.
2. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 1, characterized in that, Step 2, cleaning and standardizing the problem text, specifically includes: The process unifies capitalization and punctuation in the problem text, removes stop words, and, in conjunction with a pre-defined surgical dictionary, corrects spelling and terminology, mapping instruments, organs, stages, and steps to a unified expression, resulting in the processed problem text.
3. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 1, characterized in that, The target granularity of the processed problem text in step 2 is determined in the following way: The processed question text is identified to obtain keywords. When the keywords contain stage, step, or process, the processed question text is classified as scene-level and its initial granularity is set to 1. When the keywords contain organ or spatial location and quantity statistics of instruments, the processed question text is classified as region-organ level and its initial granularity is set to 2. When the keywords contain judgment of instrument status, action, or boundary dynamics, the processed question text is classified as pixel-level and its initial granularity is set to 3. To set up structured prompts, the processed question text, the current image of the surgical video, and the structured prompts are input into the large language model to obtain the prediction granularity and confidence score. When the confidence score is less than the preset threshold, or when the prediction granularity is inconsistent with the initial granularity, the initial granularity is incremented by 1 to become the target granularity. When the confidence score is greater than or equal to the preset threshold, and the prediction granularity is consistent with the initial granularity, the initial granularity is directly used as the target granularity.
4. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 3, characterized in that, Step 3 specifically includes: When the target granularity is 1, the current image of the surgical video is input into the fine-tuned recognition model MTMS-TCN, which outputs a scene-level posterior distribution. and global semantic summarization, Together with the global semantic summary, it serves as an intermediate representation.
5. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 4, characterized in that, Step 3 also includes: When the target granularity is 2, execute the steps for when the target granularity is 1 to generate the scene-level posterior distribution. Based on global semantic summarization, entities are extracted from the processed question text to obtain an entity set E; the current image of the surgical video is segmented using a fine-tuned segmentation model to obtain the entity mask for each entity. Instance count and relative position, Entity mask The intermediate representation is composed of instance count and relative position; Among them, entity mask Regions with a median value of 1 are entities. e Local area, entity mask All local regions constitute an entity e of Clipping Collection ,in, For entities The A local region; a local region is an instance, and the relative position is the spatial relationship between the instance and the organ or the spatial relationship between different instances.
6. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 5, characterized in that, Step 3 also includes: When the target granularity is 3, execute the steps for target granularity 2 to generate a scene-level posterior distribution. Global semantic summary, entity mask , Clipping Collection This leads to the generation of entity label distribution after scenario prior fusion. The entity label distribution after scenario prior fusion, the judgment reasons, and the evidence citations constitute the intermediate representation.
7. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 6, characterized in that, The entity label distribution after prior fusion of the scenario is generated in the following way: Define action tag space With state label space Based on the constraints of the device itself, A and S are filtered to obtain a subset of action labels. and status tag subset This leads to the construction of structured cue triples; Each The large language model is input with structured prompt triples to obtain... The corresponding entity label distribution, judgment reasoning, and evidence citation, wherein the entity label distribution is an entity action label distribution. or entity status label distribution ; Quality-weighted aggregation of entity tag distributions for the same entity yields multiple results. Entity action tag weighted aggregation or more Entity state label weighted aggregation Specifically, this is achieved through the following formula: ； ； in, For normalization operations, For the first indivual Quality weights; Will Mapping these to the action space and state space respectively, we obtain the action heuristic distribution of the scene prior mapping. State heuristic distribution mapped to scene priors ; The following formula will be used to... and Lightweight linear fusion is performed to obtain the entity action label distribution after scene prior fusion. Or, by using the following formula and Lightweight linear fusion is performed to obtain the entity state label distribution after scene prior fusion. ,Will or As the distribution of entity labels after scene prior fusion; ； ； in The fusion strength.
8. The multi-granularity surgical video visual question answering method based on hierarchical visual representation according to claim 7, characterized in that, The structured prompt triple is <context, visual evidence, criterion template>, and the superordinate and subordinate dimensions include the materialized question. The entity problem, in conjunction with global semantic summarization. It is obtained by instantiating the processed problem text; visual evidence includes the current image and entity mask. The superimposed image also includes Clipping Collection The criterion template includes textual judgment criteria and a subset of action tags. or a subset of status tags .