Construction of training data set for surgery video question and answer task and intelligent inference method

Abstract

The application provides a training data set construction and intelligent reasoning method for a surgical video question and answer task, and relates to the technical field of intelligent question and answer. The method comprises the following steps: inputting a video segment into a feature extraction network set to obtain multiple visual features; introducing a weighted scoring mechanism to evaluate the video segment from multiple dimensions of counting, attribution and time sequence, selecting a preferred question type, setting a question template and an answer for the video segment, obtaining a training data set, and supervising a fine-tuning multi-modal large model. In the reasoning system, a double-head ViT visual encoder is used for processing the frame sequence in the visual encoder part to obtain surgical instrument features and anatomical structure features. The visual token and the text token are compared with the similarity and aligned with the consistency by a representative token selector and a surgical knowledge base before entering the large model, and the visual and text tokens that can better represent the surgical knowledge are retained; the reasoning module adopts the trained multi-modal large model for reasoning, which can improve the accuracy and reliability of the answer.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, and more specifically, to a method for constructing a training dataset and intelligent reasoning for surgical video question answering tasks. Background Technology

[0002] While current surgical multimodal solutions have significantly improved structured reasoning capabilities and answer coherence by leveraging large-scale instruction data, demonstrating potential for applications in tasks such as surgical knowledge question answering and process analysis, their technical architecture still suffers from significant dependency bottlenecks. The core performance of these solutions heavily relies on manually constructed task-specific labeled data and offline alignment learning models. This labeling process is not only time-consuming and labor-intensive with high professional barriers, but offline learning also struggles to adapt to the dynamically changing operational procedures and complex conditions in surgical scenarios, thus limiting the model's generalization ability when processing real clinical surgical videos.

[0003] Crucially, the temporal information processing mechanism in the solution has inherent shortcomings. Currently, temporal association is mainly injected indirectly through the hierarchical relationship between task tags. This method cannot accurately capture the temporal logic and causal relationships of continuous operation steps in surgical videos, making it difficult to meet the query needs for operation details at specific time points. At the same time, the automatic question type allocation mechanism for specific video segments is not yet perfect, resulting in insufficient adaptability of the model to question-answering needs in different scenarios.

[0004] Although existing technologies have made some progress in the sub-field of "surgical video question answering," enabling some basic video content queries and operation step recognition, they have not yet broken through the technical framework that is mainly based on static image feature analysis. There are still problems such as time-consuming and labor-intensive annotation, inability to accurately capture temporal logic and causal relationships, and imperfect automatic assignment of question types. As a result, the accuracy and reliability of the model still need to be greatly improved when dealing with complex question answering tasks involving long-term operation processes and multi-step related reasoning. Summary of the Invention

[0005] The problem that this invention aims to solve is that existing technologies suffer from time-consuming and labor-intensive labeling, inability to accurately capture temporal logic and causal relationships, and imperfect automatic assignment of problem types.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a method for constructing a training dataset for surgical video question-answering tasks, comprising:

[0007] The surgical video is divided into multiple video segments, and the video segments are input into a feature extraction network set to obtain various visual features;

[0008] Based on multiple visual features and a hierarchical weighted scoring mechanism, scores for counting, attribution, and time series categories are determined.

[0009] Based on the scores for counting, attribution, and time series, and the preset adaptation weights corresponding to different question types, adaptation scores for various question types are obtained.

[0010] When the difference between the fit scores of multiple question types is greater than or equal to the preset difference threshold, the question type with the largest fit score that is greater than the preset fit threshold is selected as the preferred question type.

[0011] When the difference in fit scores among multiple question types is less than a preset difference threshold, the preferred question type is selected according to the preset priority order.

[0012] Based on the preferred question type, question templates and answers are set for the corresponding video segments to obtain a training dataset. The training dataset includes multiple data elements, and each data element includes a video segment and a corresponding question-answer pair.

[0013] We use the training dataset to supervise and fine-tune the multimodal large model to obtain a multimodal large model suitable for surgical video question answering.

[0014] Optionally, the feature extraction network set includes an instrument detection network, an anatomical segmentation network, a temporal recognition network, and a video quality assessment network;

[0015] Multiple visual features include instrument features, anatomical structure features, temporal features, and video quality features;

[0016] The instrument detection network outputs video clips containing category labels and spatial locations of various surgical instruments, which are used to form instrument features;

[0017] The anatomical segmentation network outputs segmentation masks and category labels for anatomical structures in video clips, which are used to form anatomical structure features.

[0018] The probability distribution and boundaries of each surgical stage, step or action in the output video clip of the temporal recognition network are used to form temporal features;

[0019] The video quality assessment network outputs a quality score reflecting sharpness, occlusion, and brightness stability, which is used to form video quality features.

[0020] Optionally, the hierarchical weighted scoring mechanism includes: setting up a count-based evaluation layer, a category-based evaluation layer, and a time-series evaluation layer, with each evaluation layer having multiple different indicators, and weighting and summing the multiple indicators to output a score;

[0021] In the counting evaluation layer, based on the characteristics of the device, the obtained indicators include the number of device instances score, the number of different device types score, the target salience score, and the detection confidence score; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset counting indicator weights to obtain the counting score.

[0022] In the classification evaluation layer, the indicators obtained based on instrument characteristics, anatomical structure characteristics, and video quality characteristics include instrument category confidence score, anatomical structure category confidence score, dominant target stability score, and video quality penalty item; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset classification indicator weights to obtain the classification score.

[0023] In the temporal evaluation layer, the indicators obtained based on temporal characteristics include the number of stage / step changes score, action duration score, instrument-anatomical interaction intensity score, and temporal coherence score; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset temporal indicator weights to obtain the temporal score.

[0024] Optionally, the question types include visual perception, temporal action, relational semantics, choice judgment, and basic common sense.

[0025] Optionally, the preset priority order is: sequential action class, relational semantic class, visual perception class, selection judgment class, and basic common sense class.

[0026] Secondly, the present invention also provides an intelligent reasoning method for surgical video question-answering tasks, comprising:

[0027] The surgical video was divided into multiple video segments, and the video segments were cut into frames to obtain a frame sequence;

[0028] The frame sequence is input into the dual-head ViT visual encoder to obtain the surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task, and the anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task.

[0029] The features of surgical instruments and anatomical structures are stitched together at the matrix level to obtain the fused visual features corresponding to each frame.

[0030] The question text is input into a word segmenter to obtain text features;

[0031] The visual and text features corresponding to each frame are fused in a visual-text order concatenation manner to obtain the visual-text features corresponding to each frame.

[0032] By filtering the visual-text features, compact visual-text features are obtained;

[0033] The compact visual-text features are input into a multimodal large model suitable for surgical video question answering, constructed using the training dataset construction method for the surgical video question answering task, to obtain the answer.

[0034] Optionally, the visual-text feature includes multiple visual tokens and multiple text tokens;

[0035] The process of filtering visual-text features to obtain compact visual-text features includes:

[0036] Visual similarity is obtained by analyzing the similarity between each term in the surgical knowledge base and each visual token. The terms in the surgical knowledge base include surgical instrument terms, surgical action terms, and anatomical structure terms.

[0037] The text similarity is obtained by analyzing the similarity between each term in the surgical knowledge base and each text token;

[0038] The maximum visual similarity corresponding to each visual token is taken as the final visual similarity of the visual token, and the maximum text similarity corresponding to each text token is taken as the final text similarity of the text token.

[0039] The first preset number of visual tokens are retained based on the final visual similarity from high to low, and the second preset number of text tokens are retained based on the final text similarity from high to low. At the same time, semantically duplicated neighboring tokens are removed to obtain compact visual-text features.

[0040] Thirdly, the present invention also provides an intelligent reasoning system for surgical video question-answering tasks, comprising:

[0041] The video segmentation module is used to divide the surgical video into multiple video segments and to cut the video segments into frames to obtain a frame sequence;

[0042] A dual-head ViT visual encoder is used to receive and process frame sequences and output surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task, and the anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task.

[0043] The feature stitching module is used to stitch together surgical instrument features and anatomical structure features at the matrix level to obtain the fused visual features for each frame.

[0044] A word segmenter is used to receive and process the question text and output text features;

[0045] The feature concatenation module is used to fuse the fused visual features and text features corresponding to each frame in a visual-text order concatenation manner to obtain the visual-text features corresponding to each frame.

[0046] A representative token selector filters visual-text features to obtain compact visual-text features;

[0047] The inference module is used to input compact visual-text features into a multimodal large model suitable for surgical video question answering, constructed using the training dataset construction method for the surgical video question answering task, to obtain the answer.

[0048] Fourthly, the present invention provides an electronic device, including a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is configured to, when executing the computer program, implement the training dataset construction method for surgical video question answering tasks as described in the first aspect or the intelligent reasoning method for surgical video question answering tasks as described in the second aspect.

[0051] Fifthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training dataset construction method for surgical video question-answering tasks as described in the first aspect or the intelligent reasoning method for surgical video question-answering tasks as described in the second aspect.

[0052] This invention provides a method for constructing a training dataset and intelligent reasoning for surgical video question-answering tasks. Compared with existing technologies, it has the following advantages:

[0053] By introducing a hierarchical weighted scoring mechanism, video segments can be evaluated in a refined manner from multiple dimensions such as counting, attribution, and temporal sequence. The introduction of temporal sequence scores directly quantifies the temporal dynamics of the video segments, leading to a better understanding of the surgical procedure. By calculating the fit scores for various question types and combining them with preset difference thresholds and priority orders to determine the preferred question type, the problem of imperfect automatic question type allocation in existing technologies is solved. This mechanism allows the model to adaptively select the most appropriate question-and-answer type based on the actual content of the video segment, thereby improving the model's adaptability to question-and-answer needs in different scenarios. Through the above mechanism, high-quality training datasets can be automatically generated, significantly reducing the reliance on manually constructed task-specific labeled data and overcoming the time-consuming, labor-intensive, and highly technical labeling problems of existing technologies. Simultaneously, by using this automatically generated training data for supervised fine-tuning of the multimodal large model, the model can better adapt to the dynamically changing operational procedures and complex conditions in surgical scenarios, improving the model's generalization ability when processing real clinical surgical videos. Therefore, when dealing with complex question-answering tasks involving long-term operational processes and multi-step related reasoning, it can provide higher accuracy and reliability, effectively solving many technical problems existing in the prior art. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a method for constructing a training dataset for a surgical video question-answering task, as provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the multimodal large model construction process provided in an embodiment of the present invention;

[0057] Figure 3 A flowchart illustrating an intelligent reasoning method for surgical video question-answering tasks provided in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the reasoning process for surgical video question answering provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the feature fusion process provided in an embodiment of the present invention;

[0060] Figure 6This is a schematic diagram of the visual-text feature processing process provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the structure of an intelligent reasoning system for surgical video question answering tasks provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0064] like Figure 1 As shown in the embodiment of this application, a method for constructing a training dataset for a surgical video question-answering task includes:

[0065] S11: Divide the surgical video into multiple video segments and input the video segments into a feature extraction network set to obtain various visual features.

[0066] S12: Determine the scores for count category, attribution category, and time sequence category based on multiple visual features and a hierarchical weighted scoring mechanism.

[0067] S13: Based on the scores for count type, attribution type, time series type, and the preset adaptation weights corresponding to different question types, obtain the adaptation scores for various question types.

[0068] S14: When the difference between the fit scores of multiple question types is greater than or equal to the preset difference threshold, select the question type with the largest fit score that is greater than the preset fit threshold as the preferred question type.

[0069] S15: When the difference in fit scores among multiple question types is less than a preset difference threshold, the preferred question type is selected according to the preset priority order.

[0070] S16: Based on the preferred question type, set question templates and answers for the corresponding video segments to obtain a training dataset, wherein the training dataset includes multiple data elements, and each data element includes a video segment and a corresponding question-answer pair.

[0071] S17: Use the training dataset to supervise and fine-tune the multimodal large model to obtain a multimodal large model suitable for surgical video question answering.

[0072] Specifically, a dedicated multimodal large model for surgical video question answering needs to be trained under supervision on top of a general multimodal large model to incorporate surgical knowledge. However, a single question type can cause the multimodal large model to rely on surface patterns. Therefore, the visual question answering dataset needs to cover multiple types of reasoning tasks. However, how to construct the most suitable question-answer pair for each surgical video segment based on its features is an urgent problem to be solved.

[0073] In this embodiment, by introducing a hierarchical weighted scoring mechanism, video segments can be evaluated in a refined manner from multiple dimensions such as counting, attribution, and temporal sequence. This is significantly different from the existing technologies that mainly rely on static image feature analysis or indirect injection of temporal information through task labels. Existing technologies often struggle to accurately capture the temporal logic and causal relationships of consecutive operational steps when processing surgical videos. This method, by introducing temporal class scores, directly quantifies the temporal dynamic information of video segments, thereby enabling a better understanding of the surgical procedure. By calculating the fit scores of various question types and combining them with preset difference thresholds and priority orders to determine the preferred question type, the problem of imperfect automatic question type allocation in existing technologies is solved. This mechanism allows the model to adaptively select the most appropriate question-and-answer type based on the actual content of the video segment, thereby improving the model's adaptability to question-and-answer needs in different scenarios. Through the above mechanism, high-quality training datasets can be automatically generated, significantly reducing the reliance on manually constructed task-specific labeled data and overcoming the problems of time-consuming, labor-intensive, and highly technical labeling processes in existing technologies. Meanwhile, by using these automatically generated training data to perform supervised fine-tuning of the multimodal large model, the model can better adapt to the dynamically changing operational procedures and complex conditions in surgical scenarios, improving its generalization ability when processing real clinical surgical videos. Therefore, it can provide higher accuracy and reliability when handling complex question-answering tasks involving long-term operational procedures and multi-step associative reasoning, effectively solving many technical problems existing in the prior art.

[0074] The following is a detailed description of each step.

[0075] S11: Divide the surgical video into multiple video segments and input the video segments into a feature extraction network set to obtain various visual features.

[0076] Specifically, such as Figure 2As shown, open-source laparoscopic surgery videos are divided into multiple 30-second video segments. Each video segment is sequentially input into a video feature extraction network set consisting of an instrument detection network, an anatomical segmentation network, a temporal recognition network, and a video quality assessment network. The feature extraction network set includes the instrument detection network, anatomical segmentation network, temporal recognition network, and video quality assessment network. Multiple visual features are used, including instrument features, anatomical structure features, temporal features, and video quality features.

[0077] The instrument detection network can employ a YOLOv8n architecture, outputting category labels and spatial locations of various surgical instruments in the video clip to form instrument features. The anatomical segmentation network can use a DeepLabV3+ semantic segmentation network, outputting segmentation masks and category labels for anatomical structures in the video clip to form anatomical structure features. The temporal recognition network can use a ResNet-50+TCN temporal modeling network trained on labeled surgical stage data, outputting the probability distribution and boundaries of each surgical stage, step, or action in the video clip to form temporal features. The video quality assessment network can use a ResNet-18-based no-reference video quality assessment network, outputting quality scores reflecting sharpness, occlusion, and brightness stability to form video quality features. These specific and targeted visual features can more accurately reflect the actual content of the surgical video, providing high-quality input for the subsequent hierarchical weighted scoring mechanism, thus making the calculation of count-based scores, attribution-based scores, and temporal scores more accurate. This helps to more accurately determine the fit scores corresponding to different question types and select the most suitable first question type, thereby constructing a high-quality training dataset and effectively improving the performance of multimodal large models suitable for surgical video question answering.

[0078] S12: Determine the scores for count category, attribution category, and time sequence category based on multiple visual features and a hierarchical weighted scoring mechanism.

[0079] Specifically, the hierarchical weighted scoring mechanism includes: setting up a counting-based (counting / existence) evaluation layer, a belonging-based (identification / belonging) evaluation layer, and a time-series-based (time-series / relationship) evaluation layer. Each evaluation layer sets different multiple indicators, and the multiple indicators are weighted and summed to output the score. For example... Figure 2As shown, after obtaining the instrument features, anatomical structure features, temporal features, and video quality features of each video segment, the system, based on a hierarchical weighted scoring mechanism, uses "the matching degree between the video segment and the candidate question type" as the overall target layer. It designs three criterion layers: the number / existence evaluation layer, the identification / attribution evaluation layer, and the temporal / relationship evaluation layer. Each criterion layer sets several indicators related to the instrument features, anatomical structure features, temporal features, and video quality features. The weights of each layer are determined in a manner similar to the analytic hierarchy process, and finally, the system automatically calculates the fit score of the candidate question type.

[0080] In the counting evaluation layer, for questions such as "does it exist?" and "how many are there?", indicators such as the number of device instances, the number of different device types, the target salience, and the detection confidence score are set. Based on device characteristics, the obtained indicators include the number of device instances, the number of different device types, the target salience, and the detection confidence score. After normalizing the values ​​of multiple indicators, they are weighted and summed according to preset counting indicator weights to obtain the counting score. Specifically, the number of device instances and the number of types are obtained by normalizing the number of instances and categories output by the device detection network; the target salience is calculated by the area ratio of the target in the image and whether it is located in the center region; and the detection confidence is calculated by the average confidence of the detection results. After normalizing each indicator value to the [0,1] interval, they are weighted and summed according to pre-defined weights to obtain the counting score. .

[0081] In the classification evaluation layer, for questions such as "what is it?" and "which part does it belong to?", indicators such as instrument category confidence score, anatomical structure category confidence score, dominant target stability score, and video quality penalty are set. Based on instrument characteristics, anatomical structure characteristics, and video quality characteristics, the resulting indicators include instrument category confidence score, anatomical structure category confidence score, dominant target stability score, and video quality penalty. The values ​​of multiple indicators are normalized and then weighted and summed according to preset classification indicator weights to obtain the classification score. Specifically, the instrument category confidence and anatomical structure category confidence are obtained from the category probabilities output by the instrument detection network and the anatomical segmentation network; the dominant target stability is calculated from the position and scale changes of the target in consecutive frames; and the video quality penalty is converted from quality features such as blurriness, occlusion, and brightness stability into penalty factors. Each indicator is also normalized and weighted to obtain the classification score. .

[0082] In the temporal evaluation layer, for questions such as "what action was performed," "which step is being performed," and "what is the relationship between the instrument and the anatomical structure," indicators such as the number of stage / step changes, action duration, instrument-anatomical interaction intensity, and temporal coherence are set. Based on temporal characteristics, the resulting indicators include the number of stage / step changes, action duration, instrument-anatomical interaction intensity, and temporal coherence. The values ​​of multiple indicators are normalized and then weighted and summed according to preset temporal indicator weights to obtain the temporal score. Specifically, the number of stage / step changes is obtained by statistically analyzing the stage label sequence output by the temporal recognition network; the action duration is calculated by the number of frames or duration of a specific action label; the instrument-anatomical interaction intensity is calculated by the spatial contact time and relative position change between the instrument and the anatomical structure; and temporal coherence is calculated by the jump frequency of the stage / action label. Each indicator is normalized and weighted to obtain the temporal score. .

[0083] This hierarchical, multi-dimensional, and weighted summation evaluation method ensures a comprehensive, detailed, and quantitative analysis of video segment content. This approach enables the extraction of highly generalized and discriminative scores from complex visual information, providing a solid foundation for subsequently determining the suitability for different question types, thereby significantly improving the quality and relevance of the multimodal large model training dataset.

[0084] S13: Based on the scores for counting, attribution, and temporal categories, and the preset adaptation weights corresponding to different question types, adaptation scores are obtained for various question types. These question types include visual perception, temporal action, relational semantics, choice / judgment, and basic common sense.

[0085] Specifically, for any given video segment, the counting score for that video segment is calculated. Category Score and time series scores Five candidate question types—visual perception, temporal action, relational semantics, choice judgment, and basic common sense—are each assigned a pre-defined adaptation weight. The scores for counting, attribution, and temporal categories are then weighted and summed based on these pre-defined weights to calculate the fit score for each question type. For example, a possible question type, such as "visual perception," might be assigned higher weights for counting and attribution scores, while the temporal category score might be assigned a lower weight. Conversely, a "temporal action" question might be assigned a higher weight for temporal category scores. These pre-defined adaptation weights can be set by domain experts based on experience to reflect the emphasis different question types place on each score category. The fit score, obtained by weighting and summing each score with its corresponding pre-defined adaptation weight, quantifies the degree of matching between each video segment and different question types.

[0086] S14: When the difference between the fit scores of multiple question types is greater than or equal to the preset difference threshold, select the question type with the largest fit score that is greater than the preset fit threshold as the preferred question type.

[0087] Specifically, if a video clip has a fit score of 0.8 for "visual perception" questions, while all other question types have fit scores below 0.5, and the difference between 0.8 and 0.5 is greater than a preset difference threshold (e.g., 0.2), and 0.8 is also greater than a preset fit threshold (e.g., 0.6), then "visual perception" questions are selected as the preferred question type. This mechanism ensures that when there are clearly dominant question types, the most suitable question type can be accurately identified and selected.

[0088] S15: When the difference in fit scores among multiple question types is less than a preset difference threshold, the preferred question type is selected according to a preset priority order. The preset priority order is: sequential action type, relational semantic type, visual perception type, selection judgment type, and basic common sense type.

[0089] Specifically, the system first selects the question type with the highest fit score that exceeds a preset minimum threshold as the preferred question type for the video clip. When the fit scores of multiple question types differ by less than a preset difference threshold, they are decided according to a preset priority order to automatically determine the most suitable question type for the video clip. For example, if the fit score for "visual perception" questions is 0.7 and the fit score for "temporal action" questions is 0.68, and the difference between them is less than a preset difference threshold (e.g., 0.05), the system will refer to a preset priority order (e.g., "temporal action" takes precedence over "visual perception") and select "temporal action" as the preferred question type. This mechanism provides a deterministic selection strategy when fit scores are similar, avoiding randomness in selection.

[0090] S16: Based on the preferred question type, set question templates and answers for the corresponding video segments to obtain a training dataset, wherein the training dataset includes multiple data elements, and each data element includes a video segment and a corresponding question-answer pair.

[0091] Specifically, if the preferred question type is "visual perception," the system can select a template from a pre-set question template library, such as "What instruments are included in this video clip?", and automatically or semi-automatically generate an answer based on the visual features of the video clip, such as "scissors, tweezers." If the preferred question type is "temporal action," it might select "What operation occurred in this video clip?" and generate the answer "tissue cutting." In this way, a large-scale question-and-answer dataset highly relevant to video content can be efficiently constructed.

[0092] S17: Use the training dataset to supervise and fine-tune the multimodal large model to obtain a multimodal large model suitable for surgical video question answering.

[0093] Specifically, templated questions and answers are manually constructed based on the question type for supervised fine-tuning of the multimodal large model. Through supervised fine-tuning using the training dataset constructed using the above method, the model's parameters are adjusted to better understand the complex content of surgical videos and generate accurate and relevant answers for specific question types. For example, the model can learn how to associate visual information in the video with textual questions, thus enabling it to output accurate answers when given surgical video clips and related questions.

[0094] Traditional multimodal large model vision branches typically extract common features from each frame using a single-head ViT after video is segmented. However, their attention is easily disturbed by background texture and lighting, making it difficult to stably focus on surgical instruments and anatomical structures. This invention uses a dual-head ViT vision encoder pre-trained on surgical videos.

[0095] like Figure 3 As shown in the embodiment of this application, an intelligent reasoning method for surgical video question answering tasks is provided, including:

[0096] S21: Divide the surgical video into multiple video segments, and cut the video segments into frames to obtain a frame sequence.

[0097] S22: The frame sequence is input into a dual-head ViT visual encoder to obtain the surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head, comprising two lightweight ViT visual encoders, one for the instrument head and one for the anatomical head, used to enhance the representation capabilities of surgical instruments and anatomical structures, respectively. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task; it is the instrument feature extraction branch (i.e., the first ViT encoding branch), used to visually encode the input frame sequence and output visual features representing the surgical instruments. The anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task; it is the anatomical structure feature extraction branch (i.e., the second ViT encoding branch), used to visually encode the input frame sequence and output visual features representing the anatomical structures. The visual features output by the two lightweight ViT visual encoders can be further concatenated, weighted fused, or attention-fused to form a comprehensive visual representation for subsequent multimodal inference.

[0098] Specifically, such as Figure 4 As shown, the video clip is first segmented into frames. Assume the input video is segmented into a frame sequence. H, W, and 3 represent the three-dimensional dimensions of each frame, which are fed into a dual-head ViT visual encoder consisting of an "instrument head" and an "anatomical head" to obtain the surgical instrument features focused on surgical instruments and anatomical structures, respectively. and anatomical features :

[0099]

[0100] in, This is the feature extraction function for the instrument head. Let h, w, and be the feature extraction functions for the dissected head. These represent the three dimensions of the surgical instrument features: h, w, and These are the three dimensions of anatomical structural features.

[0101] S23: Perform matrix-level stitching of surgical instrument features and anatomical structure features to obtain the fused visual features corresponding to each frame.

[0102] Specifically, such as Figure 5 As shown, the two features are matrix-level concatenated in the channel dimension to obtain the fused visual features. :

[0103]

[0104] like Figure 4 As shown, the final visual token is generated by mapping the projector (projection MLP) to a dimension aligned with the language end.

[0105] S24: Input the question text into the word segmenter to obtain text features.

[0106] S25: Fuse the visual and text features corresponding to each frame in a visual-text cascade manner to obtain the visual-text features corresponding to each frame.

[0107] Specifically, the question text is processed by the Tokenizer to obtain a text token, and the two types of tokens are merged into a complete token sequence through sequential concatenation.

[0108] S26: Filter the visual-text features to obtain compact visual-text features. The visual-text features include multiple visual tokens and multiple text tokens.

[0109] Specifically, the original visual-text token often includes video backgrounds and irrelevant text, which can easily lead to redundant calculations and latency issues, such as... Figure 4 As shown, it is therefore necessary to set a representative token selector before feeding the visual-text token into the multimodal large model (LLM) to perform a "surgical knowledge-based slimming" of the input in order to improve inference speed and achieve real-time inference during surgery. Figure 6 As shown, S26 specifically includes:

[0110] S261: Analyze the similarity between each term in the surgical knowledge base and each visual token to obtain visual similarity, wherein the terms in the surgical knowledge base include surgical instrument terms, surgical action terms, and anatomical structure terms.

[0111] S262: Analyze the similarity between each term in the surgical knowledge base and each text token to obtain the text similarity.

[0112] Specifically, visual tokens and corresponding text tokens are mapped to a semantic space consistent with the surgical knowledge base, which contains only standard terms such as surgical instruments, surgical actions, and anatomical structures, and their synonyms. Then, the similarity between each visual token and these term entries is calculated. Specifically, for token vectors mapped to the unified semantic vector space... and terminology vectors in the surgical knowledge base The similarity is defined as:

[0113]

[0114] in, Represents the dot product of two vectors. and These represent the Euclidean norms of the corresponding vectors. Using the above formula, the similarity between each visual token or text token and each term in the surgical knowledge base can be calculated.

[0115] S263: Take the maximum visual similarity corresponding to each visual token as the final visual similarity of the visual token, and take the maximum text similarity corresponding to each text token as the final text similarity of the text token.

[0116] S264: Retain a first preset number of visual tokens in descending order of final visual similarity, and retain a second preset number of text tokens in descending order of final text similarity, while removing semantically repetitive nearest neighbor tokens to obtain compact visual-text features.

[0117] Specifically, K visual tokens and k text tokens are retained based on similarity from high to low, while semantically repetitive nearest neighbor tokens and tokens with low similarity are removed to form a compact token sequence that prioritizes coverage of the three elements: "instrument-action-anatomy". This effectively removes redundant and irrelevant tokens from the original visual-text features, significantly reducing the feature dimensionality and computational complexity input to the multimodal large model. This intelligent filtering mechanism based on the surgical knowledge base allows the model to focus more on core, domain-related information in the surgical video and question text, thereby improving the processing efficiency and question-answering accuracy of the multimodal large model in surgical video question-answering tasks. By removing semantically repetitive nearest neighbor tokens, the conciseness of the features is further optimized, avoiding information overload and ensuring the effectiveness and high quality of the input features.

[0118] S27: Input the compact visual-text features into a multimodal large model suitable for surgical video question answering, constructed using the training dataset construction method described above, to obtain the answer.

[0119] Specifically, a compact visual-text token is directly input into a supervised, fine-tuned multimodal large model to generate reliable answers through reasoning. This allows for obtaining the final answer for the current video segment and question with less computational overhead under equal computing power conditions, enabling real-time in-surgery reasoning.

[0120] In this embodiment, by employing refined feature extraction and optimized visual-text feature filtering in a dual-head ViT visual encoder, the temporal dynamic information in surgical videos is accurately captured, and the quality of feature representation is improved. This enhances the model's understanding of surgical procedures and improves question-answering accuracy. Specifically, this method achieves independent feature extraction of surgical instruments and anatomical structures through parallel processing of the instrument head and dissecting head, avoiding the feature confusion problem of traditional single-head encoders. Matrix-level splicing and sequential visual-text cascading fusion ensure the effective integration of multimodal information. The feature filtering mechanism further eliminates redundant information and highlights key semantic content. Through the above technical solutions, the shortcomings of existing technologies, such as insufficient temporal information processing mechanisms and imperfect automatic question type allocation, are effectively overcome. This enables the model to more accurately identify causal relationships between operational steps when processing real clinical surgical videos, significantly improving its generalization ability and reliability in complex question-answering tasks.

[0121] like Figure 7 As shown in the embodiment of this application, an intelligent reasoning system for surgical video question answering tasks is provided, comprising:

[0122] The video segmentation module 10 is used to divide the surgical video into multiple video segments and to cut the video segments into frames to obtain a frame sequence.

[0123] A dual-head ViT visual encoder 20 is used to receive and process frame sequences and output surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task, and the anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task.

[0124] The feature stitching module 30 is used to stitch together surgical instrument features and anatomical structure features at the matrix level to obtain the fused visual features corresponding to each frame.

[0125] The word segmenter 40 is used to receive and process the question text and output text features.

[0126] The feature concatenation module 50 is used to fuse the fused visual features and text features corresponding to each frame in a visual-text order concatenation manner to obtain the visual-text features corresponding to each frame.

[0127] Representative token selector 60 filters visual-text features to obtain compact visual-text features.

[0128] The inference module 70 is used to input compact visual-text features into a multimodal large model suitable for surgical video question answering, constructed using the training dataset construction method for the surgical video question answering task, to obtain the answer.

[0129] In this embodiment, the beneficial effects of the intelligent reasoning system for surgical video question answering tasks are similar to those of the intelligent reasoning method for surgical video question answering tasks described above, and will not be repeated here.

[0130] In summary, compared with existing technologies, it has the following beneficial effects:

[0131] This invention, through a design of "template-based scoring allocation—dual-head ViT decoupling and focusing—representative token selection," significantly improves the accuracy, real-time performance, and interpretability of video-level surgical question answering under the same computing power and memory constraints.

[0132] Using quantifiable instrument, anatomical, temporal, and quality features, the fragments are scored and automatically assigned questions, directly generating question types that fit the semantics of the fragments. Finally, templated question-answer pairs are constructed manually based on the question types, which reduces the cost of manual annotation and subjective bias, and improves sample coverage and utilization.

[0133] A dual-head ViT is used to learn instrument and anatomical semantics separately and then project them into visual tokens aligned with the language end after channel-level fusion. This enhances the focus on key entities and surgical scenes, suppresses background noise, and provides traceable dual semantic fusion visual features, making the generation process more robust and interpretable.

[0134] Before inputting into a large model, visual-text tokens are screened for similarity based on surgical knowledge. Only highly relevant, concise evidence sequences are retained to participate in cross-attention reasoning. This effectively compresses irrelevant computations, reduces latency, and maintains or improves the self-consistency of the answer with the relationship between terminology and instrument-action-anatomy. This enables highly reliable applications in intraoperative low-latency deployment, postoperative teaching, and quality control scenarios, and allows for continuous benefit without modifying the model structure when the knowledge base is updated.

[0135] An electronic device provided in this application includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, the training dataset construction method for surgical video question answering tasks as described above or the intelligent reasoning method for surgical video question answering tasks as described above.

[0136] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the training dataset construction method for surgical video question-answering tasks as described above, or the intelligent reasoning method for surgical video question-answering tasks as described above.

[0137] In this embodiment, the beneficial effects of the electronic device and the computer-readable storage medium are similar to those of the above-described method for constructing a training dataset for surgical video question answering tasks or the above-described intelligent reasoning method for surgical video question answering tasks, and will not be repeated here.

[0138] The present invention describes electronic devices that can serve as servers or clients of this application, which are examples of hardware devices that can be applied to various aspects of this application. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistant devices, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0139] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0140] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the separately described modules may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application according to actual needs. Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a training dataset for surgical video question-answering tasks, characterized in that, include: The surgical video is divided into multiple video segments, and the video segments are input into a feature extraction network set to obtain various visual features; Based on multiple visual features and a hierarchical weighted scoring mechanism, scores for counting, attribution, and time series categories are determined. Based on the scores for counting, attribution, and time series, and the preset adaptation weights corresponding to different question types, adaptation scores for various question types are obtained. When the difference between the fit scores of multiple question types is greater than or equal to the preset difference threshold, the question type with the largest fit score that is greater than the preset fit threshold is selected as the preferred question type. When the difference in fit scores among multiple question types is less than a preset difference threshold, the preferred question type is selected according to the preset priority order. Based on the preferred question type, question templates and answers are set for the corresponding video segments to obtain a training dataset. The training dataset includes multiple data elements, and each data element includes a video segment and a corresponding question-answer pair. The multimodal large model was fine-tuned under supervision using the training dataset to obtain a multimodal large model suitable for surgical video question answering; Among them, various visual features include instrument features, anatomical structure features, temporal features, and video quality features; The hierarchical weighted scoring mechanism includes: setting up a count-based evaluation layer, a category-based evaluation layer, and a time-series evaluation layer. Each evaluation layer sets up multiple different indicators, and the multiple indicators are weighted and summed to output the score. In the counting evaluation layer, based on the characteristics of the device, the obtained indicators include the number of device instances score, the number of different device types score, the target salience score, and the detection confidence score; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset counting indicator weights to obtain the counting score. In the classification evaluation layer, the indicators obtained based on instrument characteristics, anatomical structure characteristics, and video quality characteristics include instrument category confidence score, anatomical structure category confidence score, dominant target stability score, and video quality penalty item; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset classification indicator weights to obtain the classification score. In the temporal evaluation layer, the indicators obtained based on temporal characteristics include the number of stage / step changes score, action duration score, instrument-anatomical interaction intensity score, and temporal coherence score; after normalizing the values ​​of multiple indicators, they are weighted and summed according to the preset temporal indicator weights to obtain the temporal score.

2. The method for constructing a training dataset for surgical video question answering tasks as described in claim 1, characterized in that, The feature extraction network set includes an instrument detection network, an anatomical segmentation network, a temporal recognition network, and a video quality assessment network; The instrument detection network outputs video clips containing category labels and spatial locations of various surgical instruments, which are used to form instrument features; The anatomical segmentation network outputs segmentation masks and category labels for anatomical structures in video clips, which are used to form anatomical structure features. The probability distribution and boundaries of each surgical stage, step or action in the output video clip of the temporal recognition network are used to form temporal features; The video quality assessment network outputs a quality score reflecting sharpness, occlusion, and brightness stability, which is used to form video quality features.

3. The method for constructing a training dataset for surgical video question-answering tasks as described in claim 1, characterized in that, The question types include visual perception, temporal action, relational semantics, choice and judgment, and basic common sense.

4. The method for constructing a training dataset for surgical video question-answering tasks as described in claim 1, characterized in that, The preset priority order is: sequential action type, relational semantic type, visual perception type, selection judgment type, and basic common sense type.

5. An intelligent reasoning method for surgical video question-answering tasks, characterized in that, include: The surgical video was divided into multiple video segments, and the video segments were cut into frames to obtain a frame sequence; The frame sequence is input into the dual-head ViT visual encoder to obtain the surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task, and the anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task. The features of surgical instruments and anatomical structures are stitched together at the matrix level to obtain the fused visual features corresponding to each frame. The question text is input into a word segmenter to obtain text features; The visual and text features corresponding to each frame are fused in a visual-text order concatenation manner to obtain the visual-text features corresponding to each frame. By filtering the visual-text features, compact visual-text features are obtained; The compact visual-text features are input into a multimodal large model suitable for surgical video question answering, constructed using any one of the training dataset construction methods for surgical video question answering tasks described in 1-4, to obtain the answer.

6. The intelligent reasoning method for surgical video question answering tasks as described in claim 5, characterized in that, The visual-text features include multiple visual tokens and multiple text tokens; The process of filtering visual-text features to obtain compact visual-text features includes: Visual similarity is obtained by analyzing the similarity between each term in the surgical knowledge base and each visual token. The terms in the surgical knowledge base include surgical instrument terms, surgical action terms, and anatomical structure terms. The text similarity is obtained by analyzing the similarity between each term in the surgical knowledge base and each text token; The maximum visual similarity corresponding to each visual token is taken as the final visual similarity of the visual token, and the maximum text similarity corresponding to each text token is taken as the final text similarity of the text token. The first preset number of visual tokens are retained based on the final visual similarity from high to low, and the second preset number of text tokens are retained based on the final text similarity from high to low. At the same time, semantically duplicated neighboring tokens are removed to obtain compact visual-text features.

7. An intelligent reasoning system for surgical video question-answering tasks, characterized in that, include: The video segmentation module is used to divide the surgical video into multiple video segments and to cut the video segments into frames to obtain a frame sequence; A dual-head ViT visual encoder is used to receive and process frame sequences and output surgical instrument features and anatomical structure features corresponding to each frame. The dual-head ViT visual encoder includes an instrument head and an anatomical head. The instrument head is a lightweight ViT visual encoder trained using a surgical instrument dataset for a classification task, and the anatomical head is a lightweight ViT visual encoder trained using a surgical anatomy dataset for a classification task. The feature stitching module is used to stitch together surgical instrument features and anatomical structure features at the matrix level to obtain the fused visual features for each frame. A word segmenter is used to receive and process the question text and output text features; The feature concatenation module is used to fuse the fused visual features and text features corresponding to each frame in a visual-text order concatenation manner to obtain the visual-text features corresponding to each frame. A representative token selector filters visual-text features to obtain compact visual-text features; The inference module is used to input compact visual-text features into a multimodal large model suitable for surgical video question answering, constructed using any one of the training dataset construction methods for surgical video question answering tasks described in 1-4, to obtain the answer.

8. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the training dataset construction method for surgical video question answering tasks as described in any one of claims 1 to 4, or implement the intelligent reasoning method for surgical video question answering tasks as described in any one of claims 5 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the training dataset construction method for surgical video question answering tasks as described in any one of claims 1 to 4, or the intelligent reasoning method for surgical video question answering tasks as described in any one of claims 5 to 6.

Images