A system for surgical visual question localization and answering
Patent Information
- Application Number
- CN202510782015.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-12
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2045-06-12
AI Technical Summary
然而，这一问题仍然带来重大挑战
[0045]与现有技术相比，本发明的专家外科混合模型是一种通过混合专家(MoE)策略集成大小语言和视觉模型的新型架构，本发明的模型显著提高了手术视觉问题定位和回答的性能。本发明的模型通过引入LoRA-MoE模块进行微调，混合交叉注意力模块(MCAM)学习交叉模态特征，MoE融合模块进行交叉特征交互，MoE位置头和MoE回答头进行专家系统决策，与现有的先进模型相比，取得了更好的结果。实验结果表明，在相应的数据集上，本发明的模型优于其他模型，在准确性、F评分和mIoU方面有显著提高，验证了其在医疗手术机器人领域的巨大潜力。此外，相关研究证实了MoE相关模块的重要贡献，尤其是LoRA-MoE模块和MoE融合模块，这对模型性能有重大影响。这些结果证明了本发明的模型在增强手术视觉问答任务中的推理能力、准确性和视觉一致性方面的有效性，为个性化手术指导中更先进和自动化的解决方案铺平了道路。
Smart Images
Figure CN120671836B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotic surgery technology, and in particular to a system for locating and responding to surgical visual problems. Background Technology
[0002] Recent advances in Surgical Visual Question Localization and Response (SQLA) have demonstrated its significant potential for application in medical surgical robotics, addressing the urgent need for automated methods in personalized surgical guidance. However, this problem still presents significant challenges. For example, medical students and junior surgeons often rely on senior surgeons and experts to answer questions they encounter during surgical learning, but these experts are often busy with clinical and academic work and find it difficult to provide guidance. Existing deep learning-based Surgical Visual Question Response (VQLA) systems can only provide simple answers without offering locational information for the answers. Furthermore, research on visual-language (ViL) embeddings in this type of task is limited. Therefore, a system capable of providing visual question localization and response (VQLA) would be highly beneficial for medical students and junior surgeons learning and understanding surgical videos. Summary of the Invention
[0003] The purpose of this invention is to provide a system for locating and answering visual questions in surgery. This invention enhances reasoning ability, accuracy, and visual consistency in surgical visual question-answering tasks, exhibiting excellent question location and answering capabilities.
[0004] The technical solution of this invention is as follows: A system for locating and answering surgical visual problems, comprising constructing an expert surgical hybrid model composed of a large language visual model and a small language visual model; in the surgical hybrid model, the large language visual model is used to extract image features and extract text features respectively, and then the image features and text features are subjected to cross-modal feature extraction and fusion to output features, and then the output features are converted into text embeddings; the small language visual model is used to obtain the image features extracted by the large language visual model and the text embeddings output by the large language visual model, and then the image features and text embeddings are subjected to cross-modal feature extraction and fusion to output fused features, and finally the fused features are used to locate and answer surgical visual problems.
[0005] The aforementioned system for locating and answering surgical visual problems includes a large language visual model comprising a ViT network, a word segmenter, a hybrid cross-attention module, a MoE fusion module, and a MoE model. The ViT network is used to extract basic image features. The word segmenter tokenizes the input text and then extracts basic text features. The hybrid cross-attention module and the MoE fusion module sequentially extract and fuse cross-modal features to obtain output features. The MoE model is used to convert the output features into text embeddings.
[0006] The small language visual model includes a hybrid cross-attention module, a MoE fusion module, a decoder, a MoE location head, and a MoE response head. The hybrid cross-attention module and the MoE fusion module sequentially extract and fuse cross-modal features to obtain fused features. The decoder is used to receive the fused features and uses the MoE location head and the MoE response head to locate and answer surgical visual questions.
[0007] The aforementioned system for locating and answering surgical visual problems includes a hybrid cross-attention module consisting of three MoA layers: two layers for extracting image and text features, and one layer for feature interaction. Each layer includes a routing network G and N attention experts; where, for a query vector q... t The routing network G selects a subset of attention experts and assigns weights. The output of MoA is represented as:
[0008]
[0009] In the formula: i is the length of the routing network G, w i，t For normalized weights, E i The function for calculating the output of the attention expert is: K is the key feature input of the previous layer, V is the value feature input of the previous layer, and t is the routing time.
[0010] Among them, the routing network G calculates the normalized weight w. i，t First, use linear weights W g The routing probability p at routing time t is calculated using softmax. i :
[0011] p i，t =Softmax(q t ·Wg);
[0012] In the formula: W g Indicates linear weights;
[0013] Select the top k attention experts and renormalize their probabilities:
[0014]
[0015] In the formula: p j This represents the renormalized route probability;
[0016] Each attention expert uses a projection matrix W k W v , The weights are calculated as follows:
[0017]
[0018] In the formula: Attention weights; d h Let be the dimension of the projection matrix, and normalize it using the dimension. T represents a mathematical operation, which is to transform the rank.
[0019] The weighted sum of the values is:
[0020]
[0021] In the formula: o i，t For intermediate output;
[0022] The final output from the attention expert is:
[0023]
[0024] In the aforementioned system for locating and answering surgical visual problems, the MoE fusion module takes text and image features as input, processes them through an attention network, and learns the weights 'a' of an attention expert. i Each attention expert is obtained through a linear network and a tanh activation function; where, for text features and image features, feature fusion is performed using the following formula:
[0025]
[0026] In the formula: y is the final fused feature, bilinear is the bilinear operator; Expert(I) i For image features, Expert(T) i For text features.
[0027] The aforementioned system for locating and answering surgical visual problems includes a LoRA-MoE module in its MoE model. The LoRA-MoE module combines hybrid expert MoE with low-rank adapter LoRA, increasing parameters without increasing computational cost. Specifically, for input x, the LoRA-MoE module selects network G(x) to generate weights.
[0028] G(x) = [G(x)1, ..., G(x)] n ];
[0029] The hybrid expert MoE output is a weighted sum of the attention expert outputs:
[0030]
[0031] In the formula: E i (x) represents the output of the attention expert;
[0032] Since only k < n experts are active during the calculation process, sparsity is ensured through Top-k routing;
[0033] The low-rank adapter LoRA reduces the dimension using A(x) and restores the dimension using B(x):
[0034] A(x)=xA, B(x)=xB, LoRA(x)=xAB
[0035] In the formula: A(x) and B(x) are both computational functions. d represents the dimension of input A, and r represents the dimension of input A;
[0036] By integrating these modules by treating the low-rank adapter LoRA as an expert, the output is:
[0037]
[0038] Where: LoRA i (x) is the low-rank adapter LoRA function;
[0039] Finally, convolution, activation, normalization, and projection operations are performed to convert the output features into text embeddings.
[0040] x=Linear(Conv1D(HardSwish(CoordDWConv(x0))))+x0;
[0041] In the formula: Linear is a linear layer, Conv 1D is a one-dimensional convolutional layer, HardSwish is the Hard Swish activation function, CoordDWConv is a coordinate grouped convolution, and X0 is the input.
[0042] The aforementioned system for locating and answering surgical visual problems uses a MoE location head that employs a MoE network, linear projection layers, and sigmoid activation to model the coordinates of bounding boxes, and softmax activation to achieve classification.
[0043] In the aforementioned system for locating and answering surgical visual questions, the MoE answer head achieves answering through a MoE network and a linear projection layer.
[0044] The aforementioned system for locating and responding to surgical visual problems, wherein the MoE location head and MoE response head use the GIoU loss function L GIoU Perform bounding box regression; where the position head MoE uses cross-entropy loss L CE For location classification, the MoE answer head uses the L1 norm loss function for question answering.
[0045] Compared to existing technologies, the expert surgical hybrid model of this invention is a novel architecture that integrates large and small language and visual models through a hybrid expert (MoE) strategy. This model significantly improves the performance of surgical visual question localization and answering. The model is fine-tuned by introducing a LoRA-MoE module, learns cross-modal features using a hybrid cross-attention module (MCAM), performs cross-feature interaction using a MoE fusion module, and uses a MoE location head and a MoE answer head for expert system decision-making, achieving better results than existing state-of-the-art models. Experimental results show that on the corresponding dataset, the model of this invention outperforms other models, significantly improving accuracy, F-score, and mIoU, validating its great potential in the field of medical surgical robotics. Furthermore, related research confirms the important contributions of MoE-related modules, especially the LoRA-MoE module and the MoE fusion module, which have a significant impact on model performance. These results demonstrate the effectiveness of the model of this invention in enhancing reasoning ability, accuracy, and visual consistency in surgical visual question answering tasks, paving the way for more advanced and automated solutions in personalized surgical guidance. Attached Figure Description
[0046] Figure 1 This is a schematic diagram illustrating the principle of the expert surgical hybrid model of the present invention;
[0047] Figure 2 This is a schematic diagram illustrating the principle of the hybrid cross-attention module;
[0048] Figure 3 This is a schematic diagram of the principle of MoA;
[0049] Figure 4 This is a schematic diagram of the MoE fusion module.
[0050] Figure 5 This is a schematic diagram of the MoE position header and the MoE response header. Detailed Implementation
[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.
[0052] Example: A system for locating and responding to surgical visual problems, such as Figure 1As shown, the steps are as follows: Construct a Surgical Mixed of Experts (Surgical MoE) model consisting of a large language and vision model (LLVM) and a small language and vision model (SLVM); in the surgical mixed model, the LLVM is used to extract image features and text features respectively, and then cross-modal feature extraction and fusion are performed on the image features and text features to output features, which are then converted into text embeddings; the small language and vision model is used to obtain the image features extracted by the LLVM and the text embeddings of the LLVM, and then cross-modal feature extraction and fusion are performed on the image features and text embeddings to output fused features, which are finally used to locate and answer surgical vision questions.
[0053] In this embodiment, the large language vision model includes a ViT network, a tokenizer, a hybrid cross-attention module, a MoE fusion module, and a MoE model. The ViT network is used to extract basic image features. The ViT network (VisionTransformer) is a model based on the Transformer architecture, mainly used for image recognition and classification tasks. The ViT network segments the image into multiple patches, then linearly embeds these patches into fixed-dimensional vectors, encodes them through a series of Transformer layers, and finally classifies them through a multilayer perceptron (MLP) head. The tokenizer tokenizes the input text and then extracts basic text features using a tokenizer. The tokenizer is a key tool in natural language processing (NLP), whose main function is to segment the input text data into smaller, easier-to-process units, usually words, subwords, or characters. In NLP models, the tokenizer is a crucial step in text preprocessing, directly impacting the model's input format and performance. The Mixture of Cross-Attention Module (MCAM) and MoE Fusion Module sequentially extract and fuse cross-modal features to obtain output features. The MoE model is used to convert the output features into text embeddings. The MoE model is Qwen1.5-MoE-A2.7B, an open-source MoE model from the Tongyi Qianwen team.
[0054] The small language visual model includes a hybrid cross-attention module, a MoE fusion module, a decoder, a MoE location head, and a MoE response head. The hybrid cross-attention module and the MoE fusion module sequentially extract and fuse cross-modal features to obtain fused features. The decoder (DeiT decoder) is used to receive the fused features and uses the MoE location head and the MoE response head to locate and answer surgical visual questions.
[0055] In this embodiment, as Figure 2 As shown, the hybrid cross-attention module consists of three layers of MoA (Mixture of Attention), two layers for extracting image and text features, and one layer for feature interaction. Each layer includes a routing network G and N attention experts; where, for the query vector q t The routing network G selects a subset of attention experts and assigns weights to them to generate the output of MoA:
[0056]
[0057] In the formula: i is the length of the routing network G, w i，t For normalized weights, E i The function for calculating the output of the attention expert is: K is the key feature input of the previous layer, V is the value feature input of the previous layer, and t is the routing time.
[0058] like Figure 3 As shown, the routing network G calculates the normalized weight w. i，t First, use linear weights W g The routing probability p at routing time t is calculated using softmax. i :
[0059] p i，t =Softmax(q t ·Wg);
[0060] In the formula: W g Indicates linear weights;
[0061] Then select the top k attention experts and renormalize their probabilities:
[0062]
[0063] In the formula: p j This represents the renormalized route probability;
[0064] Each attention expert uses a projection matrix W k W v , The weights are calculated as follows:
[0065]
[0066] In the formula: Attention weights; d h Let be the dimension of the projection matrix, and normalize it using the dimension. T represents a mathematical operation, which is to transform the rank.
[0067] The weighted sum of the values is:
[0068]
[0069] In the formula: o i，t This is the intermediate output; the final attention expert output is:
[0070]
[0071] Shared W k and W v Allows pre-calculation of KW k and VW v This reduces computational and memory overhead.
[0072] In this embodiment, as Figure 4 As shown, the MoE fusion module takes text features and image features as input, processes them through an attention network, and learns the weights 'a' of the attention expert. i Each attention expert is obtained through a linear network and a tanh activation function; where, for text features and image features, feature fusion is performed using the following formula:
[0073]
[0074] In the formula: y is the final fused feature, bilinear is the bilinear operator; Expert(I) i For image features, Expert(T) i For text features.
[0075] In this embodiment, the MoE model includes a LoRA-MoE module, which combines hybrid expert MoE with low-rank adapter LoRA to increase parameters without increasing computational cost. Specifically, for input x, the LoRA-MoE module selects network G(x) to generate weights.
[0076] G(x)=[G(x)1,...,G(x)n];
[0077] The hybrid expert MoE output is a weighted sum of the attention expert outputs:
[0078]
[0079] In the formula: E i (x) represents the output of the attention expert;
[0080] Since only k < n experts are active during the calculation process, sparsity is ensured through Top-k routing;
[0081] The low-rank adapter LoRA reduces the dimension using A(x) and restores the dimension using B(x):
[0082] A(x)=xA, B(x)=xB, LoRA(x)=xAB
[0083] In the formula: A(x) and B(x) are both computational functions. d represents the dimension of input A, and r represents the dimension of input A;
[0084] By integrating these modules by treating the low-rank adapter LoRA as an expert, the output is:
[0085]
[0086] Where: LoRA i (x) is the low-rank adapter LoRA function;
[0087] Finally, convolution, activation, normalization, and projection operations are performed to convert the output features into text embeddings.
[0088] x=Linear(Conv1D(HardSwish(CoordDWConv(x0))))+x0;
[0089] In the formula: Linear is a linear layer, Conv 1D is a one-dimensional convolutional layer, HardSwish is the Hard Swish activation function, CoordDWConv is a coordinate grouped convolution, and x0 is the input.
[0090] In this embodiment, as Figure 5 As shown, the MoE location head uses a MoE network, a linear projection layer, and sigmoid activation to model the bounding box coordinates, and uses softmax activation for classification. The MoE response head uses a MoE network and a linear projection layer to answer questions. Both the MoE location head and the MoE response head use the GIoU loss function L. GIoU Perform bounding box regression; where the position head MoE uses cross-entropy loss L CE For location classification, the MoE answer head uses the L1 norm loss function for question answering.
[0091] Based on the above model, this embodiment conducts experiments on surgical visual question localization and response. The EndoVis-18-VQLA dataset contains 14 robotic surgical videos from the MICCAI 2018 challenge. VQLA annotations provide instrument location, bounding boxes, and question-answer pairs for each surgical step.
[0092] The EndoVis-17-VQLA dataset includes 97 frames from surgical videos in the MICCAI 2017 challenge, with annotations for surgical actions and instrument positions. It serves as an external validation set to evaluate model generalization across various surgical scenarios.
[0093] The EndoVis dialogue dataset combines the EndoVis-18-VQLA and EndoVis-17-VQLA datasets, along with 19,020 training and 2,151 test image QA pairs. Through various QA tasks, it helps improve the model's understanding of surgical scenarios. In experiments, the Qwen1.5-MoE-A2.7B model based on MoE-LoRA was fine-tuned using the EndoVis dialogue dataset, and then the surgical MoE of this invention was further trained using the training set of the EndoVis-18-VQLA dataset. The model was evaluated on the EndoVis-18-VQLA dataset and the test set of the entire EndoVis-17-VQLA dataset.
[0094] The performance comparison of different models on the ENDOVIS-18-VQLA and ENDOVIS-17-VQLA datasets is shown in Table 1:
[0095]
[0096] Table 1
[0097] The experiment was implemented using the Python-PyTorch framework and conducted on an nvidia10a100-gpu server. The epochs, batch size, and learning rate were set to 200, 32, and 1×10⁻⁶, respectively. -5The evaluation metrics used in the experiments included accuracy, F-score, and mean IoU (mIoU) on the joint datasets. As shown in Table 1, the experimental results demonstrate that the Surgical MOE of this invention performs exceptionally well on the EndoVis-18-VQLA and EndoVis-17-VQLA datasets. On the EndoVis-18-VQLA dataset, the Surgical MOE of this invention outperforms all other models in terms of accuracy (0.7086), F-score (0.3790), and mIoU (0.8552), surpassing models such as GVLE LViT, Surgical VQLA++, and Surgical LVLM. Furthermore, on the EndoVis-17-VQLA dataset, the Surgical MOE of this invention also achieves impressive results with mIoU (0.7960), outperforming other competing models. These experimental results indicate that the Surgical MOE significantly improves the reasoning ability, accuracy, and visual consistency of the surgical visual question answering task, demonstrating a clear advantage over existing state-of-the-art methods and showcasing its strong potential.
[0098] Furthermore, to verify the effectiveness of the MoE-related modules, experiments were conducted by removing the MoE head, MoE fusion, MoA, and MoE LoRA from the network. Removing the MoE head means replacing the MoE structure in the MoE head with a simple fully connected network. Removing the MoE fusion means replacing the entire MoE fusion module with a simple addition operation. Removing the MoA means directly replacing the MoA in the MCAM with self-attention. Finally, removing the MoE LoRA means not fine-tuning the Qwen1.5-MoE-A2.7B model. The results are shown in Table 2.
[0099]
[0100] Table 2
[0101] Table 2 shows the effectiveness of all MoE-related modules, especially MoE LoRA and MoE fusion, which have the most significant impact on the performance of surgical MoE.
[0102] In summary, the expert surgical hybrid model of this invention is a novel architecture that integrates large and small language and visual models through a hybrid expert (MoE) strategy. This model significantly improves the performance of surgical visual question localization and answering. The model achieves better results compared to existing state-of-the-art models by introducing a LoRA-MoE module for fine-tuning, a hybrid cross-attention module (MCAM) for learning cross-modal features, a MoE fusion module for cross-feature interaction, and a MoE location head and MoE answer head for expert system decision-making. Experimental results show that on the corresponding dataset, the model of this invention outperforms other models, significantly improving accuracy, F-score, and mIoU, validating its great potential in the field of medical surgical robotics. Furthermore, related research confirms the important contributions of MoE-related modules, especially the LoRA-MoE module and the MoE fusion module, which have a significant impact on model performance. These results demonstrate the effectiveness of the model of this invention in enhancing reasoning ability, accuracy, and visual consistency in surgical visual question answering tasks, paving the way for more advanced and automated solutions in personalized surgical guidance.
Claims
1. A system for locating and responding to surgical visual problems, characterized in that: This includes an expert surgical hybrid model consisting of a large language vision model and a small language vision model; In the surgical hybrid model, the large language vision model is used to extract image features and text features respectively. Then, cross-modal feature extraction and fusion are performed on the image features and text features to output features. The output features are then converted into text embeddings. The small language vision model is used to obtain the image features extracted by the large language vision model and the text embeddings output by the large language vision model. Then, cross-modal feature extraction and fusion are performed on the image features and text embeddings to output fused features. Finally, the fused features are used to locate and answer surgical visual questions. The large language vision model includes a ViT network, a word segmenter, a hybrid cross-attention module, a MoE fusion module, and a MoE model. The ViT network is used to extract basic image features. The word segmenter tokenizes the input text and then extracts basic text features. The hybrid cross-attention module and the MoE fusion module sequentially extract and fuse cross-modal features to obtain output features. The MoE model is used to convert the output features into text embeddings. The small language visual model includes a hybrid cross-attention module, a MoE fusion module, a decoder, a MoE location head, and a MoE response head. The hybrid cross-attention module and the MoE fusion module sequentially extract and fuse cross-modal features to obtain fused features. The decoder is used to receive the fused features and uses the MoE location head and the MoE response head to locate and answer surgical visual questions. The MoE fusion module takes text and image features as input, processes them through an attention network, and learns the weights of an attention expert. Each attention expert is obtained through a linear network and a tanh activation function; where, for text features and image features, feature fusion is performed using the following formula: In the formula: It is a characteristic of the final fusion. It is a bilinear operator; For image features, For text features.
2. The system for locating and responding to surgical visual problems according to claim 1, characterized in that: The hybrid cross-attention module consists of three layers of MoA, two of which are used to extract image and text features, and one layer is used for feature interaction. Each layer includes a routing network G and N attention experts. For query vectors The routing network G selects a subset of attention experts and assigns weights to them. The output of MoA is represented as: ； In the formula: Let G be the length of the routing network. To normalize the weights, To compute the function output by the attention expert, The key feature of the previous layer is used as input. The value feature is input to the previous layer. For routing time; In this context, the routing network G calculates the normalized weights. First, use linear weights. Calculating routing time with softmax Routing probability at time : ； In the formula: Indicates linear weights; Select the top k attention experts and renormalize their probabilities: ； In the formula: This represents the renormalized route probability; Each attention expert uses a projection matrix , , , The attention weights are calculated as follows: ； In the formula: Attention weights; Let be the dimension of the projection matrix, and normalization be performed using this dimension. This represents a mathematical operation, specifically the transformation of rank. The weighted sum of the values is: ； In the formula: For intermediate output; The final output from the attention expert is: 。 3. The system for locating and responding to surgical visual problems according to claim 1, characterized in that: The MoE model includes a LoRA-MoE module, which combines hybrid expert MoE with low-rank adapter LoRA to increase parameters without increasing computational cost; the LoRA-MoE module, for input... Select network Generate weights: ； The hybrid expert MoE output is a weighted sum of the attention expert outputs: ； In the formula: Output for attention experts; Since only k≪n experts are active during the calculation process, sparsity is ensured through Top-k routing; Low-rank adapter LoRA via Reduce dimensionality and use Recovery Dimension: ； In the formula: and All are computational functions. , , For the dimension of input A, Let A be the low-rank dimension of the output. By integrating these modules by treating the low-rank adapter LoRA as an expert, the output is: ； In the formula: LoRA function for low-rank adapter; Finally, convolution, activation, normalization, and projection operations are performed to convert the output features into text embeddings. ； In the formula: For linear layers, It is a one-dimensional convolutional layer. For Hard Swish activation function, Grouping convolutions by coordinates For input.
4. The system for locating and responding to surgical visual problems according to claim 1, characterized in that: The MoE location head uses a MoE network, a linear projection layer, and sigmoid activation to model the coordinates of the bounding box, and uses softmax activation to achieve classification.
5. The system for locating and responding to surgical visual problems according to claim 1, characterized in that: The MoE response head implements the answering function through the MoE network and a linear projection layer.
6. The system for locating and responding to surgical visual problems according to claim 5, characterized in that: The MoE location header and MoE response header use the GIoU loss function LGIoU for bounding box regression; the location header MoE uses cross-entropy loss LCE for localization and classification, and the MoE response header uses the L1 norm loss function for question answering.