A visual language reasoning method for robot-assisted surgery

Abstract

The application belongs to the technical field of medical artificial intelligence, computer vision and robot-assisted surgery, and particularly relates to a visual language reasoning method for robot-assisted surgery, comprising the following steps: dividing an intraoperative endoscope image and a corresponding natural language question into multiple tokens respectively and performing joint coding, fusing to obtain enhanced visual language features, and inputting the enhanced visual language features into a Mamba sequence modeling module based on a state space model to model the text tokens and the visual tokens; obtaining global question semantics according to enhanced text token features obtained after Mamba sequence modeling, calculating attention weights between the global question semantics and each visual token, selecting visual tokens corresponding to top-k attention weights as question-related local visual regions, constructing visual memory token representation, and further constructing a compact visual language memory, and inputting the compact visual language memory into a lightweight text decoder to generate a text answer.

Description

Technical Field

[0001] This invention belongs to the fields of medical artificial intelligence, computer vision and robot-assisted surgery technology, and specifically relates to a visual language reasoning method for robot-assisted surgery. Background Technology

[0002] With the rapid development of robot-assisted surgery, minimally invasive endoscopic surgery, and artificial intelligence technologies, intelligent perception and semantic understanding of intraoperative scenes have become important research directions for intelligent surgical systems. Intraoperative endoscopic images contain crucial information such as surgical instrument types, instrument status, tissue structures, operational behaviors, and spatial relationships between instruments and tissues. Visual question answering technology can combine intraoperative images with natural language questions to generate corresponding text answers, supporting applications such as intraoperative navigation, risk warning, intelligent decision support, human-computer interaction, and surgical teaching. However, compared with visual question answering of ordinary medical images, intraoperative visual question answering faces complex interferences such as dynamic changes in the surgical scene, instrument occlusion, tissue deformation, reflection, smoke, bleeding, and limited field of view, placing higher demands on the model's ability to perceive local details, understand spatial relationships, and perform real-time reasoning.

[0003] Existing surgical visual question answering methods largely rely on global visual features or general visual-language fusion mechanisms, making it difficult to fully focus on key local regions relevant to the current question. This can easily lead to the loss of information regarding instrument status, tissue boundaries, and instrument-tissue interactions. Furthermore, while Transformer-based visual-language models possess strong cross-modal modeling capabilities, they suffer from high computational complexity when processing large numbers of visual and text tokens, resulting in high memory consumption and inference latency, which hinders real-time deployment in robot-assisted surgery scenarios. In addition, existing methods typically lack explicit modeling of the surgical area's spatial structure, making it difficult to effectively utilize prior spatial information such as instrument positions, tissue layouts, and local regional relationships. Therefore, there is an urgent need to propose an intraoperative visual question answering method that can balance the preservation of question-related local visual information, spatial structure modeling, and lightweight inference capabilities. Summary of the Invention

[0004] In view of this, the present invention provides a visual language reasoning method for robot-assisted surgery. By using question-guided visual region selection, spatial structure modeling, and lightweight visual language fusion, it achieves efficient visual semantic reasoning and text answer generation in complex intraoperative scenarios, thereby solving the problems of insufficient attention to local key areas, inadequate utilization of spatial structure information, and large computational load of model reasoning in existing intraoperative visual question answering methods.

[0005] The technical solution for implementing the present invention is as follows: In a first aspect, the present invention provides a visual language reasoning method for robot-assisted surgery, comprising: Visual-language joint encoding: Intraoperative endoscopic images and corresponding natural language questions are divided into multiple visual patch tokens and multiple text tokens, and then jointly encoded. The hidden features of the last and penultimate layers of the encoder are extracted and fused to obtain enhanced visual-language features. ; Lightweight visual language sequence modeling: fusing to obtain enhanced visual language features Further segmentation into text features With visual features Input the Mamba sequence modeling module based on the state space model to model text tokens and visual tokens; Question-guided visual token filtering: Enhanced text token features obtained by modeling Mamba sequences. Obtain the global question semantics, calculate the attention weight between the global question semantics and each visual token, and select the visual tokens corresponding to the top-k attention weights as the question-related local visual regions; Spatial anchor point construction: weighted visual features Global visual features The top-k visual tokens and spatial anchor tokens are fused to construct a visual memory token representation. ; Lightweight text answer generation: Constructing a compact visual-linguistic memory by combining enhanced text tokens with visual memory tokens. Compact visual language memory Input a lightweight text decoder to generate a text answer.

[0006] Optionally, the present invention utilizes the VisualBERT-ResMLP encoder to jointly encode the question token and the visual token.

[0007] Optionally, the present invention describes extracting the hidden features of the last layer of the encoder. With the penultimate hidden feature And by fusing them, we obtain enhanced visual language features: , in, Indicates feature splicing, and These represent the learnable weights and bias parameters, respectively.

[0008] Optionally, the lightweight visual language sequence modeling of the present invention specifically includes: The fusion will result in enhanced visual language features. Further segmentation into text features With visual features Sequence dependency modeling is performed using the Mamba sequence modeling module based on the state-space model. , , in, and These represent problem features and visual features, respectively. and Represents the linear mapping parameters. This indicates spatial location embedding, and Mamba(·) represents sequence modeling operation based on the state-space model.

[0009] Optionally, the specific process of filtering the question-guided visual token described in this invention is as follows: First, a global problem semantic representation is obtained through average pooling: , Secondly, calculate the relevance score between the question semantics and the visual token. and corresponding attention weights : , , in, and Indicates learnable parameters; Finally, based on the relevance scores, the top-k visual tokens are selected as the question-related local visual regions, denoted as... .

[0010] Optionally, the compact visual language memory of the present invention is constructed as follows: Extract the center region token, top-left region token, and bottom-right region token from the visual token grid as spatial anchor tokens, denoted as... : Visual memory representation is constructed by combining weighted visual features, global visual features, top-k visual tokens, and spatial anchor tokens. Weighted visual features for: .

[0011] Optionally, the specific process of generating lightweight text answers according to the present invention is as follows: Enhance problem features using visual memory: , in, (·) indicates layer normalization operation. This represents learnable gating parameters, and ⊙ represents element-wise multiplication; ultimately, a compact visual-linguistic memory is constructed. for:

[0012] Compact visual language memory The lightweight decoder generates the text answer. During the decoding phase, cross-modal interaction only occurs at the first layer, and its output probability is:

[0013] in, Indicates the current decoding status. and Indicates the output layer parameters; The output probability represents, given the intraoperative image, the natural language question, and the generated answer prefix, the probability of the first... Each decoding step calculates the probability distribution of candidate answer tokens; according to a preset decoding strategy, the current answer token is determined from the probability distribution and added to the prefix of the generated answer. This process is iterated until an end marker is generated, thereby obtaining the complete text answer. Thus, after inputting intraoperative images and natural language questions, the model can directly generate the corresponding text answer, achieving intraoperative visual semantic reasoning.

[0014] Optionally, the visual language question-answering model for implementing the visual language reasoning method of the present invention includes a visual language joint encoding module, a multi-layer feature fusion module, a Mamba sequence modeling module, a question-guided visual token filtering module, a spatial anchor construction module, a compact visual language memory construction module, and a text answer generation module; during the training phase, the learnable parameters involved in the above modules are optimized end-to-end using a negative log-likelihood loss function. , in, Indicates the length of the answer sequence.

[0015] Beneficial effects: First, by constructing a question-guided visual token filtering mechanism, this invention can adaptively retain the local visual regions most relevant to the current question based on the natural language question, thereby effectively enhancing the model's ability to perceive key local information such as instrument status, tissue boundaries, and instrument-tissue interaction, and improving the accuracy of intraoperative visual question answering.

[0016] Second, this invention introduces a lightweight sequence modeling mechanism based on Mamba, which uses a state-space model to replace the traditional high-complexity self-attention calculation. While reducing the number of model parameters and computational overhead, it realizes long-distance dependency modeling between visual tokens and text tokens, thereby improving the visual language reasoning ability in complex intraoperative scenarios.

[0017] Third, by constructing spatial anchor tokens, this invention explicitly models the spatial structure information of the central and boundary areas of the surgical area, thereby enhancing the model's ability to understand the spatial positional relationships of instruments, the organizational layout relationships, and the structural information of the surgical area, and improving the performance of spatial semantic reasoning.

[0018] Fourth, this invention constructs a compact visual language memory representation by fusing weighted visual features, global visual features, top-k local visual tokens, and spatial anchor tokens. This reduces redundant visual tokens participating in cross-modal computation while retaining key visual information, thereby effectively reducing inference complexity and improving model running efficiency.

[0019] Fifth, in the decoding stage, this invention only performs cross-modal attention interaction in the first layer, and uses a lightweight sequence modeling method for text generation in subsequent decoding layers. Therefore, it can further reduce the memory usage and inference latency caused by repeated cross-modal computation, and meet the real-time intelligent question and answer application requirements in robot-assisted surgery scenarios.

[0020] This invention can be applied to scenarios such as surgical area scene understanding, intraoperative intelligent assistance, human-computer interaction, and surgical navigation in robot-assisted surgery. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the network model structure of the present invention; Detailed Implementation It should be noted that, in the absence of conflict, the following embodiments and the technical features in the embodiments can be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It should be understood that the aspects described herein can be implemented in various forms, and the specific structures and functions described herein are only for illustrating the technical solutions of the present invention, and not for limiting the scope of protection of the present invention. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently or in combination with other aspects. For example, any number of technical solutions described herein can be used to implement the corresponding device or method, and adjustments and extensions can be made based on the technical solutions described herein by combining other structures or functions, all of which fall within the scope of protection of the present invention.

[0024] like Figure 1 As shown, this embodiment of the invention provides an intraoperative visual question answering method based on spatial memory enhancement and Mamba fusion. Through question-guided visual token filtering, spatial structure modeling, and lightweight visual language fusion, it achieves efficient visual semantic reasoning and text answer generation in complex intraoperative scenarios.

[0025] Visual-Language Joint Encoding Method Based on VisualBERT-ResMLP In this embodiment, an intraoperative endoscopic image I and a corresponding natural language question Q are first acquired. The intraoperative image I originates from a video stream of robot-assisted surgery or laparoscopic surgery and contains information such as surgical instruments, tissue structures, and the spatial layout of the surgical area. B represents the batch size, and H and W represent the image height and width, respectively. The problem Q includes natural language processing tasks such as instrument status recognition, operational behavior recognition, instrument positioning, and surgical stage determination.

[0026] First, the input image is patched, and visual features are extracted using a visual encoding network to form a sequence of visual tokens:

[0027] in, This indicates the number of visual tokens. Simultaneously, natural language processing is performed on the tokens to obtain a text token sequence:

[0028] in, Indicates the number of text tokens.

[0029] Subsequently, the visual token and text token are input into the VisualBERT-ResMLP joint encoder for cross-modal feature interaction, resulting in a joint visual-language feature representation:

[0030] in, , Indicates batch size. This indicates the total number of tokens. Indicates the feature dimension.

[0031] Furthermore, to enhance the semantic information representation capability at different levels, this embodiment fuses the hidden features of the last layer and the penultimate layer of the encoder:

[0032] in, This indicates a feature concatenation operation. and These represent the learnable weight parameters and bias parameters, respectively. This approach enables joint modeling of intraoperative visual information and natural language semantic information, thereby enhancing the model's cross-modal understanding of complex surgical scenarios.

[0033] A lightweight visual language sequence modeling method based on Mamba In this embodiment, the fused visual language features Further segmentation into text features With visual features This is then mapped to a low-dimensional fusion space. To reduce the high complexity of traditional Transformer self-attention computation, this embodiment introduces a Mamba sequence modeling module based on a state-space model to perform lightweight modeling of the long-distance dependency relationship between visual tokens and text tokens. Specifically, the modeling processes for text tokens and visual tokens are represented as follows:

[0034] in, and Represents the linear mapping parameters. This indicates the spatial embedding of the visual token. This represents the state-space sequence modeling operation. and These represent the enhanced text token features and enhanced visual token features obtained after low-dimensional linear mapping and corresponding Mamba branch sequence modeling, respectively. This approach enables efficient sequence dependency modeling between visual and text tokens while reducing the number of model parameters and computational complexity, thereby improving visual language reasoning capabilities in complex intraoperative scenarios.

[0035] Question-guided visual token filtering method In this embodiment, to enhance the model's ability to focus on problem-related local areas, a problem-guided visual token filtering mechanism is constructed.

[0036] First, a global problem semantic representation is obtained through average pooling:

[0037] Subsequently, the relevance score between the question semantics and the visual token is calculated:

[0038] in, Indicates the first The relevance score of each visual token. Represents enhanced visual feature sequences The first in Each visual token is obtained by fusing visual features, performing linear mapping, adding spatial location embedding, and modeling using visual Mamba. This represents the corresponding attention weight. and This represents the learnable parameters.

[0039] Next, based on the relevance score, visual tokens are weighted by attention. Sort by size and select the top-k visual tokens as the relevant local visual regions for the problem:

[0040] in, This represents the first visual token selected from the enhanced visual feature sequence after sorting by attention weight from largest to smallest. The j-th element is the j-th enhanced visual token (j=1,…,k) after sorting by attention weights.

[0041] In this way, the model can focus on retaining the local visual information most relevant to the current problem, thereby improving the ability to recognize instrument status, instrument movement, and local tissue structure.

[0042] Compact Visual Language Memory Construction Method In this embodiment, to enhance the model's ability to understand the spatial structure information of the surgical area, multiple spatial anchor tokens are extracted from the visual token grid. Specifically, the central region token, the upper left region token, and the lower right region token are selected from the visual token grid to construct a set of spatial anchor tokens.

[0043] in, Represents the central area token. This represents the token in the top left area. This represents the token in the bottom right area.

[0044] Furthermore, weighted visual features Global visual features The top-k visual tokens and spatial anchor tokens are fused to construct a visual memory token representation. :

[0045]

[0046] in, This represents global visual features. The spatial anchor point modeling method described above enhances the model's ability to understand the spatial relationships between instruments, the layout of tissues, and the overall structural information of the surgical area.

[0047] Lightweight text answer generation method In this embodiment, the enhanced text token and the visual memory token are combined to construct a compact visual-linguistic memory. :

[0048] in, Presentation layer normalization operation, This indicates learnable gating parameters.

[0049] Subsequently, compact visual language memory Input a lightweight text decoder to generate a text answer.

[0050] In this embodiment, cross-modal attention interaction is performed only in the first layer of the decoder. Subsequent decoding layers use a lightweight sequence modeling approach based on Mamba to generate answers, thereby reducing memory usage and inference latency caused by repeated cross-modal attention computations. The model output probability is represented as:

[0051] in, Indicates the current decoding status. and This represents the output layer parameters.

[0052] During the training phase, the negative log-likelihood loss function is used for optimization:

[0053] During the reasoning phase, only intraoperative images and natural language questions need to be input to output the corresponding text answers, without the need for additional post-processing modules. Therefore, it can meet the real-time visual question answering task requirements in robot-assisted surgery scenarios.

[0054] Secondly, the present invention also provides an intraoperative visual question-answering system based on spatial memory enhancement and Mamba fusion, comprising: an image acquisition module, a visual-language joint encoding module, a multi-layer feature fusion module, a Mamba sequence modeling module, a question-guided visual token filtering module, a spatial anchor construction module, a compact visual-language memory construction module, and a text answer generation module.

[0055] Experimental verification To verify the effectiveness of the method described in this invention, experiments were conducted on the EndoVis-18-VQA dataset. This dataset is constructed from robot-assisted surgical video sequences and includes intraoperative endoscopic images, natural language questions, and corresponding text answers, which can be used to evaluate the answer generation ability of intraoperative visual question answering models. The experiments used BLEU-3, BLEU-4, CIDEr, and METEOR as evaluation metrics. BLEU measures the degree of n-gram overlap between the generated answer and the reference answer; CIDEr measures consensus-based semantic consistency; and METEOR measures word-level matching and sentence-level consistency. Higher values ​​indicate better model generation performance.

[0056] The experimental results are shown in the table below. Compared with the methods such as MedFuse, VisualBert + TD, and VisualBert ResMLP + TD, the method of this invention achieves better results in BLEU-3, BLEU-4, CIDEr, and METEOR metrics. This indicates that the proposed question-guided visual token selection, Mamba lightweight sequence modeling, and spatial memory enhancement mechanism can effectively improve the text generation quality in intraoperative visual question answering tasks.

[0057] Table 1 Comparison of results from different methods on the public dataset Endovis-18-VQA

[0058] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual-language reasoning method for robot-assisted surgery, characterized in that, include: Visual-language joint encoding: Intraoperative endoscopic images and corresponding natural language questions are divided into multiple visual patch tokens and multiple text tokens, and then jointly encoded. The hidden features of the last and penultimate layers of the encoder are extracted and fused to obtain enhanced visual-language features. ; Lightweight visual language sequence modeling: fusing to obtain enhanced visual language features Further segmentation into text features With visual features Input the Mamba sequence modeling module based on the state-space model to model text tokens and visual tokens; Question-guided visual token filtering: Enhanced text token features obtained by modeling Mamba sequences. Obtain the global question semantics, calculate the attention weight between the global question semantics and each visual token, and select the visual tokens corresponding to the top-k attention weights as question-related local visual regions; Spatial anchor point construction: weighted visual features Global visual features The top-k visual tokens and spatial anchor tokens are fused to construct a visual memory token representation. ; Lightweight text answer generation: Constructing a compact visual-linguistic memory by combining enhanced text tokens with visual memory tokens. Compact visual language memory Input a lightweight text decoder to generate a text answer.

2. The visual language reasoning method for robot-assisted surgery according to claim 1, characterized in that, The VisualBERT-ResMLP encoder is used to jointly encode the question token and the visual token.

3. The visual language reasoning method for robot-assisted surgery according to claim 1, characterized in that, The extraction of the last layer of hidden features from the encoder With the penultimate hidden feature And by fusing them, we obtain enhanced visual language features: ， in, Indicates feature splicing, and These represent the learnable weights and bias parameters, respectively.

4. The visual language reasoning method for robot-assisted surgery according to claim 3, characterized in that, The lightweight visual language sequence modeling specifically involves: The fusion will result in enhanced visual language features. Further segmentation into text features With visual features Sequence dependency modeling is performed using the Mamba sequence modeling module based on the state-space model. ， ， in, and These represent problem features and visual features, respectively. and Represents the linear mapping parameters. This indicates spatial location embedding, and Mamba(·) represents a sequence modeling operation based on a state-space model. and These represent the enhanced text token features and enhanced visual token features obtained after low-dimensional linear mapping and corresponding Mamba branch sequence modeling, respectively.

5. The visual-language reasoning method for robot-assisted surgery according to claim 4, characterized in that, The specific process for selecting visual tokens guided by the question is as follows: First, a global problem semantic representation is obtained through average pooling: ， Secondly, calculate the relevance score between the question semantics and the visual token. and corresponding attention weights : ， ， in, and Indicates learnable parameters; Finally, based on the relevance scores, the top-k visual tokens are selected as the question-related local visual regions, denoted as... .

6. The visual language reasoning method for robot-assisted surgery according to claim 5, characterized in that, The compact visual language memory is constructed as follows: Extract the center region token, top-left region token, and bottom-right region token from the visual token grid as spatial anchor tokens, denoted as... : Visual memory representation is constructed by combining weighted visual features, global visual features, top-k visual tokens, and spatial anchor tokens. Weighted visual features for: 。 7. The visual language reasoning method for robot-assisted surgery according to claim 6, characterized in that, The specific process for generating the lightweight text answer is as follows: Enhance problem features using visual memory: ， in, (·) indicates layer normalization operation. This represents learnable gating parameters, and ⊙ represents element-wise multiplication; ultimately, a compact visual-linguistic memory is constructed. for: Compact visual language memory The lightweight decoder generates the text answer. During the decoding phase, cross-modal interaction only occurs at the first layer, and its output probability is: in, Indicates the current decoding status. and Indicates the output layer parameters; The output probability represents, given the intraoperative image, the natural language question, and the generated answer prefix, the probability of the first... Each decoding step calculates the probability distribution of candidate answer tokens; the current answer token is determined from the probability distribution according to a preset decoding strategy, and added to the prefix of the generated answer. This process is iterated until an end marker is generated, thereby obtaining the complete text answer.

8. The visual language reasoning method for robot-assisted surgery according to claim 1, characterized in that, The current visual language reasoning method's visual language question answering model includes a visual language joint encoding module, a multi-layer feature fusion module, a Mamba sequence modeling module, a question-guided visual token filtering module, a spatial anchor construction module, a compact visual language memory construction module, and a text answer generation module. During the training phase, the learnable parameters involved in the above modules are optimized end-to-end using a negative log-likelihood loss function. ， in, Indicates the length of the answer sequence.

Images