Surgical intent-based medical image segmentation method and device, electronic equipment and medium

Abstract

The application provides a surgery-intention-based medical image segmentation method and device, electronic equipment and medium. The method comprises the following steps: inputting surgery-intention text and medical images into a cross-modal segmentation model to obtain a segmentation result of the medical images; the cross-modal segmentation model comprises a text encoder, an image encoder, a segmentation backbone network and a visual language guiding module; the visual language guiding module expands the text features in the spatial dimension to obtain a pixel-level semantic alignment graph that is aligned with the pixel space of the visual features, thereby establishing a direct correlation between the semantics of the surgery-intention text and the pixels of the medical images; the visual features and the pixel-level semantic alignment graph are subjected to modal coordination processing to obtain coordinated features, and a spatial attention graph is generated based on the coordinated features, so that the segmentation backbone network can process the visual features under the guidance of the spatial attention graph, thereby determining an accurate segmentation result, and significantly improving the intelligence and human-machine cooperation efficiency of the surgery robot.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and medical robotics, and in particular to a method, device, electronic device and medium for medical image segmentation based on surgical intent. Background Technology

[0002] With the rapid development of robotics technology, vascular interventional surgical robots have made significant progress in recent years. Existing technologies have achieved major breakthroughs in areas such as precise control algorithms, hand tremor filtering technology, and force feedback systems, greatly improving the accuracy and safety of surgeries. For example, current vascular interventional surgical robot systems can achieve sub-millimeter-level precision positioning, effectively filtering out physiological tremors in the surgeon's hand, providing technical support for delicate surgical procedures.

[0003] However, existing vascular interventional surgical robot technologies still have shortcomings. First, current robots operate entirely in a master-slave mode, acting merely as a mechanical extension of the surgeon's actions. They cannot understand the surgeon's intentions and struggle to achieve proactive, intelligent collaboration. Surgeons must manually select and adjust various intelligent algorithms, unable to adapt to intraoperative changes or the surgeon's intentions, resulting in a relatively low level of intelligent assistance. Second, current technologies have limited ability to recognize surgical intentions. Traditional methods, such as those based on EEG signal processing or emotion recognition, can only achieve relatively coarse-grained classification of basic actions and emotional states, failing to accurately identify and interpret nuanced surgical intentions, thus failing to meet the high-precision intention understanding requirements of vascular interventional surgery. Finally, expressing the understood surgical intentions in a multimodal environment remains a significant challenge, especially in intraoperative imaging guidance. Aligning clinical command semantics with multimodal image guidance is a complex technical problem. Existing pre-trained visual-language models, such as Contrastive Language-Image Pre-training (CLIP), can bridge the gap between text and image modalities, but their performance in intensive prediction tasks within surgical scenarios remains unsatisfactory. Summary of the Invention

[0004] This invention provides a medical image segmentation method, device, electronic device, and medium based on surgical intent, which addresses the shortcomings of existing vascular interventional surgical robot technology, such as low intelligence, limited surgical intent recognition capability, and difficulty in cross-modal information expression and intraoperative image guidance alignment.

[0005] This invention provides a medical image segmentation method based on surgical intent, comprising the following steps.

[0006] Obtain the surgical intent text representing the surgical intent, as well as the medical images to be processed; The surgical intent text and the medical image are input into a cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to acquire the text features of the surgical intent text; the image encoder is used to acquire the visual features of the medical image; the visual language guidance module is used to extend the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, perform modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordination features, and generate a spatial attention map based on the coordination features to guide the segmentation of the target region in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0007] According to a medical image segmentation method based on surgical intent provided by the present invention, the training steps of the visual language guidance module include: Obtain medical image samples, surgical intent text samples, and training segmentation labels for the medical image samples; The visual features of the medical image sample and the text features of the surgical intent text sample are input into the initial visual language guidance module to obtain the prediction spatial attention map output by the initial visual language guidance module. Based on the predicted spatial attention map and the visual features of the sample, the predicted segmentation result is determined; Based on the difference between the predicted segmentation result and the trained segmentation label, the segmentation loss is determined; The initial visual language guidance module is iterated based on the segmentation loss to obtain the visual language guidance module.

[0008] According to a medical image segmentation method based on surgical intent provided by the present invention, the step of obtaining surgical intent text representing the surgical intent includes: Obtain a long sequence of control commands characterizing the surgical intent; The long sequence of control commands is input into the surgical intent understanding model to obtain the surgical intent text output by the surgical intent understanding model; The surgical intent understanding model is trained based on sample long sequence command pairs and similarity labels that characterize the similarity between the sample long sequence command pairs and preset standard commands.

[0009] According to a medical image segmentation method based on surgical intent provided by the present invention, the training steps of the surgical intent understanding model include: The two sample long sequence commands in the sample long sequence command pair are respectively input into the initial surgical intent understanding model to obtain the first command embedding representation and the second command embedding representation output by the initial surgical intent understanding model; The target loss is determined based on the difference between the first command embedding representation and the second command embedding representation, as well as the similarity label; Based on the target loss, the parameters of the initial surgical intent understanding model are iterated to obtain the surgical intent understanding model.

[0010] According to a medical image segmentation method based on surgical intent provided by the present invention, the step of determining a target loss based on the difference between a first command embedding representation and a second command embedding representation, and the similarity label, includes... Based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label, the contrast loss is determined; The first command embedding representation and the second command embedding representation are respectively masked to obtain the first masked command embedding representation and the second masked command embedding representation; The first mask loss is determined based on the difference between the first mask command embedding representation and the first command embedding representation; The second mask loss is determined based on the difference between the second mask command embedding representation and the second command embedding representation; The target loss is determined based on the contrast loss, the first mask loss, and the second mask loss.

[0011] According to a medical image segmentation method based on surgical intent provided by the present invention, the modal coordination processing of the visual features and the pixel-level semantic alignment map to obtain coordinated features includes: The visual features and the pixel-level semantic alignment map are fused to obtain combined features; The combined features are subjected to a first convolution operation to obtain the first processed features; The first processed feature is normalized and nonlinearly transformed to obtain the second processed feature; The second convolution operation is performed on the second processed feature to obtain the coordinated feature.

[0012] According to a medical image segmentation method based on surgical intent provided by the present invention, the step of generating a spatial attention map for guiding the segmentation of target regions in the medical image based on the coordination features includes: Perform a first dimensionality reduction convolution operation on the coordinated features to obtain dimensionality reduction features; A second dimensionality reduction convolution operation is performed on the reduced features to obtain a single-channel feature map; The single-channel feature map is processed by a pixel-level activation function to obtain the spatial attention map.

[0013] The present invention also provides a medical image segmentation device based on surgical intent, comprising the following units: The acquisition unit is used to acquire surgical intent text representing surgical intent, as well as medical images to be processed; The input unit is used to input the surgical intent text and the medical image into the cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to acquire the text features of the surgical intent text; the image encoder is used to acquire the visual features of the medical image; the visual language guidance module is used to extend the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, perform modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordination features, and generate a spatial attention map based on the coordination features to guide the segmentation of the target region in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the medical image segmentation method based on surgical intent as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical image segmentation method based on surgical intent as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the medical image segmentation method based on surgical intent as described above.

[0017] This invention provides a method, apparatus, electronic device, and medium for medical image segmentation based on surgical intent. The method inputs surgical intent text and medical images into a cross-modal segmentation model to obtain the segmentation results of the medical images output by the cross-modal segmentation model. The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module. The visual language guidance module expands the text features in the spatial dimension to obtain a pixel-level semantic alignment map aligned with the pixel space of the visual features. This establishes a direct correlation between the semantics of the surgical intent text and the pixels of the medical image. Furthermore, modal coordination processing is performed on the visual features and the pixel-level semantic alignment map to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of target regions in the medical image. This allows the segmentation backbone network to process the visual features under the guidance of the spatial attention map, thereby determining an accurate segmentation result. This invention solves the technical problem in the prior art that surgical robots cannot understand the surgeon's fine surgical intentions and are difficult to achieve proactive intelligent collaboration by constructing an end-to-end guidance mechanism for surgical intent text-to-image segmentation. It can accurately segment the target region on the intraoperative image according to natural language instructions, avoiding the problem of low level of intelligent assistance caused by limited intent recognition ability and difficulty in cross-modal information expression, and significantly improving the intelligence of surgical robots and human-machine collaboration efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of the medical image segmentation method based on surgical intent provided by the present invention.

[0020] Figure 2 This is the second flowchart of the medical image segmentation method based on surgical intent provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the SurgCommand-BART model provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the overall application scenario provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the medical image segmentation device based on surgical intent provided by the present invention.

[0024] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0027] This invention provides a medical image segmentation method based on surgical intent. This method can accurately segment corresponding target regions, such as specific blood vessels, guide wires, or stents, on medical images based on surgical intent text representing the surgeon's intention, thereby significantly improving the intelligent collaboration level of surgical robots and surgical efficiency. Figure 1 This is one of the flowcharts illustrating the medical image segmentation method based on surgical intent provided by the present invention. Figure 2 This is the second flowchart of the medical image segmentation method based on surgical intent provided by the present invention, as shown below. Figure 1 , Figure 2 As shown, the method includes the following: Step 110: Obtain the surgical intent text representing the surgical intent, and the medical image to be processed.

[0028] Specifically, firstly, surgical intent text representing the surgical intent can be obtained, along with medical images to be processed.

[0029] Surgical intent is used to reflect the specific operations that the surgeon expects to perform or the anatomical structures that the surgeon wishes to focus on during the operation.

[0030] Here, surgical intent text is the textual expression of surgical intent, a type of natural language instruction that can be processed by machines. Surgical intent text can be standardized short commands directly input by the physician, such as advancing the guidewire 1 mm, dilating the balloon to 12 atmospheres, or displaying the left anterior descending coronary artery. In another possible implementation, surgical intent text can also be standardized text obtained by processing and simplifying longer, more conversational or verbose control commands generated by the physician at the main control terminal.

[0031] In one specific embodiment, the surgical intent text can be obtained by inputting a long sequence of control commands into the surgical intent understanding model.

[0032] Here, the surgical intent text can be obtained by converting the doctor's verbal instructions through a speech recognition system, or by the doctor manually inputting it through a graphical user interface, or it can be automatically retrieved from a surgical planning system or an electronic medical record system. This embodiment of the invention does not specifically limit this.

[0033] Among them, the medical images to be processed are used to reflect the current anatomical structure of the surgical area and the status of the surgical instruments.

[0034] Medical images can be in various modalities. In a specific application scenario, such as in vascular interventional surgery, the medical image to be processed can be a two-dimensional X-ray angiography image acquired in real time during the procedure. The medical image to be processed can also be a two-dimensional image such as a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, or an ultrasound image, or a three-dimensional image data volume formed by CT, MRI, etc. The embodiments of this invention do not specifically limit this.

[0035] Here, the medical image can be received directly from the output interface of the imaging device in real time, or it can be read from an image storage and transmission system; this embodiment of the invention does not specifically limit this. For example, a frame of X-ray angiography image with a resolution of 512×512 pixels can be acquired as the medical image to be processed.

[0036] Step 120: Input the surgical intent text and the medical image into the cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to acquire the text features of the surgical intent text; the image encoder is used to acquire the visual features of the medical image; the visual language guidance module is used to extend the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, perform modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordination features, and generate a spatial attention map based on the coordination features to guide the segmentation of the target region in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0037] Specifically, after obtaining the surgical intent text and medical images, the surgical intent text and medical images can be input into the cross-modal segmentation model to obtain the segmentation results of the medical images output by the cross-modal segmentation model.

[0038] Here, the design goal of the cross-modal segmentation model is to understand the surgical intent within the surgical intent text and, guided by this intent, perform accurate semantic segmentation of the input medical image. The cross-modal segmentation model can process and fuse information from text and images from different modalities, achieving cross-modal understanding and analysis. The segmentation result of the medical image output by the cross-modal segmentation model is a mask image spatially corresponding to the input medical image, clearly identifying the target region related to the surgical intent. For example, if the surgical intent text is "locating the guidewire tip," the segmentation result should be a mask image with highlighted marks only at the pixel positions corresponding to the guidewire tip.

[0039] Here, the cross-modal segmentation model may include a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module.

[0040] The text encoder is used to encode the input surgical intent text to obtain text features. The text encoder can be any neural network model capable of effectively processing natural language and extracting semantics. For example, the text encoder can be the text encoder part of a pre-trained language model based on the Transformer architecture, such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, or CLIP (Contrastive Language-Image Pre-Training) model; however, this embodiment of the invention does not specifically limit this.

[0041] Here, text features are typically a set of high-dimensional vectors. Text features are used to reflect the core instruction information in the surgical intent text, such as the object of operation, behavior, and parameters. The object of operation can be a guidewire or a balloon, the behavior can be forward movement and rotation, and the parameters can be 1 mm and 30 degrees. This embodiment of the invention does not specifically limit these parameters.

[0042] The image encoder is used to encode the input medical image to extract visual information at various levels and convert it into a set of feature maps, i.e., visual features. The image encoder can be any convolutional neural network (CNN) or vision transformer (VIT) model capable of effectively extracting image features. In a preferred embodiment, considering that instruments used in vascular interventional surgery, such as guidewires, are typically small, a CNN-based encoder, such as ResNet (Residual Network), Densely Connected Convolutional Network (DenseNet), or EfficientNet, can be used to better capture the local and fine features of these small targets. This embodiment of the invention does not specifically limit the specific implementation of this method.

[0043] Visual features are typically a series of feature maps of different sizes and number of channels, which capture hierarchical features of an image, ranging from low-level texture and edge information to high-level shape and structural information.

[0044] Here, the visual language guidance module is used to expand the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features. Modal coordination processing is performed on the visual features and the pixel-level semantic alignment map to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of target regions in medical images.

[0045] Here, a one-dimensional text feature vector can be copied or expanded into a two-dimensional feature map with the same height and width as the visual feature map through linear transformation and dimensionality reshaping operations. The resulting pixel-level semantic alignment map, in which the feature vector at each spatial location carries the semantic information of the surgical intent text, thus initially establishing a potential association between the surgical intent text and the medical image at the pixel level. In other words, the pixel-level semantic alignment map is used to reflect the mapping of the global semantic information contained in the surgical intent text to each pixel location corresponding to the visual features. The pixel-level semantic alignment map copies and expands a single text feature vector representing the global intent into a two-dimensional feature map of the same size as the visual feature map, thereby establishing an initial alignment between text semantics and visual space at the pixel level.

[0046] Modality coordination processing is performed on visual features and pixel-level semantic alignment maps to obtain coordinated features. This step aims to effectively fuse and align two modality features from different sources and with different properties, solving the potential modality gap problem and generating coordinated features. There are various specific methods for modality coordination processing. For example, the simplest method is to concatenate visual features and pixel-level semantic alignment maps along the channel dimension and then fuse them through a convolutional layer; more complex mechanisms, such as cross-attention, can also be used to allow visual features and pixel-level semantic alignment maps to mutually query and enhance each other. This embodiment of the invention does not specifically limit the specific methods used.

[0047] The spatial attention map can be numerically viewed as a heatmap. Each pixel value in the spatial attention map represents the probability or importance of the pixel location belonging to the target region specified by the surgical intent. Pixel values ​​are typically between 0 and 1. Highlighted areas have pixel values ​​close to 1, and these highlighted areas are those that the model considers to require priority attention and segmentation.

[0048] Here, the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and visual features. The segmentation backbone network can employ a classic encoder-decoder architecture, such as U-Net or U-Net++. The segmentation backbone network first uses the spatial attention map to guide and enhance visual features. Specifically, by performing element-wise multiplication operations between visual features and the spatial attention map, features relevant to the surgical intent—i.e., highlighted areas in the spatial attention map—are enhanced, while irrelevant background features are suppressed. Subsequently, this intent-guided enhanced feature map is fed into the decoder part of the segmentation backbone network. Through a series of upsampling and convolution operations, the spatial resolution of the image is gradually restored, and finally, a pixel-level binary or multi-class segmentation mask with the same size as the original medical image is output—the segmentation result of the medical image.

[0049] The method provided in this invention inputs surgical intent text and medical images into a cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model. The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module. The visual language guidance module is used to expand the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, thereby establishing a direct association between the semantics of the surgical intent text and the pixels of the medical image. Then, modal coordination processing is performed on the visual features and the pixel-level semantic alignment map to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of the target region in the medical image, so that the segmentation backbone network can process the visual features under the guidance of the spatial attention map, thereby determining an accurate segmentation result. This invention solves the technical problem in the prior art that surgical robots cannot understand the surgeon's fine surgical intentions and are difficult to achieve proactive intelligent collaboration by constructing an end-to-end guidance mechanism for surgical intent text-to-image segmentation. It can accurately segment the target region on the intraoperative image according to natural language instructions, avoiding the problem of low level of intelligent assistance caused by limited intent recognition ability and difficulty in cross-modal information expression, and significantly improving the intelligence of surgical robots and human-machine collaboration efficiency.

[0050] Among related technologies, existing methods have relatively limited coverage of medical knowledge, making it difficult to accurately understand and handle the professional terminology and operating procedures of vascular interventional surgery, which affects the actual application effect of the system in the clinical environment.

[0051] Based on the above embodiments, the training steps of the visual language guidance module include: Step 210: Obtain medical image samples, surgical intent text samples, and training segmentation labels for the medical image samples; Step 220: Input the visual features of the medical image sample and the text features of the surgical intent text sample into the initial visual language guidance module to obtain the prediction spatial attention map output by the initial visual language guidance module. Step 230: Based on the predicted spatial attention map and the visual features of the sample, determine the predicted segmentation result; Step 240: Determine the segmentation loss based on the difference between the predicted segmentation result and the trained segmentation label; Step 250: Iterate the parameters of the initial visual language guidance module based on the segmentation loss to obtain the visual language guidance module.

[0052] Specifically, the core idea of ​​the training method for the visual language guidance module in this embodiment is to adopt supervised learning. By utilizing a large amount of labeled data, the initial visual language guidance module is taught to generate a predictive spatial attention map that is as consistent as possible with the real target region, given medical images and surgical intent text. This training method may specifically include the following steps: First, the dataset required for training needs to be prepared. This dataset includes medical image samples, surgical intent text samples, and training segmentation labels for the medical image samples.

[0053] Among these, medical image samples are used to represent various medical images used in training and form the basis for the model to learn visual features. For example, a large number of X-ray angiography images during vascular interventional surgery can be collected, and each image is a medical image sample.

[0054] Surgical intent text samples are text instructions that correspond one-to-one with each medical image sample, used to inform the initial visual language guidance module what operation to perform or which target to focus on on the medical image sample. For example, for a medical image sample showing the position of a guidewire in a blood vessel, the corresponding surgical intent text sample could be "advance guidewire".

[0055] Here, the training segmentation label is used to provide ground truth for the initial visual language guidance module's learning. The training segmentation label can be a masked image, manually or semi-automatically annotated by a domain expert, with the same size as the corresponding medical image sample, precisely marking the target region pointed to by the surgical intent text sample. For example, for a sample pair where the surgical intent text sample is an advancing guidewire, the training segmentation label would be a binary image where only the pixel region containing the guidewire is highlighted, while other areas are dark.

[0056] It should be understood that the training data includes medical image samples, surgical intent text samples, and training segmentation labels for medical image samples. This entire dataset comes entirely from the specific scenario of vascular interventional surgery. Through learning from this domain data, the initial visual language guidance module naturally acquires knowledge of this domain, enabling it to accurately understand domain terminology and locate corresponding images.

[0057] Then, medical image samples and surgical intent text samples can be input into the initial visual language guidance module to obtain the prediction space attention map output by the initial visual language guidance module. The initial visual language guidance module is the initial model used for training.

[0058] The medical image samples are first processed by an image encoder to extract their visual features, while the surgical intent text samples are processed by a text encoder to extract their text features. Then, the visual features of the medical image samples and the text features of the surgical intent text samples are input into the initial visual language guidance module to obtain the prediction spatial attention map output by the initial visual language guidance module.

[0059] Furthermore, the predicted segmentation result can be determined based on the predicted spatial attention map and the visual features of the samples. Here, the segmentation backbone network uses the predicted spatial attention map to guide and weight the visual features of the samples, and then generates the final predicted segmentation result through the decoder part of the segmentation backbone network.

[0060] Furthermore, the segmentation loss can be determined based on the difference between the predicted segmentation result and the trained segmentation label. Segmentation loss is a quantitative metric used to measure the gap between the model's prediction and the standard answer.

[0061] Understandably, the greater the difference between the predicted segmentation result and the training segmentation label, the greater the segmentation loss; conversely, the smaller the difference between the predicted segmentation result and the training segmentation label, the smaller the segmentation loss.

[0062] Here, the segmentation loss can be a loss function commonly used in the art, such as Binary Cross-Entropy Loss (BCE Loss), Dice Loss, Focal Loss, or a weighted combination of Binary Cross-Entropy Loss, Dice Loss, and Focal Loss. This embodiment of the invention does not specifically limit this.

[0063] For example, Dess loss can be used to optimize the overlap of segmented regions, while cross-entropy loss can be combined to focus on pixel-level classification accuracy.

[0064] Here, the formula for the segmentation loss is as follows: in, Indicates the partition loss. Indicates focal loss. Represents the binary cross-entropy loss. Indicates the loss due to the Desce coefficient. , and These represent the weights, which can take values ​​of (0.4, 0.3, 0.3).

[0065] Finally, the initial visual language guidance module can be iterated based on the segmentation loss, and the initial visual guidance module after parameter iteration can be used as the visual language guidance module.

[0066] Here, the cross-modal segmentation model is specifically implemented as CMFA-RepNet. This embodiment of the invention constructs a dedicated cardiovascular image dataset. The cardiovascular image dataset originates from 426 complete surgical cases from XX Hospital, including common surgical instrument types such as guidewires, balloons, and stents. The image resolution of all images is uniformly adjusted to 512×512 pixels. To ensure training accuracy, all training segmentation labels are precisely annotated by professional physicians.

[0067] Figure 3 This is a schematic diagram of the SurgCommand-BART model provided by the present invention, as shown below. Figure 3 As shown, this model aims to efficiently extract core semantics from long command sequences. Its data processing flow and training / inference logic are as follows: The process begins with the input of a long sequence of surgical instructions. This long sequence can consist of a string of different morphemes, including adverbs, verbs, and nouns. The long sequence is fed into a Bidirectional and Auto-Regressive Transformer (BART) tokenizer to convert it into a numerical sequence. This numerical sequence then enters a BART encoder, which acts as a freeze layer, for initial semantic understanding. The output feature map is then processed by a Cov3×3 convolutional layer for local feature extraction, and then fed into a deeper Transformer encoder. This encoder explicitly contains four attention heads for accurately modeling long-distance dependencies in the sequence. Finally, through masked mean pooling, sequences of different lengths are processed into a fixed-dimensional vector representation.

[0068] The resulting vector representation serves two purposes. During inference, it's used to match against a predefined command library containing standardized commands such as Adv (forward), v (verb), and n (noun). The matching process involves L2 normalization of the vector, followed by calculating cosine similarity to determine which command in the library it most closely matches. During training, the model optimizes using contrastive learning. Input training samples are labeled as similar / dissimilar. Based on these labels, the model calculates the cosine similarity of the pairwise command embeddings, applies a threshold to the results to determine prediction accuracy, and then calculates contrastive loss to update model parameters. This closed-loop design ensures the model's strong ability to distinguish between different surgical intentions.

[0069] The method provided in this invention can effectively train a visual language guidance module, enabling it to learn the complex mapping relationship from text semantics to visual space, thereby generating high-quality spatial attention maps and improving the accuracy and reliability of language-guided medical image segmentation.

[0070] In related technologies, during vascular interventional surgery, the control commands issued by doctors are typically characterized by long sequences, numerous steps, redundant information, and similar expressions. Traditional command recognition methods often struggle to effectively understand the core intent of long sequences of commands specifically designed for vascular interventional surgery robots, especially when the understanding and conversion process is not standardized, making it difficult to effectively transmit standardized information.

[0071] Based on the above embodiments, obtaining the surgical intent text representing the surgical intent in step 110 includes: Step 1101: Obtain a long sequence of control commands characterizing the surgical intent; Step 1102: Input the long sequence control command into the surgical intent understanding model to obtain the surgical intent text output by the surgical intent understanding model; The surgical intent understanding model is trained based on sample long sequence command pairs and similarity labels that characterize the similarity between the sample long sequence command pairs and preset standard commands.

[0072] Specifically, in real-world surgical scenarios, surgeons may issue lengthy, repetitive, or non-standardized instructions, such as a series of verbal commands like "move forward a little," "move forward a little more," or "turn a little to the left," or a long string of control signals generated at the main operating terminal. Directly processing such commands increases the burden on subsequent cross-modal segmentation models and may introduce ambiguity.

[0073] To address this issue, this embodiment proposes a method for obtaining surgical intent text. This method first introduces a surgical intent understanding model, specifically designed to parse and simplify these complex, long-sequence control commands.

[0074] Here, the surgical intent understanding model is used to reflect the ability to transform raw, verbose sequences of user commands into standardized, concise core surgical intent text.

[0075] First, long-sequence control commands representing the surgical intent are acquired. For example, long-sequence control commands can be obtained by recording the operation log of the surgical robot over a period of time, or by capturing the doctor's continuous verbal instructions through a speech recognition module.

[0076] Then, long sequence control commands can be input into the surgical intent understanding model to obtain the surgical intent text output by the model. For example, after inputting a long sequence control command including a series of operations such as "advance, advance, stop, slight rotation" into the model, the surgical intent understanding model can understand its core intent and output a standardized surgical intent text, such as "advance guidewire 2mm and rotate 15 degrees".

[0077] The surgical intent understanding model is trained based on sample long-sequence command pairs and similarity labels representing the similarity between sample long-sequence command pairs and preset standard commands. This suggests that its training process employs the idea of ​​contrastive learning, aiming to teach the surgical intent understanding model to distinguish whether different command sequences are semantically equivalent.

[0078] The method provided in this invention decouples the complex intent understanding task from the subsequent cross-modal segmentation task by introducing a dedicated surgical intent understanding model. This design not only improves the robustness of the system and enables it to handle more diverse and conversational user input, but also ensures that the text intent input to the segmentation model is clear, explicit, and consistent through command standardization, thereby enhancing the overall intelligence level of the system.

[0079] Based on the above embodiments, the training steps of the surgical intent understanding model include: Step 310: Input the two sample long sequence commands in the sample long sequence command pair into the initial surgical intent understanding model respectively to obtain the first command embedding representation and the second command embedding representation output by the initial surgical intent understanding model; Step 320: Determine the target loss based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label; Step 330: Based on the target loss, perform parameter iteration on the initial surgical intent understanding model to obtain the surgical intent understanding model.

[0080] Specifically, the training method employs a contrastive learning framework. Instead of having the surgical intent understanding model learn the specific classification of each command, it learns a metric space in which semantically similar commands are close to each other in vector representation, while semantically dissimilar commands are far apart.

[0081] First, the two sample long sequence commands in the sample long sequence command pair are input into the initial surgical intent understanding model to obtain the first command embedding representation and the second command embedding representation output by the initial surgical intent understanding model.

[0082] A sample long command pair consists of two long commands. If the two long commands have the same final surgical intent, they constitute a positive sample pair; if the two long commands have different intents, they constitute a negative sample pair. For example, pushing the guidewire forward and advancing the guidewire slightly can constitute a positive sample pair. Pushing the guidewire forward and dilating balloon, on the other hand, constitute a negative sample pair.

[0083] Here, the similarity label is a label associated with each sample long sequence command pair, used to indicate whether the sample long sequence command pair is a positive sample pair or a negative sample pair. For example, a similarity label of 0 indicates that the two long sequence commands in the sample long sequence command pair are similar, while a similarity label of 1 indicates that the two long sequence commands in the sample long sequence command pair are not similar.

[0084] In this embodiment of the invention, the surgical intent understanding model is specifically implemented as a model called SurgCommand-BART, which is specifically responsible for parsing and simplifying the complex long-sequence control commands issued by the surgeon into standardized surgical intent text.

[0085] To train and run the SurgCommand-BART model, this embodiment of the invention constructs a surgical long-sequence command database and a predefined command library. The surgical long-sequence command database is a dataset used for training the model. In one specific embodiment, the surgical long-sequence command database contains 600 pairs of command sequences, of which 400 pairs are used for training and 200 pairs for testing. Each command sequence is between 15 and 30 words in length and is constructed through random sampling to simulate the diversity of commands in real-world scenarios. The core of the surgical long-sequence command database is similarity tags, provided by medical experts, used to label whether the intents of any two command sequences are similar. The commands in the surgical long-sequence command database are divided into 5 categories and sampled according to specific sampling weights to ensure that the model can learn all types of operations in a balanced manner. These 5 categories of commands and their sampling weights are as follows: Motion control commands, such as “advance / rotate / retract”, have a sampling weight of 0.2.

[0086] Capture control commands such as "tighten guidewire / balloon / stent", with a sampling weight of 0.25.

[0087] Release control command: such as "release guidewire / balloon / stent", with a sampling weight of 0.25.

[0088] Robot speed control commands: such as "slowly / quickly", with a sampling weight of 0.2.

[0089] Fine-grained operation control commands: such as "1mm / 30 degrees / 5mm", with a sampling weight of 0.1.

[0090] Here, the predefined command library serves as the reference standard for the output of the SurgCommand-BART model. It is a standard library containing a series of standardized, unambiguous short commands. During the inference phase, the model converts long command sequences into embedded representations and then performs similarity matching with commands in the predefined command library to ultimately output the most matching standardized short command. The construction of the predefined command library maintains the same classification and weighting ratios as the aforementioned database.

[0091] SurgCommand-BART's network architecture is ingeniously designed to efficiently extract core semantics from long command sequences. Its specific data processing flow is as follows: The original long sequence of control command text is first segmented and positionally encoded by a BART tokenizer, converting it into a machine-processable numerical sequence.

[0092] The digit sequence is input into a pre-trained BART encoder. To preserve its powerful general language understanding capabilities and reduce training overhead, the encoder's parameters are frozen and not used in training. It is responsible for initial bidirectional semantic understanding.

[0093] The output of the Bidirectional and Auto-Regressive Transformers (BART) is fed into two consecutive CNN layers. CNN layer 1 reduces the feature dimension to 128 to capture local grammatical patterns in the commands; CNN layer 2 further reduces the dimension to 64 to capture short-range word dependencies.

[0094] The features are then fed into a trainable Transformer layer. This layer utilizes its self-attention mechanism to model long-distance semantic relationships in the command sequence, capturing global intent.

[0095] Finally, a mapping layer reduces the feature dimension to 32, forming the final command embedding representation. Before similarity matching, this embedding representation is L2 normalized to make it a unit-length vector, thereby eliminating the influence of the magnitude on similarity calculation.

[0096] The initial surgical intent understanding model refers to the model to be trained. The two input long sequence commands are processed by the encoder inside the initial surgical intent understanding model, such as an architecture based on a bidirectional autoregressive transformer or Transformer, and are each mapped into a high-dimensional vector, namely the first command embedding representation and the second command embedding representation. The embedding representation is used to reflect the semantic information of the long sequence commands.

[0097] Then, the target loss can be determined based on the difference between the first command embedding representation and the second command embedding representation, as well as the similarity label. For positive sample pairs, the target loss will reduce the distance between their embedding representations, such as Euclidean distance or cosine distance; for negative sample pairs, the target loss will increase the distance between their embedding representations, and this distance is usually required to be greater than a preset boundary value m.

[0098] Finally, based on the target loss, the initial surgical intent understanding model can be iterated through parameters, and the initial surgical intent understanding model after parameter iteration can be used as the surgical intent understanding model.

[0099] The method provided in this invention enables the surgical intent understanding model to autonomously learn deep semantic similarities of surgical commands from data, rather than relying on surface word matching. This gives the surgical intent understanding model strong generalization ability, allowing it to accurately understand the true intent even for new command expressions not seen during training, thus improving the accuracy and reliability of surgical intent understanding.

[0100] Based on the above embodiments, step 320 includes: Step 3201: Determine the contrast loss based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label; Step 3202: Perform masking processing on the first command embedding representation and the second command embedding representation respectively to obtain the first masked command embedding representation and the second masked command embedding representation; Step 3203: Determine the first mask loss based on the difference between the first mask command embedding representation and the first command embedding representation; Step 3204: Determine the second mask loss based on the difference between the second mask command embedding representation and the second command embedding representation; Step 3205: Determine the target loss based on the contrast loss, the first mask loss, and the second mask loss.

[0101] Specifically, firstly, the contrastive loss can be determined based on the difference between the first command embedding representation and the second command embedding representation, as well as the similarity label, as shown in the following formula: ; in, Indicates comparative loss, E1 and E2 These represent the first command embedding representation and the second command embedding representation, respectively. The similarity label is represented by 0 (0 indicates similarity, 1 indicates dissimilarity). It is the Euclidean distance between the first command embedding representation and the second command embedding representation. m It is a margin hyperparameter, representing the minimum distance threshold that dissimilar sample pairs should maintain. This indicates the number of long sequence command pairs in the sample.

[0102] Then, the first command embedding representation and the second command embedding representation can be masked to obtain the first masked command embedding representation and the second masked command embedding representation. This step introduces a self-supervised learning mechanism. Masking refers to randomly masking or zeroing out some dimensions of the first command embedding representation and the second command embedding representation. The processed first command embedding representation and the second command embedding representation are the first masked command embedding representation and the second masked command embedding representation.

[0103] Furthermore, the first mask loss can be determined based on the difference between the first mask command embedding representation and the first command embedding representation, and the second mask loss can be determined based on the difference between the second mask command embedding representation and the second command embedding representation.

[0104] It is understandable that the greater the difference between the first mask command embedding representation and the first command embedding representation, the greater the first mask loss; the smaller the difference between the first mask command embedding representation and the first command embedding representation, the smaller the first mask loss.

[0105] It is understandable that the greater the difference between the second mask command embedding representation and the second command embedding representation, the greater the second mask loss; the smaller the difference between the second mask command embedding representation and the second command embedding representation, the smaller the second mask loss.

[0106] Here, the surgical intent understanding model is given an additional task: to reconstruct the original, complete command embedding representation as accurately as possible from the partially masked masked command embedding representation. The first mask loss and the second mask loss are used to measure the accuracy of the command embedding representation reconstruction.

[0107] By minimizing this loss, the surgical intent understanding model can be forced to learn more context-aware and information-redundant feature representations, because the surgical intent understanding model must use information from the unmasked dimensions to infer information from the masked dimensions.

[0108] Here, a mask pooling strategy is used to process the fixed-length input token for both the first and second command embedding representations, as shown in the following formula: ; in, Let represent the feature vector of the i-th long sequence command and the j-th token in the sample long sequence command pair. This represents the mask value at the corresponding position. This represents the first mask command embedding representation or the second mask command embedding representation. B represents the number of samples in the batch, and L represents the maximum number of tokens fixed after a single long sequence command is tokenized, which is the maximum number of tokens that a long sequence command can contain, as allowed by the surgical intent understanding model.

[0109] Finally, the target loss can be determined based on the contrast loss, the first mask loss, and the second mask loss. Here, the target loss can be determined based on the sum of the contrast loss, the first mask loss, and the second mask loss, or it can be determined based on a weighted average of the contrast loss, the first mask loss, and the second mask loss. This embodiment of the invention does not specifically limit the determination in this way.

[0110] The method provided in this invention combines a contrastive learning task with a mask representation reconstruction task, providing dual supervision for training a surgical intent understanding model. This multi-task learning strategy enables the model not only to distinguish the semantics of different long sequence commands, but also to gain a deeper understanding of the internal structure of each long sequence command, thereby learning command embedding representations of higher quality and stronger generalization ability.

[0111] Based on the above embodiments, the modal coordination processing of the visual features and the pixel-level semantic alignment map to obtain coordinated features includes: Step 410: The visual features and the pixel-level semantic alignment map are fused to obtain combined features; Step 420: Perform a first convolution operation on the combined features to obtain the first processed features; Step 430: Normalize and nonlinearly transform the first processed feature to obtain the second processed feature; Step 440: Perform a second convolution operation on the second processed feature to obtain the coordinated feature.

[0112] Specifically, firstly, visual features and pixel-level semantic alignment maps can be fused to obtain combined features. The most direct way to fuse them is to concatenate the visual features and pixel-level semantic alignment maps along the channel dimension. Assuming the dimensions of the visual features are [H, W, C1] and the dimensions of the pixel-level semantic alignment maps are [H, W, C2], then the dimensions of the combined features obtained after concatenation are [H, W, C1+C2].

[0113] Here, the pixel-level semantic alignment map is used to reflect the mapping of the global semantic information contained in the surgical intent text to each pixel position corresponding to the visual features. The pixel-level semantic alignment map copies and expands a single text feature vector representing the global intent into a two-dimensional feature map of the same size as the visual feature map, thus establishing a preliminary alignment between text semantics and visual space at the pixel level.

[0114] After obtaining the combined features, a first convolution operation can be performed on them to obtain the first processed features. The combined features are then input into a convolutional layer, such as a 3x3 convolutional layer. The purpose of this step is to allow spatially adjacent features from different modalities to begin preliminary interaction and information exchange, generating the first processed features.

[0115] Furthermore, the first processed features can be normalized and nonlinearly transformed to obtain the second processed features. The normalization process can be batch normalization, used to stabilize the training process and accelerate model convergence. The nonlinear transformation can be achieved by using the Rectified Linear Unit (ReLU) activation function, which increases the model's nonlinear expressive power, enabling it to learn more complex intermodal relationships. After these two operations, the second processed features are obtained.

[0116] Finally, a second convolution operation can be performed on the second processed features to obtain the coordinated features. These second processed features are then input into a second convolutional layer, such as another 3x3 convolutional layer. This step aims to further refine and deepen the intermodal fusion, and the final output is the coordinated features. The coordinated features reflect the deeply fused and semantically consistent cross-modal feature information formed after the visual features and pixel-level semantic alignment map have undergone modal coordination processing.

[0117] Here, the specific implementation of the modality coordination processing step is carried out in CMFA-RepNet by a component called the Semantic-Visual Feature Harmonization Module (SVFHM). Its internal implementation is consistent with the above, that is, through a structure of first convolution, normalization and nonlinear transformation, and second convolution, the stitched visual features and pixel-level semantic alignment map are deeply fused to generate coordinated features.

[0118] The method provided in this invention uses a concatenated structure of convolution-normalization-activation-convolution to avoid modal conflicts that may result from direct splicing through hierarchical and local feature interactions. This ensures that the features of the two modalities remain semantically consistent, providing a solid foundation for generating high-quality spatial attention maps and thus improving the accuracy of the final segmentation task.

[0119] In related technologies, the surgical environment demands extremely high response speeds. Existing large-scale language models require long inference times, making it difficult to meet the stringent accuracy and efficiency requirements of clinical settings. Furthermore, surgical instruments are typically small in size and have minute details, and Transformer-based architectures tend to overlook these small features, posing additional challenges to accurate instrument prediction and dense image understanding.

[0120] Based on the above embodiments, generating a spatial attention map for guiding target region segmentation in the medical image based on the coordination features includes: Step 510: Perform a first dimensionality reduction convolution operation on the coordinated features to obtain dimensionality reduction features; Step 520: Perform a second dimensionality reduction convolution operation on the reduced features to obtain a single-channel feature map; Step 530: Perform pixel-level activation function processing on the single-channel feature map to obtain the spatial attention map.

[0121] Specifically, to obtain high-quality reconciliation features, the image encoder was optimized. A ResNet-based encoder was used instead of the ViT encoder used in the original CLIP model. This is because, in scenarios such as vascular interventional surgery, the targets to be identified are often small in size and elongated in shape, such as guide wires. The ResNet network, based on convolutional operations, has a natural advantage in capturing local features and preserving fine spatial details due to its inherent inductive bias. In contrast, ViT segments the image into several image patches, which may lose or weaken the structural information of small targets during the segmentation process. Therefore, using ResNet as the image encoder can extract richer and more accurate visual features. These high-quality visual features, after modal reconciliation processing with pixel-level semantic alignment maps, result in reconciliation features that inherently carry a stronger representation of small targets.

[0122] Accordingly, firstly, a first dimensionality reduction convolution operation can be performed on the coordinating features to obtain dimensionality-reduced features. Coordinating features typically have a high number of channels, such as 256 or 512. To improve computational efficiency, a 1x1 convolutional layer is first used to reduce the channel dimensionality, for example, reducing the number of channels from 256 to 32, resulting in dimensionality-reduced features. A 1x1 convolution can integrate and compress information between channels without changing the feature map space size.

[0123] Then, a second dimensionality-reducing convolution operation can be performed on the reduced features to obtain a single-channel feature map. Next, a second 1x1 convolutional layer is used to further compress the number of channels of the reduced features to 1, resulting in a single-channel feature map. At this point, the values ​​at each spatial location of the single-channel feature map can be regarded as a preliminary quantitative indicator of the degree of attention at that location.

[0124] Finally, pixel-level activation functions are applied to the single-channel feature maps to obtain the spatial attention map. To map the values ​​in the single-channel feature maps to a standardized, easily interpretable range, such as between 0 and 1, pixel-level activation functions are needed. A typical choice is the Sigmoid function. The Sigmoid function maps any real-valued input to the (0, 1) interval, and its output value can be intuitively understood as the probability or attention weight that the pixel belongs to the target region. The result after this processing is the final spatial attention map.

[0125] Here, the step of generating the spatial attention map in the specific implementation of this invention, CMFA-RepNet, is performed by a component called the Adaptive Spatial Attention Module. The implementation of the Adaptive Spatial Attention Module is consistent with the aforementioned method, namely, using two consecutive 1x1 convolutions for efficient dimensionality reduction, and finally generating a single-channel spatial attention map through a pixel-level sigmoid activation function. Furthermore, as mentioned above, to better handle small targets such as guide wires, the image encoder upstream of this Adaptive Spatial Attention Module is preferably a ResNet-based encoder.

[0126] The method provided in this invention reduces dimensionality through two consecutive 1x1 convolutions and then generates an attention map using the sigmoid function, significantly reducing computational cost while maintaining effectiveness. This lightweight design allows the visual language guidance module to be easily integrated into various segmentation networks without incurring excessive performance overhead, meeting the real-time requirements of clinical applications.

[0127] Based on any of the above embodiments Figure 4 This is a schematic diagram of the overall application scenario provided by the present invention, such as... Figure 4As shown, the operator issues surgical commands from the main control console. These commands can be status control commands, such as "advance," "rotate," and "retract"; grasp control commands, such as "tighten the guidewire," "tighten the balloon," and "tighten the stent"; or release control commands, such as "release the guidewire," "release the balloon," and "release the stent." These commands serve as input for long-sequence command analysis and are processed by the SurgCommand-BART module. The processed, standardized intent is then represented by an aligned text-language-space segmentation network and finally visualized on the intraoperative images. The system accurately understands and identifies the operational intent of "advance the guidewire by 1 mm."

[0128] Based on any of the above embodiments, long-sequence surgical commands can be input into the SurgCommand-BART module to parse and obtain standardized surgical commands (such as "guidewire", "advance", "1 mm"). These commands are then fed into a text encoder that acts as a frozen layer. Simultaneously, the input medical image, after being scaled to 224×224, is fed into another frozen-layer image encoder. The outputs of both are then aligned using the CLIP alignment module. The image data flows through an encoder path containing multiple double convolutions and downsampling. This path specifically includes: a Conv3×3 3->64 layer, a Conv3×3 64->128 layer, a Conv3×3 128->256 layer, a Conv3×3 256->512 layer, and a Conv3×3 512->512 layer, which serves as the network bottleneck. At the bottleneck, visual features and textual features are fused in a semantic-visual feature coordination module, and the output then enters an adaptive spatial attention module.

[0129] The output of the adaptive spatial attention module is processed through a deep supervision path (512->256->1) to calculate the segmentation loss. Simultaneously, it, along with the bottleneck features processed by the Conv3×3 512->512 double convolutional layer, enters the upsampled decoder path. After upsampling at each level of the decoding path, the feature map is concatenated with the features of the corresponding layer in the encoder, and then sequentially passes through a Conv3×3 512->256 layer and a Conv3×3 256->128 layer. The adaptive spatial attention module and deep supervision (512->256->1) are also applied after the Conv3×3 512->256 layer to calculate the loss. Finally, the feature map passes through a Conv3×3 128->64 layer and the final output layer Conv3×3 64->1 to generate the segmentation result, and the segmentation loss is calculated.

[0130] The surgical intent-based medical image segmentation apparatus provided by the present invention will be described below. The surgical intent-based medical image segmentation apparatus described below and the surgical intent-based medical image segmentation method described above can be referred to in correspondence.

[0131] Based on any of the above embodiments, the present invention provides a medical image segmentation device based on surgical intent. Figure 5 This is a schematic diagram of the medical image segmentation device based on surgical intent provided by the present invention, as shown below. Figure 5 As shown, the device includes: The acquisition unit 510 is used to acquire surgical intent text representing surgical intent, as well as medical images to be processed; The input unit 520 is used to input the surgical intent text and the medical image into the cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to acquire the text features of the surgical intent text; the image encoder is used to acquire the visual features of the medical image; the visual language guidance module is used to extend the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, perform modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordination features, and generate a spatial attention map based on the coordination features to guide the segmentation of the target region in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0132] The apparatus provided in this invention inputs surgical intent text and medical images into a cross-modal segmentation model to obtain the segmentation result of the medical images output by the cross-modal segmentation model. The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module. The visual language guidance module is used to extend the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features, thereby establishing a direct association between the semantics of the surgical intent text and the pixels of the medical images. Furthermore, modal coordination processing is performed on the visual features and the pixel-level semantic alignment map to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of target regions in the medical images, enabling the segmentation backbone network to process the visual features under the guidance of the spatial attention map, thereby determining an accurate segmentation result. This invention solves the technical problem in the prior art that surgical robots cannot understand the surgeon's fine surgical intentions and are difficult to achieve proactive intelligent collaboration by constructing an end-to-end guidance mechanism for surgical intent text-to-image segmentation. It can accurately segment the target region on the intraoperative image according to natural language instructions, avoiding the problem of low level of intelligent assistance caused by limited intent recognition ability and difficulty in cross-modal information expression, and significantly improving the intelligence of surgical robots and human-machine collaboration efficiency.

[0133] Based on any of the above embodiments, a training unit is further included, wherein the training unit is specifically used for: Obtain medical image samples, surgical intent text samples, and training segmentation labels for the medical image samples; The visual features of the medical image sample and the text features of the surgical intent text sample are input into the initial visual language guidance module to obtain the prediction spatial attention map output by the initial visual language guidance module. Based on the predicted spatial attention map and the visual features of the sample, the predicted segmentation result is determined; Based on the difference between the predicted segmentation result and the trained segmentation label, the segmentation loss is determined; The initial visual language guidance module is iterated based on the segmentation loss to obtain the visual language guidance module.

[0134] Based on any of the above embodiments, the acquisition unit 510 is specifically used for: Obtain a long sequence of control commands characterizing the surgical intent; The long sequence of control commands is input into the surgical intent understanding model to obtain the surgical intent text output by the surgical intent understanding model; The surgical intent understanding model is trained based on sample long sequence command pairs and similarity labels that characterize the similarity between the sample long sequence command pairs and preset standard commands.

[0135] Based on any of the above embodiments, a parameter iteration unit is further included, wherein the parameter iteration unit specifically includes: An embedding representation unit is determined, which is used to input the two sample long sequence commands in the sample long sequence command pair into the initial surgical intent understanding model respectively, so as to obtain the first command embedding representation and the second command embedding representation output by the initial surgical intent understanding model; A target loss unit is defined to determine the target loss based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label. The parameter iteration subunit is used to perform parameter iteration on the initial surgical intent understanding model based on the target loss to obtain the surgical intent understanding model.

[0136] Based on any of the above embodiments, the target loss determination unit is specifically used for: Based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label, the contrast loss is determined; The first command embedding representation and the second command embedding representation are respectively masked to obtain the first masked command embedding representation and the second masked command embedding representation; The first mask loss is determined based on the difference between the first mask command embedding representation and the first command embedding representation; The second mask loss is determined based on the difference between the second mask command embedding representation and the second command embedding representation; The target loss is determined based on the contrast loss, the first mask loss, and the second mask loss.

[0137] Based on any of the above embodiments, the visual language guidance module is specifically used for: The visual features and the pixel-level semantic alignment map are fused to obtain combined features; The combined features are subjected to a first convolution operation to obtain the first processed features; The first processed feature is normalized and nonlinearly transformed to obtain the second processed feature; The second convolution operation is performed on the second processed feature to obtain the coordinated feature.

[0138] Based on any of the above embodiments, the visual language guidance module is specifically used for: Perform a first dimensionality reduction convolution operation on the coordinated features to obtain dimensionality reduction features; A second dimensionality reduction convolution operation is performed on the reduced features to obtain a single-channel feature map; The single-channel feature map is processed by a pixel-level activation function to obtain the spatial attention map.

[0139] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can invoke logical instructions in the memory 630 to execute a medical image segmentation method based on surgical intent. This method includes: acquiring surgical intent text representing the surgical intent, and a medical image to be processed; inputting the surgical intent text and the medical image into a cross-modal segmentation model to obtain a segmentation result of the medical image output by the cross-modal segmentation model; the cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; the text encoder is used to acquire text features of the surgical intent text; the image encoder is used to acquire visual features of the medical image; the visual language guidance module is used to extend the text features in a spatial dimension to obtain a pixel-level semantic alignment map aligned with the pixel space of the visual features; performing modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordinated features; and generating a spatial attention map based on the coordinated features to guide the segmentation of target regions in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0140] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the medical image segmentation method based on surgical intent provided by the above methods. The method includes: acquiring surgical intent text representing surgical intent and a medical image to be processed; inputting the surgical intent text and the medical image into a cross-modal segmentation model to obtain a segmentation result of the medical image output by the cross-modal segmentation model; the cross-modal segmentation model includes a text encoder, an image encoder, and a segmentation backbone network. The system comprises a network and a visual language guidance module; a text encoder for acquiring text features of the surgical intent text; an image encoder for acquiring visual features of the medical image; a visual language guidance module for expanding the text features in a spatial dimension to obtain a pixel-level semantic alignment map aligned with the pixel space of the visual features; performing modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordinated features; and generating a spatial attention map based on the coordinated features to guide the segmentation of the target region in the medical image; and a segmentation backbone network for determining the segmentation result based on the spatial attention map and the visual features.

[0142] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the medical image segmentation method based on surgical intent provided by the methods described above. This method includes: acquiring surgical intent text representing the surgical intent, and a medical image to be processed; inputting the surgical intent text and the medical image into a cross-modal segmentation model to obtain a segmentation result of the medical image output by the cross-modal segmentation model; the cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; the text encoder is used to acquire text features of the surgical intent text; the image encoder is used to acquire visual features of the medical image; the visual language guidance module is used to extend the text features in a spatial dimension to obtain a pixel-level semantic alignment map aligned with the pixel space of the visual features; performing modal coordination processing on the visual features and the pixel-level semantic alignment map to obtain coordinated features; and generating a spatial attention map based on the coordinated features to guide the segmentation of a target region in the medical image; the segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method based on surgical intent, characterized in that, include: Obtain the surgical intent text representing the surgical intent, as well as the medical images to be processed; The surgical intent text and the medical image are input into a cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to obtain the text features of the surgical intent text; the image encoder is used to obtain the visual features of the medical image. The visual language guidance module is used to expand the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features. The visual features and the pixel-level semantic alignment map are subjected to modal coordination processing to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of target regions in the medical image. The segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

2. The medical image segmentation method based on surgical intent according to claim 1, characterized in that, The training steps of the visual language guidance module include: Obtain medical image samples, surgical intent text samples, and training segmentation labels for the medical image samples; The visual features of the medical image sample and the text features of the surgical intent text sample are input into the initial visual language guidance module to obtain the prediction spatial attention map output by the initial visual language guidance module. Based on the predicted spatial attention map and the visual features of the sample, the predicted segmentation result is determined; Based on the difference between the predicted segmentation result and the trained segmentation label, the segmentation loss is determined; The initial visual language guidance module is iterated based on the segmentation loss to obtain the visual language guidance module.

3. The medical image segmentation method based on surgical intent according to claim 1, characterized in that, The acquisition of the surgical intent text representing the surgical intent includes: Obtain a long sequence of control commands characterizing the surgical intent; The long sequence of control commands is input into the surgical intent understanding model to obtain the surgical intent text output by the surgical intent understanding model; The surgical intent understanding model is trained based on sample long sequence command pairs and similarity labels that characterize the similarity between the sample long sequence command pairs and preset standard commands.

4. The medical image segmentation method based on surgical intent according to claim 3, characterized in that, The training steps for the surgical intent understanding model include: The two sample long sequence commands in the sample long sequence command pair are respectively input into the initial surgical intent understanding model to obtain the first command embedding representation and the second command embedding representation output by the initial surgical intent understanding model; The target loss is determined based on the difference between the first command embedding representation and the second command embedding representation, as well as the similarity label; Based on the target loss, the parameters of the initial surgical intent understanding model are iterated to obtain the surgical intent understanding model.

5. The medical image segmentation method based on surgical intent according to claim 4, characterized in that, The target loss is determined based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label, including... Based on the difference between the first command embedding representation and the second command embedding representation, and the similarity label, the contrast loss is determined; The first command embedding representation and the second command embedding representation are respectively masked to obtain the first masked command embedding representation and the second masked command embedding representation; The first mask loss is determined based on the difference between the first mask command embedding representation and the first command embedding representation; The second mask loss is determined based on the difference between the second mask command embedding representation and the second command embedding representation; The target loss is determined based on the contrast loss, the first mask loss, and the second mask loss.

6. The medical image segmentation method based on surgical intent according to any one of claims 1 to 5, characterized in that, The modal coordination processing of the visual features and the pixel-level semantic alignment map to obtain coordinated features includes: The visual features and the pixel-level semantic alignment map are fused to obtain combined features; The combined features are subjected to a first convolution operation to obtain the first processed features; The first processed feature is normalized and nonlinearly transformed to obtain the second processed feature; The second convolution operation is performed on the second processed feature to obtain the coordinated feature.

7. The medical image segmentation method based on surgical intent according to any one of claims 1 to 5, characterized in that, The step of generating a spatial attention map based on the coordination features to guide the segmentation of target regions in the medical image includes: Perform a first dimensionality reduction convolution operation on the coordinated features to obtain dimensionality reduction features; A second dimensionality reduction convolution operation is performed on the reduced features to obtain a single-channel feature map; The single-channel feature map is processed by a pixel-level activation function to obtain the spatial attention map.

8. A medical image segmentation device based on surgical intent, characterized in that, include: The acquisition unit is used to acquire surgical intent text representing surgical intent, as well as medical images to be processed; The input unit is used to input the surgical intent text and the medical image into the cross-modal segmentation model to obtain the segmentation result of the medical image output by the cross-modal segmentation model; The cross-modal segmentation model includes a text encoder, an image encoder, a segmentation backbone network, and a visual language guidance module; The text encoder is used to obtain the text features of the surgical intent text; the image encoder is used to obtain the visual features of the medical image. The visual language guidance module is used to expand the text features in the spatial dimension to obtain a pixel-level semantic alignment map that is aligned with the pixel space of the visual features. The visual features and the pixel-level semantic alignment map are subjected to modal coordination processing to obtain coordinated features. Based on the coordinated features, a spatial attention map is generated to guide the segmentation of target regions in the medical image. The segmentation backbone network is used to determine the segmentation result based on the spatial attention map and the visual features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the medical image segmentation method based on surgical intent as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the medical image segmentation method based on surgical intent as described in any one of claims 1 to 7.

Images