Text interactive minimally invasive surgical instrument segmentation method and system

Abstract

The invention relates to the technical field of computer vision, and discloses a text interactive minimally invasive surgery instrument segmentation method and system, and the method comprises the steps: generating the text prompt information of an instrument, and carrying out the feature extraction of a surgery image and the text prompt information based on a vision-language large model; the visual features are strengthened and refined according to the text features, and mask probability graphs corresponding to various instruments are generated; performing weighted fusion on a plurality of mask probability graphs corresponding to different text prompt information of the same instrument category; a difficult segmentation area is obtained by analyzing the corresponding relation between the fusion mask probability graph and a preset label, the difficult segmentation area in the operation image is masked and then input into a preset image encoder, and a corresponding decoder is added to execute image reconstruction. According to the method, flexible and accurate mask segmentation prediction can be carried out on different surgical instruments on the input surgical image, and the limitation that an existing method is low in segmentation precision, poor in generalization ability and fixed in prediction category is broken through.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a text interactive minimally invasive surgical instrument segmentation method and system. Background Art

[0002] Minimally invasive surgery refers to a surgical procedure that uses endoscopic equipment to enter the body through a relatively small incision. It is the preferred treatment method for disciplines such as gastroenterology, cardiothoracic surgery, general surgery, and urology. In recent years, with the deep integration of the new generation of information technology and the medical industry, minimally invasive surgery has begun to evolve in the direction of intelligence. Minimally invasive surgical robots can improve the flexibility, stability, and accuracy of doctors' operations, and assist doctors in performing more difficult and precise operations. Considering the small operating space in the patient's body and the fragility and vulnerability of the patient, robot-assisted minimally invasive surgery must use surgical instrument segmentation algorithms to obtain the precise position and posture of surgical instruments, thereby improving doctors' perception and understanding of the surgical process and ensuring the safe and stable operation of the operation.

[0003] With the development of deep neural networks and medical image analysis technology, researchers in related fields have proposed a variety of technical solutions for surgical instrument segmentation. Relevant studies in recent years have shown that although the existing segmentation models can provide relatively accurate prediction results of binary foreground masks containing instruments, it is difficult to distinguish the categories of different instruments. The reasons are as follows: The most advanced methods today (such as S3Net, TraSeTR and MATIS, etc.) all focus on the visual end. First, an image encoder is used to extract the visual features of the input image, and the extracted visual features are converted into binary surgical instrument masks using an instance segmentation decoder. The predicted masks are then classified by visual semantic features. However, since these models fail to take into account the textual semantics of the instrument description and only understand abstract category information through visual features, they are limited by the similarity of surgical instruments and the scarcity of pixel annotations. Their segmentation performance is still not ideal and cannot meet the accuracy and safety requirements of clinical surgery. In addition, with the rapid development of minimally invasive surgery, the number of instruments used has increased dramatically. The current minimally invasive surgical instrument segmentation algorithm is not enough to adapt to the increasing number of surgical instruments. Whenever a new instrument category is introduced, data needs to be re-annotated and the model needs to be trained, which seriously hinders the practical application of surgical instrument segmentation technology in the field of minimally invasive surgery. Summary of the invention

[0004] The present application provides a text-interactive minimally invasive surgical instrument segmentation method to solve the problems in the prior art that the existing methods have weak ability to distinguish between instruments and the predicted label space is fixed and immutable.

[0005] Correspondingly, the present application also provides a text interactive minimally invasive surgical instrument segmentation system, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above technical problems, the present application discloses a text interactive minimally invasive surgical instrument segmentation method, the method comprising:

[0007] Generate text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description;

[0008] Based on the visual-language model, feature extraction is performed on surgical images and text prompt information to obtain visual-text feature pairs; the visual-text feature pairs include visual features and text features;

[0009] Enhance and refine visual features based on text features to generate mask probability maps corresponding to various types of equipment;

[0010] Performing weighted fusion on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map;

[0011] The difficult segmentation areas in the surgical images are masked and input into a preset image encoder, and the corresponding decoder is added to perform image reconstruction; wherein the difficult segmentation areas are obtained by parsing the correspondence between the fused mask probability map and the preset labels.

[0012] The present application also discloses a text interactive minimally invasive surgical instrument segmentation system, the system comprising:

[0013] A text prompt generation module, used to generate text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description;

[0014] An encoder module is used to extract features from surgical images and text prompt information based on a visual-language large model to obtain visual-text feature pairs; the visual-text feature pairs include visual features and text features;

[0015] The mask decoder module is used to enhance and refine the visual features according to the text features to generate mask probability maps corresponding to various types of devices;

[0016] A hybrid prompt prediction fusion module is used to perform weighted fusion on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map;

[0017] The difficult segmentation area enhancement module is used to mask the difficult segmentation area in the surgical image and input it into a preset image encoder, and add a corresponding decoder to perform image reconstruction; wherein the difficult segmentation area is obtained by parsing the correspondence between the fused mask probability map and the preset label.

[0018] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0019] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0020] In this application, text prompt information including the instrument category name, instrument appearance description and instrument function description is generated, which helps alleviate the problem of unsatisfactory model semantic segmentation performance caused by the similarity of surgical instruments and the scarcity of pixel annotation from the input text end, and provides important text prompts for text interactive minimally invasive surgical instrument segmentation. Based on the visual-language large model, the surgical image and text prompt information are feature extracted to obtain a visual-text feature pair including visual features and text features, and the alignment of visual and text features is achieved. The visual features are strengthened and refined according to the text features, and mask probability maps corresponding to various types of instruments are generated, so that the model can more accurately identify the location and category of surgical instruments to improve the accuracy of minimally invasive surgical instrument segmentation. Multiple mask probability maps corresponding to different text prompt information of the same instrument category are weighted fused to obtain a fused mask probability map, thereby improving the model's adaptability to diversified input text. By parsing the correspondence between the fused mask probability map and the preset label, the difficult segmentation area is mined, and then the difficult segmentation area in the surgical image is masked and input into the preset image encoder, and the corresponding decoder is added to perform image reconstruction, which enhances the model's segmentation prediction accuracy for the difficult segmentation area and edge details of the surgical instrument.

[0021] The method in this application can flexibly and accurately predict the mask segmentation of different surgical instruments on the input surgical images. In addition, in addition to obtaining accurate prediction results on the basic classes trained on the standard dataset, the method in this application can also obtain considerable segmentation accuracy on new categories that have not been seen in the training phase, breaking through the limitations of existing methods such as low segmentation accuracy, poor generalization ability, and fixed prediction categories.

[0022] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1A flowchart of a text interactive minimally invasive surgical instrument segmentation method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a text prompt generation module provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the network structure provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a text interactive minimally invasive surgical instrument segmentation system provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0030] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0031] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0032] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. For the technical problems existing in the prior art, the text interactive minimally invasive surgical instrument segmentation method and system provided in this application is intended to solve at least one of the technical problems of the prior art.

[0033] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0034] The present application embodiment provides a possible implementation method, such as Figure 1 As shown, a flowchart of a text interactive minimally invasive surgical instrument segmentation method is provided, and the scheme can be executed by any electronic device, and optionally, can be executed on a server side or a terminal device.

[0035] like Figure 1 As shown in , the method may include the following steps:

[0036] Step 101, generating text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description.

[0037] In an embodiment of the present application, constructing prompt text information containing the name of each surgical instrument category, description of the instrument appearance and description of the instrument function can provide an important text prompt input set for the text interactive minimally invasive surgical instrument segmentation model constructed in the embodiment of the present application (specifically implementing the method described in steps 102-105 below).

[0038] Step 102, extracting features from the surgical image and text prompt information based on the visual-language macro model to obtain visual-text feature pairs; the visual-text feature pairs include visual features and text features.

[0039] In an embodiment of the present application, the visual features of the input surgical impact and the text features of the text prompt are first extracted to form a visual-text feature pair. In order to ensure that the extracted visual features and text features have a high semantic consistency, the image encoder and text encoder used in the embodiment of the present application are from the visual-language large model (CLIP).

[0040] Step 103, enhancing and refining the visual features according to the text features, and generating mask probability maps corresponding to various types of equipment.

[0041] The visual features are enhanced and refined under the guidance of text features, and then high-quality mask probability maps are generated from them, so that the model can more accurately identify the location and category of surgical instruments, which is of great significance to improving the accuracy of minimally invasive surgical instrument segmentation.

[0042] Step 104 , weighted fusion is performed on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map.

[0043] For an input text prompt set containing multiple text prompt information of the same instrument, multiple mask probability maps are generated after the text prompt information contained therein is input into the model. In the embodiment of the present application, weighted fusion is performed on different mask probability maps of the same instrument category, thereby fully mining the compressed knowledge in the visual-language large model encoder to improve the surgical instrument segmentation accuracy and generalization ability of the model, thereby improving the model's adaptability to diverse input texts.

[0044] Step 105, mask the difficult segmentation area in the surgical image and input it into a preset image encoder, and add a corresponding decoder to perform image reconstruction; wherein the difficult segmentation area is obtained by analyzing the correspondence between the fused mask probability map and the preset label.

[0045] In the embodiment of the present application, during the model training stage, the model's ability to represent visual features of difficult-to-segment areas is emphasized, thereby enhancing the model's segmentation prediction accuracy for difficult-to-segment areas and edge details of surgical instruments.

[0046] In the embodiment of the present application, text prompt information including the instrument category name, instrument appearance description and instrument function description is generated, which helps alleviate the problem of unsatisfactory model semantic segmentation performance caused by the similarity of surgical instruments and the scarcity of pixel annotation from the input text end, and provides important text prompts for text interactive minimally invasive surgical instrument segmentation. Based on the visual-language large model, the surgical image and text prompt information are feature extracted to obtain a visual-text feature pair including visual features and text features, and the alignment of visual and text features is achieved. The visual features are strengthened and refined according to the text features, and mask probability maps corresponding to various types of instruments are generated, so that the model can more accurately identify the location and category of surgical instruments to improve the accuracy of minimally invasive surgical instrument segmentation. Multiple mask probability maps corresponding to different text prompt information of the same instrument category are weighted fused to obtain a fused mask probability map, thereby improving the model's adaptability to diversified input text. By parsing the corresponding relationship between the fused mask probability map and the preset label, the difficult segmentation area is mined, and then the difficult segmentation area in the surgical image is masked and input into the preset image encoder, and the corresponding decoder is added to perform image reconstruction, which enhances the model's segmentation prediction accuracy for the difficult segmentation area and edge details of the surgical instrument.

[0047] Based on the above, the method in the embodiment of the present application can perform flexible and accurate mask segmentation prediction for different surgical instruments on the input surgical images. In addition, in addition to obtaining accurate prediction results on the basic classes trained on the standard data set, the method in the embodiment of the present application can also obtain considerable segmentation accuracy on new categories that have not been seen in the training stage, breaking through the limitations of low segmentation accuracy, poor generalization ability and fixed prediction categories of existing methods.

[0048] In an optional embodiment, generating text prompt information of the instrument includes:

[0049] For different equipment categories, text prompt information of the equipment is generated based on the large language model.

[0050] In the embodiment of the present application, the text prompt information of the instrument is generated based on the text prompt generation module, optionally, as Figure 2 As shown in the figure, the text prompt generation module constructs a prompt text set containing the names of each surgical instrument category and a description of the instrument appearance. This module develops three prompt words with increasing complexity. First, the inherent name of each instrument category is used as a prompt word; second, inspired by the CLIP prompt template, a prompt word template is designed, namely "surgical instrument area represented by [class name]"; third, using the large language model GPT-4 released in recent years to generate prompt words, a prompt question template is designed: "Please describe the appearance of [class name] in endoscopic surgery, change the description to a phrase with a subject, and do not use a colon.

[0051] By generating more detailed appearance description text prompts for each surgical instrument, a text prompt set {T1, T2, T3} suitable for the surgical instrument segmentation framework is finally constructed, where T1 represents the inherent name of the instrument category, which can be abbreviated as the instrument category name, such as "bipolar forceps", "monopolar curved scissors"; T2 represents the prompt information generated according to the descriptive words "surgical instrument area represented by [class name]", such as "surgical instrument area represented by [bipolar forceps]", "surgical instrument area represented by [monopolar curved scissors]"; T3 represents the prompt information generated according to the prompt template "Please describe the appearance of [class name] in endoscopic surgery, change the description to a phrase with a subject, and do not use a colon.", such as "monopolar curved scissors show a slender handle, a curved cutting edge for precise cutting, and an insulated shaft that allows controlled application of cutting and coagulation electrical energy."

[0052] In an optional embodiment, the vision-language large model includes an image encoder and a text encoder;

[0053] Based on the visual-language model, feature extraction is performed on surgical images and text prompt information to obtain visual-text feature pairs, including:

[0054] Extract features from surgical images based on image encoders to obtain visual features;

[0055] Based on the text encoder, feature extraction is performed on the text title information to obtain text features.

[0056] In the embodiment of the present application, the feature extraction of surgical images and text prompt information is performed based on the encoder module to form a visual-text feature pair. Specifically, the image encoder (visual transformer, ViT) in the visual-language model is used, and the image encoder is fine-tuned during the training process. Figure 3 , for the input surgical image I (height and width are H and W respectively), use represents the output of the first layer of the image encoder, where I ranges from 1 to 12; N is the number of visual tokens (i.e., the sequence length of the visual feature); and D is the channel dimension of the visual feature. Assuming that the length and width of the image block conversion projection window set by the image encoder are both P, the sequence length of the visual feature is:

[0057]

[0058] At the same time, in order to aggregate multi-scale visual features, the embodiment of the present application simulates the multi-scale feature fusion mechanism widely used in the convolutional neural network architecture: extracting the output features of the 4th, 8th and 12th layers of the image encoder, and then reconstructing them into a feature map structure and fusing these features step by step through the feature pyramid network (FPN).

[0059] The text encoder (Bert) in the vision-language model is used, and its parameters are kept frozen during the training process. Figure 3 As shown, for the input text prompt T, the text encoder is able to convert it into a text feature in vector form:

[0060] F T ∈R 1×D

[0061] Where D is the channel dimension of text features, and its value is equal to the channel dimension of visual features.

[0062] In an optional embodiment, visual features are enhanced and refined according to text features to generate mask probability maps corresponding to various types of equipment, including:

[0063] Decode visual features and text features based on the attention mechanism to obtain attention-based text prompt features;

[0064] The text prompt features are processed based on the convolution operation to obtain the mask probability map.

[0065] In the embodiment of the present application, a mask decoder module is designed to enhance and refine the extracted visual features according to the guidance of text features, and then generate mask probability maps corresponding to various surgical instruments. Figure 3 As shown in Figure 3, the mask decoder module contains two sub-modules: an attention-based hint module and a convolution-based hint module. The former can be regarded as a global decoding process based on attention operation, while the latter is a local decoding process based on convolution operation.

[0066] In an optional embodiment, visual features and text features are decoded based on an attention mechanism to obtain attention-based text prompt features, including:

[0067] Calculate self-attention within the visual features to obtain self-attention features;

[0068] Calculate the cross attention of the self-attention visual feature and the text prompt feature to obtain the cross attention feature;

[0069] The cross-attention features are input into the feed-forward network and the attention-based text hint features are obtained as output.

[0070] In the attention-based prompt module, first, the visual feature F ISelf-attention (SA) is calculated internally to enhance and activate the areas that may contain foreground in the visual feature. The self-attention operation includes layer normalization (LN) and jump connection. The process formula is as follows:

[0071]

[0072] Next, calculate With F T Cross-attention (CA), according to F T The guide pair The instrument area in is positioned and further activated. Similarly, this process can be written as:

[0073]

[0074] Subsequently, a feed-forward network consisting of fully connected layers is applied to further enhance the visual features guided by the text features:

[0075]

[0076] Considering SA, CA and FFN as a decoding unit, the embodiment of the present application always uses three attention-based decoding units in sequence to obtain attention-based text prompt features

[0077] In an optional embodiment, the text prompt feature is processed based on a convolution operation to obtain a mask probability map, including:

[0078] Reconstruct the text prompt features;

[0079] The text features are converted into convolution parameters, and the reconstructed text prompt features are convolved using the convolution parameters to obtain convolution output features; the convolution parameters include weights and parameters;

[0080] The Sigmoid function is used to convert the convolution output features into a mask probability map.

[0081] Specifically, the convolution-based prompt module designs a convolution-based feature enhancement scheme to further refine the text prompt features through local convolution To do this, we first need to Reconstruct and restore its spatial dimension: transform the text feature F T Convert to convolution parameters, i.e. weights and biases, and use these parameters to convolution the text prompt features This conversion process is completed through a fully-connected (FC) layer, which converts the text feature F T Converted to a vector with channel dimensions of D×k×k+1, where “k×k” represents the convolution kernel size and “+1” is an additional dimension to consider the deviation. T Decomposed into convolution weights w∈R 1×D×k×k and bias b∈R 1 , which is then used for convolution, and the final output is converted into a mask probability map S using the Sigmoid function:

[0082]

[0083] The combination of the above-mentioned attention-based prompt module and convolution-based prompt module enables the model to more accurately identify the location and category of surgical instruments, which is of great significance to improving the accuracy of minimally invasive surgical instrument segmentation.

[0084] In an optional embodiment, weighted fusion is performed on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map, including:

[0085] Multiple mask probability maps corresponding to different text prompt information of the same device category, and multiple weight maps corresponding to the output of the visual-text threshold network;

[0086] Based on the weight mapping, multiple mask probability maps are weightedly fused to obtain a fused mask probability map;

[0087] The visual-text thresholding network consists of multiple layers of residual blocks.

[0088] The embodiments of this application are based on Figure 3 The hybrid hint prediction fusion module shown in realizes the weighted fusion of multiple mask probability maps. The specific implementation method is as follows:

[0089] First, for each type of target device, a mixed prompt set (including the device name, prompt word template, device appearance description generated by the large language model, etc.) is given, and it is marked as T={T 1 ,T 2 ,T 3}, mark the text feature converted from the i-th prompt as Then each With visual features F I Input to the mask decoder module to obtain the mask probability map S iIn order to fuse the mask probability maps in the set and obtain the final segmentation mask prediction, the hybrid prompt prediction fusion module designs a visual-text threshold network G to perform weighted fusion of the mask probability maps corresponding to multiple text prompts. The visual-text threshold network G consists of 3 layers of residual blocks. I and text features After connecting, before inputting G, first input F T ∈R 1×D Copy to To match F I In the embodiment of the present application, taking the three mask probability maps corresponding to different text prompt information of the same device category as an example, the output of the visual-text threshold network G is three weight mappings corresponding to the three mask probability maps. Through weighted calculation, the final fused mask probability map S can be obtained et ∈R H×W The final mask prediction M is composed of S et Obtained by threshold processing.

[0090] The hybrid prompt prediction fusion module can fully tap the compressed knowledge in the large visual-language model encoder, enhance the model's adaptability to complex input text descriptions, and thus improve the model's segmentation accuracy and generalization ability.

[0091] In an optional embodiment, by Figure 3 The difficult segmentation area enhancement module shown in the figure implements masking of the difficult segmentation areas in the surgical image and then inputs them into a preset image encoder, and adds a corresponding decoder to perform image reconstruction.

[0092] The difficult segmentation area enhancement module in the embodiment of the present application is different from the random masking strategy based on the masked autoencoder (MAE). It aims to make the image encoder focus on challenging difficult segmentation areas. Through the non-random difficult segmentation area mining mechanism, the model's visual feature representation ability of difficult segmentation areas is enhanced during the model training stage, which can solve the performance defects of existing methods in instrument type discrimination and instrument edge segmentation. The module first parses the correspondence between the fused mask probability map and the true value label to mine the difficult segmentation area, and then masks the difficult segmentation area in the original image (i.e., the surgical image) and inputs it into the image encoder, and adds the corresponding decoder to perform image reconstruction. The difficult segmentation area enhancement module in the embodiment of the present application shares the same image encoder with the surgical instrument segmentation model.

[0093] Specifically, given a segmentation mask probability map, the difference between it and the true value label is calculated to obtain the difficult segmentation area, and then the difficult area is selectively masked based on the pre-set mask ratio to generate a mask image. The mask image is then input into the image encoder for visual feature extraction, and an image reconstruction decoder is built to reconstruct the complete image area based on the unmasked area. This module can guide the image encoder to focus on key areas that are difficult to identify in the image, improve its ability to mine and represent visual features of difficult areas of surgical instruments, and thus improve the accuracy of the model.

[0094] like Figure 2 As shown, in the training process, the embodiment of the present application calculates the loss L between the fused mask probability map and the original input image. seg And the loss L between the difficult segmentation area and the corresponding area in the original input image rec , in order to improve the training accuracy.

[0095] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a text interactive minimally invasive surgical instrument segmentation system, such as Figure 4 As shown, the system comprises:

[0096] The text prompt generation module 401 is used to generate text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description;

[0097] The encoder module 402 is used to extract features from the surgical image and text prompt information based on the visual-language large model to obtain a visual-text feature pair; the visual-text feature pair includes a visual feature and a text feature;

[0098] The mask decoder module 403 is used to enhance and refine the visual features according to the text features to generate mask probability maps corresponding to various types of equipment;

[0099] A mixed prompt prediction fusion module 404 is used to perform weighted fusion on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map;

[0100] The difficult segmentation region enhancement module 405 is used to mask the difficult segmentation region in the surgical image and input it into a preset image encoder, and add a corresponding decoder to perform image reconstruction; wherein the difficult segmentation region is obtained by parsing the correspondence between the fused mask probability map and the preset label.

[0101] In the embodiment of the present application, flexible and accurate mask segmentation prediction can be performed on different surgical instruments on the input surgical images. In addition, in addition to obtaining accurate prediction results on the basic classes trained on the standard data set, the system in the embodiment of the present application can also obtain considerable segmentation accuracy on new categories not seen in the training stage, breaking through the limitations of low segmentation accuracy, poor generalization ability and fixed prediction categories of existing methods.

[0102] The text interactive minimally invasive surgical instrument segmentation system provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0103] The text interactive minimally invasive surgical instrument segmentation system of the embodiment of the present application can execute the text interactive minimally invasive surgical instrument segmentation method provided in the embodiment of the present application, and the implementation principle is similar. The actions performed by each module and unit in the text interactive minimally invasive surgical instrument segmentation system in each embodiment of the present application correspond to the steps in the text interactive minimally invasive surgical instrument segmentation method in each embodiment of the present application. For the detailed functional description of each module of the text interactive minimally invasive surgical instrument segmentation system, please refer to the description in the corresponding text interactive minimally invasive surgical instrument segmentation method shown in the previous text, which will not be repeated here.

[0104] Based on the same principle as the method shown in the embodiments of the present application, the embodiments of the present application also provide an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the text interactive minimally invasive surgical instrument segmentation method shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the text interactive minimally invasive surgical instrument segmentation method provided in the present application can perform flexible and accurate mask segmentation predictions for different surgical instruments on input surgical images. In addition, in addition to obtaining accurate prediction results on the basic classes that have been trained on the standard data set, the method in the embodiments of the present application can also obtain considerable segmentation accuracy on new categories that have not been seen in the training stage, breaking through the limitations of the existing methods of low segmentation accuracy, poor generalization ability, and fixed prediction categories.

[0105] In an optional embodiment, an electronic device is also provided, such as Figure 5 As shown, Figure 5 The electronic device 500 shown may be a server, including: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the electronic device 500 may further include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present application.

[0106] Processor 501 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0107] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] The memory 503 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0109] The memory 503 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the contents shown in the above method embodiment.

[0110] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0111] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0112] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0113] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0114] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0115] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0116] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0117] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the text interactive minimally invasive surgical instrument segmentation method and system provided in the above-mentioned various optional implementations.

[0118] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the text prompt generation module can also be described as a "text prompt generation module for generating text prompt information of the device".

[0121] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A text interactive minimally invasive surgical instrument segmentation method, characterized in that: The method comprises: Generate text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description; Extracting features from the surgical image and the text prompt information based on a visual-language macro model to obtain a visual-text feature pair; the visual-text feature pair includes a visual feature and a text feature; Strengthening and refining the visual features according to the text features to generate mask probability maps corresponding to various types of equipment; Performing weighted fusion on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map; The difficult segmentation area in the surgical image is masked and input into a preset image encoder, and a corresponding decoder is added to perform image reconstruction; wherein the difficult segmentation area is obtained by parsing the correspondence between the fused mask probability map and the preset label.

2. The text interactive minimally invasive surgical instrument segmentation method according to claim 1, characterized in that: The text prompt information of the generating device includes: For different equipment categories, the text prompt information of the equipment is generated based on a large language model.

3. The text interactive minimally invasive surgical instrument segmentation method according to claim 1, characterized in that: The visual-language large model includes an image encoder and a text encoder; The step of extracting features from the surgical image and the text prompt information based on the visual-language macro model to obtain visual-text feature pairs includes: Extracting features from the surgical image based on the image encoder to obtain the visual features; The text feature is obtained by performing feature extraction on the text title information based on the text encoder.

4. The text interactive minimally invasive surgical instrument segmentation method according to claim 1, characterized in that: The step of enhancing and refining the visual features according to the text features to generate mask probability maps corresponding to various types of equipment includes: Decoding the visual features and the text features based on an attention mechanism to obtain an attention-based text prompt feature; The text prompt feature is processed based on a convolution operation to obtain the mask probability map.

5. The text interactive minimally invasive surgical instrument segmentation method according to claim 2, characterized in that: The step of decoding the visual features and the text features based on the attention mechanism to obtain the attention-based text prompt features includes: Calculating self-attention within the visual feature to obtain a self-attention feature; Calculating the cross attention of the self-attention visual feature and the text prompt feature to obtain a cross attention feature; The cross-attention feature is input into a feedforward network, and the attention-based text prompt feature is output.

6. The text interactive minimally invasive surgical instrument segmentation method according to claim 4, characterized in that: The processing of the text prompt feature based on the convolution operation to obtain the mask probability map includes: Reconstructing the text prompt feature; Converting the text features into convolution parameters, and convolving the reconstructed text prompt features using the convolution parameters to obtain the convolution output features; the convolution parameters include weights and parameters; The convolution output feature is converted into the mask probability map using a Sigmoid function.

7. The text interactive minimally invasive surgical instrument segmentation method according to claim 1, characterized in that: The step of weightedly fusing a plurality of mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map includes: Multiple mask probability maps corresponding to different text prompt information of the same device category, and multiple weight maps corresponding to the output of the visual-text threshold network; Based on the weight mapping, performing weighted fusion on a plurality of mask probability maps to obtain the fused mask probability map; The visual-text threshold network includes multiple layers of residual blocks.

8. A text interactive minimally invasive surgical instrument segmentation system, characterized in that: The system comprises: A text prompt generation module, used to generate text prompt information of the device; the text prompt information includes the device category name, device appearance description and device function description; An encoder module, used for extracting features of the surgical image and the text prompt information based on a visual-language macro model to obtain a visual-text feature pair; the visual-text feature pair includes a visual feature and a text feature; A mask decoder module, for enhancing and refining the visual features according to the text features, and generating mask probability maps corresponding to various types of equipment; A hybrid prompt prediction fusion module is used to perform weighted fusion on multiple mask probability maps corresponding to different text prompt information of the same device category to obtain a fused mask probability map; The difficult segmentation area enhancement module is used to mask the difficult segmentation area in the surgical image and input it into a preset image encoder, and add a corresponding decoder to perform image reconstruction; wherein the difficult segmentation area is obtained by parsing the correspondence between the fused mask probability map and the preset label.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images