Method, device, medium and equipment for vascular interventional surgery target positioning

Abstract

The present application discloses a method, device, medium and equipment for vascular interventional surgery target positioning, the method comprising: obtaining an initial positioning model, the initial positioning model comprising a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer and an initial lesion segmentation layer; extracting the image feature vector of the sample surgical image based on the image encoder; training the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector; encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector; the scene understanding layer is the initial scene understanding layer after training; training the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model; the initial scene understanding layer is used to predict the surgical status of the sample surgical image. The method and device provided by the present application can obtain accurate lesion areas and instrument locations, providing more accurate and efficient auxiliary support for interventional surgery.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method, device, medium and equipment for locating a target for vascular intervention surgery. Background Art

[0002] As a high-precision medical procedure, vascular interventional surgery relies on precise image guidance to perform various operations, such as catheter insertion, stent placement, and balloon positioning. Surgical images provide important information about blood vessels, lesion areas, and instruments used. For example, digital subtraction angiography (DSA) images are a commonly used imaging technology in interventional surgery. During surgery, doctors need to refer to DSA images for surgery. Therefore, providing doctors with clear lesion and instrument location information in DSA images will improve the safety and effectiveness of surgery. Current technology mainly relies on the doctor's experience for image analysis, which is not only time-consuming but also easily affected by subjective judgment.

[0003] Therefore, how to accurately locate the lesion and the device has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present application provides a method, device, medium and equipment for locating a target of a vascular interventional surgery, so as to solve the technical problem of how to accurately locate the lesion position and the instrument position in the prior art.

[0005] In a first aspect, the present application provides a method for locating a target for vascular intervention surgery, comprising:

[0006] Acquire an initial positioning model, wherein the initial positioning model includes a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer, and an initial lesion segmentation layer;

[0007] Extracting an image feature vector of a sample surgical image based on the image encoder;

[0008] Training the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector;

[0009] Encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result; the scene understanding layer is the initial scene understanding layer after training;

[0010] Training the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model;

[0011] Among them, the initial scene understanding layer is used to predict the surgical status of the sample surgical image, and the surgical status includes the instrument type of the surgical instrument and the steps of the current operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image.

[0012] In some embodiments, the training of the initial scene understanding layer based on the image feature vector includes:

[0013] An adapter layer based on the initial positioning model expands the dimensional mapping of the image feature vector to a dimension that meets the input requirements of the initial scene understanding layer;

[0014] Inputting the mapped and expanded image feature vector into the initial scene understanding layer;

[0015] The initial scene understanding layer is trained based on the mapped expanded image feature vector and a cross entropy loss function.

[0016] In some embodiments, the training of the initial device segmentation layer based on the image feature vector and the text feature vector includes:

[0017] After the initial scene understanding layer training is completed, the scene understanding layer is obtained;

[0018] Extracting features of the surgical status output by the scene understanding layer based on the multimodal model and the text encoder to obtain the text feature vector;

[0019] A cross-modal attention mechanism in the initial machine segmentation layer is adjusted based on the image feature vector and the text feature vector.

[0020] In some embodiments, adjusting the cross-modal attention mechanism in the initial device segmentation layer based on the image feature vector and the text feature vector includes:

[0021] Dynamically assigning weights to each feature based on the similarity between the image feature vector and the text feature vector;

[0022] Performing feature fusion on the image feature vector and the text feature vector based on a multi-layer perceptron and weights of each feature;

[0023] The cross-modal attention mechanism is adjusted based on the feature fusion result.

[0024] In some embodiments, the training of the initial lesion segmentation layer based on the image feature vector comprises:

[0025] An adapter layer based on the initial positioning model expands the dimensional mapping of the image feature vector to a dimension that meets the input requirement of the initial lesion segmentation layer;

[0026] Inputting the mapped and expanded image feature vector into the initial lesion segmentation layer;

[0027] The initial lesion segmentation layer is trained based on the mapped expanded image feature vector and the loss function.

[0028] In some embodiments, the initial lesion segmentation layer includes a lightweight self-attention mechanism module; the lightweight self-attention mechanism module is used to enhance the attention of the initial lesion segmentation layer to the image feature vector.

[0029] In some embodiments, the loss function includes a cross entropy loss function and a Dai loss function;

[0030] The cross entropy loss function is used for pixel-level classification; the Dai loss is used to measure the similarity between the predicted position and the actual position.

[0031] In a second aspect, the present application provides a vascular intervention surgery target positioning device, comprising:

[0032] An acquisition module, used for acquiring an initial positioning model, wherein the initial positioning model includes a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer, and an initial lesion segmentation layer;

[0033] An extraction module, used for extracting an image feature vector of a sample surgical image based on the image encoder;

[0034] A first training module, used for training the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector;

[0035] An encoding module, used for encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result; the scene understanding layer is an initial scene understanding layer after training;

[0036] A second training module, used for training the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model;

[0037] The initial scene understanding layer is used to predict the surgical status of the sample surgical image, and the surgical status includes the instrument type of the surgical instrument and the steps of the current surgical operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image. In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor.

[0038] In a fourth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the above method when executing the program through the computer program.

[0039] The target positioning method, device, medium and equipment for vascular interventional surgery provided in the present application construct an initial positioning model through a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer and an initial lesion segmentation layer, extract the image feature vector of the sample surgical image through the image encoder, encode the result output by the scene understanding layer through the text encoder, extract the text feature vector, thereby training the initial scene understanding layer, the initial instrument segmentation layer and the initial lesion segmentation layer to obtain a positioning model, which can obtain accurate lesion areas and instrument positions, and provide more accurate and efficient auxiliary support for interventional surgery; the surgical status corresponding to the surgical image can be predicted through the scene understanding layer, and the text feature vector can be obtained through the text encoder, thereby integrating the text description of the surgical process into the image analysis, thereby improving the accuracy and robustness of the positioning model for surgical scene understanding and target segmentation and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 One of the flowcharts of the target positioning method for vascular intervention surgery provided in the embodiment of the present application;

[0043] Figure 2 A second flowchart of the target positioning method for vascular intervention surgery provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a target positioning device for vascular intervention surgery provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0047] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or equipment.

[0048] The vascular intervention surgery target positioning method provided in the embodiment of the present application is applicable to a terminal, which may be various electronic devices having a display screen and supporting web browsing, including but not limited to servers, smart phones, tablet computers, laptop computers, and desktop computers.

[0049] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0050] Figure 1 This is one of the flow charts of the target positioning method for vascular intervention surgery provided in the embodiment of the present application, such as Figure 1 As shown, the method includes step 110, step 120, step 130, step 140 and step 150. The steps of the method flow are only a possible implementation of the present application.

[0051] Step 110: Acquire an initial positioning model, where the initial positioning model includes a text encoder, an image encoder, an initial scene understanding layer, an initial device segmentation layer, and an initial lesion segmentation layer.

[0052] Among them, the initial scene understanding layer is used to predict the surgical status of the sample surgical image, which includes the instrument type of the surgical instrument and the steps of the current operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image.

[0053] Specifically, the vascular intervention surgery target positioning method provided in the embodiment of the present application is implemented by a vascular intervention surgery target positioning device, which can be a hardware device independently provided in a terminal, or a software program running in the terminal. For example, when the terminal is a desktop computer, the vascular intervention surgery target positioning device can be embodied as an application such as target positioning software in the desktop computer.

[0054] The positioning model is a neural network model used to segment the surgical instrument area and the patient lesion area in the surgical image during interventional surgery to obtain the position of the surgical instrument and the lesion. The input of the positioning model is the surgical image, and the output is the type of surgical instrument, the current surgical step, the step currently being performed, the position of the surgical instrument, and the lesion. The surgical image can be a DSA image.

[0055] A large amount of DSA image data from interventional surgeries can be collected as training sets. These data can come from different medical institutions and include surgical intervention devices such as guidewires, stents, and balloons. All image data are preprocessed, including denoising and contrast enhancement, to improve the efficiency and effectiveness of model training.

[0056] An initial positioning model may be constructed first through a text encoder, an image encoder, an initial scene understanding layer, an initial device segmentation layer, and an initial lesion segmentation layer. The initial positioning model is a positioning model that has not yet started training or has not yet been trained.

[0057] The text encoder and the image encoder may be CLIP (Contrastive Language-Image Pretraining) encoders.

[0058] The initial scene understanding layer can understand the visual information of the surgical scene through the CLIP image encoder and the MobileNet network. MobileNet is a lightweight deep learning neural network architecture.

[0059] The initial instrument segmentation layer can segment the instrument area used in surgery and predict the location of surgical instruments by combining the image features obtained by the CLIP image encoder with the text features obtained by the CLIP text encoder and the Segformer main network provided by the scene understanding. Segformer is a neural network architecture for image segmentation.

[0060] The initial lesion segmentation layer can segment the lesion area during surgery and predict the lesion location of the patient in the surgical image through the CLIP image encoder and the TransUNet main network. TransUNet is a neural network architecture.

[0061] The initial positioning model may also include a text feature extraction layer, which is used to extract text description features of the surgical process based on the multimodal model and the CLIP text encoder. The multimodal model may be a Generative Pre-trained Transformer 4 (GPT4).

[0062] Step 120: extracting an image feature vector of the sample surgical image based on the image encoder.

[0063] Specifically, a pre-trained image encoder with rich language-visual representation capabilities is used, and its parameters are locked in order to preserve its pre-trained representation capabilities.

[0064] The image feature vector of the input sample surgical image is extracted by leveraging the rich visual-linguistic knowledge learned by the image encoder on a large-scale dataset.

[0065] Step 130: Train the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector.

[0066] Step 140: Encode the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result; the scene understanding layer is the initial scene understanding layer after training.

[0067] Step 150: Train the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model.

[0068] Specifically, a pre-trained text encoder with rich language-visual representation capabilities is used. In order to preserve its pre-trained representation capabilities, its parameters are locked.

[0069] The initial scene understanding layer, the initial device segmentation layer, and the initial lesion segmentation layer are trained, that is, the MobileNet architecture in the initial scene understanding layer, the TransUNet architecture in the initial lesion segmentation layer, and the SegFormer architecture in the initial device segmentation layer are trained. The initial positioning model can be end-to-end trained in combination with the lightweight self-attention mechanism and the feature fusion module to obtain the final positioning model.

[0070] After the positioning model is obtained, the positioning model realizes the positioning of the target through the interaction between each layer, and the target includes surgical instruments and lesion areas. The specific positioning method is as follows:

[0071] The surgical images were input into the localization model.

[0072] Combined with the MobileNet-based scene understanding layer of the CLIP image encoder, the CLIP encoder-based image encoder extracts the image feature vector of the input surgical image, and MobileNet is used as a decoder to recognize the surgical scene as the first stage of global scene understanding for interventional surgery.

[0073] Based on the text feature extraction layer of GPT4 and CLIP text encoder, GPT4 is used to generate the morphological description language for surgical instruments and the text encoder of CLIP encoder is used to extract the text features of the recognition content (i.e., surgical status features).

[0074] Combined with the TransUNet-based lesion segmentation layer of the CLIP image encoder, the image feature vector extracted by the image encoder is used, and TransUNet is used as the decoder. After each downsampling (encoding) and upsampling (decoding) process, a lightweight self-attention mechanism (LSA) module is inserted to enhance the model's attention to features, strengthen the ability to capture the features of the lesion area, and segment the lesion area.

[0075] Combining the CLIP image encoder, CLIP text encoder and the text-inspired Segformer-based instrument segmentation layer, a feature fusion (FS) module is set up. The fine-grained text features extracted from the CLIP text encoder are nonlinearly combined with the above-mentioned image features through a multi-layer perceptron. Segformer is used as a decoder. The feature fusion module is introduced before the multiscale feature fusion (MFP) module, which realizes the use of detailed text descriptions to enhance the understanding of the image, so as to achieve high-precision recognition and precise segmentation of instruments in interventional surgery images. Combined with the relative position positioning of the lesion and the instrument, the local precise positioning of the interventional surgery is used as the second stage.

[0076] The target positioning method for vascular interventional surgery provided in the embodiment of the present application constructs an initial positioning model through a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer and an initial lesion segmentation layer, extracts the image feature vector of the sample surgical image through the image encoder, encodes the result output by the scene understanding layer through the text encoder, and extracts the text feature vector, thereby training the initial scene understanding layer, the initial instrument segmentation layer and the initial lesion segmentation layer to obtain a positioning model, which can obtain accurate lesion areas and instrument positions, and provide more accurate and efficient auxiliary support for interventional surgery; the surgical status corresponding to the surgical image can be predicted through the scene understanding layer, and the text feature vector can be obtained through the text encoder, thereby integrating the text description of the surgical process into the image analysis, thereby improving the accuracy and robustness of the positioning model for surgical scene understanding and target segmentation and positioning.

[0077] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0078] In some embodiments, step 130 includes:

[0079] The adapter layer based on the initial localization model expands the dimensional mapping of the image feature vector to the dimension that meets the input requirements of the initial scene understanding layer;

[0080] Input the mapped and expanded image feature vector into the initial scene understanding layer;

[0081] The initial scene understanding layer is trained based on the mapped expanded image feature vector and the cross entropy loss function.

[0082] Specifically, combined with the CLIP image encoder with locked parameters, the MobileNet-based scene understanding layer takes advantage of its lightweight characteristics for fast feature extraction.

[0083] The dimensional mapping of the image feature vector extracted by the CLIP image encoder is expanded to the dimension that meets the input requirements of the initial scene understanding layer through the adapter layer (fully connected layer).

[0084] For example, the 512-dimensional image feature vector mapping is expanded to 7*7*32, fused with the first bottleneck layer of MobileNet V3, and trained MobileNet after the addition operation.

[0085] Figure 2 The second flowchart of the target positioning method for vascular intervention surgery provided in the embodiment of the present application is as follows: Figure 2As shown in the figure, the scene understanding layer can accurately identify different types of surgical instruments in interventional surgery images and determine which surgical step the surgery is performing, such as "delivering guidewire, releasing stent, dilating balloon, or readiness". This serves as the first stage of global scene understanding of interventional surgery and provides assistance for the guidance of clinical interventional surgery.

[0086] Figure 2 The Chinese and English explanations are as follows: Input, Image Enhancement, CLIP Image Encoder, CLIP Text Encoder, Identification Head, Current Surgical Status, Global Scene Understanding, Segmentation Result, Instrument Segmentation Head, Lesion Segmentation Head, Instrument localization, Lesion localization, Current Surgical Scenario, Combined Loss Function and Local Accurate Localization.

[0087] The loss function of the scene understanding layer can select the cross-entropy loss function, which can effectively measure the difference between the predicted probability distribution and the true label.

[0088] The target positioning method for vascular interventional surgery provided in the embodiment of the present application automatically realizes global scene understanding by utilizing advanced deep learning technology combined with text information guidance in surgical images, thereby improving the training efficiency of the positioning model.

[0089] In some embodiments, step 150 includes:

[0090] After the initial scene understanding layer training is completed, the scene understanding layer is obtained;

[0091] Based on the multimodal model and text encoder, the surgical status output by the scene understanding layer is feature extracted to obtain a text feature vector.

[0092] The cross-modal attention mechanism in the initial machine segmentation layer is adjusted based on the image feature vector and the text feature vector.

[0093] The cross-modal attention mechanism in the initial machine segmentation layer is adjusted based on the image feature vector and the text feature vector, including:

[0094] Dynamically assign weights to each feature based on the similarity between the image feature vector and the text feature vector;

[0095] The image feature vector and the text feature vector are fused based on the multi-layer perceptron and the weights of each feature;

[0096] Adjust the cross-modal attention mechanism based on feature fusion results.

[0097] Specifically, because the input of the initial instrument segmentation layer includes the surgical status, in order to ensure the training effect of the initial instrument segmentation layer, the surgical status needs to be accurate, because the initial scene understanding layer needs to be trained before the initial instrument segmentation layer is trained. At the same time, the sample surgical status can also be directly obtained, so that each layer can be trained synchronously.

[0098] The surgical status obtained in the scene understanding layer and GPT4 generate a morphological description language for surgical instruments, which is then used to extract text features using the text encoder of the CLIP encoder. The image feature vector and the text feature vector are then used to adjust the cross-modal attention mechanism of SegFormer in the instrument segmentation layer, enabling the model to adjust its focus based on the text information provided in real time, thereby improving the accuracy and real-time performance of instrument segmentation.

[0099] Based on the image feature vector extracted by the CLIP image encoder, combined with the text encoding information and detail features (i.e., surgical status) obtained in the scene understanding layer, the FM module and the main network architecture of SegFormer are combined to accurately segment the surgical instruments in the image.

[0100] For example, through the text results (surgical status) obtained in the scene understanding layer, the text encoder of CLIP is used to process the text results to obtain a text feature vector. Subsequently, the designed FM module is used to fuse the image features with the text features through a multi-layer perceptron, and a cross-modal attention mechanism is introduced to dynamically assign weights to each feature according to the similarity between the image features and the text features, so as to realize the nonlinear combination of features. The fused feature dimension is then reshaped to a total dimension of 14*14*128 through the adapter layer, and fused before the MFP module, and then these features are integrated through a 1x1 convolutional layer, the output channel is set to 128, and SegFormer is trained subsequently. Among them, in this application, the features are fused, that is, the feature vectors are fused, the features are extracted, that is, the feature vectors are extracted, and the similarity between the features is the similarity between the feature vectors.

[0101] The target localization method for vascular intervention surgery provided in the embodiment of the present application adopts a text-inspired information fusion strategy, allowing the system to understand and utilize the language description of the surgical process, and further optimize the recognition and segmentation process.

[0102] In some embodiments, step 130 includes:

[0103] The adapter layer based on the initial localization model expands the dimensional mapping of the image feature vector to the dimension that meets the input requirements of the initial lesion segmentation layer;

[0104] The mapped and expanded image feature vector is input into the initial lesion segmentation layer;

[0105] The initial lesion segmentation layer is trained based on the mapped expanded image feature vector and the loss function.

[0106] The initial lesion segmentation layer includes a lightweight self-attention mechanism module; the lightweight self-attention mechanism module is used to enhance the attention of the initial lesion segmentation layer to the image feature vector.

[0107] The loss functions include cross entropy loss function and Dai loss function;

[0108] The cross entropy loss function is used for pixel-level classification; the Dai loss is used to measure the similarity between the predicted position and the actual position.

[0109] Specifically, based on the image feature vector extracted by the CLIP image encoder, combined with the main network architecture of TransUNet and the lightweight self-attention module, the lesion area in the image is accurately segmented.

[0110] The adapter layer based on the initial localization model expands the dimensional mapping of the image feature vector to a dimension that meets the input requirements of the initial lesion segmentation layer.

[0111] For example, the 512-dimensional image feature vector mapping is expanded to 224*224*3 through the adapter layer to meet the input requirements of the TransUNet model.

[0112] TransUNet is trained by mapping the expanded image feature vector, and after each downsampling (encoding) and upsampling (decoding) process in the downstream task of TransUNet, an LSA module is inserted to enhance the model's attention to features, first through a 1x1 convolution to reduce the number of channels in the feature map, then the self-attention mechanism is applied to focus on key features, and finally another 1x1 convolution is used to restore the number of channels. This enables subsequent deep feature extraction and accurate segmentation of the lesion area.

[0113] The loss function of the lesion segmentation layer and the device segmentation layer can select a combined loss function, combining cross entropy loss and Dice Loss. Cross entropy loss is effective for pixel-level classification, while Dice Loss is used to measure the similarity between two sets (predicted segmentation positions and actual segmentation positions).

[0114] The lesion segmentation layer and the instrument segmentation layer can accurately segment different types of lesions and surgical instruments in interventional surgery images. They are suitable for processing small parts in images and for high-accuracy segmentation under complex backgrounds. The segmentation results can be used to reflect the relative position of the lesion area and the instrument, and serve as the second stage of local precise positioning of interventional surgery. They provide assistance for the guidance of clinical interventional surgery.

[0115] After the localization model is trained, it can be integrated into a user-friendly interface so that doctors can access and use it in real time during actual surgery. The performance of the localization model can be tested in interventional surgeries at multiple medical institutions, including the accuracy of scene understanding, the accuracy of lesion and device area segmentation, and the evaluation of real-time performance.

[0116] The performance of the positioning model can be further optimized based on the feedback collected in the initial practical application, especially customized adjustments for specific surgical types or special instruments to meet different surgical needs.

[0117] The target localization method for vascular interventional surgery provided in the embodiment of the present application can improve the accuracy of scene understanding and the accuracy of target segmentation and localization by combining the efficient network architecture of CLIP text encoder, CLIP image encoder, MobileNet, TransUNet and SegFormer, and then combining the independently designed lightweight self-attention mechanism and feature fusion module.

[0118] The following is a description of a vascular intervention surgery target positioning device provided in an embodiment of the present application. The vascular intervention surgery target positioning device described below and the vascular intervention surgery target positioning method described above can be referenced to each other.

[0119] Figure 3 A schematic diagram of the structure of a target positioning device for vascular intervention surgery provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the apparatus includes an acquisition module 310 , an extraction module 320 , a first training module 330 , an encoding module 340 and a second training module 350 .

[0120] An acquisition module, used for acquiring an initial positioning model, the initial positioning model including a text encoder, an image encoder, an initial scene understanding layer, an initial device segmentation layer and an initial lesion segmentation layer;

[0121] An extraction module, used for extracting an image feature vector of a sample surgical image based on an image encoder;

[0122] A first training module, used for training an initial scene understanding layer and an initial lesion segmentation layer based on an image feature vector;

[0123] The encoding module is used to encode the output of the scene understanding layer based on the text encoder to obtain the text feature vector of the result; the scene understanding layer is the initial scene understanding layer after the training is completed;

[0124] The second training module is used to train the initial device segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model;

[0125] Among them, the initial scene understanding layer is used to predict the surgical status of the sample surgical image, which includes the instrument type of the surgical instrument and the steps of the current operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image.

[0126] Specifically, according to an embodiment of the present application, any multiple modules among the acquisition module, the extraction module, the first training module, the encoding module and the second training module can be combined into one module for implementation, or any one of the modules can be split into multiple modules.

[0127] Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.

[0128] According to an embodiment of the present application, at least one of the acquisition module, the extraction module, the first training module, the encoding module and the second training module can be at least partially implemented as a hardware circuit, such as a field programmable gate array, a programmable logic array, a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit, or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them.

[0129] Alternatively, at least one of the acquisition module, the extraction module, the first training module, the encoding module and the second training module may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0130] The target positioning device for vascular interventional surgery provided in the embodiment of the present application constructs an initial positioning model through a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer and an initial lesion segmentation layer, extracts the image feature vector of the sample surgical image through the image encoder, encodes the result output by the scene understanding layer through the text encoder, and extracts the text feature vector, thereby training the initial scene understanding layer, the initial instrument segmentation layer and the initial lesion segmentation layer to obtain a positioning model, which can obtain accurate lesion areas and instrument positions, and provide more accurate and efficient auxiliary support for interventional surgery; the surgical status corresponding to the surgical image can be predicted through the scene understanding layer, and the text feature vector can be obtained through the text encoder, thereby integrating the text description of the surgical process into the image analysis, thereby improving the accuracy and robustness of the positioning model for surgical scene understanding and target segmentation and positioning.

[0131] In some embodiments, the first training module is specifically used to:

[0132] The adapter layer based on the initial localization model expands the dimensional mapping of the image feature vector to the dimension that meets the input requirements of the initial scene understanding layer;

[0133] Input the mapped and expanded image feature vector into the initial scene understanding layer;

[0134] The initial scene understanding layer is trained based on the mapped expanded image feature vector and the cross entropy loss function.

[0135] In some embodiments, the second training module is specifically used to:

[0136] After the initial scene understanding layer training is completed, the scene understanding layer is obtained;

[0137] Based on the multimodal model and text encoder, the surgical status output by the scene understanding layer is feature extracted to obtain a text feature vector.

[0138] The cross-modal attention mechanism in the initial machine segmentation layer is adjusted based on the image feature vector and the text feature vector.

[0139] In some embodiments, adjusting a cross-modal attention mechanism in an initial machine segmentation layer based on an image feature vector and a text feature vector comprises:

[0140] Dynamically assign weights to each feature based on the similarity between the image feature vector and the text feature vector;

[0141] The image feature vector and the text feature vector are fused based on the multi-layer perceptron and the weights of each feature;

[0142] Adjust the cross-modal attention mechanism based on feature fusion results.

[0143] In some embodiments, the first training module is specifically used to:

[0144] The adapter layer based on the initial localization model expands the dimensional mapping of the image feature vector to the dimension that meets the input requirements of the initial lesion segmentation layer;

[0145] The mapped and expanded image feature vector is input into the initial lesion segmentation layer;

[0146] The initial lesion segmentation layer is trained based on the mapped expanded image feature vector and the loss function.

[0147] In some embodiments, the initial lesion segmentation layer includes a lightweight self-attention mechanism module; the lightweight self-attention mechanism module is used to enhance the attention of the initial lesion segmentation layer to the image feature vector.

[0148] In some embodiments, the loss function includes a cross entropy loss function and a Dai loss function;

[0149] The cross entropy loss function is used for pixel-level classification; the Dai loss is used to measure the similarity between the predicted position and the actual position.

[0150] It should be noted here that the vascular interventional surgery target positioning device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned vascular interventional surgery target positioning method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0151] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 4As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus (Communications Bus) 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic command in the memory 430 to execute the above method, which includes:

[0152] Acquire an initial positioning model, the initial positioning model including a text encoder, an image encoder, an initial scene understanding layer, an initial device segmentation layer, and an initial lesion segmentation layer;

[0153] Extracting image feature vectors of sample surgical images based on an image encoder;

[0154] The initial scene understanding layer and the initial lesion segmentation layer are trained based on the image feature vector;

[0155] The result output by the scene understanding layer is encoded based on the text encoder to obtain the text feature vector of the result; the scene understanding layer is the initial scene understanding layer after the training is completed;

[0156] The initial device segmentation layer is trained based on the image feature vector and the text feature vector to obtain a positioning model;

[0157] Among them, the initial scene understanding layer is used to predict the surgical status of the sample surgical image, which includes the instrument type of the surgical instrument and the steps of the current operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image.

[0158] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software function modules and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0160] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0161] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0162] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0163] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating a target for vascular intervention surgery, characterized in that: include: Acquire an initial positioning model, wherein the initial positioning model includes a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer, and an initial lesion segmentation layer; Extracting an image feature vector of a sample surgical image based on the image encoder; Training the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector; Encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result; the scene understanding layer is the initial scene understanding layer after training; Training the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model; The initial scene understanding layer is used to predict the surgical status of the sample surgical image, and the surgical status includes the instrument type of the surgical instrument and the steps of the current surgical operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image. The step of encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result includes: Extracting features of the surgical status output by the scene understanding layer based on the multimodal model and the text encoder to obtain the text feature vector; The initial device segmentation layer is trained based on the image feature vector and the text feature vector, including: Adjusting a cross-modal attention mechanism in the initial machine segmentation layer based on the image feature vector and the text feature vector; Wherein, adjusting the cross-modal attention mechanism in the initial device segmentation layer based on the image feature vector and the text feature vector includes: Dynamically assigning weights to each feature based on the similarity between the image feature vector and the text feature vector; Performing feature fusion on the image feature vector and the text feature vector based on a multi-layer perceptron and weights of each feature; The cross-modal attention mechanism is adjusted based on the feature fusion result.

2. The target positioning method for vascular intervention surgery according to claim 1, characterized in that: The training of the initial scene understanding layer based on the image feature vector comprises: An adapter layer based on the initial positioning model expands the dimensional mapping of the image feature vector to a dimension that meets the input requirements of the initial scene understanding layer; Inputting the mapped and expanded image feature vector into the initial scene understanding layer; The initial scene understanding layer is trained based on the mapped expanded image feature vector and a cross entropy loss function.

3. The target positioning method for vascular intervention surgery according to claim 1, characterized in that: The training of the initial lesion segmentation layer based on the image feature vector comprises: An adapter layer based on the initial positioning model expands the dimensional mapping of the image feature vector to a dimension that meets the input requirement of the initial lesion segmentation layer; Inputting the mapped and expanded image feature vector into the initial lesion segmentation layer; The initial lesion segmentation layer is trained based on the mapped expanded image feature vector and the loss function.

4. The target positioning method for vascular intervention surgery according to claim 1, characterized in that: The initial lesion segmentation layer includes a lightweight self-attention mechanism module; the lightweight self-attention mechanism module is used to enhance the attention of the initial lesion segmentation layer to the image feature vector.

5. The target positioning method for vascular intervention surgery according to claim 3, characterized in that: The loss function includes a cross entropy loss function and a Dai loss function; The cross entropy loss function is used for pixel-level classification; the Dai loss is used to measure the similarity between the predicted position and the actual position.

6. A target positioning device for vascular intervention surgery, characterized in that: include: An acquisition module, used for acquiring an initial positioning model, wherein the initial positioning model includes a text encoder, an image encoder, an initial scene understanding layer, an initial instrument segmentation layer, and an initial lesion segmentation layer; An extraction module, used for extracting an image feature vector of a sample surgical image based on the image encoder; A first training module, used for training the initial scene understanding layer and the initial lesion segmentation layer based on the image feature vector; An encoding module, used for encoding the result output by the scene understanding layer based on the text encoder to obtain a text feature vector of the result; the scene understanding layer is an initial scene understanding layer after training; A second training module, used for training the initial instrument segmentation layer based on the image feature vector and the text feature vector to obtain a positioning model; The initial scene understanding layer is used to predict the surgical status of the sample surgical image, and the surgical status includes the instrument type of the surgical instrument and the steps of the current surgical operation; the initial instrument segmentation layer is used to predict the position of the surgical instrument; and the initial lesion segmentation layer is used to predict the lesion position of the patient in the sample surgical image. Wherein, the encoding module is specifically used for: Extracting features of the surgical status output by the scene understanding layer based on the multimodal model and the text encoder to obtain the text feature vector; Wherein, the second training module is specifically used for: Adjusting a cross-modal attention mechanism in the initial machine segmentation layer based on the image feature vector and the text feature vector; Wherein, adjusting the cross-modal attention mechanism in the initial device segmentation layer based on the image feature vector and the text feature vector includes: Dynamically assigning weights to each feature based on the similarity between the image feature vector and the text feature vector; Performing feature fusion on the image feature vector and the text feature vector based on a multi-layer perceptron and weights of each feature; The cross-modal attention mechanism is adjusted based on the feature fusion result.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for locating a target for vascular intervention surgery as described in any one of claims 1 to 5 is implemented.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the vascular intervention surgery target positioning method according to any one of claims 1 to 5 through the computer program.

Images