A liver minimally invasive surgery video segmentation method, system, computer device and storage medium

Abstract

The present application relates to the technical field of surgical video segmentation, and provides a liver minimally invasive surgery video segmentation method, system, computer device and storage medium, the method comprising: acquiring a liver minimally invasive surgery video to be processed and a corresponding video description text, and analyzing the liver minimally invasive surgery video to obtain a plurality of original video images; using a pre-trained segmentation model to process the original video images and the video description text to generate a segmentation mask; and adding the segmentation mask to the liver minimally invasive surgery video. The present application provides a liver minimally invasive surgery video segmentation method, system, computer device and storage medium, which guides the segmentation process based on the video description text, and fully integrates the text features of the video description text and the image features of the liver minimally invasive surgery video, so that the accuracy of the segmentation result is high.

Description

Technical Field

[0001] This invention relates to the field of surgical video segmentation technology, specifically to a method, system, computer equipment, and storage medium for segmenting minimally invasive liver surgery videos. Background Technology

[0002] Recent advances in artificial intelligence have led to a surge in the application of computer vision technology in surgical video analysis. Minimally invasive surgeries generate a large volume of surgical videos, providing new opportunities for improving computer vision techniques in this field. In minimally invasive liver surgery, the liver contains multiple blood vessels, such as the hepatic pedicle (GP) and hepatic vein (HV). Damage to these vessels during liver resection can cause intraoperative bleeding and other events, jeopardizing patient safety and prognosis. Deep learning models can assist surgeons in identifying these blood vessels, significantly improving surgical safety. However, the anatomical structures of liver blood vessels are similar. Current semantic segmentation frameworks used in minimally invasive liver resection, while assigning a category label to each pixel in an image, do not distinguish between different instances of the same category, making it difficult to differentiate between GP and HV blood vessels.

[0003] Currently, most video semantic segmentation tasks employ unimodal segmentation models, which heavily rely on the information inherent in that modality and the capabilities of the model's own framework. To improve segmentation performance by incorporating information from other modalities, text-image multimodal segmentation has garnered significant exploration and discussion within the deep learning community. For semantic segmentation, some studies have proposed unimodal segmentation models such as DeepLab v3+, which introduce dilated convolutions to effectively increase the receptive field without losing image information. For tasks involving image segmentation incorporating textual information, the Clip model has been proposed, employing a text-image comparison approach. The image uses VIT encoding, while the text uses BERT encoding, achieving multimodal information fusion. For instance segmentation, the Mask R-CNN model has been proposed, using bilinear interpolation to eliminate spatial misalignment between feature maps and candidate boxes, improving the accuracy of instance segmentation mask prediction.

[0004] Single-modal segmentation models, such as DeepLab v3+, introduce dilated convolutions, effectively increasing the receptive field without losing image information. However, their drawback lies in their heavy reliance on the information contained in the single modality of the image and their limitations in handling blurred instance boundaries and insensitivity to intra-class differences, making it difficult to distinguish between generalized (GP) and holographic (HV) images for instance segmentation. For tasks involving image segmentation incorporating textual information, the Clip model has been proposed, employing a contrastive learning approach between text and images. Images use VIT encoding, and text uses BERT encoding, achieving multimodal fusion of visual image and textual information. However, its drawback is that simple multimodal contrastive learning cannot fully utilize the rich information contained within the multimodal structures. Sufficient cross-modal interaction is crucial for multi-channel learning to improve text pixel alignment. In CLIP encoding, there is no interaction between the image and text; only the final global features of the two modalities are matched through contrastive learning. This is effective for image-level tasks such as classification or image-text retrieval, but insufficient for pixel-level image segmentation tasks. While the Mask R-CNN model can perform instance segmentation tasks, it relies on information from a single modality image, and its deep feature map has low resolution, making it easy to lose details of small targets and miss dense small targets. Therefore, it has insufficient accuracy in segmenting small blood vessels in minimally invasive liver surgery videos. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method, system, computer equipment, and storage medium for segmenting videos of minimally invasive liver surgery. The segmentation process is guided by video description text, and the text features of the video description text are fully integrated with the image features of the minimally invasive liver surgery video, resulting in high accuracy of the segmentation results.

[0006] To achieve the above objectives, the specific solution adopted by the present invention is as follows:

[0007] A method for video segmentation in minimally invasive liver surgery includes the following steps:

[0008] The process involves acquiring a minimally invasive liver surgery video and its corresponding video description text, and then parsing the video to obtain multiple raw video images.

[0009] A segmentation mask is generated by processing the original video image and the video description text using a pre-trained segmentation model.

[0010] Add the segmentation mask to the minimally invasive liver surgery video;

[0011] The method by which the segmentation model processes the original video image and the video description text includes:

[0012] Basic image features are obtained by performing feature extraction on the original video image, and basic text features are obtained by performing feature extraction on the video description text;

[0013] The basic image features and the basic text features are initially aligned to generate optimized text features and optimized image features;

[0014] Based on the basic image features, the basic text features, the optimized text features, and the optimized image features, multiple encodings are performed to generate encoded image features and encoded text features, wherein cross-attention mechanism is used to optimize alignment between adjacent encodings;

[0015] The encoded image features and the encoded text features are initially fused to obtain basic fused features;

[0016] The basic fusion features are enhanced multiple times based on a multi-head self-attention mechanism to obtain enhanced fusion features.

[0017] The enhanced fusion features are segmented and decoded to generate the segmentation mask.

[0018] The method for parsing the minimally invasive liver surgery video to obtain multiple original video images includes:

[0019] Frame analysis was performed on the video of the minimally invasive liver surgery to obtain multiple frame images;

[0020] The frame image is segmented based on a preset segmentation size to obtain multiple original video images.

[0021] Preferably, the segmentation model includes a hybrid encoder, which includes an image encoder and a text encoder. Both the image encoder and the text encoder use a Vision Transformer module as the backbone network. The image encoder includes an image feature extraction layer for extracting features from the original video image to obtain the basic image features. The text encoder includes a text feature extraction layer for extracting features from the original video image.

[0022] Preferably, the method for preliminary alignment of the basic image features and the basic text features includes:

[0023] The optimized image features are obtained by fusing the basic text features as Q and the basic image features as K and V.

[0024] The optimized text features are obtained by fusing the basic image features as Q and the basic text features as K and V.

[0025] Preferably, the method for performing multiple encodings based on the basic image features, the basic text features, the optimized text features, and the optimized image features includes:

[0026] During the first encoding, the basic image features and the optimized text features are combined and encoded, and the basic text features and the optimized image features are combined and encoded.

[0027] In subsequent encoding, the result of the previous encoding, after being optimized and aligned by the cross-attention mechanism, is encoded.

[0028] Preferably, the segmentation model includes a hybrid decoder, which includes multiple multi-head self-attention blocks. The multi-head self-attention blocks are used to enhance the basic fusion features based on the multi-head self-attention mechanism. The multi-head self-attention blocks include a multi-head self-attention layer, a first connection grouping layer, a feedforward layer, and a second connection grouping layer arranged sequentially.

[0029] Preferably, the method for enhancing the basic fusion features based on the multi-head self-attention block includes:

[0030] The first intermediate feature is obtained by processing the basic fusion features using the multi-head self-attention layer;

[0031] The first enhanced feature is obtained by connecting and normalizing the basic fusion feature and the first intermediate feature using the first connection layer;

[0032] The first enhanced feature is processed using the feedforward layer to obtain the second intermediate feature;

[0033] The second enhanced feature is obtained by connecting and normalizing the first enhanced feature and the second intermediate feature using the second connection normalization layer;

[0034] The second enhancement feature output by the previous multi-head self-attention block is used as the input to the next multi-head self-attention block.

[0035] A video segmentation system for minimally invasive liver surgery includes:

[0036] The data acquisition module is used to acquire the minimally invasive liver surgery video to be processed and the corresponding video description text, and to parse the minimally invasive liver surgery video to obtain multiple raw video images;

[0037] The data processing module is used to process the original video image and the video description text using a pre-trained segmentation model to generate a segmentation mask;

[0038] A video generation module is used to add the segmentation mask to the minimally invasive liver surgery video.

[0039] Computer equipment, including:

[0040] Memory, used to store computer programs;

[0041] A processor is used to read and execute the computer program to implement the above-described method for video segmentation in minimally invasive liver surgery.

[0042] A storage medium for storing a computer program that, when executed, implements the aforementioned method for segmenting video footage of minimally invasive liver surgery.

[0043] This invention combines video description text with minimally invasive liver surgery videos, using the textual information in the video description text to guide the video segmentation process. This avoids the low accuracy problem caused by traditional segmentation relying on single-modal data, effectively improving the accuracy of the segmentation results. The segmentation model of this invention is based on cross-attention mechanism and multi-head self-attention mechanism, which fully aligns and fuses text features and image features, thereby ensuring the accuracy of the segmentation results. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the segmentation method of the present invention;

[0046] Figure 2 This is a schematic diagram of the segmentation model;

[0047] Figure 3 This is a schematic diagram of the segmentation system of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of the computer device of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] like Figure 1 and Figure 2 As shown, the present invention first provides a method for video segmentation in minimally invasive liver surgery, including S1 to S3.

[0051] S1. Obtain the video of the minimally invasive liver surgery to be processed and the corresponding video description text, and parse the video to obtain multiple raw video images. The video description text is designed by the physician based on their prior knowledge to assist in segmenting the video, or it can be designed based on standard knowledge in the field of minimally invasive liver surgery. The specific content of the video description text may include the names of hepatic vessels and their corresponding morphological characteristics, where the names of hepatic vessels may include the hepatic pedicle (GP) and hepatic vein (HV), etc.

[0052] More specifically, the method for analyzing minimally invasive liver surgery videos to obtain multiple raw video images includes S11 to S12.

[0053] S11. Frame analysis is performed on the minimally invasive liver surgery video to obtain multiple frame images. The specific method for frame analysis of the minimally invasive liver surgery video utilizes mature existing technologies, such as the existing FFmpeg program. After obtaining the frame images through frame analysis, the frame images can be filtered to remove some frames that do not contain liver vessels, thereby accelerating the subsequent video segmentation process. Alternatively, video segments that clearly do not contain liver vessels can be deleted from the minimally invasive liver surgery video before frame analysis.

[0054] S12. Segment the frame image based on a preset segmentation size to obtain multiple original video images. By segmenting the frame image to obtain the original video images, the size of the original video images is significantly smaller than that of the frame image, thereby reducing the complexity of subsequent data processing. The segmentation size can be determined based on the actual resolution of the minimally invasive liver surgery video, or it can be determined based on the input constraints of the subsequent segmentation model.

[0055] Similarly, if the length of the video description text exceeds a preset length threshold, it can be segmented into multiple text segments. Segmentation can be based on both the length and semantics of the video description text. For example, if the video description text contains at least two complete sentences, it can be segmented based on periods to obtain at least two individual sentences. Segmenting the video description text also reduces the complexity of subsequent data processing.

[0056] S2. Use a pre-trained segmentation model to process the original video image and video description text to generate a segmentation mask.

[0057] More specifically, the segmentation model includes a hybrid encoder and a hybrid decoder. The hybrid encoder includes an image encoder and a text encoder, both of which use a Vision Transformer module as their backbone network. The image encoder includes an image feature extraction layer for extracting features from the original video images to obtain basic image features, and the text encoder includes a text feature extraction layer for extracting features from the video description text. The hybrid decoder includes multiple multi-head self-attention blocks, which are used to enhance the basic fused features based on a multi-head self-attention mechanism. Each multi-head self-attention block includes a multi-head self-attention layer, a first connection convergence layer, a feedforward layer, and a second connection convergence layer, arranged sequentially.

[0058] More specifically, based on the structure of the segmentation model described above, in S2, the methods by which the segmentation model processes the original video image and the video description text include S21 to S26.

[0059] S21. Basic image features are obtained by extracting features from the original video images, and basic text features are obtained by extracting features from the video description text. Since the original video images are segmented from frame images extracted from a minimally invasive liver surgery video, there are multiple original video images. During feature extraction, multiple basic image features corresponding to the original video images can be obtained, denoted as Fi1. Similarly, multiple basic text features can be obtained by extracting features from the video description text, denoted as Ft1. It should also be noted that after obtaining the basic text features Ft1, positional encoding is performed on the basic text features Ft1.

[0060] S22. Based on the attention mechanism, perform preliminary alignment of basic image features and basic text features to generate optimized text features and optimized image features.

[0061] More specifically, the methods for preliminary alignment of basic image features and basic text features include S221 to S222.

[0062] S221. Using the basic text feature Ft1 as Q and the basic image feature Fi1 as K and V, fuse the basic text feature Ft1 and the basic image feature Fi1 to obtain the optimized image feature. Here, Q is Query, K is Key, and V is Value, which are basic definitions in attention mechanisms and will not be elaborated further here. The fused optimized image feature can be denoted as Fci1.

[0063] S222. Using the basic image feature Fi1 as Q and the basic text feature Ft1 as K and V, fuse the basic image feature Fi1 and the basic text feature Ft1 to obtain the optimized text feature. The optimized text feature obtained by fusion can be denoted as Fct1.

[0064] After initial alignment, the segmentation model can initially identify text features and image features before starting encoding, avoiding excessive separation of information from two different sources, which would make subsequent fusion difficult.

[0065] S23. Based on basic image features, basic text features, optimized text features, and optimized image features, multiple encoding processes are performed to generate coded image features and coded text features. Between adjacent encoding processes, optimization alignment is performed based on a cross-attention mechanism. To fully integrate the basic text features Ft1 and the basic image features Fi1, after initial alignment, this invention further performs multiple optimization alignment processes. More specifically, the method for multiple encoding processes based on basic image features, basic text features, optimized text features, and optimized image features includes S231 to S232.

[0066] S231. During the first encoding, the basic image feature Fi1 and the optimized text feature Fct1 are combined and encoded, and the basic text feature Ft1 and the optimized image feature Fci1 are combined and encoded. Specifically, the combination of the basic image feature Fi1 and the optimized text feature Fct1 is encoded to obtain the intermediate image feature Fi2, and the combination of the basic text feature Ft1 and the optimized image feature Fci1 is encoded to obtain the intermediate text feature Ft2.

[0067] S232. In subsequent encoding, the result of the previous encoding, after optimization and alignment through the cross-attention mechanism, is encoded. That is, in any two adjacent encoding processes, optimization and alignment are performed based on the cross-attention mechanism. Multiple alignments can be performed across multiple encoding processes, thus fully realizing the alignment and fusion of text features and image features. The output of the final encoding is denoted as encoded image feature FiN and encoded text feature FtN, respectively.

[0068] S24. Perform preliminary fusion of encoded image features FiN and encoded text features FtN to obtain basic fused features.

[0069] S25. Based on a multi-head self-attention mechanism, the basic fusion features are enhanced multiple times to obtain enhanced fusion features. For example... Figure 2 As shown, the multi-head self-attention mechanism is implemented based on multi-head self-attention blocks. The structure of the multi-head sub-attention block includes a multi-head self-attention layer, a first connection grouping layer, a feedforward layer, and a second connection grouping layer. Figure 2 In this invention, the multi-head self-attention layer is represented as Multi-Head Attention, the first connection grouped into one layer and the second connection grouped into one layer are represented as Add & Norm, but in different positions, and the feedforward layer is represented as Feed Forward. Multiple multi-head self-attention blocks are set in this invention, enabling multiple enhancements of the basic fusion features to obtain enhanced fusion features.

[0070] More specifically, methods for enhancing basic fusion features based on multi-head self-attention blocks include S251 to S255.

[0071] S251. The first intermediate feature is obtained by processing the basic fused features using a multi-head attention layer. The multi-head attention layer can dynamically capture long-distance dependencies in the basic fused features.

[0072] S252. The first enhanced feature is obtained by connecting and normalizing the basic fused feature and the first intermediate feature using the first connection normalization layer. The basic fused feature and the first intermediate feature can be fused together using residual connections.

[0073] S253. The first enhanced feature is processed using a feedforward layer to obtain the second intermediate feature. The feedforward layer processes the first enhanced feature to achieve nonlinear feature transformation and information fusion.

[0074] S254. The first enhanced feature and the second intermediate feature are connected and normalized using a second connection layer to obtain the second enhanced feature. The first enhanced feature and the second intermediate feature can also be fused together using residual connections. Residual connections are a conventional technique in this field and will not be elaborated here.

[0075] In S252 and S254, gradient vanishing and depth optimization can be prevented by introducing the first connection to a layer and the second connection to a layer.

[0076] S255. The second enhancement feature output by the previous multi-head self-attention block is used as the input to the next multi-head self-attention block.

[0077] S26. Perform segmented decoding on the enhanced fusion features to generate a segmentation mask. After enhancement by multiple multi-head self-attention blocks, the output of the last multi-head self-attention block is input into the Segmentation Decoder to identify the pixels of the segmentation target in the original video image and generate a segmentation mask.

[0078] To further ensure the performance of the segmentation model and the accuracy of the output segmentation mask, this invention sets two loss functions during the training of the segmentation model. The first loss function is used to calculate between the segmentation mask and the frame image, and the second loss function is used to calculate between the original video image and the segmentation mask.

[0079] S3. Add the segmentation mask to the minimally invasive liver surgery video. Specifically, the segmentation mask can be added to the original video images first, then the original video images can be recombined into frame images with the segmentation mask, and finally all frame images can be re-fused to output a minimally invasive liver surgery video with the segmentation mask.

[0080] This invention combines video description text with minimally invasive liver surgery videos, using the textual information in the video description text to guide the video segmentation process. This avoids the low accuracy problem caused by traditional segmentation relying on single-modal data, effectively improving the accuracy of the segmentation results. The segmentation model of this invention is based on cross-attention mechanism and multi-head self-attention mechanism, which fully aligns and fuses text features and image features, thereby ensuring the accuracy of the segmentation results.

[0081] like Figure 3 As shown, the present invention further provides a video segmentation system for minimally invasive liver surgery, including a data acquisition module 21, a data processing module 22, and a video generation module 23.

[0082] The data acquisition module 21 is used to acquire the minimally invasive liver surgery video to be processed and the corresponding video description text, and to parse the minimally invasive liver surgery video to obtain multiple raw video images.

[0083] The data processing module 22 is used to process the original video image and video description text using a pre-trained segmentation model to generate a segmentation mask.

[0084] The video generation module 23 is used to add a segmentation mask to the video of minimally invasive liver surgery.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or modules may be electrical, mechanical, or other forms.

[0086] like Figure 4 As shown, the present invention also provides a computer device, including a memory 33 and a processor 32.

[0087] The memory 33 is used to store the computer program 31.

[0088] The processor 32 is used to read and execute the computer program 31 to implement the above-described method for video segmentation in minimally invasive liver surgery.

[0089] Finally, the present invention provides a storage medium for storing a computer program that, when executed, implements the above-described method for video segmentation in minimally invasive liver surgery.

[0090] It should be understood that the processor 32 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors 32, digital signal processors 32 (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 32 may be a microprocessor 32, or it may be any conventional processor 32, etc.

[0091] It should also be understood that the memory 33 in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0092] The aforementioned computer device may further include an input unit 34 and a display unit 35. The input unit 34 is used to receive signal input and user-input keywords. The input unit 34 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (e.g., operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive corresponding connection devices according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (e.g., play control buttons, power buttons), trackballs, mice, joysticks, etc. The display unit 35 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. The display unit 35 may be in the form of a liquid crystal display (LCD), organic light-emitting diode (OLED), etc. The processor 32 is the control center of the terminal device, connecting various parts of the device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 33 and calling data stored in the memory.

[0093] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage system such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0094] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0097] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for liver minimally invasive surgery video segmentation, characterized in that, The method comprises the following steps: acquiring a liver minimally invasive surgery video to be processed and a corresponding video description text, and parsing the liver minimally invasive surgery video to obtain a plurality of original video images; processing the original video images and the video description text by using a pre-trained segmentation model to generate a segmentation mask; adding the segmentation mask to the liver minimally invasive surgery video; the segmentation model comprises a hybrid encoder, the hybrid encoder comprises an image encoder and a text encoder, and the image encoder and the text encoder both take a Vision Transformer module as a backbone network, the image encoder comprises an image feature extraction layer for extracting features of the original video images, and the text encoder comprises a text feature extraction layer for extracting features of the video description text; the method for processing the original video images and the video description text by using the segmentation model comprises: extracting features of the original video images to obtain basic image features, and extracting features of the video description text to obtain basic text features; performing preliminary alignment on the basic image features and the basic text features to generate optimized text features and optimized image features; the method for performing preliminary alignment on the basic image features and the basic text features comprises: taking the basic text features as Q and the basic image features as K and V, and fusing the basic text features and the basic image features to obtain the optimized image features; taking the basic image features as Q and the basic text features as K and V, and fusing the basic image features and the basic text features to obtain the optimized text features; performing multiple encodings based on the basic image features, the basic text features, the optimized text features and the optimized image features to generate encoded image features and encoded text features, wherein, between two adjacent encodings, optimization alignment is performed based on a cross-attention mechanism; the method for performing multiple encodings based on the basic image features, the basic text features, the optimized text features and the optimized image features comprises: in the first encoding, combining the basic image features and the optimized text features and then encoding, and combining the basic text features and the optimized image features and then encoding; in subsequent encodings, encoding the results of the previous encoding after optimization alignment by the cross-attention mechanism; performing preliminary fusion on the encoded image features and the encoded text features to obtain basic fusion features; performing multiple enhancements on the basic fusion features based on a multi-head self-attention mechanism to obtain enhanced fusion features; performing segmentation decoding on the enhanced fusion features to generate the segmentation mask.

2. The liver minimally invasive surgery video segmentation method of claim 1, wherein, The method for parsing the liver minimally invasive surgery video to obtain a plurality of original video images comprises: performing frame parsing on the liver minimally invasive surgery video to obtain a plurality of frame images; segmenting the frame images based on a preset segmentation size to obtain a plurality of original video images.

3. The liver minimally invasive surgery video segmentation method of claim 1, wherein, The segmentation model comprises a hybrid decoder comprising a plurality of multi-head self-attention blocks for enhancing the basic fusion features based on a multi-head self-attention mechanism, wherein each multi-head self-attention block comprises a multi-head self-attention layer, a first connection normalization layer, a feedforward layer and a second connection normalization layer arranged in sequence.

4. The liver minimally invasive surgery video segmentation method of claim 3, wherein, The method for enhancing the basic fusion features based on the multi-head self-attention mechanism comprises: processing the basic fusion features by using the multi-head self-attention layer to obtain a first intermediate feature; connecting and normalizing the basic fusion features and the first intermediate feature by using the first connection normalization layer to obtain a first enhanced feature; processing the first enhanced feature by using the feedforward layer to obtain a second intermediate feature; connecting and normalizing the first enhanced feature and the second intermediate feature by using the second connection normalization layer to obtain a second enhanced feature; the second enhanced feature output by a previous multi-head self-attention block is taken as the input of a subsequent multi-head self-attention block.

5. Computer device, characterized in that comprise: a memory for storing a computer program; a processor for reading and executing the computer program to implement the liver minimally invasive surgery video segmentation method according to any one of claims 1-4.

6. Storage medium, characterized in that a computer program for storing, which, when executed, implements the liver minimally invasive surgery video segmentation method according to any one of claims 1-4.

Images