Liver minimally invasive surgery video segmentation method and system, computer equipment and storage medium

Abstract

The invention relates to the technical field of operation video segmentation, and provides a liver minimally invasive operation video segmentation method and system, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed liver minimally invasive operation video and a corresponding video description text, analyzing the liver minimally invasive surgery video to obtain a plurality of original video images; processing the original video image and the video description text by using a pre-trained segmentation model to generate a segmentation mask; and adding the segmentation mask to the liver minimally invasive surgery video. According to the liver minimally invasive surgery video segmentation method and system, the computer equipment and the storage medium, the segmentation process is guided based on the video description text, the text features of the video description text and the image features of the liver minimally invasive surgery video are fully fused, and the accuracy of the segmentation result is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surgical video segmentation, in particular to a liver minimally invasive surgery video segmentation method and system, computer equipment and storage medium. BACKGROUND

[0002] The latest progress of artificial intelligence has triggered a surge in the application of computer vision technology in surgical video analysis. Minimally invasive surgery generates a large amount of surgical videos, providing a new opportunity for the improvement of minimally invasive surgery computer vision technology. In liver minimally invasive surgery, due to the multiple vessels of the liver, such as the hepatic peduncle (GP) and the hepatic vein (HV), damage to the vessels during liver resection surgery will cause intraoperative bleeding and other events, endangering the safety of patients during surgery and prognosis. Artificial intelligence deep learning models can assist surgeons in identifying vessels, greatly improving surgical safety. However, the anatomical structure of liver vessels is similar, and the current semantic segmentation framework applied to minimally invasive liver resection can assign a class label to each pixel point on the image due to its characteristics, but it cannot distinguish different instances of the same class, so it is difficult to distinguish GP from HV in liver vessels.

[0003] Currently, single-modal segmentation models are mostly used for video semantic segmentation tasks, which heavily rely on the information contained in the modal and the ability of the model itself. In order to add information from other modalities to improve segmentation performance in this semantic segmentation task, text-image multi-modal segmentation has caused extensive exploration and discussion in the deep learning community. For semantic segmentation tasks, some studies have proposed single-modal based segmentation models such as DeepLab v3+, which introduces dilated convolution, effectively increasing the receptive field without losing image information. For the task of segmenting images combined with text information, some studies have proposed the Clip model, which uses a contrastive approach to text and image, where the image is encoded using the vit structure and the text is encoded using bert, achieving multi-modal information fusion of text and image. For instance segmentation tasks, some studies have proposed the Mask R-CNN model, which eliminates the spatial misalignment between the feature map and the candidate box through bilinear interpolation, improving the accuracy of instance segmentation mask prediction.

[0004] The single-modal segmentation model such as DeepLab v3+ introduces a hole convolution, which effectively increases the receptive field without losing image information. However, the defect is that it relies heavily on the information contained in the image single mode, and is still limited by the problems of blurred instance boundary, insensitive to intra-class differences, and difficult to distinguish GP and HV to achieve instance segmentation task. For the task of segmenting images combined with text information, some studies have proposed the Clip model, which uses the contrast learning method of text and image, where the image uses the vit structure encoding, and the text uses the bert encoding to realize the multi-modal fusion of visual image and text information. However, the defect is that the simple multi-modal contrast learning is difficult to fully utilize the rich information contained in the multi-modal. Sufficient cross-modal interaction is crucial for multi-channel learning to improve text pixel alignment. In the CLIP encoding, there is no interaction process between the image and the text, and only the final global features of the two modalities are matched through contrast learning, which is effective for image-level tasks such as classification or image-text retrieval, but it is insufficient for pixel-level image segmentation tasks. Although the Mask R-CNN model can achieve the instance segmentation task, it only relies on the information of a single image mode, and the resolution of the deep feature map is low, and small target details are easy to lose, and dense small targets are still easy to miss, and there is a lack of segmentation accuracy for small blood vessels in the liver minimally invasive surgery video. SUMMARY

[0005] In order to solve the problems in the prior art, the present application provides a liver minimally invasive surgery video segmentation method, system, computer device and storage medium, which is guided by video description text during segmentation process, and fully combines the text features of video description text and the image features of liver minimally invasive surgery video, so that the accuracy of segmentation result is high.

[0006] In order to achieve the above purpose, the specific scheme adopted by the present application is as follows:

[0007] A liver minimally invasive surgery video segmentation method, comprising the following steps:

[0008] Obtaining a liver minimally invasive surgery video to be processed and corresponding video description text, and analyzing the liver minimally invasive surgery video to obtain a plurality of original video images;

[0009] Processing the original video images and the video description text by using a pre-trained segmentation model to generate a segmentation mask;

[0010] Adding the segmentation mask to the liver minimally invasive surgery video;

[0011] The method for processing the original video images and the video description text by the segmentation model comprises:

[0012] extracting features from the original video images to obtain basic image features, and extracting features from the video description text to obtain basic text features;

[0013] preliminarily aligning the basic image features and the basic text features to generate optimized text features and optimized image features;

[0014] generating encoded image features and encoded text features through multiple encodings based on the basic image features, the basic text features, the optimized text features and the optimized image features, wherein the optimized alignment is performed based on a cross-attention mechanism between adjacent two encodings;

[0015] preliminarily fusing the encoded image features and the encoded text features to obtain basic fusion features;

[0016] performing multiple enhancements on the basic fusion features based on a multi-head self-attention mechanism to obtain enhanced fusion features;

[0017] segmenting and decoding the enhanced fusion features to generate the segmentation mask.

[0018] The method for parsing the liver minimally invasive surgery video to obtain a plurality of original video images comprises:

[0019] frame parsing the liver minimally invasive surgery video to obtain a plurality of frame images;

[0020] segmenting the frame images based on a preset segmentation size to obtain a plurality of original video images.

[0021] Preferably, the segmentation model comprises a hybrid encoder, the hybrid encoder comprises an image encoder and a text encoder, and the image encoder and the text encoder both take a Vision Transformer module as a backbone network, the image encoder comprises an image feature extraction layer for extracting features from the original video images to obtain the basic image features, the image encoder comprises an image feature extraction layer for extracting features from the original video images, and the text encoder comprises a text feature extraction layer for extracting features from the video description text.

[0022] Preferably, the method for preliminarily aligning the basic image features and the basic text features comprises:

[0023] taking the basic text features as Q and the basic image features as K and V, and fusing the basic text features and the basic image to obtain the optimized image features;

[0024] The base image feature is taken as Q, and the base text feature is taken as K and V, and the base image feature and the base text feature are fused to obtain the optimized text feature.

[0025] Preferably, the method of multiple encoding based on the base image feature, the base text feature, the optimized text feature and the optimized image feature comprises:

[0026] In the first encoding, the base image feature and the optimized text feature are combined and then encoded, and the base text feature and the optimized image feature are combined and then encoded;

[0027] In the subsequent encoding, the result of the previous encoding after the cross-attention mechanism optimization alignment is encoded.

[0028] Preferably, the segmentation model comprises a hybrid decoder, and the hybrid decoder comprises a plurality of multi-head self-attention blocks, the multi-head self-attention blocks are used to enhance the base fusion feature based on a multi-head self-attention mechanism, and the multi-head self-attention block comprises a multi-head self-attention layer, a first connection normalization layer, a feedforward layer and a second connection normalization layer arranged in sequence.

[0029] Preferably, the method of enhancing the base fusion feature based on the multi-head self-attention mechanism of the multi-head self-attention block comprises:

[0030] The base fusion feature is processed by the multi-head self-attention layer to obtain a first intermediate feature;

[0031] The base fusion feature and the first intermediate feature are connected and normalized by the first connection normalization layer to obtain a first enhanced feature;

[0032] The first enhanced feature is processed by the feedforward layer to obtain a second intermediate feature;

[0033] The first enhanced feature and the second intermediate feature are connected and normalized by the second connection normalization layer to obtain a second enhanced feature;

[0034] The second enhanced feature output by the previous multi-head self-attention block is taken as the input of the subsequent multi-head self-attention block.

[0035] A liver minimally invasive surgery video segmentation system comprises:

[0036] A data acquisition module is configured to acquire a liver minimally invasive surgery video to be processed and a corresponding video description text, and parse the liver minimally invasive surgery video to obtain a plurality of original video images.

[0037] a data processing module, configured to process the original video image and the video description text by using a pre-trained segmentation model to generate a segmentation mask;

[0038] a video generation module, configured to add the segmentation mask to the liver minimally invasive surgery video.

[0039] a computer device, comprising:

[0040] a memory, configured to store a computer program;

[0041] a processor, configured to read and execute the computer program to implement the above-described liver minimally invasive surgery video segmentation method.

[0042] a storage medium, configured to store a computer program, which is executed to implement the above-described liver minimally invasive surgery video segmentation method.

[0043] The present application combines video description text with liver minimally invasive surgery video, uses the text information of the video description text to guide the segmentation process of the video, can avoid the problem of low accuracy caused by traditional segmentation relying on single modal data, and effectively improves the accuracy of the segmentation result. The segmentation model of the present application is based on cross attention mechanism and multi-head self-attention mechanism, and fully aligns and fuses the text features and image features, thereby ensuring the accuracy of the segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 is a flowchart of the segmentation method of the present application;

[0046] Figure 2 is a structural schematic diagram of the segmentation model;

[0047] Figure 3 is a structural schematic diagram of the segmentation system of the present application;

[0048] Figure 4 is a structural schematic diagram of the computer device of the present application. DETAILED DESCRIPTION

[0049] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0050] As shown in Figure 1 and Figure 2 The present application first provides a liver minimally invasive surgery video segmentation method, comprising S1 to S3.

[0051] S1, obtaining a liver minimally invasive surgery video to be processed and a corresponding video description text, and analyzing the liver minimally invasive surgery video to obtain a plurality of original video images. The video description text is designed by a doctor based on his prior knowledge to assist in segmenting the liver minimally invasive surgery video, or the video description text can be designed based on standard knowledge in the field of liver minimally invasive surgery. The specific content of the video description text can include the name of the liver vessel and the corresponding morphological characteristics, wherein the name of the liver vessel can include the hepatic pedicle (GP) and the hepatic vein (HV) and the like.

[0052] More specifically, the method of analyzing the liver minimally invasive surgery video to obtain a plurality of original video images comprises S11 to S12.

[0053] S11, frame analysis is performed on the liver minimally invasive surgery video to obtain a plurality of frame images. The specific way of frame analysis on the liver minimally invasive surgery video is a mature prior art, for example, the existing FFmpeg program can be used for frame analysis. After frame analysis is performed on the liver minimally invasive surgery video to obtain frame images, the frame images can be screened, and some frame images not containing liver vessels can be removed to speed up the subsequent video segmentation process. In addition, before frame analysis is performed, the video segments in the liver minimally invasive surgery video that do not obviously contain liver vessels can be deleted, and then frame analysis is performed.

[0054] S12, based on a preset segmentation size, the frame images are segmented to obtain a plurality of original video images. By segmenting the frame images to obtain the original video images, the size of the original video images is greatly smaller than that of the frame images, thereby reducing the complexity of the subsequent data processing process. The segmentation size can be determined according to the actual resolution of the liver minimally invasive surgery video, or can be determined according to the input limit of the subsequent segmentation model.

[0055] Similarly, if the length of the video description text exceeds the preset length threshold, the video description text can also be segmented to obtain multiple text segments. When segmenting the video description text, segmentation can be performed based on the length and semantics of the video description text. For example, when the video description text contains at least two complete sentences, the video description text can be segmented based on the period in the video description text to obtain at least two single sentences. By segmenting the video description text, the complexity of subsequent data processing can also be reduced.

[0056] S2, processing the original video image and the video description text by using the pre-trained segmentation model to generate a segmentation mask.

[0057] More specifically, the segmentation model includes a hybrid encoder and a hybrid decoder. The hybrid encoder includes an image encoder and a text encoder, and the image encoder and the text encoder both use a Vision Transformer module as a backbone network. The image encoder includes an image feature extraction layer for extracting basic image features from the original video image. The image encoder includes an image feature extraction layer for extracting features from the original video image. The text encoder includes a text feature extraction layer for extracting features from the video description text. The hybrid decoder includes multiple multi-head self-attention blocks. The multi-head self-attention block is used to enhance the basic fusion features based on the multi-head self-attention mechanism. The multi-head self-attention block includes a multi-head self-attention layer, a first connection normalization layer, a feedforward layer, and a second connection normalization layer arranged in sequence.

[0058] More specifically, based on the structure of the segmentation model described above, in S2, the method for processing the original video image and the video description text by the segmentation model includes S21 to S26.

[0059] S21, extracting features from the original video image to obtain basic image features, and extracting features from the video description text to obtain basic text features. Because the original video image is obtained by segmenting the frame image extracted from the liver minimally invasive surgery video, the number of original video images is multiple. When performing feature extraction, multiple basic image features corresponding to the original video images can also be obtained, which can be denoted as Fii. Similarly, when performing feature extraction on the video description text, multiple basic text features can also be obtained, which can be denoted as Fti. It should be noted that after obtaining the basic text features Fti, the basic text features Fti are positionally encoded.

[0060] S22, aligning the basic image features and the basic text features based on the attention mechanism to generate optimized text features and optimized image features.

[0061] More specifically, the method of preliminary alignment of the base image features and the base text features includes S221 to S222.

[0062] S221, fuse the base text features Ft1 and the base image features Fi1 to obtain optimized image features, taking the base text features Ft1 as Q and the base image features Fi1 as K and V. Q is Query, K is Key, and V is Value, which are basic definitions in the attention mechanism and will not be described here. The fused optimized image features can be denoted as Fci1.

[0063] S222, fuse the base image features Fi1 and the base text features Ft1 to obtain optimized text features, taking the base image features Fi1 as Q and the base text features Ft1 as K and V. The fused optimized text features can be denoted as Fct1.

[0064] After preliminary alignment, the segmentation model can preliminarily identify text features and image features before starting encoding, avoiding excessive fragmentation of information from different sources, which leads to subsequent fusion difficulties.

[0065] S23, multiple encodings are performed based on the base image features, the base text features, the optimized text features, and the optimized image features to generate encoded image features and encoded text features, and the cross-attention mechanism is used for optimization alignment between adjacent two encodings. In order to fully fuse the base text features Ft1 and the base image features Fi1, after preliminary alignment, the present application further performs multiple optimization alignments. More specifically, the method of multiple encodings based on the base image features, the base text features, the optimized text features, and the optimized image features includes S231 to S232.

[0066] S231, in the first encoding, the base image features Fi1 and the optimized text features Fct1 are combined and then encoded, and the base text features Ft1 and the optimized image features Fci1 are combined and then encoded. Wherein, the base image features Fi1 and the optimized text features Fct1 are combined and then encoded to obtain intermediate image features Fi2, and the base text features Ft1 and the optimized image features Fci1 are combined and then encoded to obtain intermediate text features Ft2.

[0067] S232, in subsequent encoding, the result of the previous encoding after optimization alignment by the cross-attention mechanism is encoded. That is, in any two adjacent encoding processes, optimization alignment is performed based on the cross-attention mechanism, and multiple alignment can be performed in the multiple encoding processes, so as to fully realize the alignment and fusion of text features and image features. The output of the last encoding is denoted as encoded image features FiN and encoded text features FtN.

[0068] S24. Perform preliminary fusion of encoded image features FiN and encoded text features FtN to obtain basic fused features.

[0069] S25. The basic fusion features are enhanced multiple times based on a multi-head self-attention mechanism to obtain enhanced fusion features. For example... Figure 2 As shown, the multi-head self-attention mechanism is implemented based on multi-head self-attention blocks. The structure of the multi-head sub-attention block includes a multi-head self-attention layer, a first connection grouping layer, a feedforward layer, and a second connection grouping layer. Figure 2 In this invention, the multi-head self-attention layer is represented as Multi-Head Attention, the first connection grouped into one layer and the second connection grouped into one layer are represented as Add & Norm, but in different positions, and the feedforward layer is represented as Feed Forward. Multiple multi-head self-attention blocks are set in this invention, enabling multiple enhancements of the basic fusion features to obtain enhanced fusion features.

[0070] More specifically, methods for enhancing basic fusion features based on multi-head self-attention blocks include S251 to S255.

[0071] S251. The first intermediate feature is obtained by processing the basic fused features using a multi-head attention layer. The multi-head attention layer can dynamically capture long-distance dependencies in the basic fused features.

[0072] S252. The first enhanced feature is obtained by connecting and normalizing the basic fused feature and the first intermediate feature using the first connection normalization layer. The basic fused feature and the first intermediate feature can be fused together using residual connections.

[0073] S253. The first enhanced feature is processed using a feedforward layer to obtain the second intermediate feature. The feedforward layer processes the first enhanced feature to achieve nonlinear feature transformation and information fusion.

[0074] S254. The first enhanced feature and the second intermediate feature are connected and normalized using a second connection layer to obtain the second enhanced feature. The first enhanced feature and the second intermediate feature can also be fused together using residual connections. Residual connections are a conventional technique in this field and will not be elaborated here.

[0075] In S252 and S254, gradient vanishing and depth optimization can be prevented by introducing the first connection to a layer and the second connection to a layer.

[0076] S255, the second enhancement feature output by the previous multi-head self-attention block is used as the input to the next multi-head self-attention block.

[0077] S26, segment decoding is performed on the enhanced fusion features to generate a segmentation mask. After enhancement by multiple multi-head self-attention blocks, the output of the last multi-head self-attention block is input into the segmentation decoder to identify the pixels of the segmentation target in the original video image and generate a segmentation mask.

[0078] To further ensure the performance of the segmentation model and the accuracy of the output segmentation mask, two loss functions are set during the training of the segmentation model, wherein the first loss function is used to calculate between the segmentation mask and the frame image, and the second loss function is used to calculate between the original video image and the segmentation mask.

[0079] S3, adding the segmentation mask to the liver minimally invasive surgery video. Specifically, the segmentation mask can be added to the original video image first, and then the original video image is recombined into a frame image with the segmentation mask, and then all the frame images are re-fused to output a liver minimally invasive surgery video with the segmentation mask.

[0080] The present application combines video description text with liver minimally invasive surgery video, uses the text information of the video description text to guide the segmentation process of the video, can avoid the problem of low accuracy caused by traditional segmentation relying on single modal data, and effectively improves the accuracy of the segmentation result; the segmentation model of the present application is based on cross attention mechanism and multi-head self-attention mechanism, and fully aligns and fuses the text features and image features, thereby ensuring the accuracy of the segmentation result.

[0081] As shown in Figure 3 The present application further provides a liver minimally invasive surgery video segmentation system, comprising a data acquisition module 21, a data processing module 22 and a video generation module 23.

[0082] The data acquisition module 21 is used to acquire the liver minimally invasive surgery video to be processed and the corresponding video description text, and analyze the liver minimally invasive surgery video to obtain a plurality of original video images.

[0083] The data processing module 22 is used to process the original video image and the video description text by using the pre-trained segmentation model to generate a segmentation mask.

[0084] The video generation module 23 is used to add the segmentation mask to the liver minimally invasive surgery video.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and the device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there can be another division in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed modules can be indirect coupling or communication connection through some communication interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0086] As shown in Figure 4 The present application further provides a computer device, comprising a memory 33 and a processor 32.

[0087] The memory 33 is used for storing a computer program 31.

[0088] The processor 32 is used for reading and executing the computer program 31 to realize the above-mentioned liver minimally invasive surgery video segmentation method.

[0089] The present application further provides a computer device, comprising a memory 33 and a processor 32.

[0090] It should be understood that the processor 32 in the embodiments of the present application can be a central processing unit (CPU), and the processor 32 can also be other general-purpose processors 32, digital signal processors 32 (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor 32 can be a microprocessor or the processor 32 can also be any conventional processor.

[0091] It should also be understood that the memory 33 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0092] The computer device described above can further include an input unit 34 and a display unit 35. The input unit 34 is configured to receive input of a signal and receive a keyword input by a user. The input unit 34 can include a touch panel and other input devices. The touch panel can collect a touch operation of a user thereon or adjacent thereto (such as an operation of the user using a finger, a stylus, or any suitable object or accessory on or adjacent to the touch panel) and drive a corresponding connection device according to a preset program; the other input devices can include, but are not limited to, one or more of a physical keyboard, function keys (such as play control buttons, switch buttons, etc.), a trackball, a mouse, a joystick, etc. The display unit 35 can be configured to display information input by a user or information provided to a user and various menus of the terminal device. The display unit 35 can take the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 32 is a control center of the terminal device, connects all parts of the entire device through various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 33 and calling data stored in the memory.

[0093] The above-described embodiments can be implemented, in whole or in part, by software, hardware (e.g., circuitry), firmware or any combination thereof. When implemented in software, the above-described embodiments can be implemented, in whole or in part, by a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs cause, at least in part, the processes or functions described in the embodiments of the present application to be performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage system such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0094] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, another division manner can be used. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0095] If the functions are realized in the form of software function units and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer system (which can be a personal computer, a server, or a network system, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0096] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0097] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for video segmentation in minimally invasive liver surgery, characterized in that, Includes the following steps: The process involves acquiring a minimally invasive liver surgery video and its corresponding video description text, and then parsing the video to obtain multiple raw video images. A segmentation mask is generated by processing the original video image and the video description text using a pre-trained segmentation model. Add the segmentation mask to the minimally invasive liver surgery video; The method by which the segmentation model processes the original video image and the video description text includes: Basic image features are obtained by performing feature extraction on the original video image, and basic text features are obtained by performing feature extraction on the video description text; The basic image features and the basic text features are initially aligned to generate optimized text features and optimized image features; Based on the basic image features, the basic text features, the optimized text features, and the optimized image features, multiple encodings are performed to generate encoded image features and encoded text features, wherein cross-attention mechanism is used to optimize alignment between adjacent encodings; The encoded image features and the encoded text features are initially fused to obtain basic fused features; The basic fusion features are enhanced multiple times based on a multi-head self-attention mechanism to obtain enhanced fusion features. The enhanced fusion features are segmented and decoded to generate the segmentation mask.

2. The method for video segmentation in minimally invasive liver surgery as described in claim 1, characterized in that, The method for parsing the minimally invasive liver surgery video to obtain multiple original video images includes: Frame analysis was performed on the video of the minimally invasive liver surgery to obtain multiple frame images; The frame image is segmented based on a preset segmentation size to obtain multiple original video images.

3. The method for video segmentation in minimally invasive liver surgery as described in claim 1, characterized in that, The segmentation model includes a hybrid encoder, which comprises an image encoder and a text encoder. Both the image encoder and the text encoder use a Vision Transformer module as their backbone network. The image encoder includes an image feature extraction layer for extracting features from the original video image to obtain the basic image features. The text encoder includes a text feature extraction layer for extracting features from the video description text.

4. The method for video segmentation in minimally invasive liver surgery as described in claim 3, characterized in that, The method for preliminary alignment of the basic image features and the basic text features includes: The optimized image features are obtained by fusing the basic text features as Q and the basic image features as K and V. The optimized text features are obtained by fusing the basic image features as Q and the basic text features as K and V.

5. The method for video segmentation in minimally invasive liver surgery as described in claim 3, characterized in that, The method for performing multiple encodings based on the basic image features, the basic text features, the optimized text features, and the optimized image features includes: During the first encoding, the basic image features and the optimized text features are combined and encoded, and the basic text features and the optimized image features are combined and encoded. In subsequent encoding, the result of the previous encoding, after being optimized and aligned by the cross-attention mechanism, is encoded.

6. The method for video segmentation in minimally invasive liver surgery as described in claim 1, characterized in that, The segmentation model includes a hybrid decoder, which includes multiple multi-head self-attention blocks. The multi-head self-attention blocks are used to enhance the basic fusion features based on the multi-head self-attention mechanism. The multi-head self-attention blocks include a multi-head self-attention layer, a first connection grouping layer, a feedforward layer, and a second connection grouping layer arranged sequentially.

7. The method for video segmentation in minimally invasive liver surgery as described in claim 6, characterized in that, The method for enhancing the basic fusion features based on the multi-head self-attention block using a multi-head self-attention mechanism includes: The first intermediate feature is obtained by processing the basic fusion features using the multi-head self-attention layer; The first enhanced feature is obtained by connecting and normalizing the basic fusion feature and the first intermediate feature using the first connection layer; The first enhanced feature is processed using the feedforward layer to obtain the second intermediate feature; The second enhanced feature is obtained by connecting and normalizing the first enhanced feature and the second intermediate feature using the second connection normalization layer; The second enhancement feature output by the previous multi-head self-attention block is used as the input to the next multi-head self-attention block.

8. A video segmentation system for minimally invasive liver surgery, characterized in that, include: The data acquisition module is used to acquire the minimally invasive liver surgery video to be processed and the corresponding video description text, and to parse the minimally invasive liver surgery video to obtain multiple raw video images; The data processing module is used to process the original video image and the video description text using a pre-trained segmentation model to generate a segmentation mask; A video generation module is used to add the segmentation mask to the minimally invasive liver surgery video.

9. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for reading and executing the computer program to implement a video segmentation method for minimally invasive liver surgery as described in any one of claims 1-7.

10. A storage medium, characterized in that, Used to store a computer program, which, when executed, implements a method for video segmentation in minimally invasive liver surgery as described in any one of claims 1-7.

Images