A method and device for segmenting Cooper's ligament based on mastectomy surgery video

Abstract

The present invention discloses a method and device for segmenting Cooper's ligament based on a mastectomy video, comprising: continuously collecting a plurality of first images from a mastectomy video; decoupling the foreground and background of each first image based on the optical flow features between each first image and its adjacent images, and generating a decoupled feature stream of each first image according to the decoupling result and the spatial feature information of each first image; extracting the boundary features of each first image based on the multi-scale feature map and the decoupled feature stream of each first image to obtain a boundary feature estimation result, and then inputting the image hidden features, the multi-scale feature map, the boundary feature estimation result and the decoupled feature stream of each first image into a multi-scale feature decoder for decoding, and obtaining the Cooper's ligament segmentation result of each first image. The present invention utilizes the optical flow features between two adjacent frames of images to decouple the foreground and background of the image, thereby reducing the influence of surgical instrument occlusion on the accuracy of Cooper's ligament segmentation.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a Cooper ligament segmentation method and device based on a mastectomy surgery video. Background Art

[0002] Breast cancer is a type of cancer that develops in the cells of breast tissue. It is one of the most common cancers affecting women worldwide. Breast cancer can originate in different parts of the breast, including the ducts, lobules, or connective tissue. Treatment for breast cancer depends on a variety of factors, such as the type and stage of breast cancer, as well as the individual's overall health. However, mastectomy remains the most common treatment for breast cancer. Cooper's ligaments, also known as Cooper's suspensory ligaments or fibrocollagen septa, are structures within the breast tissue that provide support and help maintain the shape of the breast. They are found throughout the breast tissue and allow the breast tissue to be fixed to the chest wall and skin. During surgery, surgeons need to cut these ligaments in order to completely remove the breast tissue. During mastectomy surgery, doctors need to use an endoscope to obtain video images of the inside of the breast so that doctors can observe the anatomical structure of the surgical site in real time, thereby achieving precise manipulation of the tissue.

[0003] With the advancement of science and technology and the rapid development of computers, it is possible to delineate the region of interest (ROI) in images and videos with the help of deep learning models, thereby helping surgeons to quickly locate the target area to reduce the workload of doctors. For mastectomy, the surgical video is very different from the natural image data. Since the boundary between the Cooper ligament and the surrounding tissue is irregular, unclear and variable in position, the boundary of the Cooper ligament in the surgical video requires very professional doctors to accurately identify and delineate it. Moreover, each frame in a video contains a complex and diverse Cooper ligament area. Based on the delineation results of the Cooper ligament in each frame of a video, it is an extremely time-consuming and laborious task to assist in determining the next operation of the operation, and the labeling cost is very high. In addition, the current technical solution of delineating the region of interest (ROI) in images and videos with the help of deep learning models is easily affected by the occlusion of surgical instruments, and the features extracted during the delineation process are not comprehensive enough. Summary of the invention

[0004] The embodiment of the present invention provides a Cooper ligament segmentation method and device based on mastectomy surgery video, which reduces the influence of surgical instrument occlusion on the accuracy of Cooper ligament segmentation and comprehensively considers different types of image features such as foreground features, background features, and hidden features of depth to improve the accuracy of Cooper ligament segmentation.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a Cooper ligament segmentation method based on a mastectomy surgery video, comprising:

[0006] Continuously acquiring a plurality of first images from a mastectomy surgery video to be detected;

[0007] Based on the optical flow features between each of the first images and the adjacent images of each of the first images, the foreground and the background of each of the first images are decoupled, and according to the decoupling results and the spatial feature information of each of the first images, a decoupled feature flow corresponding to each of the first images is generated; wherein the adjacent image of the first image is a first image adjacent to the first image;

[0008] Based on the multi-scale feature map and the decoupled feature stream corresponding to each of the first images, boundary features are extracted from each of the first images to obtain boundary feature estimation results corresponding to each of the first images, and then the image hidden features corresponding to each of the first images, the multi-scale feature map, the boundary feature estimation results and the decoupled feature stream are input into a multi-scale feature decoder for decoding to obtain the Cooper ligament segmentation results of each of the first images; wherein the multi-scale feature map and the image hidden features are obtained by extracting features from the first images.

[0009] In the implementation of the embodiment of the present invention, a plurality of first images are continuously collected from the mastectomy video to be detected, and then the foreground and background of each first image are decoupled by using the optical flow features between each first image and the adjacent images of each first image, so that attention can be focused on the deformation of human tissue rather than the movement of surgical instruments, so as to reduce the influence of surgical instrument occlusion on the accuracy of Cooper ligament segmentation. Then, according to the decoupling results of the foreground and background and the spatial feature information of each first image, a decoupled feature stream corresponding to each first image is generated, and then based on the multi-scale feature map and the decoupled feature stream corresponding to each first image, the boundary feature of each first image is extracted to obtain the boundary feature estimation result corresponding to each first image. In addition, the image hidden features, multi-scale feature maps, boundary feature estimation results and decoupled feature streams corresponding to each first image are input into the multi-scale feature decoder for decoding. By combining the above contents, the foreground features, background features, convolutional layer features of different scales, irregular and unclear boundary features, and deep hidden features can be fully considered, thereby improving the accuracy of Cooper ligament segmentation.

[0010] As a preferred solution, based on the multi-scale feature map corresponding to each of the first images and the decoupled feature flow, boundary feature extraction is performed on each of the first images to obtain boundary feature estimation results corresponding to each of the first images, specifically:

[0011] Traversing each first image, extracting a cross-attention unit through boundary features, performing a cross-attention operation on the multi-scale feature map and the decoupled feature stream corresponding to the currently traversed first image, and obtaining a spatiotemporal fusion feature corresponding to the currently traversed first image;

[0012] Performing global and local feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image to obtain global features and local features corresponding to the currently traversed first image;

[0013] The global features and the local features corresponding to the currently traversed first image are fused to obtain the global-local fusion features corresponding to the currently traversed first image, and then the global-local fusion features corresponding to the currently traversed first image are multiplied element by element with the boundary feature estimation result corresponding to the first image traversed last time, and then the Sigmad function is used to activate the operation result to obtain the boundary feature estimation result corresponding to the currently traversed first image.

[0014] A preferred solution of the embodiment of the present invention is to traverse each first image, extract a cross-attention unit through boundary features, and perform a cross-attention operation on the multi-scale feature map and decoupled feature stream corresponding to the currently traversed first image, so as to fuse the convolutional layer features of the currently traversed first image with the temporal features between adjacent images, and obtain the spatiotemporal fusion features corresponding to the currently traversed first image. In addition, considering that there is a certain correlation between the features of images at adjacent acquisition moments, the boundary features of the last traversed first image (i.e., the adjacent image of the currently traversed first image) and the boundary features of the currently traversed first image are highly similar, and the boundary feature estimation result corresponding to the last traversed first image is used to perform an element-by-element multiplication operation with the global-local fusion features corresponding to the currently traversed first image, so as to better fuse the multi-scale feature information and improve the accuracy of Cooper ligament segmentation.

[0015] As a preferred solution, the global and local feature extraction is performed on the spatiotemporal fusion feature corresponding to the currently traversed first image to obtain the global feature and local feature corresponding to the currently traversed first image, specifically:

[0016] Using the Mamba layer, respectively performing forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, to obtain the global features corresponding to the currently traversed first image;

[0017] A 3×3 convolutional layer is used to extract local features of the spatiotemporal fusion features corresponding to the currently traversed first image to obtain local features corresponding to the currently traversed first image.

[0018] A preferred solution for implementing the embodiment of the present invention utilizes a novel architecture of parallel 3×3 convolutional layers and Mamba layers to simultaneously extract local information and global information of the first image to enhance the ability to recognize and delineate irregular long boundaries.

[0019] As a preferred solution, based on the optical flow features between each of the first images and the adjacent images of each of the first images, the foreground and background of each of the first images are decoupled, and according to the decoupling results and the spatial feature information of each of the first images, a decoupled feature flow corresponding to each of the first images is generated, specifically:

[0020] Traverse each first image, extract the optical flow features between the current traversed first image and the adjacent image of the current traversed first image through SpyNet, and respectively extract the spatial feature information of the current traversed first image and the spatial feature information of the adjacent image of the current traversed first image through the convolution layer;

[0021] Inputting the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground features, and inputting the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background features;

[0022] The decoupled foreground feature is subtracted from the optical flow feature between the currently traversed first image and the adjacent image of the currently traversed first image, and then the subtraction result is added element by element to the decoupled background feature to obtain the decoupled optical flow feature. Then, based on the decoupled optical flow feature, the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image are sampled respectively to obtain the first decoupled feature corresponding to the currently traversed first image and the second decoupled feature corresponding to the adjacent image of the currently traversed first image. Then, the first decoupled feature and the second decoupled feature are input into the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow corresponding to the currently traversed first image.

[0023] According to the preferred solution of the embodiment of the present invention, the foreground and background in the currently traversed first image can be effectively separated by the foreground decoupling unit and the background decoupling unit, so as to facilitate the subsequent extraction of the Cooper ligament boundary feature. In addition, the decoupled feature stream is mapped by the output mapping unit, which helps to further extract and utilize the useful information in the image, thereby improving the understanding and processing capabilities of the image content.

[0024] As a preferred solution, the multi-scale feature map and the image hidden features are obtained as follows:

[0025] Downsampling each of the first images to obtain a third image corresponding to each of the first images;

[0026] Traverse each third image, extract the multi-scale convolutional layer features of the current traversed third image through the convolutional network, and generate a multi-scale feature map of the first image corresponding to the current traversed third image based on the multi-scale convolutional layer features of the current traversed third image, and then filter the deepest convolutional layer features from the multi-scale convolutional layer features of the current traversed third image as the input of the Transformer layer, so that the Transformer layer extracts hidden features and outputs the image hidden features of the first image corresponding to the current traversed third image.

[0027] A preferred solution for implementing the embodiment of the present invention is to select the deepest convolutional layer features from the multi-scale convolutional layer features currently traversing the third image as the input of the Transformer layer to further extract deeper hidden features.

[0028] As a preferred solution, the method of continuously acquiring a plurality of first images from the mastectomy surgery video to be detected is specifically as follows:

[0029] receiving a mastectomy video to be detected uploaded by a user, and continuously extracting frames of the mastectomy video to be detected for a plurality of times according to a preset extraction frequency to obtain a plurality of second images;

[0030] Down-sampling each of the second images is performed to obtain a first image corresponding to each of the second images.

[0031] According to a preferred solution of the embodiment of the present invention, the received mastectomy surgery video to be detected is continuously frame-sampled several times according to a preset sampling frequency to obtain several second images, and then each second image is downsampled, so that the size of the image can be unified, which is convenient for the subsequent image-based Cooper ligament segmentation task.

[0032] In order to solve the same technical problem, an embodiment of the present invention further provides a Cooper ligament segmentation device based on a mastectomy surgery video, comprising:

[0033] An image acquisition system, used for continuously acquiring a plurality of first images from a mastectomy surgery video to be detected;

[0034] a foreground-background decoupling system, configured to decouple the foreground and background of each of the first images based on the optical flow features between each of the first images and the adjacent images of each of the first images, and to generate a decoupled feature flow corresponding to each of the first images according to the decoupling result and the spatial feature information of each of the first images; wherein the adjacent image of the first image is a first image adjacent to the first image;

[0035] A boundary feature extraction system, configured to extract boundary features from each of the first images based on the multi-scale feature maps corresponding to each of the first images and the decoupled feature flows, to obtain boundary feature estimation results corresponding to each of the first images;

[0036] A feature decoding system is used to input the image hidden features corresponding to each of the first images, the multi-scale feature map, the boundary feature estimation result and the decoupled feature stream into a multi-scale feature decoder for decoding, so as to obtain the Cooper ligament segmentation result of each of the first images; wherein the multi-scale feature map and the image hidden features are obtained by extracting features from the first images.

[0037] As a preferred solution, the boundary feature extraction system specifically includes:

[0038] A spatiotemporal fusion feature acquisition module, used to traverse each first image, extract a cross-attention unit through boundary features, perform a cross-attention operation on the multi-scale feature map and the decoupled feature stream corresponding to the currently traversed first image, and obtain the spatiotemporal fusion feature corresponding to the currently traversed first image;

[0039] A second feature extraction module is used to extract global and local features of the spatiotemporal fusion features corresponding to the currently traversed first image, so as to obtain global features and local features corresponding to the currently traversed first image;

[0040] The boundary feature estimation module is used to fuse the global features and the local features corresponding to the currently traversed first image to obtain the global-local fusion features corresponding to the currently traversed first image, and then perform element-by-element multiplication operation on the global-local fusion features corresponding to the currently traversed first image and the boundary feature estimation result corresponding to the first image traversed last time, and then use the Sigmoid function to activate the operation result to obtain the boundary feature estimation result corresponding to the currently traversed first image.

[0041] As a preferred solution, the second feature extraction module specifically includes:

[0042] A global feature extraction unit, used to use the Mamba layer to perform forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, to obtain the global features corresponding to the currently traversed first image;

[0043] The local feature extraction unit is used to use a 3×3 convolution layer to perform local feature extraction on the spatiotemporal fusion feature corresponding to the currently traversed first image to obtain the local feature corresponding to the currently traversed first image.

[0044] As a preferred solution, the foreground-background decoupling system specifically includes:

[0045] A first feature extraction module is used to traverse each first image, extract the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image through SpyNet, and respectively extract the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image through a convolution layer;

[0046] A foreground-background decoupling module is used to input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground features, and input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background features;

[0047] A feature interaction module is used to subtract the decoupled foreground feature from the optical flow feature between the currently traversed first image and the adjacent image of the currently traversed first image, and then perform element-by-element addition operation on the subtraction result and the decoupled background feature to obtain the decoupled optical flow feature; then, based on the decoupled optical flow feature, respectively sample the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image to obtain the first decoupled feature corresponding to the currently traversed first image and the second decoupled feature corresponding to the adjacent image of the currently traversed first image; then, input the first decoupled feature and the second decoupled feature into the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow corresponding to the currently traversed first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1: A flowchart of a method for segmenting Cooper's ligaments based on a mastectomy surgery video provided in Example 1 of the present invention;

[0049] Figure 2 : A schematic diagram of the structure of the Cooper ligament segmentation model in the surgical video provided in Example 1 of the present invention;

[0050] Figure 3 : A schematic diagram of the structure of a foreground-background decoupling module provided in Embodiment 1 of the present invention;

[0051] Figure 4 : A schematic diagram of the structure of a boundary feature extraction module provided in the first embodiment of the present invention;

[0052] Figure 5 : A schematic diagram of the structure of the Mamba layer provided in the first embodiment of the present invention;

[0053] Figure 6 : A schematic diagram of the structure of a Cooper ligament segmentation device based on a mastectomy surgery video provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Embodiment one:

[0056] It should be noted that during mastectomy, surgeons need to accurately cut the Cooper ligament. However, there are several challenges in accurately identifying the Cooper ligament: ① The Cooper ligament is translucent and is adjacent to the mammary lobules. Due to the differences in the appearance of different mammary lobules, the boundary between the Cooper ligament and the mammary lobules is fuzzy and complex; ② The Cooper ligament is connected to the pectoralis fascia at one end and to the fibrous septum of the breast skin at the other end. It is generally in the shape of a curved cord. This shape of target area is very rare in computer vision segmentation tasks; ③ During the operation, surgical instruments often block the Cooper ligament area, affecting the identification of the Cooper ligament area. Existing video processing models such as DeepLabV3+, PraNet, MFI Net, and C3D can complete the recognition and scene segmentation of natural scene videos and surgical videos in some scenarios, but they are not well adapted to the segmentation task of Cooper ligament in videos. This is because these existing methods still have some shortcomings: ① It is impossible to accurately identify the boundary area of ​​Cooper's ligament. Cooper's ligament has relatively unique structural characteristics (for example, light color, long and irregular boundaries), which makes it difficult for existing models to accurately identify its boundaries; ② It is difficult to eliminate the influence of surgical instrument occlusion. During the operation, the doctor needs to cut the Cooper's ligament, which will inevitably cause occlusion of the target area. Existing methods are difficult to respond accurately to the occlusion behavior of the instrument, which in turn affects the segmentation accuracy of the model.

[0057] Please refer to Figure 1 , a Cooper ligament segmentation method based on a mastectomy surgery video provided by an embodiment of the present invention, the method includes steps S1 to S4, each step is specifically as follows:

[0058] Step S1, continuously acquiring a plurality of first images from a mastectomy surgery video to be detected.

[0059] It should be noted that surgical videos are different from natural image videos and require professionals to use special equipment to collect them. Therefore, in this embodiment, the mastectomy surgery video to be detected is a real-time video collection of the surgical process by professional breast surgeons.

[0060] As a preferred solution, step S1 includes step S11 to step S12, and each step is specifically as follows:

[0061] Step S11, receiving a mastectomy video to be detected uploaded by a user, and continuously extracting frames of the mastectomy video to be detected for several times according to a preset extraction frequency to obtain several second images.

[0062] Step S12: downsample each second image to obtain a first image corresponding to each second image.

[0063] In this embodiment, for the mastectomy video to be detected, video frame extraction is performed several times at a sampling frequency of 4 frames / second to obtain several second images, and then the size of each second image is downsampled to 384*384, and then data enhancement operations are performed on the downsampled second images. The data enhancement operations include but are not limited to scaling, rotation, flipping, and random noise addition operations.

[0064] It should be noted that, for a plurality of first images continuously acquired from the mastectomy video to be detected, the Cooper ligament segmentation of the first image can be implemented by the video Cooper ligament segmentation model. The implementation process of implementing the Cooper ligament segmentation of the first image by the video Cooper ligament segmentation model can refer to steps S2 to S4 of this embodiment. In this embodiment, the construction process of the video Cooper ligament segmentation model is as follows:

[0065] 1) Surgical video collection: Professional breast surgeons collected videos of the surgery during the operation. The video collection subjects were 10 different patients. A video of about 1 hour was collected during the operation of each patient, and a total of 10 surgical videos of about 1 hour were collected. Then, professional surgeons edited the surgical videos, and edited 10 short videos of about 15-20 seconds in each 1-hour video. It was necessary to ensure that every frame in each short video contained the Cooper ligament area. Finally, a total of 100 surgical videos of 15-20 seconds were obtained.

[0066] 2) Data annotation: First, all 15-20s short videos are framed, that is, each frame in the video is extracted and saved as an image in PNG format. Then, because when the frame rate of the video is high, the difference between adjacent frames is very small, which will cause the segmentation model to not be able to capture the temporal characteristics between adjacent frames well, and the high frame rate will increase the burden of model training, so the extracted frame sequence is downsampled. After downsampling, the frame rate of the video changes from 60FPS to 4FPS, that is, 4 frames per second. After downsampling, a total of 6,000 images were obtained, and each image contains the Cooper ligament area. Then, experienced breast surgeons annotated these images to mark the boundaries of the Cooper ligament with a broken line. Because the boundaries of the Cooper ligament are very vague and irregular, even experienced doctors cannot be 100% sure of the accuracy of the annotation results. Therefore, after the annotation is completed, the annotation results are evaluated by the professor of breast surgery to ensure the effectiveness of the annotation and the quality of the data set.

[0067] 3) Model structure: please refer to Figure 2,The Cooper ligament segmentation model for surgical videos includes a convolutional and Transformer hybrid backbone network encoder, a foreground-background decoupling module, a boundary feature extraction module, and a multi-scale feature encoder.

[0068] Please refer to Figure 3 The foreground-background decoupling module includes an optical flow feature extraction network, a 3×3 convolutional layer, a foreground decoupling unit, a background decoupling unit, and an output mapping unit. The input of the foreground-background decoupling module is two adjacent frames, F o is the optical flow feature between two adjacent frames. Two 3×3 convolutional layers are used to extract the spatial feature information of two adjacent frames respectively. The spatial feature information of two adjacent frames and the optical flow feature between two adjacent frames are simultaneously used as the input of the foreground decoupling unit and the background decoupling unit to achieve the decoupling of the foreground and the background and obtain the decoupled foreground feature F n and the decoupled background feature F p After the decoupling operation is completed, the optical flow feature F between the two frames is used o Subtract the decoupled foreground feature F n , and then with the decoupled background feature F p Perform element-by-element addition to obtain the decoupled optical flow feature F w Next, decouple the optical flow feature F w The spatial features of two adjacent frames are sampled as weights to obtain the decoupled features F i and F i+1 , and the decoupled feature F i and F i+1 As the input of the output mapping unit, the final output is the decoupled feature flow F f .

[0069] Please refer to Figure 4 The boundary feature extraction module includes a 3×3 convolutional layer, a boundary feature extraction cross attention unit, and a Mamba layer. is the multi-scale feature map output by the encoder, For is the output feature of the input convolution layer, F f To decouple the characteristic flow, S i+1 The boundary features output by the deep boundary extraction module, S i is the boundary feature output by the current boundary feature extraction module; in the boundary feature extraction cross attention unit, F f After linear mapping, feature Q is generated, which contains the temporal features between different frames. Features K and V are generated through two different linear mapping layers, K and V contain the spatial features of the image; then, the weight W is obtained by cross-attention operation on Q and V, W integrates the temporal-spatial feature information (i.e., temporal features); finally, W and V are matrix multiplied, and the boundary features are further extracted with the help of the temporal-spatial feature information to obtain the output feature F. It should be noted that the Mamba layer is a new type of network architecture, which uses a new selective state space model. The Mamba layer can extract the global context features of the image in both positive and negative directions, thereby more efficiently modeling the global features of the image, while the classic 3×3 convolution layer is used to model the local features of the image.

[0070] Please refer to Figure 5 (a), Conv1D in the Mamba layer is a one-dimensional convolution operation, and SSM is a selective state space model. For a schematic diagram of Forward SSM, please refer to Figure 5 (b), Forward SSM will follow Figure 5 Each image block is traversed in the order shown by the arrows in (b) to extract global context features in forward order. Figure 5 (c), Forward SSM will follow Figure 5 Each image block is traversed in the order shown by the arrows in (c) to extract global context features in reverse order.

[0071] 4) Model training: Since the size and shape of Cooper's ligament vary greatly among different individuals, and the boundary between it and the surrounding tissue is not clear, the video Cooper's ligament segmentation task is very difficult. At present, the segmentation of medical images is mostly based on U-Net and Transformer architectures. However, these traditional architectures cannot effectively utilize the timing information between adjacent frames of the video, which affects the segmentation performance of the model. Before model training, all images are downsampled to 384*384 size, and then the entire data set is divided into training set and test set according to 8:2. After the segmentation model is constructed, it is iteratively trained for 20 cycles to finally obtain the video Cooper's ligament segmentation model.

[0072] It should be noted that, after testing, the video Cooper ligament segmentation model in this embodiment can process 70-75 frames of images per second, and the frame rate of the current surgical camera in the endoscope is generally 30-60 frames per second. This shows that the video Cooper ligament segmentation model in this embodiment can segment the video images generated during the operation in real time, and finally automatically outline the predicted segmentation results in the original video image through the algorithm, so as to achieve a visual effect of segmentation. According to the prediction results of the segmentation model, surgeons can be assisted in making more accurate judgments on the positioning of the Cooper ligament during the operation, so as to save operation time and improve the efficiency of the operation.

[0073] Step S2, based on the optical flow features between each first image and its adjacent images, decouple the foreground and background of each first image, and generate a decoupled feature flow corresponding to each first image according to the decoupling result and the spatial feature information of each first image.

[0074] The adjacent image of the first image is a first image adjacent to the first image.

[0075] As a preferred solution, step S2 includes step S21 to step S23, and each step is specifically as follows:

[0076] Step S21, traverse each first image, use SpyNet as the optical flow feature extraction network, and extract the current traversed first image I through SpyNet. i With the current traversal of the first image I i+1 The optical flow feature F between adjacent images o , and through a 3×3 convolutional layer, respectively extract the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent images of the currently traversed first image.

[0077] Step S22: the first image I currently traversed i The adjacent image I of the first image currently traversed i+1 The optical flow feature F between o , the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image are input into the foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground feature F n , and the first image I currently traversed i The adjacent image I of the first image currently traversed i+1 The optical flow feature F between o , the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image are input into the background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background feature Fp .

[0078] Step S23: the first image I currently being traversed i The adjacent image I of the first image currently traversed i+1 The optical flow feature F between o Subtract the decoupled foreground feature F n , and then the subtraction result is combined with the decoupled background feature F p Perform element-by-element addition to obtain the decoupled optical flow feature F w , the decoupling operation of foreground-background features helps to focus attention on the area to be segmented in this embodiment, namely Cooper's ligament, thereby reducing the influence of irrelevant information (such as the optical flow features generated by the movement of surgical instruments) on the segmentation effect; then, the decoupled optical flow feature F w As weights, the first image I traversed is i The spatial feature information of the first image and the adjacent image I of the current traversal i+1 The spatial feature information of the image is sampled to obtain the first image I i The corresponding first decoupling feature F i , and the adjacent image I of the current traversal first image i+1 The corresponding second decoupling feature F i+1 , and then the first decoupling feature I i and the second decoupling feature I i+1 The input is sent to the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow F corresponding to the current traversal of the first image f .

[0079] Step S3: extracting boundary features of each first image based on the multi-scale feature graph and the decoupled feature flow corresponding to each first image, and obtaining boundary feature estimation results corresponding to each first image.

[0080] As a preferred solution, step S3 includes step S31 to step S33, and each step is specifically as follows:

[0081] Step S31, traverse each first image, extract the cross attention unit through the boundary feature, and perform the cross attention unit on the current traversed first image I i The corresponding multi-scale feature map F m i and decoupled feature flow F f Perform a cross-attention operation to obtain the spatiotemporal fusion features corresponding to the first image currently traversed.

[0082] In this embodiment, the multi-scale feature map F corresponding to the currently traversed first image is m i Input the convolutional layer so that the convolutional layer outputs the feature Fd i m , then the feature F d i m and decoupled feature flow F f As the input of the boundary feature extraction cross attention unit, to achieve the multi-scale feature map F corresponding to the current traversal of the first image m i and decoupled feature flow F f Cross-attention operation.

[0083] Step S32, extracting global and local features of the spatiotemporal fusion features corresponding to the currently traversed first image, and obtaining global features and local features corresponding to the currently traversed first image.

[0084] As a preferred solution, step S32 includes step S321 to step S322, and each step is specifically as follows:

[0085] Step S321, please refer to Figure 5 (a)(b)(c) uses the Mamba layer to perform forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, and obtains the global features corresponding to the currently traversed first image.

[0086] Step S322: Using a 3×3 convolutional layer, extract local features of the spatiotemporal fusion features corresponding to the currently traversed first image to obtain local features corresponding to the currently traversed first image.

[0087] Step S33, fusing the global features and local features corresponding to the currently traversed first image to obtain the global-local fusion features corresponding to the currently traversed first image, and then performing element-by-element multiplication operation on the global-local fusion features corresponding to the currently traversed first image and the boundary feature estimation result corresponding to the last traversed first image, and then activating the operation result using the Sigmoid function to obtain the currently traversed first image I i The corresponding boundary feature estimation result S i .

[0088] Step S4, inputting the image hidden features, multi-scale feature maps, boundary feature estimation results and decoupled feature streams corresponding to each first image into a multi-scale feature decoder for decoding to obtain the Cooper ligament segmentation results of each first image.

[0089] The multi-scale feature map and the image hidden features are obtained by extracting features from the first image.

[0090] As a preferred solution, the process of obtaining the multi-scale feature map and the image hidden features includes steps S01 to S02, and each step is specifically as follows:

[0091] Step S01: down-sample each first image to obtain a third image corresponding to each first image.

[0092] In this embodiment, the frame rate of the original video is 60 FPS, and after downsampling, the frame rate is 4 FPS, that is, 4 frames per second.

[0093] Step S02, traverse each third image, extract the multi-scale convolutional layer features of the currently traversed third image through the convolutional network, and generate a multi-scale feature map of the first image corresponding to the currently traversed third image based on the multi-scale convolutional layer features of the currently traversed third image, and then filter the deepest convolutional layer features from the multi-scale convolutional layer features of the currently traversed third image as the input of the Transformer layer, so that the Transformer layer extracts hidden features and outputs the image hidden features of the first image corresponding to the currently traversed third image.

[0094] Please refer to Figure 6 , is a schematic diagram of the structure of a Cooper ligament segmentation device based on a mastectomy video provided by an embodiment of the present invention. The device includes an image acquisition system T1, a foreground-background decoupling system T2, a boundary feature extraction system T3 and a feature decoding system T4. The specifics of each system are as follows:

[0095] The image acquisition system T1 is used to continuously acquire a plurality of first images from the mastectomy surgery video to be detected;

[0096] The foreground-background decoupling system T2 is used to decouple the foreground and background of each first image based on the optical flow features between each first image and the adjacent image of each first image, and generate a decoupled feature flow corresponding to each first image according to the decoupling result and the spatial feature information of each first image; wherein the adjacent image of the first image is a first image adjacent to the first image;

[0097] The boundary feature extraction system T3 is used to extract boundary features of each first image based on the multi-scale feature map and decoupled feature flow corresponding to each first image, so as to obtain boundary feature estimation results corresponding to each first image;

[0098] The feature decoding system T4 is used to input the image hidden features, multi-scale feature maps, boundary feature estimation results and decoupled feature streams corresponding to each first image into the multi-scale feature decoder for decoding, so as to obtain the Cooper ligament segmentation results of each first image; wherein the multi-scale feature map and the image hidden features are obtained by extracting features from the first image.

[0099] As a preferred solution, the boundary feature extraction system T3 specifically includes a spatiotemporal fusion feature acquisition module M31, a second feature extraction module M32 and a boundary feature estimation module M33, and each module is specifically as follows:

[0100] The spatiotemporal fusion feature acquisition module M31 is used to traverse each first image, extract the cross-attention unit through the boundary feature, perform the cross-attention operation on the multi-scale feature map and the decoupled feature flow corresponding to the currently traversed first image, and obtain the spatiotemporal fusion feature corresponding to the currently traversed first image;

[0101] The second feature extraction module M32 is used to extract global and local features of the spatiotemporal fusion features corresponding to the currently traversed first image, so as to obtain global features and local features corresponding to the currently traversed first image;

[0102] The boundary feature estimation module M33 is used to fuse the global features and local features corresponding to the currently traversed first image to obtain the global-local fusion features corresponding to the currently traversed first image, and then perform element-by-element multiplication operation on the global-local fusion features corresponding to the currently traversed first image and the boundary feature estimation result corresponding to the first image traversed last time, and then use the Sigmoid function to activate the operation result to obtain the boundary feature estimation result corresponding to the currently traversed first image.

[0103] As a preferred solution, the second feature extraction module M32 specifically includes a global feature extraction unit 321 and a local feature extraction unit 322, and the specific details of each unit are as follows:

[0104] The global feature extraction unit 321 is used to use the Mamba layer to perform forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, so as to obtain the global features corresponding to the currently traversed first image;

[0105] The local feature extraction unit 322 is used to use a 3×3 convolution layer to perform local feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image to obtain the local features corresponding to the currently traversed first image.

[0106] As a preferred solution, the foreground-background decoupling system T2 specifically includes a first feature extraction module M21, a foreground-background decoupling module M22 and a feature interaction module M23, and the specific details of each module are as follows:

[0107] A first feature extraction module M21 is used to traverse each first image, extract the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image through SpyNet, and respectively extract the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image through a convolution layer;

[0108] A foreground-background decoupling module M22 is used to input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground features, and input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background features;

[0109] The feature interaction module M23 is used to subtract the decoupled foreground feature from the optical flow feature between the currently traversed first image and the adjacent image of the currently traversed first image, and then perform element-by-element addition operation on the subtraction result and the decoupled background feature to obtain the decoupled optical flow feature. Then, based on the decoupled optical flow feature, the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image are sampled respectively to obtain the first decoupled feature corresponding to the currently traversed first image and the second decoupled feature corresponding to the adjacent image of the currently traversed first image. Then, the first decoupled feature and the second decoupled feature are input into the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow corresponding to the currently traversed first image.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0112] The present invention provides a method and device for segmenting Cooper's ligament based on a mastectomy video. A plurality of first images are continuously collected from a mastectomy video to be detected. Then, the optical flow features between each first image and the adjacent images of each first image are used to decouple the foreground and background of each first image, so that attention can be focused on the deformation of human tissue rather than the movement of surgical instruments, so as to reduce the influence of surgical instrument occlusion on the accuracy of Cooper's ligament segmentation. Then, according to the decoupling results of the foreground and background and the spatial feature information of each first image, a decoupled feature stream corresponding to each first image is generated. Then, based on the multi-scale feature map and the decoupled feature stream corresponding to each first image, the boundary feature of each first image is extracted to obtain the boundary feature estimation result corresponding to each first image. In addition, the image hidden features, multi-scale feature maps, boundary feature estimation results and decoupled feature streams corresponding to each first image are input into a multi-scale feature decoder for decoding. By combining the above contents, the foreground features, background features, convolutional layer features of different scales, irregular and unclear boundary features, and deep hidden features can be fully considered, thereby improving the accuracy of Cooper's ligament segmentation.

[0113] Furthermore, considering that there is a certain correlation between the features of images at adjacent acquisition moments, the boundary features of the last traversed first image (i.e., the adjacent image of the current traversed first image) and the current traversed first image have a high similarity. The boundary feature estimation result corresponding to the last traversed first image and the global-local fusion feature corresponding to the current traversed first image are used to perform element-by-element multiplication, which can better fuse multi-scale feature information to improve the accuracy of Cooper ligament segmentation. In addition, the new architecture of parallel 3×3 convolutional layers and Mamba layers is used to extract local and global information of the first image at the same time to enhance the recognition and delineation capabilities of irregular long boundaries.

[0114] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Cooper ligament segmentation method based on mastectomy surgery video, characterized in that: include: Continuously acquiring a plurality of first images from a mastectomy surgery video to be detected; Based on the optical flow features between each of the first images and the adjacent images of each of the first images, the foreground and the background of each of the first images are decoupled, and according to the decoupling results and the spatial feature information of each of the first images, a decoupled feature flow corresponding to each of the first images is generated; wherein the adjacent image of the first image is a first image adjacent to the first image; Based on the multi-scale feature map and the decoupled feature stream corresponding to each of the first images, boundary feature extraction is performed on each of the first images to obtain boundary feature estimation results corresponding to each of the first images, and then the image hidden features corresponding to each of the first images, the multi-scale feature map, the boundary feature estimation results and the decoupled feature stream are input into a multi-scale feature decoder for decoding to obtain the Cooper ligament segmentation results of each of the first images; wherein the multi-scale feature map and the image hidden features are obtained by extracting features from the first images; The step of extracting boundary features from each of the first images based on the multi-scale feature maps and the decoupled feature flows corresponding to each of the first images to obtain boundary feature estimation results corresponding to each of the first images is specifically as follows: Traversing each first image, extracting a cross-attention unit through boundary features, performing a cross-attention operation on the multi-scale feature map and the decoupled feature stream corresponding to the currently traversed first image, and obtaining a spatiotemporal fusion feature corresponding to the currently traversed first image; Performing global and local feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image to obtain global features and local features corresponding to the currently traversed first image; The global features and the local features corresponding to the currently traversed first image are fused to obtain the global-local fusion features corresponding to the currently traversed first image, and then the global-local fusion features corresponding to the currently traversed first image are multiplied element by element with the boundary feature estimation result corresponding to the first image traversed last time, and then the operation result is activated using the Sigmoid function to obtain the boundary feature estimation result corresponding to the currently traversed first image.

2. The Cooper ligament segmentation method based on mastectomy surgery video according to claim 1, characterized in that: The global and local feature extraction is performed on the spatiotemporal fusion feature corresponding to the currently traversed first image to obtain the global feature and the local feature corresponding to the currently traversed first image, specifically: Using the Mamba layer, respectively performing forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, to obtain the global features corresponding to the currently traversed first image; A 3×3 convolutional layer is used to extract local features of the spatiotemporal fusion features corresponding to the currently traversed first image to obtain local features corresponding to the currently traversed first image.

3. The method for segmenting Cooper's ligaments based on mastectomy surgery video according to claim 1, characterized in that: The foreground and background of each first image are decoupled based on the optical flow features between each first image and its adjacent images, and a decoupled feature flow corresponding to each first image is generated according to the decoupling result and the spatial feature information of each first image, specifically: Traverse each first image, extract the optical flow features between the current traversed first image and the adjacent image of the current traversed first image through SpyNet, and respectively extract the spatial feature information of the current traversed first image and the spatial feature information of the adjacent image of the current traversed first image through the convolution layer; Inputting the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground features, and inputting the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background features; The decoupled foreground feature is subtracted from the optical flow feature between the currently traversed first image and the adjacent image of the currently traversed first image, and then the subtraction result is added element by element to the decoupled background feature to obtain the decoupled optical flow feature. Then, based on the decoupled optical flow feature, the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image are sampled respectively to obtain the first decoupled feature corresponding to the currently traversed first image and the second decoupled feature corresponding to the adjacent image of the currently traversed first image. Then, the first decoupled feature and the second decoupled feature are input into the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow corresponding to the currently traversed first image.

4. The Cooper ligament segmentation method based on mastectomy surgery video according to claim 1, characterized in that: The acquisition of the multi-scale feature map and the image hidden features is specifically as follows: Downsampling each of the first images to obtain a third image corresponding to each of the first images; Traverse each third image, extract the multi-scale convolutional layer features of the current traversed third image through the convolutional network, and generate a multi-scale feature map of the first image corresponding to the current traversed third image based on the multi-scale convolutional layer features of the current traversed third image, and then filter the deepest convolutional layer features from the multi-scale convolutional layer features of the current traversed third image as the input of the Transformer layer, so that the Transformer layer extracts hidden features and outputs the image hidden features of the first image corresponding to the current traversed third image.

5. The Cooper ligament segmentation method based on mastectomy surgery video according to claim 1, characterized in that: The method of continuously acquiring a plurality of first images from the mastectomy surgery video to be detected is specifically as follows: receiving a mastectomy video to be detected uploaded by a user, and continuously extracting frames of the mastectomy video to be detected for a plurality of times according to a preset extraction frequency to obtain a plurality of second images; Down-sampling each of the second images is performed to obtain a first image corresponding to each of the second images.

6. A Cooper ligament segmentation device based on a mastectomy video, characterized in that: include: An image acquisition system, used for continuously acquiring a plurality of first images from a mastectomy surgery video to be detected; a foreground-background decoupling system, configured to decouple the foreground and background of each of the first images based on the optical flow features between each of the first images and the adjacent images of each of the first images, and to generate a decoupled feature flow corresponding to each of the first images according to the decoupling result and the spatial feature information of each of the first images; wherein the adjacent image of the first image is a first image adjacent to the first image; A boundary feature extraction system, configured to extract boundary features from each of the first images based on the multi-scale feature maps corresponding to each of the first images and the decoupled feature flows, to obtain boundary feature estimation results corresponding to each of the first images; A feature decoding system, used for inputting the image hidden features corresponding to each of the first images, the multi-scale feature map, the boundary feature estimation result and the decoupled feature stream into a multi-scale feature decoder for decoding, so as to obtain the Cooper ligament segmentation result of each of the first images; wherein the multi-scale feature map and the image hidden features are obtained by extracting features from the first images; Wherein, the boundary feature extraction system specifically includes: A spatiotemporal fusion feature acquisition module, used to traverse each first image, extract a cross-attention unit through boundary features, perform a cross-attention operation on the multi-scale feature map and the decoupled feature stream corresponding to the currently traversed first image, and obtain the spatiotemporal fusion feature corresponding to the currently traversed first image; A second feature extraction module is used to extract global and local features of the spatiotemporal fusion features corresponding to the currently traversed first image, so as to obtain global features and local features corresponding to the currently traversed first image; The boundary feature estimation module is used to fuse the global features and the local features corresponding to the currently traversed first image to obtain the global-local fusion features corresponding to the currently traversed first image, and then perform element-by-element multiplication operation on the global-local fusion features corresponding to the currently traversed first image and the boundary feature estimation result corresponding to the first image traversed last time, and then use the Sigmoid function to activate the operation result to obtain the boundary feature estimation result corresponding to the currently traversed first image.

7. The Cooper ligament segmentation device based on mastectomy surgery video according to claim 6, characterized in that: The second feature extraction module specifically includes: A global feature extraction unit, used to use the Mamba layer to perform forward global context feature extraction and reverse global context feature extraction on the spatiotemporal fusion features corresponding to the currently traversed first image, to obtain the global features corresponding to the currently traversed first image; The local feature extraction unit is used to use a 3×3 convolution layer to perform local feature extraction on the spatiotemporal fusion feature corresponding to the currently traversed first image to obtain the local feature corresponding to the currently traversed first image.

8. The Cooper ligament segmentation device based on mastectomy surgery video according to claim 6, characterized in that: The foreground-background decoupling system specifically includes: A first feature extraction module is used to traverse each first image, extract the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image through SpyNet, and respectively extract the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image through a convolution layer; A foreground-background decoupling module is used to input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a foreground decoupling unit, so that the foreground decoupling unit decouples the foreground of the currently traversed first image to obtain the decoupled foreground features, and input the optical flow features between the currently traversed first image and the adjacent image of the currently traversed first image, the spatial feature information of the currently traversed first image, and the spatial feature information of the adjacent image of the currently traversed first image into a background decoupling unit, so that the background decoupling unit decouples the background of the currently traversed first image to obtain the decoupled background features; A feature interaction module is used to subtract the decoupled foreground feature from the optical flow feature between the currently traversed first image and the adjacent image of the currently traversed first image, and then perform element-by-element addition operation on the subtraction result and the decoupled background feature to obtain the decoupled optical flow feature; then, based on the decoupled optical flow feature, respectively sample the spatial feature information of the currently traversed first image and the spatial feature information of the adjacent image of the currently traversed first image to obtain the first decoupled feature corresponding to the currently traversed first image and the second decoupled feature corresponding to the adjacent image of the currently traversed first image; then, input the first decoupled feature and the second decoupled feature into the output mapping unit for mapping processing, so that the output mapping unit outputs the decoupled feature flow corresponding to the currently traversed first image.

Images