Endoscopic surgery video multi-target segmentation method based on basic segmentation large model and mixed expert fine tuning

Abstract

The invention provides an endoscopic surgery video multi-target segmentation method based on a basic segmentation large model and mixed expert fine tuning. The method comprises the following steps: preprocessing a data set, establishing an SAM2 baseline segmentation network, constructing a hierarchical hybrid expert module, constructing a stage gating network, establishing an endoscopic surgery video segmentation network, training the endoscopic surgery video segmentation network, and automatically segmenting an endoscopic surgery video by the segmentation network. According to the invention, accurate segmentation of different operation scenes can be realized. The hierarchical hybrid expert module and the stage gating network are introduced, and the problems of scene difference and multi-organization segmentation commonly existing in endoscopic surgery video segmentation tasks can be well solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, in particular to an endoscopic surgery video multi-object segmentation method based on a basic segmentation large model and mixed expert fine-tuning. BACKGROUND

[0002] Medical image segmentation, as a core technology of medical image analysis, plays a key role in computer-aided diagnosis systems. For example, in the field of endoscopic surgery video analysis, different objects such as surgical instruments, tissues, and lesions are segmented. Accurate identification and division of endoscopic surgery video images provide a technical basis for real-time navigation during surgery, intelligent analysis of surgical scenes, and other functions, which is of great significance to the development of computer-aided surgery systems.

[0003] Currently, endoscopic surgery video segmentation technology based on deep learning architecture has become the mainstream of research, among which models based on Transformer architecture (such as Vision Transformer, basic segmentation large model, etc.) have shown significant advantages in video-level segmentation tasks due to their strong feature modeling capabilities. Although these methods have achieved excellent performance on multiple public datasets, there are still challenges in segmenting endoscopic surgery videos: different surgical scenes have large differences, and the generalization ability of the model is tested; multiple anatomical tissues in endoscopic images are often closely related in space, with blurred boundaries, and similar appearances between instruments, which are mutually occluded, making accurate segmentation of multiple objects challenging. Therefore, it is of great significance to research and design a multi-scene, multi-tissue, and multi-object medical image segmentation algorithm for endoscopic surgery video segmentation. SUMMARY

[0004] To achieve the above-mentioned purposes and other advantages of the present application, the first object of the present application is to provide an endoscopic surgery video multi-object segmentation method based on a basic segmentation large model and mixed expert fine-tuning, comprising the following steps:

[0005] An endoscopic surgery video is obtained and preprocessed;

[0006] A basic segmentation large model baseline segmentation network is established to handle segmentation tasks for any image and video;

[0007] A hierarchical mixed expert module is constructed to improve the self-adaptive ability and task processing ability of the model for complex and diverse surgical scenes and different segmentation objects;

[0008] A stage gating network is constructed to realize dynamic control and context collaboration of the hierarchical mixed expert module in different task stages;

[0009] Insert the hierarchical hybrid expert module and the stage gating network on the baseline segmentation network to establish an endoscopic surgery video segmentation network;

[0010] Input the preprocessed video images and segmentation gold standards into the endoscopic surgery video segmentation network for supervised learning;

[0011] After the network is trained, the network weights are saved, and an endoscopic surgery video to be tested is inputted to segment the segmentation objects in the video.

[0012] Further, the endoscopic surgery video includes a laparoscopic surgery video, a thoracoscopic surgery video, a colonoscopy video, a pituitary endoscope surgery video, and a cataract surgery video.

[0013] Further, the preprocessing step of the endoscopic surgery video includes:

[0014] Adjust the size of the endoscopic surgery video to a preset size while maintaining the original aspect ratio of the image;

[0015] An online data enhancement method is used to adjust the endoscopic surgery video.

[0016] Further, the basic segmentation large model is composed of an image encoder, a prompt encoder, a mask decoder, a memory encoder, and a memory attention module; the image encoder is composed of a multi-layer multi-head self-attention layer, a feedforward fully connected layer, and a layer normalization layer; the prompt encoder includes a two-dimensional convolution layer and an embedding mapping layer; the mask decoder includes a multi-head self-attention layer, a cross-attention layer, a feedforward fully connected layer, a convolution layer, a fully connected layer, and a multi-layer perceptron module; the memory encoder is composed of several convolution layers; and the memory attention module is composed of a stacked multi-head self-attention layer, a cross-attention layer, and a feedforward fully connected layer.

[0017] Further, the hierarchical hybrid expert module includes a multi-scene hybrid expert module and a multi-target hybrid expert module; the multi-scene hybrid expert module identifies and adapts to different surgical scene distributions by a guide model to adapt to different surgical scenes; and the multi-target hybrid expert module captures structural differences by fine-grained modeling of specific segmentation objects to improve the performance of the model in multi-target recognition and segmentation tasks.

[0018] Further, each expert subnetwork in the multi-scene hybrid expert module is constructed based on a LoRA structure and is integrated in the attention module of the backbone network in parallel; each expert is composed of a dimension reduction matrix and a dimension increase matrix as its exclusive adaptation increment to structurally compensate the original weight matrix in a low-rank form;

[0019] The calculation formula of the subnetwork module in the multi-scene hybrid expert module is:

[0020]

[0021] wherein h represents the output of the expert subnetwork, x represents the input of the expert subnetwork, A represents a dimension reduction matrix, B represents a dimension increase matrix, and a / r is a scaling factor.

[0022] Further, the expert subnetworks of the multi-target hybrid expert module respectively add the feature outputs from the image encoder and the memory attention module element by element after multi-scale compression processing by convolution layers to realize feature fusion, splice the obtained results with the intermediate mask representation of the mask decoder, then realize dimension adjustment and channel fusion through a 1x1 convolution layer, and finally introduce a group of learnable target prototype vectors, which are initialized in the form of parameters and automatically updated during the training process, representing the typical feature distribution of different target structures, and used to guide the expert submodules to focus on feature modeling of different structure regions; the fusion features output by the expert module cross attention with the prototype vectors, and output the reinforced target representation.

[0023] The calculation formula of the subnetwork module in the multi-target hybrid expert module is:

[0024] F 1 =Conv(F i )+Conv(F mem )

[0025] F 2 =Conv([F 1 ;F mask ])

[0026] F attn =Attn(F 2 ,P,P)

[0027] wherein F i represents the feature output from the image encoder, F mem represents the feature output from the memory attention module, F 1 represents the output result of feature fusion, F mask represents the mask representation from the mask decoder, F 2 represents the result after dimension adjustment and channel fusion, P represents the introduced target prototype vector, and F attn represents the output reinforced target representation.

[0028] Further, the stage gating network adopts a convolutional network and a multi-layer perceptron as the main body. In the multi-scene mixed expert module stage, the stage gating network generates a learnable scene selection weight vector with a dimension consistent with the number of sub-modules in the multi-scene mixed expert module. After the weight vector is normalized by a softmax function, it is sent to the multi-scene mixed expert module, and the weighted sum of the outputs of each module is used as the final task adaptive parameter to be incorporated into the backbone network through a residual link.

[0029] In the multi-target mixed expert module stage, the stage gating network receives the gating weight vector output by the previous stage and maps it into the control weight of the target expert through a cross-stage gating mapping module. The cross-stage gating mapping module is composed of a linear layer and a nonlinear activation function, and includes a softmax normalization layer. The input stage gating vector is mapped into a weight vector consistent with the number of sub-modules in the multi-target mixed expert module. The weight generated by the stage gating network gates the output of the corresponding expert sub-network to form the final task adaptive parameter into the backbone network, and its calculation formula is as follows:

[0030]

[0031] Where y represents the final output result, h t represents the output of the expert sub-network t, G t represents the weight of the weight vector corresponding to the sub-network, and N is the total number of experts.

[0032] Further, the step of inserting the hierarchical mixed expert module and the stage gating network on the baseline segmentation network to establish an endoscopic surgery video segmentation network comprises:

[0033] In the hierarchical mixed expert module, the multi-scene mixed expert module is embedded in the image encoder stage in parallel with the attention projection layer of the image encoder, and the multi-target mixed expert module is located before the mask decoder.

[0034] The stage gating network is located in the path between the backbone network and the hierarchical mixed expert module, and connects the two parts of the hierarchical mixed expert module, which is used to dynamically perceive the semantic features of the task in the training and inference process, generate the selection weights required by the encoding and decoding stage expert module, and adjust the weight distribution of the mixed expert module.

[0035] Further, the step of inputting the preprocessed video image and the segmentation gold standard into the endoscopic surgery video segmentation network for supervised learning comprises:

[0036] Load the pre-training weights of the baseline segmentation network and initialize them to the original image encoder, prompt encoder, mask decoder, memory encoder, and memory attention module.

[0037] During training, the weight parameters of the backbone network are frozen, and training is performed only on the hierarchical hybrid expert module and the stage-gated network. The total loss of the network consists of the sum of cross-entropy loss and Dice loss. Backpropagation is used to update the parameters and guide the optimization of the entire network. The loss function is defined as follows:

[0038]

[0039] L total =L dice +λL ce

[0040] Among them, L dice P represents the Dice loss, where i represents each training sample. i G represents the network output prediction probability graph. i This represents the gold standard for this sample, where N is the batch size and L is the value of the sample. ce L represents the cross-entropy loss. total This represents the total loss of the entire network, where λ is the weight parameter.

[0041] A second objective of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0042] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] This invention provides a medical image segmentation method for endoscopic surgical videos, covering multiple scenes, tissues, and objects, enabling accurate segmentation of different surgical scenarios. This invention introduces a hierarchical hybrid expert module and a stage-gated network, effectively addressing the common challenges of scene differences and multi-tissue segmentation in endoscopic surgical video segmentation tasks.

[0045] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0047] Figure 1 Flowchart of a multi-object segmentation method for endoscopic surgical videos based on a basic segmentation model and hybrid expert fine-tuning;

[0048] Figure 2 The network architecture diagram of a multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning is shown.

[0049] Figure 3 This is a schematic diagram of a computer device.

[0050] Figure 4 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0051] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0052] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0053] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0055] This invention employs the Segment Anything Model 2 (SAM2) model as the basic segmentation network framework, constructing a hierarchical hybrid expert model and a stage-gated network to achieve multi-scene, multi-tissue, and multi-object segmentation of endoscopic surgical videos. The specific scheme is as follows:

[0056] Example 1

[0057] A multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning, such as... Figures 1-2As shown, it includes the following steps:

[0058] S1. Acquire endoscopic surgery video and preprocess the endoscopic surgery video;

[0059] The data used in this embodiment includes laparoscopic surgical videos (including cholecystectomy videos, hysterectomy videos, endometriosis treatment surgical videos, appendectomy videos, sleeve gastrectomy videos, etc.), thoracoscopic surgical videos, colonoscopy videos, pituitary endoscopic surgical videos, and cataract surgery videos. The laparoscopic surgical videos are mostly 1920×1080 pixels, the thoracoscopic surgical videos are 1280×720 pixels, the colonoscopy videos are mainly 320×448 pixels, the pituitary endoscopic surgical videos are 1280×720 pixels, and the cataract surgery videos are 1920×1080 pixels. The video lengths range from several seconds to several minutes.

[0060] In some embodiments, the preprocessing step of the endoscopic surgery video includes:

[0061] Adjust all endoscopic surgery video sizes to preset sizes. For example, to speed up network reading and computation efficiency, adjust all video sizes in the dataset to 512×512 while maintaining the original aspect ratio of the images.

[0062] Online data augmentation methods were used to adjust endoscopic surgery videos. These methods included random horizontal flipping, random translation, rotation, scaling, and color enhancement. To maintain temporal consistency, all augmentation operations supported inter-frame consistency constraints.

[0063] S2. Establish a baseline segmentation network for a large-scale basic segmentation model to handle segmentation tasks for arbitrary images and videos;

[0064] In this embodiment, the segmentation network model uses the SAM2 model as the baseline network for segmentation. SAM2 is a general segmentation model based on the Transformer architecture, whose core advantage lies in its ability to handle segmentation tasks for arbitrary images and videos. The original SAM2 model consists of an image encoder, a cue encoder, a mask decoder, a memory encoder, and a memory attention module. The image encoder consists of multiple multi-head self-attention layers, feedforward fully connected layers, and layer normalization layers. The cue encoder includes two-dimensional convolutional layers and embedding mapping layers. The mask decoder includes multi-head self-attention layers, cross-attention layers, feedforward fully connected layers, convolutional layers, fully connected layers, and a multilayer perceptron (MLP) module. The memory encoder consists of several convolutional layers, and the memory attention module consists of stacked multi-head self-attention layers, cross-attention layers, and feedforward fully connected layers. In this embodiment, the overall structure of the baseline network and the original modules and memory mechanism remain unchanged.

[0065] S3. Construct a hierarchical hybrid expert module to enhance the model's adaptability and task processing capabilities to complex and diverse surgical scenarios and different segmentation objects;

[0066] This embodiment constructs a hierarchical hybrid expert module, aiming to fully leverage multi-expert collaboration mechanisms to enhance the model's adaptability and task processing capabilities to complex and diverse surgical scenarios and different segmentation objects. This module can dynamically adapt to different scenarios or tissue features, thereby achieving robust modeling and enhanced generalization capabilities across tasks.

[0067] The hierarchical hybrid expert module includes two functional sub-modules: the multi-scenario hybrid expert module and the multi-objective hybrid expert module.

[0068] In terms of network structure design, this embodiment constructs multiple parallel expert subnetworks. Each expert is regarded as an independent submodule with its own parameter learning path, which is used to learn and express parameter adjustment strategies for specific scenarios or task domains.

[0069] Specifically, each expert sub-network in the multi-scenario hybrid expert module is constructed based on the LoRA (Low-Rank Adaptation) structure and integrated in parallel with the attention module of the backbone network. Each expert consists of a reduced-dimensional matrix and an increased-dimensional matrix, which serve as their exclusive adaptation increments. These increments structurally compensate the original weight matrix in a low-rank form, thereby completing personalized parameter updates for specific scenarios without significantly increasing the model parameter scale. This enables lightweight fine-tuning and functional expansion of the backbone network.

[0070] The calculation formula for the sub-network module in the multi-scenario hybrid expert module is as follows:

[0071]

[0072] Where h represents the output of the expert subnetwork, x represents the input of the expert subnetwork, A represents the dimension reduction matrix, B represents the dimension increase matrix, and α / r is the scaling factor.

[0073] In this embodiment, a multi-scene hybrid expert module is embedded in the image encoder stage, running in parallel with the attention projection layer of the image encoder. The output of the hybrid expert module is selected by a gating network and then input into the backbone network in the form of residual connections.

[0074] The expert sub-network of the multi-objective hybrid expert module performs multi-scale compression processing on the feature outputs from the image encoder and the memory attention module through convolutional layers, and then adds them element-wise to achieve feature fusion. The result is concatenated with the intermediate mask representation of the mask decoder, and then dimensionality adjustment and channel fusion are achieved through 1×1 convolutional layers. Finally, a set of learnable target prototype vectors is introduced. These prototype vectors are initialized in the form of parameters and automatically updated during the training process, representing the typical feature distribution of different target structures. They are used to guide the expert sub-module to focus on feature modeling of different structural regions. The fused features output by the expert module are cross-attentioned with the prototype vectors to output the enhanced target representation.

[0075] The calculation formula for the sub-network module in the multi-objective hybrid expert module is as follows:

[0076] F 1 =Conv(F i )+Conv(F mem )

[0077] F 2 =Conv([F 1 ;F mask ])

[0078] F attn =Attn(F 2 ,P,P)

[0079] Among them, F i F represents the feature output from the image encoder. mem F represents the feature output from the memory attention module. 1 F represents the output of feature fusion. mask F represents the mask representation from the mask decoder. 2 Representing the result after dimension adjustment and channel fusion, P represents the introduced target prototype vector, and F... attn This represents the enhanced target representation of the output.

[0080] Each expert sub-network has a consistent structure but independent function, focusing on the relevant characteristics of different organizational types. Specific parameter updates are performed for different experts based on different organizations and target learning tasks. This embodiment introduces a multi-target hybrid expert module in the decoding stage. Its inputs come from the image encoder, memory attention, and mask decoder, respectively. The outputs of different sub-modules are selected by a gating network, then fused with the output of the memory attention through feature channel concatenation and 1×1 convolution compression before being fed into the mask decoder.

[0081] The multi-scenario hybrid expert module guides the model to identify and adapt to different surgical scenario distributions, improving its generalization ability for multi-scenario tasks in endoscopic surgery and exhibiting good scalability and training stability. The multi-object hybrid expert module enhances the model's performance in multi-object recognition and segmentation tasks by performing fine-grained modeling and capturing structural differences of specific segmentation objects.

[0082] S4. Construct a phased gating network to enable dynamic control and contextual collaboration of the hierarchical hybrid expert module at different task stages;

[0083] This embodiment introduces a stage-gated network to achieve dynamic control and contextual collaboration of the expert module at different task stages. The stage-gated network continuously perceives the current task feature distribution during training and inference, dynamically allocates task paths based on the input video images, and activates the corresponding expert sub-networks, realizing a responsive scheduling strategy across multiple scenarios and objectives. In this embodiment, the stage-gated network operates on two parts of the hierarchical hybrid expert module.

[0084] In terms of network architecture design, the stage-gated network adopts convolutional networks and multilayer perceptrons (MLPs) as its main components to achieve a stage-by-stage, task-aware dynamic expert selection mechanism. Specifically, in the multi-scene hybrid expert module stage, the stage-gated network generates a learnable scene selection weight vector with the same dimension as the number of sub-modules in the multi-scene hybrid expert module. After being normalized by the softmax function, the weight vector is fed into the multi-scene hybrid expert module, and the weighted sum with the outputs of each module is used as the final task adaptive parameter, which is then incorporated into the backbone network through residual connections. This weight vector is not only used for the fusion calculation of the current module, but also serves as a priori guidance, passed to subsequent multi-objective hybrid expert modules as the basis for their gating modulation.

[0085] In the multi-objective hybrid expert module stage, the stage gating network receives the gating weight vector output from the previous stage and maps it to the control weights of the target experts through the cross-stage gating mapping module. The cross-stage gating mapping module consists of linear layers, nonlinear activation functions, and includes a softmax normalization layer, mapping the input stage gating vector to a weight vector consistent with the number of sub-modules in the multi-objective hybrid expert module. This weight vector enables the activation of more relevant target segmentation paths and the suppression of redundant computation paths in specific scenarios, thereby achieving scene-driven target-aware dynamic routing. The weights generated by the stage gating network will gate the outputs of the corresponding expert sub-networks, forming the final task adaptive parameters that enter the backbone network. The calculation formula is as follows:

[0086]

[0087] Where y represents the final output result, ht G represents the output of the expert subnetwork t. t This represents the weight vector corresponding to the weights of the subnetwork, where N is the total number of experts.

[0088] like Figure 2 As shown, S5, insert the hierarchical hybrid expert module and the stage gating network into the baseline segmentation network to establish an endoscopic surgery video segmentation network;

[0089] Further, the step of inserting the hierarchical hybrid expert module and the stage-gated network into the baseline segmentation network to establish an endoscopic surgery video segmentation network includes:

[0090] In the hierarchical hybrid expert module, the multi-scene hybrid expert module is embedded in the image encoder stage, running parallel to the attention projection layer of the image encoder. The multi-target hybrid expert module is located before the mask decoder. The hybrid expert module can model the feature distribution of complex surgical scenes and multi-class segmentation targets respectively, enhancing the model's expressive power and generalization performance in multi-task and multi-structure scenarios.

[0091] The stage-gated network is located in the path between the backbone network and the hierarchical hybrid expert module, and connects the two parts of the hierarchical hybrid expert module. It is used to dynamically perceive the semantic features of the task during training and inference, generate the selection weights required by the encoding and decoding stage expert modules, and adjust the weight allocation of the hybrid expert module.

[0092] S6. To train the entire endoscopic surgery video segmentation network, the preprocessed video images and the segmentation gold standard need to be input into the endoscopic surgery video segmentation network for supervised learning.

[0093] Furthermore, the step of inputting the preprocessed video images and the segmentation gold standard into the endoscopic surgery video segmentation network for supervised learning includes:

[0094] Load the pre-trained weights of the baseline segmentation network and initialize them into the original image encoder, cue encoder, mask decoder, memory encoder, and memory attention module;

[0095] During training, the weight parameters of the backbone network, including the cue encoder, mask decoder, and memory encoder, are frozen, and training is performed only on the hierarchical hybrid expert module and the stage gating network. The total loss of the network consists of the sum of the cross-entropy loss and the Dice loss. Backpropagation is used to update the parameters and guide the optimization of the entire network. The loss function is defined as follows:

[0096]

[0097]

[0098] L total =L dice +λL ce

[0099] Among them, L dice P represents the Dice loss, where i represents each training sample. i G represents the network output prediction probability graph. i This represents the gold standard for this sample, where N is the batch size and L is the value of the sample. ce L represents the cross-entropy loss. total This represents the total loss of the entire network, where λ is the weight parameter.

[0100] S7. After the network is trained, save the network weights. When testing the segmentation network, only the endoscopic surgery video to be tested needs to be input; no gold standard for segmentation is required. The segmentation network will automatically segment the objects in the video image based on the endoscopic surgery video to be tested.

[0101] Example 2

[0102] A computer device 800, such as Figure 3 As shown, the system includes a memory 810, a processor 820, and a computer program 830 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a multi-target segmentation method for endoscopic surgical videos based on a large-scale basic segmentation model and hybrid expert fine-tuning. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0103] Example 3

[0104] A computer-readable storage medium, such as Figure 4 As shown, a computer program is stored thereon. When executed by a processor, this computer program implements the steps of a multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0105] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0106] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0107] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0108] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0109] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0110] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0117] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning, characterized in that, Includes the following steps: Acquire endoscopic surgery videos and preprocess the endoscopic surgery videos; Establish a baseline segmentation network for a large-scale basic segmentation model to handle segmentation tasks of arbitrary images and videos; A hierarchical hybrid expert module is constructed to improve the model's adaptability and task processing capabilities to complex and diverse surgical scenarios and different segmentation objects; A phased gating network is constructed to enable dynamic control and contextual collaboration of the hierarchical hybrid expert module at different task stages; The hierarchical hybrid expert module and the stage-gated network are inserted into the baseline segmentation network to establish an endoscopic surgery video segmentation network; The preprocessed video images and the gold standard for segmentation are input into the endoscopic surgery video segmentation network for supervised learning. After the network is trained, the network weights are saved, and the endoscopic surgery video to be tested is input to segment the objects in the video.

2. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 1, characterized in that: The endoscopic surgery videos include laparoscopic surgery videos, thoracoscopic surgery videos, colonoscopy videos, pituitary endoscopic surgery videos, and cataract surgery videos.

3. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 1, characterized in that, The preprocessing steps for the endoscopic surgery video include: Adjust all endoscopic surgery video sizes to the preset size while maintaining the original aspect ratio of the images; Online data augmentation methods were used to adjust endoscopic surgery videos.

4. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 1, characterized in that: The basic segmentation model consists of an image encoder, a cue encoder, a mask decoder, a memory encoder, and a memory attention module. The image encoder consists of multiple multi-head self-attention layers, feedforward fully connected layers, and layer normalization layers. The cue encoder includes two-dimensional convolutional layers and embedding mapping layers. The mask decoder includes multi-head self-attention layers, cross-attention layers, feedforward fully connected layers, convolutional layers, fully connected layers, and a multilayer perceptron module. The memory encoder consists of several convolutional layers. The memory attention module consists of stacked multi-head self-attention layers, cross-attention layers, and feedforward fully connected layers.

5. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 4, characterized in that: The hierarchical hybrid expert module includes a multi-scenario hybrid expert module and a multi-objective hybrid expert module. The multi-scenario hybrid expert module guides the model to identify and adapt to different surgical scenario distributions to adapt to different surgical scenarios. The multi-objective hybrid expert module improves the model's performance in multi-object recognition and segmentation tasks by fine-grained modeling and structural difference capture of specific segmentation objects.

6. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 5, characterized in that: In the multi-scenario hybrid expert module, each expert sub-network is built based on the LoRA structure and integrated in parallel with the attention module of the backbone network. Each expert consists of a reduced-dimensional matrix and an increased-dimensional matrix, which serve as their exclusive adaptation increments to structurally compensate the original weight matrix in a low-rank form. The calculation formula for the sub-network module in the multi-scenario hybrid expert module is as follows: Where h represents the output of the expert subnetwork, x represents the input of the expert subnetwork, A represents the dimension reduction matrix, B represents the dimension increase matrix, and α / r is the scaling factor.

7. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 6, characterized in that: The expert sub-network of the multi-objective hybrid expert module performs multi-scale compression processing on the feature outputs from the image encoder and the memory attention module through convolutional layers, and then adds them element-wise to achieve feature fusion. The result is concatenated with the intermediate mask representation of the mask decoder, and then dimensionality adjustment and channel fusion are achieved through 1×1 convolutional layers. Finally, a set of learnable target prototype vectors is introduced. These prototype vectors are initialized in the form of parameters and automatically updated during the training process, representing the typical feature distribution of different target structures. They are used to guide the expert sub-module to focus on feature modeling of different structural regions. The fused features output by the expert module are cross-attentioned with the prototype vectors to output the enhanced target representation. The calculation formula for the sub-network module in the multi-objective hybrid expert module is as follows: F 1 =Conv(F i )+Conv(F mem ) F 2 =Conv([F 1 ;F mask ]) F attn =Attn(F 2 ,P,P) Among them, F i F represents the feature output from the image encoder. mem F represents the feature output from the memory attention module. 1 F represents the output of feature fusion. mask F represents the mask representation from the mask decoder. 2 Representing the result after dimension adjustment and channel fusion, P represents the introduced target prototype vector, and F... attn This represents the enhanced target representation of the output.

8. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 7, characterized in that: The stage-gated network uses convolutional networks and multilayer perceptrons as the main components. In the multi-scene hybrid expert module stage, the stage-gated network generates a learnable scene selection weight vector with the same dimension as the number of sub-modules in the multi-scene hybrid expert module. The weight vector is normalized by the softmax function and then fed into the multi-scene hybrid expert module. The weighted sum of the weighted sum with the output of each module is used as the final task adaptive parameter and then incorporated into the backbone network through residual connections. In the multi-objective hybrid expert module stage, the stage gating network receives the gating weight vector output from the previous stage and maps it to the control weights of the target experts through the cross-stage gating mapping module. The cross-stage gating mapping module consists of linear layers, nonlinear activation functions, and includes a softmax normalization layer. It maps the input stage gating vector to a weight vector with the same number of sub-modules as the multi-objective hybrid expert module. The weights generated by the stage gating network gate the outputs of the corresponding expert sub-networks, forming the final task adaptive parameters that enter the backbone network. The calculation formula is as follows: Where y represents the final output result, h t G represents the output of the expert subnetwork t. t This represents the weight vector corresponding to the weights of the subnetwork, where N is the total number of experts.

9. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 8, characterized in that, The step of inserting the hierarchical hybrid expert module and the stage-gated network into the baseline segmentation network to establish an endoscopic surgery video segmentation network includes: In the hierarchical hybrid expert module, the multi-scene hybrid expert module is embedded in the image encoder stage, running in parallel with the attention projection layer of the image encoder, and the multi-target hybrid expert module is located before the mask decoder; The stage-gated network is located in the path between the backbone network and the hierarchical hybrid expert module, and connects the two parts of the hierarchical hybrid expert module. It is used to dynamically perceive the semantic features of the task during training and inference, generate the selection weights required by the encoding and decoding stage expert modules, and adjust the weight allocation of the hybrid expert module.

10. The multi-object segmentation method for endoscopic surgical videos based on a basic segmentation large model and hybrid expert fine-tuning as described in claim 4, characterized in that, The step of inputting the preprocessed video images and the segmentation gold standard into the endoscopic surgery video segmentation network for supervised learning includes: Load the pre-trained weights of the baseline segmentation network and initialize them into the original image encoder, cue encoder, mask decoder, memory encoder, and memory attention module; During training, the weight parameters of the backbone network are frozen, and training is performed only on the hierarchical hybrid expert module and the stage-gated network. The total loss of the network consists of the sum of cross-entropy loss and Dice loss. Backpropagation is used to update the parameters and guide the optimization of the entire network. The loss function is defined as follows: THE total =L dice +λL ce Among them, L dice P represents the Dice loss, where i represents each training sample, and P represents the Dice loss. i G represents the network output predicted probability graph. i This represents the gold standard for this sample, where N is the batch size and L is the value of the sample. ce L represents the cross-entropy loss. total This represents the total loss of the entire network, where λ is the weight parameter.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

Images