Method and system for training an instantiated semantic segmentation model for a surgical scene

Abstract

The application relates to the technical field of medical image processing and artificial intelligence, and provides a method and system for training an instantiation semantic segmentation model for a surgical scene, which comprises the following steps: acquiring training data; constructing a teacher network and a student network; inputting video images into the teacher network and the student network simultaneously, extracting a multi-scale feature map of the teacher network and a serialized feature vector of the student network respectively, and acquiring a pixel-level segmentation prediction map output by the student network; obtaining projected features of the serialized feature vector through a learnable linear projection layer; performing weighted summation on a spatial distillation loss, a temporal distillation loss and a main task segmentation loss to obtain a total loss function; and updating network parameters of the student network based on the total loss function until convergence is achieved, so that an instantiation semantic segmentation model applicable to a real-time surgical scene is obtained. In this way, the time cost and the labor cost of model training are reduced.

Description

Technical Field

[0001] This application relates to the fields of medical image processing and artificial intelligence technology, specifically to a method and system for training an instantiated semantic segmentation model for surgical scenarios. Background Technology

[0002] Colorectal cancer is a prevalent malignant tumor worldwide, and surgical resection is the core treatment. With the development of surgical robotics technology, robot-assisted total mesorectal excision (R-TME) has significantly improved the ability to perform delicate anatomical operations due to its advantages such as high-definition three-dimensional vision, multi-degree-of-freedom robotic arms, and tremor filtering. However, R-TME surgery faces the challenge of real-time identification and localization of key structures in the complex pelvic anatomical environment. Current methods still heavily rely on surgeon experience and lack intelligent visual navigation support.

[0003] In recent years, deep learning-based instance segmentation technology has shown great potential in medical image analysis. However, instance segmentation in surgical scenarios faces two major bottlenecks: first, pixel-level annotation of medical images is extremely costly, making it difficult to obtain large-scale, high-quality labeled data; second, real-time surgical navigation requires models to have lightweight, low-latency inference capabilities, while high-precision models have a huge number of parameters and cannot be directly deployed on intraoperative computing platforms.

[0004] It is evident that instance segmentation techniques in related technologies suffer from high training costs. Summary of the Invention

[0005] This application aims to address the problem of high training costs in instance segmentation techniques in related technologies, and provides a method and system for training an instanced semantic segmentation model for surgical scenarios.

[0006] To solve the above problems, this application is implemented as follows:

[0007] Firstly, this application provides a method for training an instantiated semantic segmentation model for surgical scenarios, including: Acquire training data, which includes video images captured by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. A teacher network and a student network are constructed. The teacher network is a pre-trained general image segmentation model, and its network parameters are frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The video image is simultaneously input into the teacher network and the student network. The multi-scale feature map of the teacher network and the serialized feature vector of the student network are extracted respectively, and the pixel-level segmentation prediction map output by the student network is obtained. The serialized feature vector is mapped to the same spatial dimension as the multi-scale feature map by a learnable linear projection layer to obtain the projected features of the serialized feature vector; Calculate the spatial distillation loss between the projected features and the multi-scale feature map, calculate the temporal distillation loss between the serialized feature vectors of adjacent video frames, and calculate the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation; The total loss function is obtained by weighted summing of the spatial distillation loss, the temporal distillation loss, and the main task segmentation loss. The network parameters of the student network are updated based on the total loss function through backpropagation until convergence, resulting in an instantiated semantic segmentation model that can be used in real-time surgical scenarios. In one embodiment, Secondly, this application provides an instantiated semantic segmentation model training system for surgical scenarios, comprising: The acquisition module is used to acquire training data, which includes video images collected by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. The module is used to build a teacher network and a student network. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The feature extraction module is used to simultaneously input the video image into the teacher network and the student network, extract the multi-scale feature map of the teacher network and the serialized feature vector of the student network respectively, and obtain the pixel-level segmentation prediction map output by the student network. The mapping module is used to map the serialized feature vector to the same spatial dimension as the multi-scale feature map through a learnable linear projection layer, so as to obtain the projected features of the serialized feature vector; The first calculation module is used to calculate the spatial distillation loss between the projected features and the multi-scale feature map, the temporal distillation loss between the serialized feature vectors of adjacent video frames, and the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation. The second calculation module is used to perform a weighted summation of the spatial distillation loss, the temporal distillation loss, and the main task segmentation loss to obtain the total loss function. The update module is used to update the network parameters of the student network based on the total loss function through backpropagation until convergence, thereby obtaining an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0008] Thirdly, this application provides a terminal device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.

[0011] In a sixth aspect, this application provides a computer program product stored in a storage medium, which is executed by at least one processor to perform the steps of the method described in the first aspect.

[0012] Compared with existing technologies, this application has the following advantages: by using a knowledge distillation mechanism, a pre-trained general image segmentation model is used as the teacher network, and its rich semantic knowledge can be transferred to a lightweight student network with only a small amount of labeled data. In this way, the demand for labeled data is greatly reduced, and the time and manpower costs of model training are significantly reduced. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a method for training an instantiated semantic segmentation model for a surgical scenario, provided in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of an instantiated semantic segmentation model training system for surgical scenarios provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0017] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0018] The training method for the instantiated semantic segmentation model for surgical scenarios provided in this application is described below.

[0019] See Figure 1 , Figure 1 This is a flowchart illustrating a method for training an instantiated semantic segmentation model for a surgical scenario, provided in an embodiment of this application. Figure 1 The instantiated semantic segmentation model training method shown for surgical scenarios can be executed by terminal devices such as mobile phones and computers.

[0020] like Figure 1 As shown, the instantiated semantic segmentation model training method for surgical scenarios provided in this application may include the following steps: Step 101: Obtain training data, which includes video images collected by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures.

[0021] In this embodiment, the training data can be derived from a surgical robot system. Specifically, the da Vinci surgical robot system can be used to acquire intraoperative video recordings of total mesorectal excision. The surgical robot system records high-definition 3D video at a frame rate of 30 frames per second and a resolution of 1920×1080 pixels to obtain the original video images.

[0022] In some embodiments, the aforementioned high-definition three-dimensional video needs to cover the entire surgical process, including key surgical stages such as dissection of the rectal mesentery, operation of surgical instruments, and exposure of blood vessels and nerves.

[0023] In some embodiments, the surgical instruments are labeled as one or more of the following: electric hook, Maryland forceps, irrigation tube, and intestinal forceps; the anatomical structures are labeled as one or more of the following: mesenteric space, blood vessel, nerve, and ureter.

[0024] Step 102: Construct a teacher network and a student network. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter, which is a lightweight model to be trained.

[0025] The aforementioned general image segmentation model (Segment Anything Model, SAM) is a large-scale pre-trained model based on Transformer, possessing powerful zero-shot segmentation capabilities and able to generate high-quality segmentation masks for any target in any image.

[0026] By keeping the network parameters of the teacher network frozen during training, resource consumption can be calculated, requiring only forward propagation. This also ensures the knowledge stability of the teacher network and prevents the training of the student network from interfering with teacher characteristics.

[0027] Step 103: Input the video image into the teacher network and the student network simultaneously, extract the multi-scale feature map of the teacher network and the serialized feature vector of the student network respectively, and obtain the pixel-level segmentation prediction map output by the student network.

[0028] In some embodiments, the student network uses a decoder structure to generate the pixel-level segmentation prediction map, wherein the decoder is a feature pyramid network or a lightweight Transformer decoder.

[0029] Step 104: Map the serialized feature vector to the same spatial dimension as the multi-scale feature map through a learnable linear projection layer to obtain the projected features of the serialized feature vector.

[0030] In some embodiments, the learnable linear projection layer consists of one or more fully connected layers, and the output dimension of the learnable linear projection layer is consistent with the dimension of the multi-scale feature map after spatial flattening.

[0031] Step 105: Calculate the spatial distillation loss between the projected features and the multi-scale feature map, calculate the temporal distillation loss of the serialized feature vectors between adjacent video frames, and calculate the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation.

[0032] In some embodiments, the temporal distillation loss is used to constrain the consistency of the serialized feature vectors between consecutive video frames.

[0033] Step 106: The spatial distillation loss, the temporal distillation loss, and the main task segmentation loss are weighted and summed to obtain the total loss function.

[0034] Step 107: Update the network parameters of the student network through backpropagation based on the total loss function until convergence, and obtain an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0035] This application utilizes a knowledge distillation mechanism, employing a pre-trained general image segmentation model as the teacher network. Only a small amount of labeled data is needed to transfer its rich semantic knowledge to a lightweight student network. In this way, the requirement for labeled data is greatly reduced, and the time and manpower costs of model training are significantly reduced.

[0036] By introducing temporal distillation loss into the training of surgical scene instance segmentation, and by constraining the consistency of student network features between adjacent video frames, the model is forced to learn temporally smooth feature representations, which significantly reduces the inter-frame variation rate of the segmentation mask.

[0037] By designing a learnable linear projection layer, the sequential feature vectors of the student network are mapped to the same spatial dimension as the multi-scale feature maps of the teacher network, thus solving the architectural alignment problem between sequential features and spatial features. Furthermore, the addition of the projection layer effectively improves the convergence speed of the spatial distillation loss and enhances the final segmentation accuracy.

[0038] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an instantiated semantic segmentation model training system for surgical scenarios provided in an embodiment of this application. Figure 2 As shown, system 200 includes: The acquisition module 201 is used to acquire training data, which includes video images collected by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. Module 202 is used to construct a teacher network and a student network. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are kept frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The feature extraction module 203 is used to simultaneously input the video image into the teacher network and the student network, extract the multi-scale feature map of the teacher network and the serialized feature vector of the student network respectively, and obtain the pixel-level segmentation prediction map output by the student network. The mapping module 204 is used to map the serialized feature vector to the same spatial dimension as the multi-scale feature map through a learnable linear projection layer, so as to obtain the projected features of the serialized feature vector; The first calculation module 205 is used to calculate the spatial distillation loss between the projected features and the multi-scale feature map, calculate the temporal distillation loss of the serialized feature vector between adjacent video frames, and calculate the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation. The second calculation module 206 is used to perform a weighted summation of the spatial distillation loss, the temporal distillation loss and the main task segmentation loss to obtain the total loss function. The update module 207 is used to update the network parameters of the student network based on the total loss function through backpropagation until convergence, thereby obtaining an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0039] Optionally, the learnable linear projection layer consists of one or more fully connected layers, and the output dimension of the learnable linear projection layer is consistent with the dimension of the multi-scale feature map after spatial flattening.

[0040] Optionally, the temporal distillation loss is used to constrain the consistency of the serialized feature vectors between consecutive video frames.

[0041] Optionally, the student network uses a decoder structure to generate the pixel-level segmentation prediction map, wherein the decoder is a feature pyramid network or a lightweight Transformer decoder.

[0042] Optionally, the surgical instruments may be categorized as one or more of the following: electric hook, Maryland forceps, irrigation tube, and intestinal forceps; the anatomical structures may be categorized as one or more of the following: mesenteric space, blood vessel, nerve, and ureter.

[0043] The instantiated semantic segmentation model training system for surgical scenarios provided in this application can achieve the goals outlined in this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.

[0044] like Figure 3 As shown, this application also provides a terminal device, including a processor 301 and a memory 302. The memory 302 stores a program or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they implement the various steps of the above-described embodiment of the instantiated semantic segmentation model training method for surgical scenarios and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0045] It should be noted that the terminal device in this application can be a terminal or other devices besides a terminal. For example, the terminal device can be a mobile phone, tablet computer, laptop computer, etc., and this application does not make any specific limitation.

[0046] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described embodiment of the instantiated semantic segmentation model training method for surgical scenarios, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0047] The processor is the processor in the terminal device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory). Only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc.

[0048] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the instantiated semantic segmentation model training method for surgical scenarios, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0049] It should be understood that the chip mentioned in this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0050] This application provides a computer program product stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-described embodiment of the instantiated semantic segmentation model training method for surgical scenarios, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0051] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0052] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as a read-only memory). The device includes a number of instructions in a ROM (random access memory), RAM (magnetic disk), or optical disk to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0053] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training an instantiated semantic segmentation model for surgical scenarios, characterized in that, include: Acquire training data, which includes video images captured by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. A teacher network and a student network are constructed. The teacher network is a pre-trained general image segmentation model, and its network parameters are frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The video image is simultaneously input into the teacher network and the student network. The multi-scale feature map of the teacher network and the serialized feature vector of the student network are extracted respectively, and the pixel-level segmentation prediction map output by the student network is obtained. The serialized feature vector is mapped to the same spatial dimension as the multi-scale feature map by a learnable linear projection layer to obtain the projected features of the serialized feature vector; Calculate the spatial distillation loss between the projected features and the multi-scale feature map, calculate the temporal distillation loss between the serialized feature vectors of adjacent video frames, and calculate the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation; The total loss function is obtained by weighted summing of the spatial distillation loss, the temporal distillation loss, and the main task segmentation loss. The network parameters of the student network are updated by backpropagation based on the total loss function until convergence, resulting in an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

2. The method according to claim 1, characterized in that, The learnable linear projection layer consists of one or more fully connected layers, and the output dimension of the learnable linear projection layer is consistent with the dimension of the multi-scale feature map after spatial flattening.

3. The method according to claim 1, characterized in that, The time distillation loss is used to constrain the consistency of the serialized feature vectors between consecutive video frames.

4. The method according to any one of claims 1 to 3, characterized in that, The student network uses a decoder structure to generate the pixel-level segmentation prediction map, where the decoder is a feature pyramid network or a lightweight Transformer decoder.

5. The method according to any one of claims 1 to 3, characterized in that, The surgical instruments are categorized as follows: electric hook, Maryland forceps, irrigation tube, intestinal forceps, or one or more of the following. The anatomical structures are categorized as follows: mesenteric space, blood vessels, nerves, ureter, or one or more of the following.

6. A training system for an instantiated semantic segmentation model for surgical scenarios, characterized in that, include: The acquisition module is used to acquire training data, which includes video images collected by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. The module is used to build a teacher network and a student network. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The feature extraction module is used to simultaneously input the video image into the teacher network and the student network, extract the multi-scale feature map of the teacher network and the serialized feature vector of the student network respectively, and obtain the pixel-level segmentation prediction map output by the student network. The mapping module is used to map the serialized feature vector to the same spatial dimension as the multi-scale feature map through a learnable linear projection layer, so as to obtain the projected features of the serialized feature vector; The first calculation module is used to calculate the spatial distillation loss between the projected features and the multi-scale feature map, the temporal distillation loss between the serialized feature vectors of adjacent video frames, and the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation. The second calculation module is used to perform a weighted summation of the spatial distillation loss, the temporal distillation loss, and the main task segmentation loss to obtain the total loss function. The update module is used to update the network parameters of the student network based on the total loss function through backpropagation until convergence, thereby obtaining an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

7. The system according to claim 6, characterized in that, The learnable linear projection layer consists of one or more fully connected layers, and the output dimension of the learnable linear projection layer is consistent with the dimension of the multi-scale feature map after spatial flattening.

8. The system according to claim 6, characterized in that, The time distillation loss is used to constrain the consistency of the serialized feature vectors between consecutive video frames.

9. The system according to any one of claims 6 to 8, characterized in that, The student network uses a decoder structure to generate the pixel-level segmentation prediction map, where the decoder is a feature pyramid network or a lightweight Transformer decoder.

10. The system according to any one of claims 6 to 8, characterized in that, The surgical instruments are categorized as follows: electric hook, Maryland forceps, irrigation tube, intestinal forceps, or one or more of the following. The anatomical structures are categorized as follows: mesenteric space, blood vessels, nerves, ureter, or one or more of the following.

Images