Training method of target model based on multi-stage incremental learning and image processing method

Abstract

The application provides a training method and an image processing method of a target model based on multi-stage incremental learning, which can be applied to the technical field of medical image processing. The method comprises the following steps: performing training operations in N stages based on training samples to obtain a target model, which comprises the following steps: determining a (n-1)th sample buffer image from a (n-1)th sample intraoperative image set based on a clustering result of the (n-1)th sample intraoperative image set; performing feature extraction on the (n-1)th sample buffer image by using a feature extraction model of an (n-1)th stage to obtain a (n-1)th sample buffer feature; performing feature extraction on the (n)th sample intraoperative image set and the (n-1)th sample buffer image respectively by using an attention network model to obtain an (n)th sample visual feature and an (n)th sample buffer feature; training the attention network model according to the (n)th sample visual feature, the (n-1)th sample buffer feature and the (n)th sample buffer feature to obtain a feature extraction model of an (n)th stage; and determining the feature extraction model corresponding to the Nth stage as the target model.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, specifically to a training method and image processing method for a target model based on multi-stage incremental learning. Background Technology

[0002] Laparoscopic and other minimally invasive surgeries generate a large amount of intraoperative video data. Intraoperative videos not only record tissue morphology, instrument movement, and changes in the surgical field, but also reflect structural and functional information of the surgical field through different imaging modes. Rich features extracted from intraoperative video data can be used for downstream detection tasks such as intraoperative smoke removal and surgical instrument segmentation. However, a large amount of intraoperative video data involves multiple modalities. Directly mixing samples from different modalities to train a model can easily lead to differences in feature distribution and modal conflicts, resulting in low accuracy of the model's extracted feature representation capabilities. Summary of the Invention

[0003] In view of the above problems, this application provides a training method and image processing method for a target model based on multi-stage incremental learning.

[0004] According to the first aspect of this application, a training method for a target model based on multi-stage incremental learning is provided. The training process of the target feature extraction model includes N stages, including: acquiring training samples, where N intraoperative image sets in the training samples correspond to N modality types, and the intraoperative image sets represent the structural and functional information of the target object tissue during the surgery; performing the following N stages of training operations based on the training samples to obtain the target model; the training operation in the nth stage includes: determining the (n-1)th sample buffer image from the (n-1)th sample intraoperative image set based on the clustering results of the (n-1)th sample intraoperative image set, where the (n-1)th sample intraoperative image has the (n-1)th modality type, and using the (n-1)th sample intraoperative image set to perform the (n-1)th stage of training operations to obtain the (n-1)th feature extraction model. The feature extraction model of stage n-1 is used to extract features from the buffer image of sample n-1 to obtain the buffer features of sample n-1. The attention network model is used to extract features from the intraoperative image set of sample n and the buffer image of sample n-1 to obtain the visual features of sample n and the buffer features of sample n. The attention network model is trained based on the visual features of sample n, the buffer features of sample n-1 and the buffer features of sample n to obtain the feature extraction model of stage n. The feature extraction model corresponding to stage N is determined as the target model.

[0005] According to an embodiment of this application, determining the (n-1)th sample buffer image from multiple (n-1)th sample intraoperative images based on the clustering result of the (n-1)th sample intraoperative image set includes: using the feature extraction model of the (n-1)th stage to extract features from multiple (n-1)th sample intraoperative images in the (n-1)th sample intraoperative images respectively, to obtain multiple (n-1)th sample visual features; clustering the multiple (n-1)th sample visual features to obtain clustering results, the clustering results including multiple clusters; determining the (n-1)th sample intraoperative image corresponding to the (n-1)th sample visual feature whose distance from the center point of the cluster is less than a preset distance threshold as the (n-1)th sample buffer image, the number of (n-1)th sample buffer images being less than a preset number of sample buffers.

[0006] According to an embodiment of this application, an attention network model is trained based on the visual features of the nth sample, the buffer features of the (n-1)th sample, and the buffer features of the nth sample to obtain a feature extraction model corresponding to the nth stage. This includes: obtaining a contrastive loss value based on the difference between the buffer features of the (n-1)th sample and the buffer features of the nth sample; processing the visual features of the nth sample and the sample label using a task loss function to obtain an incremental modal sample loss value, where the sample label represents the true detection category corresponding to the detection task; and training the attention network model based on the incremental modal sample loss value and the contrastive loss value to obtain a feature extraction model corresponding to the nth stage.

[0007] According to an embodiment of this application, an attention network model is trained based on the incremental modal sample loss value and the contrast loss value to obtain a feature extraction model corresponding to the nth stage. This includes: training the attention network model based on the incremental modal sample loss value, the contrast loss value, and the reconstructed image loss value, wherein the reconstructed image loss value is determined based on the following operations: masking the nth intraoperative image in the nth sample intraoperative image set to obtain a sample mask image; extracting features from the sample mask image using the attention network model to obtain sample mask features; reconstructing the sample mask features to obtain a sample reconstructed image; and determining the reconstructed image loss value based on the sample reconstructed image and the nth sample intraoperative image.

[0008] According to embodiments of this application, the modal type includes at least one of white light imaging type, fluorescence imaging type, narrowband imaging type, near-infrared imaging type, and multispectral imaging type.

[0009] According to an embodiment of this application, the training method further includes: sorting N modal types according to the data volume, structural information coverage, and tissue boundary information coverage of the sample intraoperative image set to obtain a training order, the training order including training the white light imaging type first and then training the fluorescence imaging type; determining the first sample buffer image from the first sample intraoperative image set based on the clustering result of the first sample intraoperative image set of the white light imaging type, the (n-1)th sample intraoperative image has the (n-1)th modal type, the first sample intraoperative image set is used to perform the first stage training operation to obtain the first feature extraction model; using the first stage feature extraction model to extract features from the first sample buffer image to obtain the first sample buffer features; using an attention network model to extract features from the second sample intraoperative image set and the first sample buffer image of the fluorescence imaging type respectively to obtain the second sample visual features and the second buffer features; training the attention network model according to the second sample visual features, the first sample buffer features, and the second buffer image features to obtain the second stage feature extraction model; and determining the feature extraction model corresponding to the second stage as the target model.

[0010] A second aspect of this application provides an image processing method, comprising: acquiring an intraoperative image set of a target object, the intraoperative image set representing the structural and functional information of the target object's tissues during the surgical procedure; extracting features from the intraoperative image set using a target model to obtain visual features, wherein the target model is trained according to the above method; and detecting the visual features according to a detection model corresponding to the target detection task to obtain a detection result.

[0011] According to embodiments of this application, the target detection task includes at least one of the following: tissue segmentation task, intraoperative smoke removal task, polyp classification, and lymph node metastasis detection task.

[0012] The third aspect of this application provides a target model training device based on multi-stage incremental learning. The training process of the target feature extraction model includes N stages. The device includes: a first acquisition module for acquiring training samples, wherein the N intraoperative image sets in the training samples correspond to N modalities, and the intraoperative image sets represent the structural and functional information of the target object tissue during the surgery; a training module for performing the following N stages of training operations based on the training samples to obtain the target model; the training operation in the nth stage includes: determining the (n-1)th sample buffer image from the (n-1)th sample intraoperative image set based on the clustering results of the (n-1)th sample intraoperative image set, wherein the (n-1)th sample intraoperative image has the (n-1)th modality, and the (n-1)th sample intraoperative image set is used to perform the (n-1)th stage of training operations to obtain the (n-1)th feature extraction model. The feature extraction model of stage n-1 is used to extract features from the buffer image of sample n-1 to obtain the buffer features of sample n-1. The attention network model is used to extract features from the intraoperative image set of sample n and the buffer image of sample n-1 to obtain the visual features of sample n and the buffer features of sample n. The attention network model is trained based on the visual features of sample n, the buffer features of sample n-1 and the buffer features of sample n to obtain the feature extraction model of stage n. The feature extraction model corresponding to stage N is determined as the target model.

[0013] A fourth aspect of this application provides an image processing apparatus, comprising: a second acquisition module for acquiring an intraoperative image set of a target object, the intraoperative image set representing the structural and functional information of the target object's tissues during surgery; an extraction module for extracting features from the intraoperative image set using a target model to obtain visual features, wherein the target model is trained according to the above method; and a detection module for detecting the visual features according to a detection model corresponding to a target detection task to obtain detection results.

[0014] According to the training method and image processing method of the target model based on multi-stage incremental learning provided in this application, by gradually introducing the training of the intraoperative image set of multimodal types in stages, a small number of highly representative sample buffer images that can accurately cover the typical distribution area of ​​the modality type in the feature space and retain the key distribution structure are selected in each stage. Then, when training for the next stage for the new modality type, a joint mechanism is progressively constructed using the sample buffer images of the old modality type and the intraoperative image set of the new modality type to achieve a balance between the absorption of the new modality and the retention of knowledge of the old modality. In addition, the distribution distance between the nth sample buffer feature and the (n-1)th sample buffer feature is constrained. By minimizing the distance between the sample buffer features output by the feature extraction models of adjacent stages for the same sample buffer image, the attention network model in the current training stage is constrained to not deviate from the embedding space established by the old modality type while learning the new modality type. By retaining the key sample visual features of the old modality type that have been selected, the network to be trained can maintain the feature representation of the old modality type while learning the new modality, thereby reducing the risk of forgetting the old modality type information. Multi-stage incremental learning strategies can mitigate modality conflict, reduce the forgetting of old modality information during training, achieve stable multimodal representation learning, and solve the modality conflict problem caused by training mechanisms that use mixed multimodal samples at one time. Attached Figure Description

[0015] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0016] Figure 1A flowchart is shown of a training method for a target model based on multi-stage incremental learning according to an embodiment of this application.

[0017] Figure 2 A flowchart of an image processing method according to an embodiment of this application is shown.

[0018] Figure 3 A schematic diagram illustrating the training and application of a target model according to an embodiment of this application is shown.

[0019] Figure 4A A structural block diagram of a training apparatus for a target model based on multi-stage incremental learning according to an embodiment of this application is shown.

[0020] Figure 4B A structural block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0021] Figure 5 A block diagram of an electronic device suitable for implementing a training method and an image processing method for a target model based on multi-stage incremental learning, according to an embodiment of this application, is shown. Detailed Implementation

[0022] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] The training process of related models suffers from the following problems: First, direct mixing of different modalities during training easily leads to differences in feature distribution and modal conflicts; second, when gradually introducing new modalities, the model is prone to forgetting information from existing modalities, resulting in a decline in its ability to retain knowledge of existing modalities; and third, existing task-specific models struggle to form a unified basic representation of intraoperative vision. Research indicates that current artificial intelligence methods for surgical scenarios generally suffer from insufficient generalization ability, strong dependence on labeled data, and difficulty in characterizing complex intraoperative visual features.

[0027] In view of this, embodiments of this application provide a training method for a target model based on multi-stage incremental learning. The method includes: performing N stages of training operations based on training samples to obtain the target model, including: determining a buffer image of sample n-1 from the intraoperative image set of sample n-1 based on the clustering results of the sample n-1; extracting features from the buffer image of sample n-1 using a feature extraction model of stage n-1 to obtain buffer features of sample n-1; extracting features from the intraoperative image set of sample n and the buffer image of sample n-1 using an attention network model to obtain visual features of sample n and buffer features of sample n; training the attention network model based on the visual features of sample n, buffer features of sample n-1, and buffer features of sample n to obtain a feature extraction model of stage n; and determining the feature extraction model corresponding to stage N as the target model.

[0028] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] Figure 1 A flowchart is shown of a training method for a target model based on multi-stage incremental learning according to an embodiment of this application.

[0030] like Figure 1As shown, the method includes operations S110 to S160.

[0031] In operation S110, training samples are acquired. The N intraoperative image sets in the training samples correspond to N modal types. The intraoperative image sets represent the structural and functional information of the target tissue during the operation.

[0032] In operation S120, the (n-1)th sample buffer image is determined from the (n-1)th sample intraoperative image set based on the clustering results of the (n-1)th sample intraoperative image set.

[0033] In operation S130, the feature extraction model of the (n-1)th stage is used to extract features from the (n-1)th sample buffer image to obtain the (n-1)th sample buffer features.

[0034] In operation S140, the attention network model is used to extract features from the intraoperative image set of the nth sample and the buffer image of the (n-1)th sample, respectively, to obtain the visual features of the nth sample and the buffer features of the nth sample.

[0035] In operation S150, an attention network model is trained based on the visual features of the nth sample, the buffer features of the (n-1)th sample, and the buffer features of the nth sample to obtain the feature extraction model for the nth stage.

[0036] In operation S160, the feature extraction model corresponding to the Nth stage is determined as the target model.

[0037] Laparoscopic and other minimally invasive surgeries generate a large amount of intraoperative video data. Intraoperative videos not only record tissue morphology, instrument movement, and changes in the surgical field, but also reflect structural and functional information of the surgical field through different imaging modalities. Laparoscopic and other surgeries can produce intraoperative video frames in various modalities, such as white light, fluorescence, and angiography, providing complementary visual information.

[0038] The training samples can come from intraoperative video data generated during 250 surgeries. The N sample intraoperative image sets in the training samples correspond to N modal types. The sample intraoperative image set includes multiple sample intraoperative images. The sample intraoperative images represent the structural and functional information of the target tissue during the operation.

[0039] Modality type is used to define the technical attributes of intraoperative images of a sample, such as white light frame images and fluorescence frame images. White light frame images are visible light reflectance imaging, while fluorescence frame images are contrast agent excitation imaging.

[0040] After noise filtering and duplicate frame removal of the N sample intraoperative image sets respectively, uniform sampling and image enhancement are performed at 2 frames per second to obtain the preprocessed N sample intraoperative image sets.

[0041] The target model is obtained by performing the following N stages of training operations based on the training samples.

[0042] In the first stage, unsupervised training of the attention network model using the first sample intraoperative image set of the first modality type includes: using the attention network model with initial parameters to extract features from the first sample intraoperative image set to obtain the visual features of the first sample; then using the training objective to calculate the loss value of the visual features of the first sample; if the loss value is greater than a preset threshold, adjusting the parameters of the attention network model until the loss value is less than or equal to the preset threshold and training stops, thus obtaining the feature extraction model of the first stage.

[0043] The attention network model can be constructed based on the self-supervised learning framework of Distillation with Improved Noisy student and Optimized for visual features v2 (DINOv2), where the visual backbone network is a visual encoder (Vision Transformer Large model, ViT-L / 14). The visual backbone network includes a teacher network and a student network with corresponding structures.

[0044] The first sample buffer image was determined from the first sample intraoperative image set based on the clustering results of the first sample intraoperative image set.

[0045] Clustering is performed on multiple first visual features corresponding to the first sample intraoperative image set to obtain clustering results. Representative samples close to the cluster centers are selected from the clustering results to form the first sample buffer images. For example, the top 10% of high-information samples from the first sample intraoperative image set are selected as the first sample buffer images.

[0046] The feature extraction model trained in the first stage is used to extract features from the first sample buffer image to obtain the first sample buffer features.

[0047] After the first phase of training, the student network weights obtained from the training are retained to form the initial basic model. This model is used to extract the buffer features of the first sample.

[0048] In the second stage, an attention network model with initial parameters is used to extract features from the second sample intraoperative image set and the first sample buffer image, respectively, to obtain the second sample visual features corresponding to the second sample intraoperative image set and the second sample buffer features corresponding to the first sample buffer image. The second sample intraoperative image set and the first sample buffer image are respectively processed by the attention network model to obtain corresponding features, and then jointly optimized and trained by sharing student network parameters and joint loss.

[0049] The difference between the buffer features of the second sample and the buffer features of the first sample is used to calculate the loss value of the visual features of the second sample, and the parameters of the attention network model are adjusted until the loss value is less than or equal to the preset threshold to stop training, thus obtaining the feature extraction model of the second stage.

[0050] In the (n-1)th stage, the feature extraction model trained in the (n-1)th stage is obtained by training the intraoperative image set of the (n-1)th samples and the buffer images of the (n-2)th samples. Based on the clustering results of the intraoperative image set of the (n-1)th samples, the buffer images of the (n-1)th samples are determined from the intraoperative image set of the (n-1)th samples. The feature extraction model trained in the (n-1)th stage is used to extract features from the buffer images of the (n-1)th samples, thus obtaining the features of the buffer images of the (n-1)th samples. .

[0051] In the nth stage, the attention network model with initial parameters is used to extract features from the intraoperative image set of the nth sample and the buffer image of the (n-1)th sample, respectively, to obtain the visual features of the nth sample and the buffer features of the nth sample.

[0052] The key anchor information of the (n-1)th sample buffer image represents the overall sample data distribution statistical characteristics of the (n-1)th modality type, and is used to retain the knowledge representation of the (n-1)th modality type during the subsequent nth stage of training.

[0053] The (n-1)th sample buffer features represent the stable and representative distribution characteristics of the (n-1)th modality type in the trained feature space, serving as a reference benchmark for the attention network model to maintain consistency of knowledge of the old modalities during the nth stage of training.

[0054] The nth sample buffer feature represents the current knowledge embedding representation of the (n-1)th modality type in the attention network model during the nth stage of training, and is used to compare it with the (n-1)th sample buffer feature.

[0055] The difference comparison value between the features of the nth sample buffer and the features of the (n-1)th sample buffer represents the knowledge distillation loss information. Specifically, it is a measure of the distribution difference between the features extracted by the attention network model in the nth stage and the feature extraction model in the (n-1)th stage for the same set of (n-1)th sample buffer images. This difference comparison value is used to constrain the current training model to maintain the consistency of knowledge representation for the (n-1)th modality while learning the visual features of the newly added nth modality, thereby reducing information forgetting.

[0056] The visual feature representation of the nth sample is a high-dimensional representation of the intraoperative image set of the nth sample in the current feature space, including frame-level spatial visual representations such as tissue texture, instrument morphology, anatomical structure and local regions, which is used for knowledge learning of new modal types.

[0057] By using the difference comparison value between the buffer features of the nth sample and the buffer features of the (n-1)th sample, and by calculating the loss value of the visual features of the nth sample using the training objective, the parameters of the attention network model are adjusted until the loss value is less than or equal to a preset threshold, and training is stopped to obtain the feature extraction model of the nth stage.

[0058] The feature extraction model obtained from the Nth stage of training is selected as the target model.

[0059] In the embodiments of this application, by progressively introducing training of multimodal type sample intraoperative image sets in stages, a small number of highly representative sample buffer images that can accurately cover the typical distribution area of ​​the modality type in the feature space and retain the key distribution structure are selected at each stage. Then, when training for the next stage for the new modality type, a joint mechanism is progressively constructed using the sample buffer images of the old modality type and the sample intraoperative image set of the new modality type to achieve a balance between the absorption of the new modality and the retention of knowledge of the old modality. In addition, the distribution distance between the nth sample buffer feature and the (n-1)th sample buffer feature is constrained. By minimizing the distance between the sample buffer features output by the feature extraction models of adjacent stages for the same sample buffer image, the attention network model in the current training stage is constrained to not deviate from the embedding space already established by the old modality type while learning the new modality type. By retaining the key sample visual features of the old modality type that have been selected, the network to be trained can maintain the feature representation of the old modality type while learning the new modality, thereby reducing the risk of forgetting the old modality type information. Multi-stage incremental learning strategies can mitigate modality conflict, reduce the forgetting of old modality information during training, achieve stable multimodal representation learning, and solve the modality conflict problem caused by training mechanisms that use mixed multimodal samples at one time.

[0060] According to an embodiment of this application, determining the (n-1)th sample buffer image from multiple (n-1)th sample intraoperative images based on the clustering result of the (n-1)th sample intraoperative image set includes: using the feature extraction model of the (n-1)th stage to extract features from multiple (n-1)th sample intraoperative images in the (n-1)th sample intraoperative images respectively, to obtain multiple (n-1)th sample visual features; clustering the multiple (n-1)th sample visual features to obtain clustering results, the clustering results including multiple clusters; determining the (n-1)th sample intraoperative image corresponding to the (n-1)th sample visual feature whose distance from the center point of the cluster is less than a preset distance threshold as the (n-1)th sample buffer image, the number of (n-1)th sample buffer images being less than a preset number of sample buffers.

[0061] The K-means clustering algorithm divides the visual features of multiple (n-1)th samples into several data groups. Features within the same cluster are close in distance in the vector space, representing similar visual patterns.

[0062] The center point of a cluster is the mean center of the visual features of all samples in each cluster, representing the typical or average visual features of that cluster.

[0063] The preset distance threshold is used as the upper limit of the distance for filtering buffer samples. Only samples whose distance from the center point is less than the preset distance threshold are considered to have representative visual features.

[0064] The (n-1)th sample buffer image is a representative subset of images selected from the (n-1)th modality type, used to retain key distribution information of the old modality during subsequent training of the nth modality.

[0065] The preset sample buffer size is the upper limit of the total number of sample buffer images, ensuring that the size of the buffer set is controllable. For example, the preset sample buffer size is 10%-20% of the total number of intraoperative images of the (n-1)th sample.

[0066] First, calculate the distance between the visual feature of each (n-1)th sample and the center point of its cluster. Then, select the intraoperative images of the (n-1)th samples corresponding to features whose distance is less than a preset distance threshold, and use them as buffer images for the (n-1)th samples. These images, because their features are close to the cluster center, can represent high-information, highly representative samples of this modality type.

[0067] In addition, it is necessary to control the total number of sample buffer images to be less than the preset number of sample buffers, so as to control storage and computational overhead while preserving the representativeness of the distribution.

[0068] According to the embodiments of this application, the visual features of the (n-1)th sample are divided into multiple clusters by clustering, and the nearest neighbor samples are selected as sample buffer images by using the cluster center distance threshold. This allows the sample buffer images to accurately cover the typical distribution areas of the modality type in the feature space, retain the key distribution structure of the old modality with a small number of highly representative samples, and then use the sample buffer images to achieve cross-stage knowledge distillation when training new modalities, effectively reducing the forgetting of old modality information and cross-modal conflicts.

[0069] According to an embodiment of this application, an attention network model is trained based on the visual features of the nth sample, the buffer features of the (n-1)th sample, and the buffer features of the nth sample to obtain a feature extraction model corresponding to the nth stage. This includes: obtaining a contrastive loss value based on the difference between the buffer features of the (n-1)th sample and the buffer features of the nth sample; processing the visual features of the nth sample and the sample label using a task loss function to obtain an incremental modal sample loss value, where the sample label represents the true detection category corresponding to the detection task; and training the attention network model based on the incremental modal sample loss value and the contrastive loss value to obtain a feature extraction model corresponding to the nth stage.

[0070] In one embodiment, the loss values ​​are compared. As shown in formula (1):

[0071] (1).

[0072] in, Characterize the buffer features of the nth sample. Characterize the buffer features of the (n-1)th sample. Characterize the buffer image of the (n-1)th sample. Characterize the intraoperative image set of the (n-1)th sample.

[0073] The contrastive loss is the distribution distance between the buffer features of the nth sample and the buffer features of the (n-1)th sample. By forcing the new and old models to output similar feature vectors for the same set of buffered images, the current network is constrained to not deviate from the embedding space already established by the old modes while learning new modes. This is a distillation constraint that reduces the forgetting of information about old mode types.

[0074] In one embodiment, the task loss function can be a binary classification segmentation task, which uses a segmentation head to segment the visual features of the nth sample to obtain a predicted category, which is either tumor tissue or non-tumor tissue.

[0075] The sample labels are real, manually labeled categories.

[0076] The cross-entropy loss function is used to process the sample labels and predicted categories to obtain the incremental modality sample loss value. The incremental modality sample loss value is the learning loss of the current attention model for newly added samples of the nth modality type.

[0077] The incremental modality sample loss value drives the network to learn the features of the newly added nth intraoperative image, enabling it to grasp the structural and functional information unique to this modality. At the same time, the contrastive loss value constrains the network's output of the (n-1)th buffer image to be consistent with the output of the (n-1)th stage feature extraction model, thereby anchoring the embedding space distribution of the old modality while learning new modality knowledge. Through backpropagation of the two loss values ​​and joint gradient update, the parameters of the attention network model are optimized, and finally, the nth stage feature extraction model that can represent the features of the new modality and retain the knowledge of the existing modality is obtained.

[0078] According to an embodiment of this application, an attention network model is trained based on the incremental modal sample loss value and the contrast loss value to obtain a feature extraction model corresponding to the nth stage. This includes: training the attention network model based on the incremental modal sample loss value, the contrast loss value, and the reconstructed image loss value, wherein the reconstructed image loss value is determined based on the following operations: masking the nth intraoperative image in the nth sample intraoperative image set to obtain a sample mask image; extracting features from the sample mask image using the attention network model to obtain sample mask features; reconstructing the sample mask features to obtain a sample reconstructed image; and determining the reconstructed image loss value based on the sample reconstructed image and the nth sample intraoperative image.

[0079] Data augmentation is performed on the nth intraoperative image to form a global view and a local view. Each view is then divided into image patches and labeled before being input into the teacher network and the student network. The local view labels input into the student network are randomly occluded to obtain a sample mask image. The teacher network receives the unoccluded, more complete global view and provides the training objective. The occlusion task is used to learn high-level features of the occluded regions.

[0080] Feature extraction is performed on the sample mask image to obtain the sample mask features; the sample mask features are then reconstructed to obtain the reconstructed sample image.

[0081] The loss value of the reconstructed image is determined by processing the sample reconstructed image and the nth intraoperative image using the mask prediction loss function.

[0082] In one embodiment, the sum of Euclidean distances between a sample visual feature and its nearest neighbor features can also be processed using the nearest neighbor distance regularization loss function to obtain a uniformity loss value. The uniformity loss value characterizes the geometric sparsity of the sample visual features in the embedding space. By maximizing the sum of Euclidean distances, feature collapse is penalized, which promotes the uniform distribution of sample visual features in the feature space.

[0083] In one embodiment, the incremental total loss value is obtained based on the uniformity loss value, the reconstructed image loss value, and the incremental modal sample loss value. As shown in formula (2):

[0084] (2).

[0085] in, Characterizing the incremental modal sample loss value, Characterizing the uniformity loss value, The loss value for reconstructing the image is represented by α, β, and γ, which are preset weight coefficients for the three loss terms. These coefficients are used to adjust the relative impact of image-level consistency learning, occluded image patch learning, and feature distribution regularization on the update of student network parameters.

[0086] Incremental modal sample loss is used to reduce the image-level output of the teacher network and the student network; reconstructed image loss compares the student network's representation at the location of the occluded image patch with the target representation provided by the teacher network; uniformity loss is used to adjust the distribution of features in the representation space and reduce the risk of feature collapse.

[0087] In one embodiment, the overall loss value during the training phase As shown in formula (3):

[0088] (3).

[0089] Where λ represents the weight of the old mode type's sample buffer feature preservation loss in the overall loss value.

[0090] The student network is updated using backpropagation based on the overall loss value, while the current teacher network is updated using the Exponential Moving Average (EMA) algorithm. The network parameters of the feature extraction model from the previous stage remain frozen.

[0091] The student network is updated through backpropagation of the overall loss value, as shown in Equation (4). The teacher network does not directly accept gradient updates, but updates using the exponential moving average of the student network parameters.

[0092] (4).

[0093] in, This represents the teacher network in the q-th iteration. This represents the teacher network in the (q-1)th iteration. The parameter represents the student network parameter in the q-th iteration, where q represents the iteration number and m represents the EMA momentum coefficient, which is used to control the smoothness of the change in teacher network parameters with student network parameters.

[0094] According to embodiments of this application, the modal type includes at least one of white light imaging type, fluorescence imaging type, narrowband imaging type, near-infrared imaging type, and multispectral imaging type.

[0095] White light imaging uses visible light to illuminate and image the surgical field. It forms an image by observing the difference in the reflection of visible light by the tissue surface, and characterizes the macroscopic morphological and structural information of the target tissue, including tissue outline, surface texture, color features and anatomical layer relationships. It is the most basic imaging mode in laparoscopic surgery and provides an overall visual benchmark for the surgical field.

[0096] Fluorescence imaging utilizes exogenous or endogenous fluorescent substances to be irradiated with exogenous or endogenous fluorescent light at a specific wavelength. By detecting the emitted fluorescence signal, it forms an image that characterizes the functional and metabolic information of the target tissue, including blood perfusion status, lymphatic drainage pathways, biliary tract imaging, and tumor boundary identification. It can highlight specific biological processes against a background where white light cannot distinguish them.

[0097] Narrowband imaging utilizes two narrowband blue and green lights, 415nm and 540nm, for illumination to enhance the contrast of light absorption by hemoglobin on the mucosal surface and characterize the microvascular morphology information of the target tissue, including the distribution density, course pattern, and morphological abnormalities of the capillary network on the mucosal surface.

[0098] Near-infrared imaging utilizes the 700-1000nm near-infrared band for illumination imaging. Taking advantage of the strong tissue penetration capability of this band and the characteristic absorption properties of hemoglobin / water, it characterizes the deep structure and blood oxygenation function of the target tissue, including the distribution of subcutaneous blood vessels, changes in tissue blood oxygen saturation, and the location of deep lesions, thus compensating for the limitations of insufficient penetration depth of visible light.

[0099] Multispectral imaging utilizes multiple discrete narrow bands to acquire spectral data of the surgical field sequentially or simultaneously, and then uses spectral analysis to invert tissue characteristics.

[0100] According to an embodiment of this application, the training method further includes: sorting N modal types according to the data volume, structural information coverage, and tissue boundary information coverage of the sample intraoperative image set to obtain a training order, the training order including training the white light imaging type first and then training the fluorescence imaging type; determining the first sample buffer image from the first sample intraoperative image set based on the clustering result of the first sample intraoperative image set of the white light imaging type, the (n-1)th sample intraoperative image has the (n-1)th modal type, the first sample intraoperative image set is used to perform the first stage training operation to obtain the first feature extraction model; using the first stage feature extraction model to extract features from the first sample buffer image to obtain the first sample buffer features; using an attention network model to extract features from the second sample intraoperative image set and the first sample buffer image of the fluorescence imaging type respectively to obtain the second sample visual features and the second buffer features; training the attention network model according to the second sample visual features, the first sample buffer features, and the second buffer image features to obtain the second stage feature extraction model; and determining the feature extraction model corresponding to the second stage as the target model.

[0101] The data volume of the sample intraoperative image set is the number of sample intraoperative image sets for each modality type. Modalities with sufficient data are prioritized for early-stage training to ensure that the basic model converges fully on a large number of samples and establishes a stable feature space.

[0102] Structural information coverage characterization assesses the ability of each modality to represent the macroscopic morphology, anatomical layers, and spatial relationships of the target tissue. White light imaging typically provides the richest structural information; prioritizing its training allows the model to establish a complete cognitive framework of anatomical structures.

[0103] Tissue boundary information coverage characterization assesses the ability of each modality to distinguish tissue boundaries, lesion edges, and instrument-tissue interfaces. Narrow-band imaging and fluorescence modalities highlight boundary information, but they need to be introduced after the structural framework is established to avoid boundary features lacking anatomical context support.

[0104] When sorting, the modal types with large data volume and high structural information coverage are selected as the training data for the first stage, such as white light imaging. Subsequent stages are introduced in order of increasing tissue boundary information coverage and decreasing data volume, such as narrowband imaging, fluorescence imaging, near-infrared imaging, and multispectral imaging, forming a progressive training path from macroscopic structure to microscopic function and from basic framework to fine features.

[0105] According to embodiments of this application, by multi-dimensional quantitative ranking based on data volume, structural information coverage, and tissue boundary information coverage, incremental training is ensured to start from the modality with the most abundant data and the most complete structure. This enables the basic model to establish a stable and highly generalizable feature space representation in the early stages, providing a reliable embedding framework for knowledge fusion of subsequent modalities. Furthermore, training of subsequent modal types is introduced in order of increasing boundary information coverage, allowing the fine features of the newly added modal types to be orderly superimposed in the context of existing anatomical structures, avoiding feature drift or modal conflict caused by the lack of structural anchors in functional modalities. Finally, the ranking strategy and the sample buffer image distillation mechanism work together to gradually integrate each modal type into the unified feature space in the optimal topological order. While reducing information forgetting, this achieves complementary fusion of cross-modal information such as white light structural base and functional enhancements such as fluorescence and narrowband, providing a clear and semantically consistent visual representation foundation for multimodal intraoperative navigation and lesion identification.

[0106] In one embodiment, the training sequence includes training the white light imaging type first and then training the fluorescence imaging type.

[0107] First, low-quality or irrelevant segments such as external environment and blurred images were removed from the training samples, followed by the removal of frames with duplicate content. Then, white light imaging and fluorescence imaging types were distinguished, and uniform sampling was performed at 2 frames / second to form 1,451,670 intraoperative images of the first sample (white light imaging type) and 189,280 intraoperative images of the second sample (fluorescence imaging type). In the first stage, the attention network model was trained using the first sample intraoperative images. The student network in the first stage was updated via backpropagation based on the incremental total loss value, while the teacher network was updated using the exponential moving average of the student network parameters. After the first stage of training, the student network weights were retained, resulting in the first feature extraction model capable of extracting visual features of the white light imaging type.

[0108] Subsequently, the visual features of the first sample are extracted using the first feature extraction model. K-means clustering is performed on the extracted features, and high-information samples accounting for 10% of the original white light imaging type samples are selected to form the first sample buffer image. At the same time, the parameters of the first feature extraction model in the first stage are frozen.

[0109] In the second stage, the attention network model is trained using the second intraoperative image and the first buffer image. The student network in the second stage is updated via backpropagation based on the overall loss value, composed of the incremental total loss and the contrast loss. The teacher network is updated using the exponential moving average of the student network parameters. After the second stage training is completed, the student network weights are retained, resulting in a second feature extraction model applicable to both white light and fluorescence imaging types. This second feature extraction model is designated as the target model.

[0110] Figure 2 A flowchart of an image processing method according to an embodiment of this application is shown.

[0111] like Figure 3 As shown, the process includes operations S310 to S330.

[0112] In operation S310, an intraoperative image set of the target object is acquired. The intraoperative image set represents the structural and functional information of the target object's tissues during the operation.

[0113] During the operation of S320, the target model is used to extract features from the intraoperative image set to obtain visual features.

[0114] When operating S330, visual features are detected according to the detection model corresponding to the object detection task, and the detection results are obtained.

[0115] Load the weights of the target model that has been trained.

[0116] For example, for a single-modal classification task, the target model is used to extract features from the intraoperative image set to obtain visual features. The visual features are then detected using the detection model corresponding to the target detection task to obtain the detection results.

[0117] For example, for multimodal classification tasks involving white light imaging and fluorescence imaging, visual features of white light imaging and fluorescence imaging are extracted using the target model, the two visual features are stitched together, and the stitched visual features are detected using the detection model corresponding to the target detection task to obtain the detection results.

[0118] The target detection task is a downstream task connected to the target model and can be used for intraoperative smoke removal, surgical instrument segmentation, polyp segmentation, polyp detection, polyp classification, and lymph node metastasis prediction.

[0119] When the detection model is a segmentation head, the visual features extracted from the last layer of the target model are input into the segmentation head. Spatial resolution is restored through stepwise upsampling, and the detection results are output. The detection results are segmentation masks for instruments, polyps or other target regions.

[0120] According to embodiments of this application, the target detection task includes at least one of the following: tissue segmentation task, intraoperative smoke removal task, polyp classification, and lymph node metastasis detection task.

[0121] The tissue segmentation task involves using a computer vision model to perform pixel-level classification of different tissue types in an intraoperative image set, outputting a spatial mask for each tissue category. The detection results characterize the ability to identify the fine boundaries of the anatomical structures of the target object, providing pixel-level spatial semantic information for surgical navigation, lesion localization, and the definition of safe resection areas.

[0122] The intraoperative smoke removal task involves using image enhancement or generative models to detect and clarify areas obscured by smoke generated during surgery due to electrocautery cutting and tissue burning, thereby restoring the surgical field content that was obscured by smoke.

[0123] The polyp classification task involves using a deep learning model to determine the benign or malignant nature of polyp regions detected in an intraoperative image dataset, or to classify their pathological type. The detection results provide a semantic assessment of the pathological nature of the lesions, assisting clinical decisions on whether resection or follow-up is necessary, and reducing unnecessary tissue sampling and surgical trauma.

[0124] The lymph node metastasis detection task utilizes multimodal visual features or pathological image analysis to identify and predict the presence of tumor cell metastasis in regional lymph nodes. The detection results characterize the pixel-level identification of lymph node cells.

[0125] When the target model provided in this application is transferred to the task of predicting intraoperative lymph node metastasis in colorectal cancer, the area under the receiver operating characteristic curve (ROC) can reach 0.817.

[0126] Figure 3 A schematic diagram illustrating the training and application of a target model according to an embodiment of this application is shown.

[0127] like Figure 3As shown, the training samples include a first sample intraoperative image of white light imaging and a second sample intraoperative image of fluorescence imaging. The training process consists of two stages. In the first stage, the first sample intraoperative image set is input into the attention network model to train the first feature extraction model. Based on clustering, buffer images of the first sample are selected, and the first sample buffer features are extracted using the first feature extraction model. In the second stage, the second sample intraoperative image set and the first sample buffer images are input into a new attention network model to extract the second sample visual features and the second sample buffer features respectively. The contrast loss value determined by the first and second sample buffer features constrains the consistency of the output of the new and old models to the buffer samples. The learning of new modalities is driven by the incremental total loss value determined by the second sample visual features. The model parameters are optimized by jointly using the contrast loss value and the incremental total loss value to obtain the second feature extraction model, which is then determined as the target model. In the model application stage, the intraoperative image set is input into the target model to extract visual features, and then input into five downstream target detection tasks: intraoperative smoke removal, instrument and tissue segmentation, polyp detection, polyp classification, and lymph node metastasis prediction, to achieve multi-task collaborative intraoperative assistance.

[0128] Figure 4A A structural block diagram of a training apparatus for a target model based on multi-stage incremental learning according to an embodiment of this application is shown.

[0129] like Figure 4A As shown, the training device 410 for the target model based on multi-stage incremental learning in this embodiment includes a first acquisition module 401 and a training module 402.

[0130] The first acquisition module 401 is used to acquire training samples. The N intraoperative image sets in the training samples correspond to N modal types. The intraoperative image sets represent the structural and functional information of the target tissue during the operation.

[0131] Training module 402 is used to perform the following N-stage training operations based on training samples to obtain the target model; the nth stage training operation includes: determining the (n-1)th sample buffer image from the (n-1)th sample intraoperative image set based on the clustering results of the (n-1)th sample intraoperative image set, the (n-1)th sample intraoperative image having the (n-1)th modality type, and the (n-1)th sample intraoperative image set being used to perform the (n-1)th stage training operation to obtain the (n-1)th feature extraction model. The feature extraction model of stage n-1 is used to extract features from the buffer image of sample n-1 to obtain the buffer features of sample n-1. The attention network model is used to extract features from the intraoperative image set of sample n and the buffer image of sample n-1 to obtain the visual features of sample n and the buffer features of sample n. The attention network model is trained based on the visual features of sample n, the buffer features of sample n-1 and the buffer features of sample n to obtain the feature extraction model of stage n. The feature extraction model corresponding to stage N is determined as the target model.

[0132] Figure 4B A structural block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0133] like Figure 4B As shown, the image processing apparatus 420 of this embodiment includes a second acquisition module 403, an extraction module 404, and a detection module 405.

[0134] The second acquisition module 403 is used to acquire the intraoperative image set of the target object, which represents the structural and functional information of the target object's tissues during the operation.

[0135] The extraction module 404 is used to extract features from the intraoperative image set using the target model to obtain visual features, wherein the target model is trained according to the above method.

[0136] The detection module 405 is used to detect visual features according to the detection model corresponding to the target detection task and obtain the detection result.

[0137] Optionally, any multiple modules among the first acquisition module 401 and training module 402, second acquisition module 403, extraction module 404, and detection module 405 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. Optionally, at least one of the first acquisition module 401 and training module 402, second acquisition module 403, extraction module 404, and detection module 405 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 401 and training module 402, the second acquisition module 403, the extraction module 404 and the detection module 405 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0138] Figure 5 A block diagram of an electronic device suitable for implementing a training method and an image processing method for a target model based on multi-stage incremental learning, according to an embodiment of this application, is shown.

[0139] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0140] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 (Read-Only Memory) or a program loaded from storage portion 508 into RAM 503 (Random Access Memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0141] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0142] In embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0143] The embodiments of this application, and the method flow according to the embodiments of this application, can be implemented as a computer software program. For example, an embodiment of this application includes a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of this application. In the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0144] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the training method for the target model based on multi-stage incremental learning according to the embodiments of this application.

[0145] In embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0146] For example, in embodiments of this application, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0147] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the training method of the target model based on multi-stage incremental learning provided in the embodiments of this application.

[0148] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. In the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0149] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0150] In embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0152] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A training method for a target model based on multi-stage incremental learning, characterized in that, The training process of the target feature extraction model includes N stages, and the method includes: Acquire training samples, wherein the N intraoperative image sets in the training samples correspond to N modal types, and the intraoperative image sets represent the structural and functional information of the target tissue during the operation; Based on the training samples, the following N stages of training operations are performed to obtain the target model; the nth stage of training operations includes: Based on the clustering results of the (n-1)th sample intraoperative image set, the (n-1)th sample buffer image is determined from the (n-1)th sample intraoperative image set. The (n-1)th sample intraoperative image has an (n-1)th modality type. The (n-1)th sample intraoperative image set is used to perform the (n-1)th stage training operation to obtain the (n-1)th feature extraction model. ; The feature extraction model of the (n-1)th stage is used to extract features from the (n-1)th sample buffer image to obtain the (n-1)th sample buffer features. The attention network model was used to extract features from the intraoperative image set of the nth sample and the buffer image of the (n-1)th sample, respectively, to obtain the visual features of the nth sample and the buffer features of the nth sample. Based on the visual features of the nth sample, the buffer features of the (n-1)th sample, and the buffer features of the nth sample, the attention network model is trained to obtain the feature extraction model for the nth stage. The feature extraction model corresponding to the Nth stage is determined as the target model.

2. The method according to claim 1, characterized in that, The determination of the (n-1)th sample buffer image from multiple (n-1)th sample intraoperative images based on the clustering results of the (n-1)th sample intraoperative image set includes: The feature extraction model of the (n-1)th stage is used to extract features from multiple intraoperative images of the (n-1)th sample, respectively, to obtain multiple visual features of the (n-1)th sample. Clustering is performed on the visual features of multiple n-1th samples to obtain clustering results, which include multiple clusters; The (n-1)th sample intraoperative image corresponding to the visual feature of the (n-1)th sample that is less than the distance between the distance to the center point of the cluster is a preset distance threshold is determined as the (n-1)th sample buffer image, and the number of the (n-1)th sample buffer images is less than the preset number of sample buffers.

3. The method according to claim 1, characterized in that, The step of training the attention network model based on the visual features of the nth sample, the buffer features of the (n-1)th sample, and the buffer features of the nth sample to obtain the feature extraction model corresponding to the nth stage includes: The contrast loss value is obtained based on the difference between the (n-1)th sample buffer feature and the nth sample buffer feature; The visual features and sample labels of the nth sample are processed using a task loss function to obtain an incremental modal sample loss value, wherein the sample label represents the true detection category corresponding to the detection task. Based on the incremental modal sample loss value and the contrastive loss value, the attention network model is trained to obtain the feature extraction model corresponding to the nth stage.

4. The method according to claim 3, characterized in that, Based on the incremental modality sample loss value and the contrastive loss value, the attention network model is trained to obtain the feature extraction model corresponding to the nth stage, including: The attention network model is trained based on the incremental modality sample loss value, the contrast loss value, and the reconstructed image loss value, wherein the reconstructed image loss value is determined based on the following operation: Mask the nth intraoperative image in the nth intraoperative image set to obtain a sample mask image; The sample mask image is used to extract features using an attention network model to obtain sample mask features; The sample mask features are reconstructed to obtain the sample reconstructed image; The reconstructed image loss value is determined based on the reconstructed image of the sample and the intraoperative image of the nth sample.

5. The method according to claim 1, characterized in that, The modal types include at least one of white light imaging, fluorescence imaging, narrowband imaging, near-infrared imaging, and multispectral imaging.

6. The method according to claim 5, characterized in that, The method further includes: Based on the data volume, structural information coverage, and tissue boundary information coverage of the sample intraoperative image set, the N modal types are sorted to obtain the training order, which includes the order of training the white light imaging type first and then the fluorescence imaging type. Based on the clustering results of the first sample intraoperative image set of white light imaging type, the first sample buffer image is determined from the first sample intraoperative image set, the (n-1)th sample intraoperative image has the (n-1)th modality type, and the first sample intraoperative image set is used to perform the first stage training operation to obtain the first feature extraction model; The first sample buffer image is used to extract features using the feature extraction model of the first stage to obtain the features of the first sample buffer. The attention network model was used to extract features from the intraoperative image set of the second sample and the buffer image of the first sample in the fluorescence imaging type, respectively, to obtain the visual features of the second sample and the features of the second buffer. Based on the visual features of the second sample, the buffer features of the first sample, and the image features of the second buffer, the attention network model is trained to obtain the feature extraction model for the second stage. The feature extraction model corresponding to the second stage is determined as the target model.

7. An image processing method, characterized in that, The method includes: Acquire an intraoperative image set of the target object, wherein the intraoperative image set characterizes the structural and functional information of the target object's tissues during the surgical procedure; The target model is used to extract features from the intraoperative image set to obtain visual features, wherein the target model is trained by the method according to any one of claims 1-6; The visual features are detected using the detection model corresponding to the target detection task to obtain the detection results.

8. The method according to claim 7, characterized in that, The target detection task includes at least one of the following: tissue segmentation task, intraoperative smoke removal task, polyp classification task, and lymph node metastasis detection task.

9. A target model training device based on multi-stage incremental learning, characterized in that, The training process of the target feature extraction model includes N stages, and the device includes: The first acquisition module is used to acquire training samples, wherein the N intraoperative image sets of the training samples correspond to N modal types, and the intraoperative image sets of the samples represent the structural and functional information of the target tissue of the sample during the operation. The training module is used to perform the following N-stage training operations based on the training samples to obtain the target model; the nth stage training operation includes: determining the (n-1)th sample buffer image from the (n-1)th sample intraoperative image set based on the clustering results of the (n-1)th sample intraoperative image set, wherein the (n-1)th sample intraoperative image has an (n-1)th modality type, and the (n-1)th sample intraoperative image set is used to perform the (n-1)th stage training operation to obtain the (n-1)th feature extraction model. The feature extraction model of the (n-1)th stage is used to extract features from the (n-1)th sample buffer image to obtain the (n-1)th sample buffer features. The attention network model is used to extract features from the nth intraoperative image set and the (n-1)th sample buffer image to obtain the nth sample visual features and the nth sample buffer features. The attention network model is trained based on the nth sample visual features, the (n-1)th sample buffer features and the nth sample buffer features to obtain the nth stage feature extraction model. The feature extraction model corresponding to the Nth stage is determined as the target model.

10. An image processing apparatus, characterized in that, The device includes: The second acquisition module is used to acquire an intraoperative image set of the target object, wherein the intraoperative image set represents the structural and functional information of the target object's tissues during the surgical procedure. An extraction module is used to extract features from the intraoperative image set using a target model to obtain visual features, wherein the target model is trained by the method according to any one of claims 1-6; The detection module is used to detect the visual features according to the detection model corresponding to the target detection task, and obtain the detection result.

Images