A model-independent difficult sample generation method and system

Abstract

The application provides a model-independent difficult sample generation method and system, relating to the technical field of anomaly detection and image segmentation, which comprises: obtaining a real image and a segmentation mask thereof; constructing a real data prototype by using a pre-trained feature extraction network; fine-tuning a conditional latent diffusion model to adapt to a target segmentation task; in the model sampling process, generating a difference between the features and the prototype by target region cropping and diversity loss maximization to generate difficult samples; iteratively updating the prototype and repeatedly generating multiple rounds of samples; finally mixing the difficult samples with real data to obtain an augmented training set for improving the performance of a downstream segmentation model. The application does not need to rely on feedback from a specific downstream model, can generate samples that are real, diverse and challenging, and effectively enhances the generalization ability and robustness of various segmentation models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection and image segmentation, and particularly relates to a model-independent difficult sample generation method and system. BACKGROUND

[0002] In the technical field of anomaly detection and image segmentation, especially in application scenarios such as medical image analysis and industrial nondestructive testing where high-quality labeled data are scarce, using a generative model to synthesize training samples with accurate labels has become an important technical means to improve the generalization ability and robustness of downstream task models (such as segmentation models). Traditional methods usually rely on generative artificial intelligence technologies such as generative adversarial networks or diffusion models to generate image and segmentation mask pairs for expanding the training set.

[0003] A typical solution in the prior art is to use a conditional diffusion model (such as a latent diffusion model) or a control generation network (such as ControlNet). This type of method inputs the segmentation mask of the target region as a spatial condition to guide the generation process, so as to ensure that the generated image is structurally aligned with the conditional mask. To further improve the training effectiveness of the generated samples on downstream models, some advanced methods introduce a feedback mechanism of the downstream segmentation model itself. Specifically, these methods calculate the prediction error of the generated samples on a specific downstream model (such as a segmentation model with the CTNet or PVT architecture), dynamically adjust the generation process, and aim to produce samples that are more “challenging” or “difficult” for the specific model, so as to improve the performance of the model through adversarial training.

[0004] However, this type of prior art has obvious limitations. First, the generated samples lack diversity, especially in the morphology, texture, and spatial distribution of key target regions (such as polyps and lesion regions in medical images), which are difficult to cover the complexity and diversity of anomalies in real scenarios, leading to overfitting of downstream models and limited improvement in generalization performance. Secondly, and more critically, existing methods rely heavily on the structural preferences and error patterns of specific downstream models. Since different model architectures have different feature extraction mechanisms, attention patterns, and sensitivity to errors, the “difficult samples” generated for a specific model often cannot be effectively transferred to models with other architectures, i.e., lack of cross-model generality. This greatly limits the practical value of the generated data in real-world applications, as multiple models with different architectures are often deployed or compared in real scenarios. In addition, some methods require a large number of iterative calculation steps in the generation process, resulting in low computational efficiency and difficulty in meeting the demand for large-scale data generation.

[0005] Therefore, there is an urgent need in the art for a method capable of generating sample data with both high authenticity and wide diversity and essential challenges, and the method should be independent of any specific downstream model, so as to universally enhance the performance of various segmentation models. SUMMARY

[0006] To this end, the embodiments of the present application provide a model-independent difficult sample generation method and system, which are used to solve the problems of strong model specificity, poor cross-model universality and insufficient sample diversity caused by relying on specific downstream model feedback in the prior art.

[0007] To solve the above technical problems, the embodiments of the present application provide a model-independent difficult sample generation method, which comprises:

[0008] Step S1: obtaining a real data set corresponding to a target segmentation task, the real data set containing real images and segmentation masks corresponding to each real image;

[0009] Step S2: using a pre-trained feature extraction backbone network to extract features of all real images in the real data set, obtaining feature vectors of each real image, calculating the mean of the feature vectors, and constructing an initial real data prototype;

[0010] Step S3: loading a pre-trained conditional latent diffusion model, inputting the segmentation mask into the conditional latent diffusion model as a condition, fine-tuning the conditional latent diffusion model by minimizing a set conditional denoising score matching loss, and obtaining a fine-tuned diffusion model adapted to the target segmentation task;

[0011] Step S4: sampling images based on the fine-tuned diffusion model, and performing the following operations in the sampling process:

[0012] Step S41: according to the current predicted clean image and the set conditional mask in the sampling process, cropping a target region image block in the clean image;

[0013] Step S42: using the feature extraction backbone network to extract a feature vector of the target region image block, calculating the distance between the feature vector and the real data prototype, and maximizing the distance by a set diversity loss to generate a difficult sample;

[0014] Step S5: combining the difficult sample generated in step S4 with the real data set, updating the real data prototype, repeating step S4 for a preset number of rounds, and obtaining multiple rounds of difficult samples;

[0015] Step S6: mixing the multiple rounds of difficult samples with the real data set to obtain an augmented training set, which is used to train a downstream segmentation model.

[0016] Preferably, the real dataset in step S1 is a medical image segmentation dataset, which includes at least one of the ETIS dataset, CVC-ClinicDB dataset, CVC-ColonDB dataset, EndoScene dataset, and Kvasir dataset, and the target segmentation task is the segmentation task of lesion regions in medical images.

[0017] Preferably, the pre-trained feature extraction backbone network in step S2 is a Vision Transformer network or a ResNet network, and the parameters of the feature extraction backbone network remain fixed during the feature extraction process and are only used to perform forward propagation to output feature vectors.

[0018] Preferably, step S4 further includes a multi-objective loss joint optimization step: constructing a total loss function. The total loss function The conditional denoising score matching loss With the aforementioned diversity loss The weighted sum, i.e. ,in As preset weighting coefficients, the Experiments were conducted to determine a balance between realism and diversity in the generated images;

[0019] The conditional denoising score matching loss The calculation formula is:

[0020] ;

[0021] in, For real images, for The image after adding noise at different times. For conditional mask, This is real noise. For the noise predicted by the model, For mathematical expectation, It follows a standard normal distribution. It is the identity matrix. The square of the L2 norm;

[0022] The loss of diversity The calculation formula is:

[0023] ;

[0024] in, As a preset marginal value, Based on real data prototypes, a feature vector of a target region image block, represents the Euclidean distance calculation.

[0025] Preferably, the specific process of updating the real data prototype in step S5 includes:

[0026] Suppose the first iteration, merge the real data set generated in the previous iteration to obtain a merged data set ; ;

[0027] Extract the feature vectors of all samples in the merged data set using the feature extraction backbone network, calculate the mean of the feature vectors, and obtain the real data prototype of the first iteration: :

[0028] ;

[0029] wherein, is a sample in the merged data set, is the feature vector of the sample , and represents the number of samples in the set.

[0030] Preferably, the current predicted clean image in step S41 is calculated by the following formula:

[0031] ;

[0032] wherein, is the noisy image at the moment in the sampling process, is the diffusion coefficient at the moment , and is the noise predicted by the fine-tuned diffusion model based on the noisy image and the conditional mask at the moment .

[0033] Preferably, the fine-tuning of the pre-trained conditional latent diffusion model in step S3 adopts a parameter-efficient fine-tuning method, which is a low-rank adaptive fine-tuning method, and only a set of parameters in the conditional latent diffusion model are updated to adapt to the target segmentation task.

[0034] Preferably, the step S4 generates a difficult sample also including a corresponding difficult sample segmentation mask, which is generated by calculating the pixel-level difference between the generated difficult sample and the clean image, for directly integrating into the augmented training set as an image-mask pair.

[0035] The embodiment of the present application also provides a model-independent difficult sample generation system, which is used for implementing the model-independent difficult sample generation method described above, and comprises:

[0036] A data preparation module is configured to acquire a real data set corresponding to a target segmentation task, wherein the real data set comprises real images and segmentation masks corresponding to the real images;

[0037] A prototype construction module is configured to perform feature extraction on all the real images in the real data set by using a pre-trained feature extraction backbone network, to obtain feature vectors of the real images, to calculate the mean of the feature vectors, and to construct an initial real data prototype;

[0038] A model fine-tuning module is configured to load a pre-trained conditional latent diffusion model, to inject the conditional latent diffusion model by taking the segmentation mask as a conditional input, to fine-tune the conditional latent diffusion model by minimizing a set conditional denoising score matching loss, and to obtain a fine-tuned diffusion model adapted to the target segmentation task;

[0039] A sample generation module is configured to generate images by sampling based on the fine-tuned diffusion model, wherein the sample generation module comprises a target cropping unit and a loss constraint unit, the target cropping unit is configured to crop a target region image block in the clean image according to a currently predicted clean image in a sampling process and a set conditional mask, and the loss constraint unit is configured to extract a feature vector of the target region image block by using the feature extraction backbone network, to calculate a distance between the feature vector and the real data prototype, and to generate a difficult sample by maximizing the distance through a set diversity loss;

[0040] An iterative optimization module is configured to combine the difficult sample generated by the sample generation module with the real data set, to update the real data prototype, and to return the sample generation module for execution for a preset number of rounds, so as to obtain a plurality of rounds of difficult samples;

[0041] A sample application module is configured to mix the plurality of rounds of difficult samples with the real data set to obtain an augmented training set, which is used for training a downstream segmentation model.

[0042] The embodiment of the present application also provides a computer storage medium, which stores a computer software product, and the computer software product comprises a plurality of instructions for enabling a computer device to execute the model-independent difficult sample generation method described above.

[0043] From the above technical solutions, the present application has the following beneficial effects:

[0044] (1) Strong generality independent of model: The present application defines and generates difficult samples by introducing a feature prototype comparison mechanism, completely independent of the feedback of any specific downstream segmentation model. This method guides the generation process based on the feature distribution (prototype) of the data itself, making the generated difficult samples universally applicable to various downstream models of different architectures (such as CTNet, PVT, etc.), effectively solving the problem of poor sample transferability and low practical value caused by dependence on specific models in the prior art.

[0045] (2) Significant improvement in sample diversity and challenge: By focusing on key semantic regions through target region attention cropping, and combining with a diversity loss function to actively maximize the distance between generated sample features and real data prototypes in feature space, the present application can systematically generate samples that are highly diverse in form, texture, and appearance, and are essentially challenging. Iterative optimization strategy enables dynamic updating of prototypes, forcing the generation model to continuously explore new feature spaces, thereby continuously improving the diversity and complexity of generated samples, effectively preventing overfitting of downstream models.

[0046] (3) High generation efficiency and practicality: The use of parameter-efficient fine-tuning methods (such as LoRA) for rapid adaptation of pre-trained diffusion models significantly reduces computational costs. At the same time, the method automatically generates pixel-level accurate segmentation masks, forming complete image-mask pairs with difficult samples, which can be directly used for downstream model training, eliminating the cumbersome manual annotation process, achieving automated generation of high-quality augmented datasets, and having good practicality and scalability. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly describe the drawings needed in the embodiments, which will be clearer to understand the features and advantages of the present application by referring to the drawings. The drawings are schematic and should not be understood as any limitation on the present application. For those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings. Among them:

[0048] Figure 1 is a flowchart of a model-independent difficult sample generation method provided by the present application;

[0049] Figure 2 is an architecture diagram of the method of the present application in the present application;

[0050] Figure 3 is a block diagram of a model-independent difficult sample generation system provided by the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] Embodiment one: In order to solve the problems of strong model specificity, poor cross-model universality and insufficient sample diversity of generated samples caused by relying on specific downstream model feedback in the prior art, the present application aims to provide a model-independent difficult sample generation method and system, which generates samples that are both real and have a wide range of challenges through feature prototype comparison, iterative optimization and target area focusing strategy, without relying on any downstream model feedback, and can effectively adapt to various segmentation architectures.

[0053] To avoid ambiguity, the key terms in this embodiment are defined first:

[0054] Latent Diffusion Model (LDM): a generative model based on diffusion process, which generates images by gradually denoising in latent space, and can receive external conditions (such as segmentation masks) to control the generation direction; in this embodiment, the pre-trained Stable Diffusion v1.5 is used as the base model.

[0055] Feature extraction backbone network: a pre-trained network used to extract deep semantic features of images, with fixed parameters and only used for forward propagation; in this embodiment, Vision Transformer (ViT-B / 16) or ResNet-50 can be selected.

[0056] Real data prototype: a vector representing the feature distribution of real data, obtained by calculating the mean of the feature vectors of real images (or real + historical generated images), used to guide the diversity generation of difficult samples.

[0057] Difficult sample: a synthetic sample that is significantly different from the real data prototype in the feature space and can cover complex situations in real scenarios, with an accurate segmentation mask, which can effectively improve the robustness of downstream models.

[0058] Low Rank Adaptation (LoRA): a parameter-efficient fine-tuning method, which inserts a low-rank matrix adapter into the key layers of the model to update only a small number of parameters to adapt the model, avoiding the high computational cost of full fine-tuning.

[0059] As Figure 1As shown, the present application proposes a model-independent difficult sample generation method, which comprises:

[0060] Step S1: obtaining a real data set corresponding to a target segmentation task, the real data set containing real images and segmentation masks corresponding to each real image;

[0061] Step S2: using a pre-trained feature extraction backbone network to extract features of all real images in the real data set, obtaining feature vectors of each real image, calculating the mean of the feature vectors, and constructing an initial real data prototype;

[0062] Step S3: loading a pre-trained conditional latent diffusion model, inputting the segmentation mask into the conditional latent diffusion model as a condition, and fine-tuning the conditional latent diffusion model by minimizing the set conditional denoising score matching loss, obtaining a fine-tuned diffusion model adapted to the target segmentation task;

[0063] Step S4: sampling images based on the fine-tuned diffusion model, and performing the following operations in the sampling process:

[0064] Step S41: according to the current predicted clean image in the sampling process and the set conditional mask, the target region image block in the clean image is cropped;

[0065] Step S42: using the feature extraction backbone network to extract the feature vector of the target region image block, calculating the distance between the feature vector and the real data prototype, and generating difficult samples by maximizing the distance through the set diversity loss;

[0066] Step S5: combining the difficult samples generated in step S4 with the real data set, updating the real data prototype, returning to step S4 to repeat the execution for a preset number of rounds, and obtaining multiple rounds of difficult samples;

[0067] Step S6: mixing the multiple rounds of difficult samples with the real data set to obtain an augmented training set, which is used to train a downstream segmentation model.

[0068] From the above technical solution can be known, the present application proposes a kind of model-independent difficult sample generation method, the method is first by obtaining real image and its segmentation mask constructs initial data set, and utilizes pre-training feature extraction backbone network to extract image feature mean to construct real data prototype, establishes the reference benchmark of feature space for the generation of difficult sample;Subsequently, load pre-training conditional latent diffusion model and with segmentation mask as condition fine-tuning, make it adapt to target task and guarantee the structural rationality of generated image;In sampling generation stage, by cutting the target area in current predicted clean image, and calculate its feature distance with real data prototype, utilize diversity loss to maximize the distance, to guide generation to deviate from common mode in semantic feature, with challenging difficult sample;Further, the difficult sample generated and real data are merged and iteratively updated real data prototype, continuously expand the coverage of feature space by multi-round self-game mechanism, constantly improve the diversity and difficulty of sample;Finally, the difficult sample generated in multiple rounds and original data are mixed to form augmented training set, provide rich and high-challenge training data for downstream segmentation model, effectively improve its generalization ability and robustness.

[0069] The present embodiment takes the segmentation task of lesion area in medical image as an example, combines Figure 2 The implementation steps of the method of the present application are described in detail, and the parameters, operation details and logical relationships of each step are as follows:

[0070] In step S1, the real data set corresponding to the segmentation task of the lesion area is obtained, and the real data set needs to include "real image-segmentation mask pair", which is used to construct initial feature prototype and fine-tuning diffusion model.

[0071] Dataset selection: multiple public medical image segmentation datasets are used, including ETIS, CVC-ClinicDB, CVC-ColonDB, EndoScene and Kvasir, all images are normalized to 256x256 pixels, and the gray value is mapped to the [0,1] interval, and the segmentation mask is a binary image (the foreground polyp area is 1, and the background is 0).

[0072] Data preprocessing: remove image noise by Gaussian filtering, and use random horizontal flip for data enhancement (only used for subsequent downstream model training, does not affect prototype construction), to ensure consistent data quality.

[0073] In step S2, the pre-trained feature extraction backbone network is used to extract real image features, and the mean of the feature vector is calculated to construct the initial real data prototype , the specific operation is as follows:

[0074] Backbone network configuration: select the pre-trained ViT-B / 16 network on the ImageNet dataset, fix all its parameters (only for forward propagation, not involved in subsequent training).

[0075] Feature extraction and prototype calculation: traverse the images in the real dataset, input each image into the ViT-B / 16 network to obtain the corresponding feature vector; calculate the element-level mean of all feature vectors to obtain the initial real data prototype, and store it in the prototype database.

[0076] In step S3, the pre-trained conditional latent diffusion model is loaded to fine-tune it with the mask as the conditional input, so that it adapts to the polyp image generation task, and the specific process is as follows:

[0077] Pre-trained model loading: load the Stable Diffusion v1.5 model, which includes the VAE encoder-decoder, the CLIP text encoder (only used for compatible model structure in this embodiment, and the core condition is the segmentation mask), and the UNet network; the UNet network receives the segmentation mask input through the cross-attention mechanism.

[0078] Parameter efficient fine-tuning (LoRA): insert the LoRA adapter in the attention layer of the UNet network, only update the low-rank matrix parameters of the LoRA adapter, and reduce the calculation cost.

[0079] Loss function and training configuration: conditional denoising score matching loss as the fine-tuning target, the formula is as follows:

[0080] ;

[0081] where, is the real image, is the image with noise at time t, is the conditional mask, is the real noise, is the model predicted noise, is the mathematical expectation, is the standard normal distribution, is the identity matrix, is the square of the L2 norm.

[0082] In step S4, the diffusion model after fine-tuning is used for sampling (inference) to generate images, and the "target area cropping" and "diversity loss constraint" are used to guide the generation process to generate difficult samples, and the specific operation is as follows:

[0083] Sampling process initialization: set the number of sampling steps, randomly select a certain number from the segmentation mask of the real data set as the condition input, input the UNet network of the fine-tuned diffusion model, start the reverse denoising sampling.

[0084] Step S41: at a certain time step of the sampling process, according to the clean image predicted by the model at the current time , crop the target lesion area (such as polyp):

[0085] Calculate the clean image : through the denoising formula of the diffusion model, specifically:

[0086] ;

[0087] Where, is the noisy image at time in the sampling process, is the diffusion coefficient at time (obtained from the pre-defined parameter table of Stable Diffusion v1.5), is the noise predicted by the fine-tuned diffusion model based on the noisy image and the condition mask at time .

[0088] Crop target area: determine the minimum bounding box of the polyp area according to the condition mask , extract the image block corresponding to the bounding box from to avoid background feature interference.

[0089] This operation aims to solve the problem that the target area occupies a small proportion in medical images and the features are easily covered by the background, so as to focus the model's learning on the key semantic area.

[0090] Step S42: maximize the difference between the generated feature and the real data prototype through prototype contrast learning, specifically:

[0091] Feature extraction: input the cropped target area image block into the fixed ViT-B / 16 network to obtain the feature vector .

[0092] Distance calculation: calculate the Euclidean distance between and the real data prototype .

[0093] Diversity loss optimization: adopt diversity loss Maximizing the above distance, in turn, encourages the model to generate difficult samples that are very different from the prototype in the feature space, and the calculation formula is as follows:

[0094] ;

[0095] wherein, is a preset margin value (set to 50 in the experiment), as a threshold for controlling the intensity of diversity, used to represent the maximum allowed distance between two feature calculations, is the feature vector of the target region image block, represents the Euclidean distance calculation. The is back-propagated to the LoRA adapter of the UNet, fine-tuning the sampling direction to generate difficult sample images.

[0096] Further, step S4 also includes a multi-objective loss joint optimization step: constructing a total loss function , the total loss function is the weighted sum of the conditional denoising score matching loss and the diversity loss , that is, , wherein is a preset weighting coefficient, determined through experiments (for example ) to balance the authenticity and diversity of generated images; too large may cause semantic distortion of the image, and too small will not significantly enhance the diversity.

[0097] Further, step S4 also includes difficult sample mask generation: generating a segmentation mask corresponding to the difficult sample through pixel-level difference calculation: calculating the gray difference between the difficult sample image and the clean image , setting a threshold (0-255 gray scale range), and marking the polyp region (mask value 1) for the region with a gray difference greater than the threshold, and the background (mask value 0) otherwise, to obtain a difficult sample image-mask pair.

[0098] In step S5, to continuously generate challenging samples and prevent the model performance from plateauing, an iterative generation and prototype updating process is adopted, which is implemented as follows:

[0099] Iteration round setting: preset the iteration rounds, and generate a certain number of difficult samples in each round. First round: use the initial real data prototype (only based on real data calculation) to guide the generation of the first batch of difficult samples; second round and later: combine the difficult samples generated in the previous round with the real data set, update the real data prototype, and use the updated real data prototype to guide the generation of new difficult samples in the next round.

[0100] Prototype update (take the first Wheel for example): merge real data set With the previous Wheel generated difficult sample set , get the merged data set ; using ViT-B / 16 to extract the feature vector of all samples in the merged data set, calculate the mean to get the real data prototype of the first Wheel , the formula is as follows:

[0101] ;

[0102] Among them, is the sample in the merged data set, is the feature vector of sample , Indicates the number of samples in the set.

[0103] Sample regeneration: take the updated As a new prototype constraint, repeat step S4 to generate the first Round of difficult samples; each round of samples needs to maintain differences with historical samples in the feature space, forming a self-game diversity expansion mechanism.

[0104] In step S6, the multi-round difficult samples are mixed with the real data to generate an expanded training set for training the downstream segmentation model, specifically:

[0105] Data set mixing: mix the real data and the generated data according to the set proportion to obtain the expanded training set, without the need for data screening or phased processing.

[0106] Downstream model training: input the expanded training set into different architectures of the segmentation model (CTNet, PVT), use the Adam optimizer, and train for 50 rounds; without introducing course learning or complex preprocessing, directly adapt the standard input format of the model, which can significantly improve the generalization performance, robustness and segmentation accuracy of the model on unseen data.

[0107] Embodiment two: as shown in Figure 3 The present application provides a model-independent difficult sample generation system, which is used to realize the model-independent difficult sample generation method of the above-mentioned embodiment one, specifically comprising:

[0108] The data preparation module 100 is used for obtaining the real data set corresponding to the target segmentation task, and the real data set contains real images and segmentation masks corresponding to each real image.

[0109] The prototype construction module 200 is configured to perform feature extraction on all real images in the real data set by using the pre-trained feature extraction backbone network to obtain feature vectors of the real images, calculate the mean of the feature vectors, and construct an initial real data prototype.

[0110] The model fine-tuning module 300 is configured to load the pre-trained conditional latent diffusion model, inject a segmentation mask as a conditional input into the conditional latent diffusion model, fine-tune the conditional latent diffusion model by minimizing a set conditional denoising score matching loss, and obtain a fine-tuned diffusion model adapted to the target segmentation task.

[0111] The sample generation module 400 is configured to generate images by sampling based on the fine-tuned diffusion model. The sample generation module includes a target cropping unit and a loss constraint unit. The target cropping unit is configured to crop a target region image block in a clean image predicted in a current sampling process according to a set conditional mask. The loss constraint unit is configured to extract a feature vector of the target region image block by using the feature extraction backbone network, calculate a distance between the feature vector and the real data prototype, and generate a difficult sample by maximizing the distance through a set diversity loss.

[0112] The iterative optimization module 500 is configured to combine the difficult sample generated by the sample generation module with the real data set, update the real data prototype, and return the sample generation module to perform a preset number of rounds to obtain a plurality of rounds of difficult samples.

[0113] The sample application module 600 is configured to mix the plurality of rounds of difficult samples with the real data set to obtain an augmented training set for training a downstream segmentation model.

[0114] The model-independent difficult sample generation system of the embodiment is used to implement the model-independent difficult sample generation method described above, and therefore the specific embodiments of the model-independent difficult sample generation system can be seen from the embodiment part of the model-independent difficult sample generation method. For example, the data preparation module 100, the prototype construction module 200, the model fine-tuning module 300, the sample generation module 400, the iterative optimization module 500, and the sample application module 600 are respectively used to implement steps S1, S2, S3, S4, S5, and S6 in the model-independent difficult sample generation method described above. Therefore, the specific embodiments can be referred to the description of the respective embodiments, and the details are not repeated here to avoid redundancy.

[0115] Embodiment three: The embodiment of the present application provides a computer storage medium, which stores a computer software product. The computer software product includes a plurality of instructions for causing a computer device to execute the model-independent difficult sample generation method described above.

[0116] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one

[0117] The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0118] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0119] Obviously, the above-described embodiments are only examples for clarity of description and are not limiting on the implementation. For people of ordinary skill in the art, other different forms and changes can be made on the basis of the above description. Here, it is not necessary and impossible to enumerate all the implementations. The obvious changes or changes derived therefrom are still within the protection scope of the present application.

Claims

1. A model-independent method for generating difficult samples, characterized in that, include: Step S1: Obtain the real dataset corresponding to the target segmentation task. The real dataset includes real images and segmentation masks corresponding to each real image. Step S2: Use a pre-trained feature extraction backbone network to extract features from all real images in the real dataset, obtain feature vectors for each real image, calculate the mean of the feature vectors, and construct the initial real data prototype. Step S3: Load the pre-trained conditional latent diffusion model, inject the segmentation mask as a conditional input into the conditional latent diffusion model, and fine-tune the conditional latent diffusion model by minimizing the set conditional denoising score matching loss to obtain a fine-tuned diffusion model adapted to the target segmentation task. Step S4: Generate an image by sampling based on the fine-tuned diffusion model. During the sampling process, perform the following operations: Step S41: Based on the currently predicted clean image and the set conditional mask during the sampling process, crop the target region image block in the clean image; Step S42: Use the feature extraction backbone network to extract the feature vector of the target region image patch, calculate the distance between the feature vector and the real data prototype, maximize the distance by setting the diversity loss, and generate hard samples; Step S4 also includes a multi-objective loss joint optimization step: constructing the total loss function. The total loss function The conditional denoising score matching loss With the aforementioned diversity loss The weighted sum, i.e. ,in As preset weighting coefficients, the Experiments were conducted to determine a balance between realism and diversity in the generated images; Step S5: Merge the hard samples generated in step S4 with the real dataset, update the real data prototype, and return to step S4 to repeat the preset number of rounds to obtain multiple rounds of hard samples; the update process of the real data prototype is as follows: Let the first... During each iteration, the real dataset is merged. Compared to the past The set of difficult samples generated in each round Result in the merged dataset The feature extraction backbone network is used to extract feature vectors from all samples in the merged dataset, and the mean of the feature vectors is calculated to obtain the first... The real data prototype of the wheel : ； in, To merge samples in the dataset, For the sample eigenvectors, Indicates the number of samples in the set; Step S6: Mix the multi-round difficult samples with the real dataset to obtain an augmented training set for training the downstream segmentation model.

2. The model-independent hard sample generation method according to claim 1, characterized in that, The real dataset in step S1 is a medical image segmentation dataset, which includes at least one of the ETIS dataset, CVC-ClinicDB dataset, CVC-ColonDB dataset, EndoScene dataset, and Kvasir dataset. The target segmentation task is the segmentation task of lesion regions in medical images.

3. The model-independent hard sample generation method according to claim 1, characterized in that, The pre-trained feature extraction backbone network in step S2 is a Vision Transformer network or a ResNet network, and the parameters of the feature extraction backbone network remain fixed during the feature extraction process, and are only used to perform forward propagation to output feature vectors.

4. The model-independent hard sample generation method according to claim 1, characterized in that, The conditional denoising score matching loss The calculation formula is: ； in, For real images, for The image after adding noise at different times. For conditional mask, This is real noise. For the noise predicted by the model, For mathematical expectation, It follows a standard normal distribution. It is the identity matrix. The square of the L2 norm; The loss of diversity The calculation formula is: ； in, As a preset marginal value, Based on real data prototypes, For the feature vector of the target region image patch, For the target region image patch, This indicates the calculation of Euclidean distance.

5. The model-independent hard sample generation method according to claim 1, characterized in that, The currently predicted clean image in step S41 Calculated using the following formula: ； in, During the sampling process Noisy images at different times, for The diffusion coefficient at time t. To fine-tune the diffusion model in Time based on the noisy image and condition mask Predicted noise.

6. The model-independent hard sample generation method according to claim 1, characterized in that, In step S3, the pre-trained conditional latent diffusion model is fine-tuned using a parameter-efficient fine-tuning method. This parameter-efficient fine-tuning method is a low-rank adaptive fine-tuning method, which only updates the set parameters in the conditional latent diffusion model to adapt to the target segmentation task.

7. The model-independent hard sample generation method according to claim 1, characterized in that, The difficult samples generated in step S4 also include corresponding difficult sample segmentation masks. The difficult sample segmentation masks are obtained by calculating the pixel-level differences between the generated difficult samples and the clean image. They are used to form image-mask pairs with the difficult samples and are directly integrated into the amplified training set.

8. A model-independent hard sample generation system, characterized in that, The system is used to implement the model-independent hard sample generation method according to any one of claims 1 to 7, comprising: The data preparation module is used to obtain the real dataset corresponding to the target segmentation task. The real dataset includes real images and segmentation masks corresponding to each real image. The prototype construction module is used to extract features from all real images in the real dataset using a pre-trained feature extraction backbone network, obtain feature vectors for each real image, calculate the mean of the feature vectors, and construct an initial real data prototype. The model fine-tuning module is used to load a pre-trained conditional latent diffusion model, inject the segmentation mask as a conditional input into the conditional latent diffusion model, and fine-tune the conditional latent diffusion model by minimizing the set conditional denoising score matching loss to obtain a fine-tuned diffusion model adapted to the target segmentation task. The sample generation module is used to sample and generate images based on the fine-tuned diffusion model. The sample generation module includes a target cropping unit and a loss constraint unit. The target cropping unit is used to crop the target region image patch in the clean image according to the currently predicted clean image and the set condition mask during the sampling process. The loss constraint unit is used to extract the feature vector of the target region image patch using the feature extraction backbone network, calculate the distance between the feature vector and the real data prototype, and maximize the distance through the set diversity loss to generate hard samples. The iterative optimization module is used to merge the difficult samples generated by the sample generation module with the real dataset, update the real data prototype, and return to the sample generation module to execute a preset number of rounds to obtain multiple rounds of difficult samples. The sample application module is used to mix the multi-round difficult samples with the real dataset to obtain an augmented training set for training the downstream segmentation model.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions for causing a computer device to execute the model-independent hard sample generation method according to any one of claims 1 to 7.

Images