Pathological image data amplification method and system based on prompt sampling

Through the pathological image data amplification method based on prompt sampling, the loss function is constructed using the Gaussian distribution estimator and amplification mask, efficient characterization learning at the image block level and controllable amplification of full-slice data, solving the existing technology The problem of insufficient data amplification control in the medium improves the performance of full slice classification.

CN119091247APending Publication Date: 2024-12-06BEIHANG UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411105697.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In full-slice data amplification, existing methods are difficult to achieve efficient characterization learning and data amplification at the image block level, and the generated enhancement results lack control, affecting model performance.

Method used

Using the pathological image data amplification method based on prompt sampling, the initial view of the image block, the student view and the teacher view are set, the Gaussian distribution estimator is constructed, and the final loss function is constructed using the amplification mask and probability distribution, and the teacher encoder is constructed. Training with the Gaussian distribution estimator to generate a controllable set of amplified features.

Benefits of technology

This improves the image block characterization quality and full-slice classification performance, realizes controllable full-slice data amplification, and improves the generalization ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091247A_ABST
    Figure CN119091247A_ABST
Patent Text Reader

Abstract

The invention discloses a pathological image data amplification method based on prompt sampling. The method comprises the following steps: acquiring a pathological full-slice image block, and preprocessing to obtain an initial view; setting an amplification operation set to pre-process the initial view to obtain a student view and a teacher view; obtaining student probability distribution based on the student view; inputting the teacher view into a teacher encoder to obtain teacher features; obtaining teacher probability distribution based on the teacher features; constructing an estimator; after the initial view is processed, an amplification mask is introduced and input to an estimator to obtain sampling probability distribution; constructing a final loss function based on the student probability distribution, the teacher probability distribution and the sampling probability distribution, and training a teacher encoder, a Gaussian distribution estimator and an amplification mask; inputting all the image blocks into a trained teacher encoder to obtain an image feature set; and inputting the image feature set into the trained Gaussian distribution estimator, and obtaining an amplified feature set based on the trained amplification mask. And efficient amplification of full-slice data is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a method and system for amplifying pathological image data based on prompt sampling. Background Art

[0002] Data augmentation, as an effective strategy to expand data for training deep models, is also important in the field of whole-slice analysis of histopathology. In the field of natural image analysis, emerging methods for feature space data augmentation have been proposed. Although these methods have potential, they are difficult to apply when analyzing billion-pixel images such as whole slices. In order to achieve data augmentation for whole-slice images, some studies have proposed using generative methods or Mixup techniques to create data augmentation in feature space. However, these methods have the problem of introducing additional computational costs by training another parameterized model, or causing loss of semantic information; in addition, the enhanced results generated by these methods usually lack control, and the changes are more visually intuitive and easier to manage than data augmentation applied directly in image space.

[0003] To address the problem of full-slice data augmentation, ReMix mixes category-specific prototypes determined by K-means clustering. Although it can enhance data diversity, it may sometimes generate representations that deviate from the true data distribution, which may weaken the performance of the model on actual data sets. DAGAN creates synthetic data augmentation in the feature space by training models similar to generative adversarial networks. However, these generative models require an additional training phase independent of the representation model and require additional computing resources in the inference phase. At the same time, these methods cannot control the augmentation, resulting in insufficient amplification flexibility.

[0004] Therefore, how to achieve image block-level representation learning and efficient amplification of full-slice data is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a pathological image data amplification method and system based on prompt sampling, which realizes image block-level representation learning and efficient amplification of full-slice data, and helps to improve the full-slice classification performance.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for amplifying pathological image data based on prompt sampling, comprising:

[0008] Obtaining image blocks of the pathological full-slice image and preprocessing them to obtain an initial view;

[0009] Setting an augmentation operation set and preprocessing the initial view to obtain a student view and a teacher view;

[0010] Obtaining a student embedding probability distribution based on the student view;

[0011] Input the teacher view to a teacher encoder to obtain teacher features;

[0012] Obtaining a teacher embedding probability distribution based on the teacher characteristics;

[0013] Constructing a Gaussian distribution estimator; introducing an augmentation mask after the initial view processing and inputting it into the Gaussian distribution estimator to obtain a sampling probability distribution;

[0014] Constructing a final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution;

[0015] Training the teacher encoder, the Gaussian distribution estimator, and the augmentation mask based on the final loss function;

[0016] Inputting all the image blocks into a trained teacher encoder to obtain an image feature set;

[0017] The image feature set is input into a trained Gaussian distribution estimator and based on the trained augmentation mask, a corresponding representation distribution is obtained; and an augmented feature set is obtained based on the representation distribution.

[0018] Preferably, the pretreatment specifically comprises: performing a pre-amplification operation;

[0019] The pre-amplification operation includes: flipping and color perturbation.

[0020] Preferably, obtaining the student view and the teacher view specifically includes:

[0021] Perform random sampling and combination based on the augmented data set to obtain student augmentation prompts and teacher augmentation prompts;

[0022] Performing an expansion operation on the initial view based on the student expansion prompt to obtain the student view;

[0023] The initial view is augmented based on the teacher augmentation prompt to obtain the teacher view.

[0024] Preferably, the student embedding probability distribution is obtained by:

[0025] Based on the student view, the student view is input to a first student encoder to obtain student features;

[0026] The student features are input into the first student projection head to obtain the student embedding probability distribution.

[0027] Preferably, the sampling probability distribution is obtained by:

[0028] Inputting the initial view to a second student encoder to obtain encoding features;

[0029] Inputting the encoding feature into the Gaussian distribution estimator to obtain an image block mean and an image block standard deviation;

[0030] Introducing the augmentation mask and the teacher augmentation prompt based on the image block standard deviation to obtain a mask standard deviation;

[0031] Obtaining a sampling feature based on the mask standard deviation and the image block mean value;

[0032] The sampling features are input to the second student projection head to obtain the sampling probability distribution.

[0033] Preferably, constructing the final loss function specifically includes:

[0034] Constructing a minimization loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution

[0035]

[0036] Based on the minimization loss function Construct the final loss function

[0037]

[0038] Among them, y t represents the teacher embedding probability distribution, y s represents the student embedding probability distribution, y v represents the sampling probability distribution, represents the KL divergence constraint, represents the sparse loss, represents the variance regularization term, and β1, β2, and β3 represent the weight coefficients for controlling the corresponding term losses.

[0039] Preferably, the characterization distribution acquisition method is:

[0040] Randomly initialize an intermediate augmentation hint based on each image feature in the image feature set;

[0041] Obtaining a corresponding intermediate mask in the trained augmentation mask based on the intermediate augmentation hint;

[0042] The image features are input into the trained Gaussian distribution estimator and feature mean and feature standard deviation are obtained based on the intermediate mask;

[0043] The representation distribution corresponding to all the image features is obtained based on the feature mean and the feature standard deviation.

[0044] Preferably, the amplified feature set is obtained, specifically including:

[0045] Based on all the representation distributions, virtual image block features are randomly sampled to replace corresponding features in the original set to obtain the expanded feature set.

[0046] In a second aspect, the present invention provides a pathological image data augmentation system based on prompt sampling, comprising: an image processing module, a first probability distribution acquisition module, a second probability distribution acquisition module, a loss function construction module, a training module and a data augmentation module;

[0047] The image processing module is used to obtain image blocks of the pathological full-slice image and perform preprocessing to obtain an initial view; set an augmentation operation set and perform preprocessing on the initial view to obtain a student view and a teacher view;

[0048] The first probability distribution acquisition module is used to obtain the student embedding probability distribution based on the student view; input the teacher view into the teacher encoder to obtain the teacher feature; and obtain the teacher embedding probability distribution based on the teacher feature;

[0049] The second probability distribution acquisition module is used to construct a Gaussian distribution estimator; after the initial view processing, an augmentation mask is introduced and input into the Gaussian distribution estimator to obtain a sampling probability distribution;

[0050] The loss function construction module is used to construct a final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution;

[0051] The training module is used to train the teacher encoder, the Gaussian distribution estimator and the augmentation mask based on the final loss function;

[0052] The data augmentation module is used to input all the image blocks into a trained teacher encoder to obtain an image feature set; input the image feature set into a trained Gaussian distribution estimator and obtain a corresponding representation distribution based on a trained augmentation mask; and obtain an augmented feature set based on the representation distribution.

[0053] In a third aspect, the present invention provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0054] Memory, used to store computer programs;

[0055] The processor, when used to execute the program stored in the memory, can implement the pathological image data amplification method based on prompt sampling as described in the first aspect.

[0056] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a pathological image data amplification method and system based on prompt sampling, which effectively improves the image block representation quality by introducing amplification prompts. At the same time, the representation amplification can be controlled based on the trained distribution estimator and amplification mask to achieve controllable full-slice amplification, which helps to improve the generalization ability of the full-slice classifier. In the full-slice data amplification, prompts are used to control the amplification, and the full-slice level data amplification is achieved by feature sampling of the typing after mask restriction. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A flow chart of a pathological image data amplification method based on prompt sampling provided by the present invention.

[0059] Figure 2 This is a flow chart of the first stage of the method provided by the present invention.

[0060] Figure 3 This is a flowchart of the prompt image amplification provided by the present invention.

[0061] Figure 4 A flowchart of amplification provided for the present invention is provided.

[0062] Figure 5 This is a flow chart of the second stage of the method provided by the present invention.

[0063] Figure 6 This is a flow chart of the characterization distribution acquisition method provided by the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Example 1

[0066] like Figure 1 As shown, an embodiment of the present invention discloses a pathological image data amplification method based on prompt sampling, comprising:

[0067] Obtaining image blocks of the pathological full-slice image and preprocessing them to obtain an initial view;

[0068] Set an augmentation operation set and preprocess the initial view to obtain a student view and a teacher view;

[0069] Get the student embedding probability distribution based on the student view;

[0070] Based on the teacher view, input it into the teacher encoder to obtain the teacher features;

[0071] Get the teacher embedding probability distribution based on teacher characteristics;

[0072] Construct a Gaussian distribution estimator; after the initial view processing, introduce the augmentation mask input to the Gaussian distribution estimator to obtain the sampling probability distribution;

[0073] Construct the final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution;

[0074] Train the teacher encoder, Gaussian distribution estimator, and augmentation mask based on the final loss function;

[0075] Input all image blocks into the trained teacher encoder to obtain the image feature set;

[0076] The image feature set is input into the trained Gaussian distribution estimator and based on the trained augmentation mask, the corresponding representation distribution is obtained; and the augmented feature set is obtained based on the representation distribution.

[0077] Example 2

[0078] The embodiment of the present invention discloses a method for amplifying pathological image data based on prompt sampling, comprising:

[0079] like Figure 2 As shown, the first stage: prompt representation distribution learning:

[0080] Obtain the image blocks of the pathological full-slice image and preprocess them to obtain the initial view

[0081] Preferably, the pretreatment specifically comprises: performing a pre-amplification operation;

[0082] Pre-amplification operations include: inversion and color perturbation.

[0083] Set the augmentation operation set and preprocess the initial view to obtain the student view and the teacher view.

[0084] Preferably, Figure 3 As shown, the student view and teacher view are obtained, including:

[0085] Based on augmented data set Perform random sampling and combination to obtain student amplification prompts p s and teacher amplification tips t ;

[0086] Based on student amplification tips p s Initial view Perform an expansion operation to obtain the student view v s ;

[0087] Based on teacher amplification tips p t Initial view Perform the expansion operation to obtain the teacher view v t .

[0088] Preferably, the expansion operation set By K augmentation operations o1,…,o K The composition,augmentation operations include cropping, flipping, color perturbation, grayscale transformation, Gaussian blur, and exposure.

[0089] Preferably, students amplify the prompts p s ∈{0,1} K , Teacher Amplification Tips t ∈{0,1} K The value of the i-th bit in each amplification hint is 1, indicating that the i-th amplification operation is in an activated state; each amplification operation corresponds to a mask m in the amplification mask M, activation means using this m, and inactivation means not using it.

[0090] Get the student embedding probability distribution based on the student views.

[0091] Preferably, the student embedding probability distribution is obtained by:

[0092] Based on student view s Input to the first student encoder Get student characteristics

[0093] The student characteristics z s Input to the first student projection head Convert to embedding and pass the softmax function to get the student embedding probability distribution y s :

[0094]

[0095] Among them, τ s is the temperature coefficient that controls the sharpness of the Student probability distribution.

[0096] Based on the teacher's view t Input to teacher encoder Get teacher characteristics

[0097] Based on teacher characteristics t Get the teacher embedding probability distribution y t .

[0098] Preferably, the teacher feature z t Input to teacher projection head Convert to embedding and pass the softmax function to get the student embedding probability distribution y t :

[0099]

[0100] Among them, τ t is the temperature coefficient that controls the sharpness of the teacher probability distribution.

[0101] Construct a Gaussian distribution estimator; after the initial view processing, introduce the augmentation mask input to the Gaussian distribution estimator to obtain the sampling probability distribution.

[0102] Preferably, a Gaussian distribution estimator is used to predict the distribution of the image patch representation, which includes a mean head h μ and variance head h σ , mean head h μ and variance head h σ They are all composed of fully connected layers.

[0103] Preferably, Figure 4 As shown, the sampling probability distribution is obtained by:

[0104] Based on the initial view Input to the second student encoder Get the encoding features

[0105] The encoded feature z is fed into the mean head h of the Gaussian distribution estimator μ and variance head hσ , we get the image block mean μ and image block standard deviation σ:

[0106] μ=h μ (z)

[0107] σ=exp(h σ (z) / 2)

[0108] Introducing augmentation mask M and teacher augmentation hint p based on image patch standard deviation t , by introducing teacher amplification prompts t , we can restrict the distribution to a more explicit augmentation space and obtain the mask standard deviation

[0109]

[0110] φ(p t )=p t M / |p t |1

[0111] Where ⊙ represents the Hadamard product, φ(p t ) indicates that it corresponds to p t The mask of |p t |1 means p t L1 regularization.

[0112] Based on mask standard deviation and the image block mean μ, and use the reparameterization technique to achieve back propagation during training. The representation is sampled under the assumption of representation independence to obtain the sampled feature z v :

[0113]

[0114] Among them, ∈ represents the features sampled from the standard Gaussian distribution, represents Gaussian distribution, I D Represents the D-dimensional identity matrix.

[0115] Based on the sampling feature z v Input to second student projection head Convert to embedding and pass through the softmax function to get the sampling probability distribution y v :

[0116]

[0117] Among them, τ v Represents the temperature coefficient that controls the sharpness of the sampled probability distribution.

[0118] Preferably, the image block mean μ and the image block standard deviation σ together constitute a Gaussian distribution

[0119] Preferably, in order to enable the representation obtained by distribution sampling to correspond to different amplification operations, a series of amplification masks M=[m1, m2, ..., m K ] T , each mask corresponds to a set of augmentation operations An amplification operation in .

[0120] The final loss function is constructed based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution:

[0121] Preferably, constructing the final loss function specifically includes:

[0122] Based on the student embedding probability distribution y s , teacher embedding probability distribution y t and the sampling probability distribution y v Constructing a function to minimize the loss

[0123]

[0124] Based on minimizing the loss function Constructing the final loss function

[0125]

[0126] Among them, y t represents the teacher embedding probability distribution, y s represents the student embedding probability distribution, y v represents the sampling probability distribution, represents the KL divergence constraint, represents the sparse loss, represents the variance regularization term, and β1, β2, and β3 represent the weight coefficients for controlling the corresponding term losses.

[0127] Preferably, since the present invention models the representation distribution based on Gaussian prior, a Kullback-Leibler (KL) divergence constraint is added to the distribution estimation. It is expressed as:

[0128]

[0129] Among them, D KL represents the KL divergence.

[0130] Preferably, the hint mask aims to identify specific feature dimensions associated with different augmentations, and L1 regularization is used to guide the sparsity of the augmentation mask, adding a sparse loss

[0131]

[0132] Among them, ||φ(p t )||1 means φ(p t ) with L1 regularization.

[0133] Preferably, the present invention introduces a variance regularization term into the representation standard deviation in the feature dimension to alleviate the problem that the augmentation mask tends to zero. The variance regularization term for:

[0134]

[0135] Here, γ is a small scalar used to prevent numerical instability.

[0136] like Figure 5 As shown, the second stage: full-slice level data amplification:

[0137] Based on the final loss function, the teacher encoder Gaussian distribution estimator and augmentation mask are trained to obtain the trained teacher encoder Trained Gaussian distribution estimator and trained augmentation mask M'.

[0138] Input all image patches to the trained teacher encoder Get the image feature set Z.

[0139] Preferably, all image blocks of the pathological full-slice image are input into the trained teacher encoder Encoding is performed to obtain multiple image features z, which constitute the image feature set Z.

[0140] The image feature set Z is input into the trained Gaussian distribution estimator and based on the trained augmentation mask, the corresponding representation distribution is obtained.

[0141] Preferably, the characterization distribution acquisition method is:

[0142] Based on each image feature z in the image feature set, an intermediate augmentation hint p∈{0, 1} is randomly initialized K ;

[0143] Based on the intermediate augmentation hint p, the corresponding intermediate mask φ(p) is obtained in the trained augmentation mask M':

[0144] φ(p)=pM′ / |p|1

[0145] The image feature z is input into the trained Gaussian distribution estimator and the feature mean μ′ and feature standard deviation σ′ are obtained based on the intermediate mask:

[0146] μ′=hμ(z)

[0147] σ′=exp(h σ (z) / 2)⊙φ(p)

[0148] Based on the feature mean μ′ and feature standard deviation σ′, the representation distribution corresponding to all image features is obtained.

[0149] The expanded feature set is obtained based on the representation distribution.

[0150] Preferably, the amplified feature set specifically includes:

[0151] Based on all representation distributions, randomly sample virtual image block features to replace the corresponding features in the original set to obtain the expanded feature set Z v .

[0152] Preferably, all image features in the feature set Z are processed to obtain multiple amplified image features, thereby achieving full-slice-level data amplification:

[0153]

[0154] Among them, Z v represents the augmented feature set, represents the virtual feature of the i-th image block, represents the feature distribution of the i-th image block predicted by the augmentation mask M corresponding to the random augmentation hint, and ~ represents sampling from the distribution.

[0155] Example 3

[0156] The present invention proposes a pathological image data augmentation method based on hint sampling, which can effectively improve the representation quality of the image block level, and improve the generalization performance of the whole slice classifier through controllable whole slice augmentation; this method uses hint-controlled augmentation in whole slice data augmentation for the first time, and realizes whole slice level data augmentation by feature sampling of the classification after mask restriction. Table 1 shows the effect of the method of the present invention:

[0157] Table 1 Whole-slice classification results

[0158]

[0159] Among them, Micro-AUC represents the micro-average of the ROC curve area, Macro-F1 represents the macro-average of the F1 score, and ACC represents the accuracy.

[0160] This method was compared with DINO for representation learning and ReMix and DAGAN for whole-slice amplification. The results showed that this method can improve the representation quality and whole-slice classification performance.

[0161] Example 4

[0162] like Figure 6 As shown, a pathological image data augmentation system based on prompt sampling includes: an image processing module, a first probability distribution acquisition module, a second probability distribution acquisition module, a loss function construction module, a training module and a data augmentation module;

[0163] An image processing module is used to obtain image blocks of the pathological full-slice image and perform preprocessing to obtain an initial view; set an augmentation operation set and perform preprocessing on the initial view to obtain a student view and a teacher view;

[0164] The first probability distribution acquisition module is used to obtain the student embedding probability distribution based on the student view; input to the teacher encoder based on the teacher view to obtain the teacher feature; and obtain the teacher embedding probability distribution based on the teacher feature;

[0165] The second probability distribution acquisition module is used to construct a Gaussian distribution estimator; after the initial view processing, the augmentation mask is introduced into the Gaussian distribution estimator to obtain a sampling probability distribution;

[0166] The loss function construction module is used to construct the final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution;

[0167] A training module for training the teacher encoder, Gaussian distribution estimator, and augmentation mask based on the final loss function;

[0168] The data augmentation module is used to input all image blocks into the trained teacher encoder to obtain an image feature set; input the image feature set into the trained Gaussian distribution estimator and obtain the corresponding representation distribution based on the trained augmentation mask; and obtain the augmented feature set based on the representation distribution.

[0169] Example 5

[0170] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0171] Memory, used to store computer programs;

[0172] The processor, when used to execute the program stored in the memory, can implement a pathological image data amplification method based on prompt sampling as in Embodiment 1 or 2.

[0173] The electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute a pathological image data amplification method based on prompt sampling in embodiment 1 or 2:

[0174] Obtaining image blocks of the pathological full-slice image and preprocessing them to obtain an initial view;

[0175] Set an augmentation operation set and preprocess the initial view to obtain a student view and a teacher view;

[0176] Get the student embedding probability distribution based on the student view;

[0177] Based on the teacher view, input it into the teacher encoder to obtain the teacher features;

[0178] Get the teacher embedding probability distribution based on teacher characteristics;

[0179] Construct a Gaussian distribution estimator; after the initial view processing, introduce the augmentation mask input to the Gaussian distribution estimator to obtain the sampling probability distribution;

[0180] Construct the final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution;

[0181] Train the teacher encoder, Gaussian distribution estimator, and augmentation mask based on the final loss function;

[0182] Input all image blocks into the trained teacher encoder to obtain the image feature set;

[0183] The image feature set is input into the trained Gaussian distribution estimator and based on the trained augmentation mask, the corresponding representation distribution is obtained; and the augmented feature set is obtained based on the representation distribution.

[0184] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0185] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a pathological image data amplification method and system based on prompt sampling, which effectively improves the image block representation quality by introducing amplification prompts. At the same time, the representation amplification can be controlled based on the trained distribution estimator and amplification mask to achieve controllable full-slice amplification, which helps to improve the generalization ability of the full-slice classifier. In the full-slice data amplification, prompts are used to control the amplification, and the full-slice level data amplification is achieved by feature sampling of the typing after mask restriction.

[0186] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0187] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pathological image data augmentation method based on prompt sampling, characterized in that: include: Obtaining image blocks of the pathological full-slice image and preprocessing them to obtain an initial view; Setting an augmentation operation set and preprocessing the initial view to obtain a student view and a teacher view; Obtaining a student embedding probability distribution based on the student view; Input the teacher view to a teacher encoder to obtain teacher features; Obtaining a teacher embedding probability distribution based on the teacher characteristics; Constructing a Gaussian distribution estimator; introducing an augmentation mask after the initial view processing and inputting it into the Gaussian distribution estimator to obtain a sampling probability distribution; Constructing a final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution; Training the teacher encoder, the Gaussian distribution estimator, and the augmentation mask based on the final loss function; Inputting all the image blocks into a trained teacher encoder to obtain an image feature set; The image feature set is input into a trained Gaussian distribution estimator and based on the trained augmentation mask, a corresponding representation distribution is obtained; and an augmented feature set is obtained based on the representation distribution.

2. A pathological image data augmentation method based on prompt sampling according to claim 1, characterized in that: The pretreatment specifically includes: performing a pre-amplification operation; The pre-amplification operation includes: flipping and color perturbation.

3. The method for pathological image data amplification based on prompt sampling according to claim 1, characterized in that: Get the student view and teacher view, including: Perform random sampling and combination based on the augmented data set to obtain student augmentation prompts and teacher augmentation prompts; Performing an expansion operation on the initial view based on the student expansion prompt to obtain the student view; The initial view is augmented based on the teacher augmentation prompt to obtain the teacher view.

4. The method for pathological image data amplification based on prompt sampling according to claim 1, characterized in that: The student embedding probability distribution is obtained by: Based on the student view, the student view is input to a first student encoder to obtain student features; The student features are input into the first student projection head to obtain the student embedding probability distribution.

5. The method for pathological image data amplification based on prompt sampling according to claim 3, characterized in that: The sampling probability distribution is obtained by: Inputting the initial view to a second student encoder to obtain encoding features; Inputting the encoding feature into the Gaussian distribution estimator to obtain an image block mean and an image block standard deviation; Introducing the augmentation mask and the teacher augmentation prompt based on the image block standard deviation to obtain a mask standard deviation; Obtaining a sampling feature based on the mask standard deviation and the image block mean value; The sampling features are input to the second student projection head to obtain the sampling probability distribution.

6. The method for pathological image data augmentation based on prompt sampling according to claim 1, characterized in that: Construct the final loss function, including: Constructing a minimization loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution Based on the minimization loss function Construct the final loss function Among them, y t represents the teacher embedding probability distribution, y s represents the student embedding probability distribution, y v represents the sampling probability distribution, represents the KL divergence constraint, represents the sparse loss, represents the variance regularization term, and β1, β2, and β3 represent the weight coefficients that control the corresponding term losses.

7. The method for pathological image data augmentation based on prompt sampling according to claim 1, characterized in that: The method for obtaining the representation distribution is: Randomly initialize an intermediate augmentation hint based on each image feature in the image feature set; Obtaining a corresponding intermediate mask in the trained augmentation mask based on the intermediate augmentation hint; The image features are input into the trained Gaussian distribution estimator and feature mean and feature standard deviation are obtained based on the intermediate mask; The representation distribution corresponding to all the image features is obtained based on the feature mean and the feature standard deviation.

8. The method for pathological image data augmentation based on prompt sampling according to claim 1, characterized in that: The amplified feature set is obtained, including: Based on all the representation distributions, virtual image block features are randomly sampled to replace corresponding features in the original set to obtain the expanded feature set.

9. A pathological image data augmentation system based on prompt sampling, characterized in that: include: An image processing module, a first probability distribution acquisition module, a second probability distribution acquisition module, a loss function construction module, a training module and a data augmentation module; The image processing module is used to obtain image blocks of the pathological full-slice image and perform preprocessing to obtain an initial view; Setting an augmentation operation set and preprocessing the initial view to obtain a student view and a teacher view; The first probability distribution acquisition module is used to obtain the student embedding probability distribution based on the student view; input into the teacher encoder based on the teacher view to obtain the teacher feature; Obtaining a teacher embedding probability distribution based on the teacher characteristics; The second probability distribution acquisition module is used to construct a Gaussian distribution estimator; after the initial view processing, an augmentation mask is introduced and input into the Gaussian distribution estimator to obtain a sampling probability distribution; The loss function construction module is used to construct a final loss function based on the student embedding probability distribution, the teacher embedding probability distribution and the sampling probability distribution; The training module is used to train the teacher encoder, the Gaussian distribution estimator and the augmentation mask based on the final loss function; The data augmentation module is used to input all the image blocks into a trained teacher encoder to obtain an image feature set; input the image feature set into a trained Gaussian distribution estimator and obtain a corresponding representation distribution based on a trained augmentation mask; and obtain an augmented feature set based on the representation distribution.

10. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor, when used to execute the program stored in the memory, can implement a pathological image data amplification method based on prompt sampling as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Pathological image representation learning method based on feature sampling, medium and equipment

    CN117557855A

  • Pathological full-slice feature learning method based on self-supervised learning

    CN117557856A

  • Self-Supervised Machine Learning for Medical Image Analysis

    US20230260652A1