A generative model based visual image data augmentation and synthesis system

Abstract

The application discloses a kind of visual image data enhancement and synthesis system and method based on generative model.It relates to the field of visual image technology, the system includes: a generation module, for generating synthetic data pair;A task utility evaluation module is used to calculate a dynamic task utility loss directly reflecting the performance of downstream task;And a style consistency evaluation module is used to calculate a label style consistency loss reflecting the difference between generated label and standard style.The generation module is trained according to the composite loss function composed of the above two losses.The application also discloses a kind of dynamic guiding mechanism based on course learning, which can dynamically adjust the training strategy according to the learning state of downstream model.The application can generate more effective, more stable and reliable synthetic visual data for downstream tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual image, in particular to a visual image data enhancement and synthesis system based on a generative model. BACKGROUND

[0002] In the key fields of medical images, autonomous driving, the performance of deep learning models is highly dependent on large-scale and high-quality labeled data, and the acquisition of such data is costly and limited. Therefore, the field has extensively studied the use of generative models such as generative adversarial networks to synthesize new data to expand the training set. This conventional method strives to improve the visual fidelity of generated images through the adversarial game between the generator and the discriminator. However, the existing technology has exposed several deep technical limitations in practice. First, the core of its training target is "visual fidelity", but this is fundamentally different from the "task utility" pursued by downstream specific tasks such as lesion segmentation or object detection, resulting in low efficiency of synthetic data enhancement. Second, existing methods indiscriminately learn and amplify the "style inconsistency" in the data set due to subjective differences in manual labeling, resulting in inconsistent and blurred generated labels, introducing noise, and limiting the upper limit of the accuracy of downstream models. Therefore, how to make synthetic data shift from pursuing "good-looking" to pursuing "good use", while ensuring the "high quality" of the label and realizing the "dynamic" of the training process, is a technical problem that needs to be solved in the field. SUMMARY

[0003] The present application provides a visual image data enhancement and synthesis system based on a generative model to solve the technical problems of low task utility of synthetic data and unstable quality of generated labels in the prior art.

[0004] In view of the above problems, the present application provides a visual image data enhancement and synthesis system based on a generative model, comprising:

[0005] A generation module for generating a synthetic data pair comprising a synthetic image and a synthetic label;

[0006] A task utility evaluation module for inputting the synthetic image into a downstream task agent model to obtain a predicted label, and calculating a dynamic task utility loss based on the predicted label and the synthetic label;

[0007] A style consistency evaluation module for calculating a label style consistency loss based on the synthetic label and a pre-calculated label style prototype;

[0008] Wherein, the generation module is configured to update its own network parameters according to a composite loss function composed of the dynamic task utility loss and the label style consistency loss.

[0009] The technical scheme provided by the application has at least the following technical effects or advantages:

[0010] The application introduces a task utility loss directly driven by a downstream task agent model, and changes the training target of the generator from traditional visual fidelity to task utility directly related to the final application. This enables the system to tend to generate data most valuable for improving the performance of downstream models, thereby reducing the amount of synthetic data required to enhance the performance of equivalent-scale downstream models and improving the overall efficiency of data enhancement.

[0011] The application introduces a style consistency loss based on a labeled style prototype to explicitly constrain the output of the generator. This mechanism effectively suppresses the learning of inconsistent labeling styles in the source data set due to human subjective differences, making the generated synthetic labels more stable and clear in key details such as edge morphology, reducing noise introduced by labeling quality problems, and helping to improve the precision upper limit of downstream segmentation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A visual image data enhancement and synthesis system architecture based on a generative model. DETAILED DESCRIPTION

[0013] The above technical scheme will be described in detail below in conjunction with the accompanying drawings and specific embodiments to better understand the above technical scheme. Obviously, the described embodiments are only a part of the embodiments of the application, not all embodiments of the application, and it should be understood that the application is not limited to the example embodiments for explaining the application. Based on the embodiments of the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application. In addition, it should be noted that, for convenience of description, only parts related to the application are shown in the drawings, not all.

[0014] Please refer to Figure 1 A visual image data enhancement and synthesis system based on a generative model, comprising:

[0015] A generation module for generating a synthetic data pair comprising a synthetic image and a synthetic label;

[0016] A task utility evaluation module for inputting the synthetic image into a downstream task agent model to obtain a predicted label, and calculating a dynamic task utility loss based on the predicted label and the synthetic label;

[0017] a style consistency evaluation module configured to calculate a label style consistency loss based on a synthetic label and a pre-computed label style prototype;

[0018] wherein the generation module is configured to update its own network parameters according to a composite loss function composed of a dynamic task utility loss and a label style consistency loss.

[0019] Embodiment: Efficient utility synthesis system applied to high-resolution colorectal polyp endoscopic image segmentation data

[0020] This embodiment is deployed in a specific clinical auxiliary diagnosis engineering scenario: using artificial intelligence models to analyze high-definition colonoscopy video streams in real time, automatically detecting and accurately segmenting precancerous lesions of colorectal cancer, polyps, to assist doctors in diagnosis and reduce the rate of missed diagnosis.

[0021] However, when building this system, there are multiple challenges from the data level that are technically interrelated, which existing data enhancement techniques cannot effectively solve:

[0022] 1. Heterogeneity problem of multi-source devices: Medical centers usually use multiple types of endoscopic devices from different manufacturers (e.g., Olympus, Fujifilm, Pentax). Models trained directly using these mixed data will have a sharp performance decline when faced with images generated by specific devices they have not "seen" before.

[0023] 2. High variability of lesion morphology and scarcity of edge cases: Colorectal polyps have various shapes, from easily identifiable protruding types (Yamada I-III types) to flat and depressed types (Yamada IIb type) with blurred boundaries with the surrounding mucosa. The latter is a type that is easily missed in clinical practice. In existing data sets, the number of samples of such edge cases is small, leading to insufficient recognition ability of traditional models. According to baseline tests, a model trained on a regular data set can have a 20% miss rate for flat polyps, which constitutes a performance limitation in clinical application scenarios.

[0024] 3. Subjective inconsistency of manual annotation: Even experienced gastroenterologists have unavoidable minor deviations when pixel-level outlining the boundaries of the same polyp image. This "annotation style" difference between experts is indiscriminately learned by traditional generation models, resulting in unstable and artifact-laden annotation edges of the generated segmentation maps, introducing harmful noise for the training of downstream segmentation models and limiting the upper limit of their accuracy.

[0025] The visual image data augmentation and synthesis system based on a generative model according to the present application can be divided into three logically continuous stages in the life cycle of a complete engineering application:

[0026] Stage 1: This is an offline preparation process, mainly including:

[0027] Data preparation and preprocessing: Collect original endoscopic images and their corresponding expert annotations from multiple sources, perform unified size normalization (e.g., scale to 512x512 pixels), data cleaning, and format conversion.

[0028] Annotation style prototype calculation: Run the "annotation style prototype calculation unit" to statistically analyze all real annotations, generate and persist a standard annotation style prototype P.

[0029] Model loading and configuration: Load the pre-trained and frozen "downstream task agent model B'"; initialize the network parameters of the "generator A'"; load and configure the "curriculum learning scheduling module", set its initial state and related thresholds.

[0030] Stage 2: This stage is an automated, closed-loop iterative process:

[0031] The system starts an unsupervised training loop. In this loop, the "generator A'" continuously generates synthetic "image-annotation pairs" under the dynamic guidance of the "curriculum learning scheduling module".

[0032] Each batch of generated synthetic data is used to calculate "dynamic task utility loss L_task" and "annotation style consistency loss L_style", and the generator's parameters are updated accordingly.

[0033] This process continues until the curriculum learning scheduling module determines that the learning of the downstream task agent model B' has reached saturation, or the pre-set amount of synthetic data is reached.

[0034] Stage 3: This stage focuses on the management and utilization of synthetic data assets:

[0035] All generated "image-annotation pairs" are not discarded at will, but are stored in a structured manner with a set of rich metadata (Metadata) in a database or file system.

[0036] Metadata at least includes: a unique sample ID, a timestamp when the sample is generated, its "curriculum learning phase" (basic construction, hard core mining or leak repair), the calculated L_task and L_style loss values, and the version number of the generator model that generates it.

[0037] This storage mechanism with rich metadata allows subsequent researchers to conduct fine-grained data tracing and analysis. For example, all samples generated during the "hard-core mining period" and with the highest L_task value can be screened out for the purpose of strengthening training or testing of the final deployed segmentation model, thereby maximizing the value of data.

[0038] So far, a macro framework from problem definition, environment configuration to complete life cycle planning has been constructed. The next part goes deep into the internal of each technical component of the framework and makes a detailed description.

[0039] In a specific embodiment of the present application, the generator A' is a deep neural network module, the function of which is defined as: receiving a random noise vector of a predetermined dimension as input, and finally outputting a multi-channel tensor containing pairs of size-matched synthetic images and their pixel-level labels through a series of neural network layers.

[0040] In a preferred embodiment, the internal network structure of the generator A' adopts the U-Net architecture, which is selected for its superior ability to preserve spatial details in medical image segmentation tasks.

[0041] The data structure processed by the generator A' is defined as follows: its input, the random noise vector z, is a two-dimensional tensor of data form (B, 128) sampled from a standard normal distribution, where B represents the batch size and 128 is the preset noise dimension. Its output, the synthetic data pair G_pair, is a four-dimensional tensor of data form (B, 2, 512, 512). In this output tensor, the slice G_pair[:, 0, :, :] along the second dimension (channel dimension) with index 0 is defined as the synthetic endoscopic image G_img, whose pixel values are normalized to the interval [0, 1]; the slice G_pair[:, 1, :, :] with index 1 is defined as the synthetic polyp segmentation label G_mask, which is passed through a Sigmoid activation function at the end of the network, so that each pixel value in the output tensor also falls within the interval [0, 1], which represents the probability that the pixel belongs to the polyp region in the physical sense.

[0042] The downstream task agent model B' is another deep neural network module. Its role is defined as: a stable performance evaluation agent for simulating real downstream segmentation tasks.

[0043] In a preferred embodiment, the internal network structure of the downstream task agent model B' also adopts the U-Net architecture, but its network depth or channel number is reduced compared to the final deployed segmentation model to achieve higher computational efficiency.

[0044] The data structure processed by the downstream task agent model B' is defined as follows: its input is the synthesized image G_img separated from G_pair. Its output is a three-dimensional tensor with a shape of (B, 1, 512, 512), which is defined as the predicted segmentation mask Pred_mask, also processed by the Sigmoid activation function, representing the segmentation prediction result of the input image G_img by the model B'.

[0045] The annotation style prototype module is a functional module responsible for defining and measuring standard annotation styles in a statistical sense. This module logically consists of an annotation style prototype calculation unit and an online constraint unit.

[0046] The core data structure of the annotation style prototype module, namely the annotation style prototype P, is defined as a three-dimensional binary tensor with a shape of (1, 1, 512, 512), whose internal element values are only 0 or 1, representing the "consensus" or "average shape" of all real expert annotations in the data set in the morphological sense.

[0047] The curriculum learning scheduling module is an algorithmic process module, and its function is defined as: by periodically monitoring the learning state of the model B', and according to the pre-set rules, dynamically selecting and applying different training strategies to guide the generator A'.

[0048] The curriculum learning scheduling module manages a key data structure, namely the feature history buffer. This data structure is defined as a first-in, first-out (FIFO) queue, and its capacity is set to store the feature vectors extracted by the synthesized image G_img at the end of the encoder path of the model B' in the last K training batches.

[0049] For example, if K=100, the feature vector dimension output by B' encoder is 512, and the batch size is 4, then the logical form of this buffer is a queue that can accommodate 400 feature vectors with a dimension of 512.

[0050] Further, the offline calculation unit: the generation algorithm of the annotation style prototype P is executed offline once before the system starts training, and its step-by-step logical process is as follows:

[0051] Step 1: Initialization. The system first creates a two-dimensional matrix with the same size as the input annotation image in memory, with a data type of float, and initializes all its elements to zero. This matrix is named the probability map prob_map.

[0052] Second step: Accumulation. The system will iterate through a set of N ground truth expert annotations C_mask. For each annotation image in the set, the system will perform a pixel-wise addition operation between it and the prob_map, and store the result back into prob_map. After this step, each pixel value in prob_map is equal to the sum of all ground truth values at that pixel position.

[0053] Third step: Averaging. The system will divide each pixel value in prob_map by the total number of ground truth annotations N. After this step, each pixel value in prob_map represents the statistical probability of that pixel position being labeled as "polyp" (i.e. value of 1) in all ground truth annotations.

[0054] Fourth step: Thresholding and generation. The system creates a zero matrix of the same size as the prototype P to store the final result. Then, the system iterates through each pixel in prob_map. If a pixel's probability value is greater than a pre-defined threshold (in a preferred embodiment, the threshold is 0.5), then assign a value of 1 to the corresponding position in the prototype P; otherwise, keep it as 0. After this step, the binary matrix is the desired annotation style prototype P.

[0055] Further, the internal logic of a single training iteration: the following is the step-by-step logic flow of the interaction between various algorithmic modules within the system during a training iteration.

[0056] Course learning scheduling module: learning progress monitoring and stage determination, triggered and executed at the beginning of each training round (Epoch).

[0057] First step: The course learning scheduling module first loads the downstream task agent model B' and performs a complete evaluation on an independent, pure real validation set D_val that does not participate in training. Through this evaluation, the Dice Similarity Coefficient of the model B' on the validation set is calculated, and this value is named as the current performance indicator P_current. Then, by subtracting the historical performance indicator P_prev recorded in the last round from P_current, the performance growth rate P_growth_rate is obtained. Here, the current performance indicator P_current and the performance growth rate P_growth_rate are equivalent quantitative indicators that together constitute the comprehensive judgment basis for the "learning state of the downstream task agent model".

[0058] Second step: The course learning scheduling module determines the course stage to be adopted in the current round according to a set of pre-defined rules with clear logical order. The judgment logic is as follows:

[0059] Rule (a): If it is the first training epoch, or, if the value of P_current is less than a preset base performance threshold (e.g. 0.70), the curriculum learning scheduling module sets the current curriculum phase current_phase to be "1", i.e. "base building period strategy".

[0060] Rule (b): If rule (a) is not satisfied, and the value of P_current is greater than or equal to the base performance threshold, and the value of P_growth_rate is greater than a preset stagnation judgment threshold (e.g. 0.001), the curriculum learning scheduling module sets current_phase to be "2", i.e. "hard core mining period strategy".

[0061] Rule (c): If the aforementioned rules are not satisfied, and the value of P_growth_rate is less than or equal to the stagnation judgment threshold, or, if the loss distribution entropy H_loss_prev recorded in the last epoch is less than a preset diversity warning threshold (e.g. 0.2), the curriculum learning scheduling module sets current_phase to be "3", i.e. "leakage checking period".

[0062] In all other cases, the curriculum learning scheduling module sets current_phase to be "2" by default.

[0063] Further, the calculation of the dynamic task utility loss L_task, which is performed in each training batch (Batch).

[0064] Step 1: The system first calculates the Dice similarity coefficient between the predicted segmentation map Pred_mask of model B' and the "pseudo ground truth" label G_mask provided by generator A', then subtracts 1 from the coefficient value to obtain the base Dice loss base_dice_loss.

[0065] Step 2: The system selects one of the following three calculation paths according to the current phase current_phase:

[0066] Path (a) - Stage 1: If it is currently in the "base building period", the calculation method of L_task is as follows: first calculate the Dice similarity coefficient (i.e. 1 - base_dice_loss), then take its natural logarithm, and finally take the reciprocal. This calculation method aims to guide the generator to generate samples that enable B' to obtain a high Dice coefficient.

[0067] Path (b) - Stage 2: If the current is "hard-core digging period", the calculation of L_task is: the value of base_dice_loss is raised to a preset focus parameter γ (for example, 2.0), and then multiplied by base_dice_loss itself. This calculation aims to weight the high-loss, difficult samples.

[0068] Path (c) - Stage 3: If the current is "leakage filling period", the calculation of L_task contains two parts. The first part is the same as the calculation of path (b). The second part is the calculation of the diversity regularization term, which is: first, extract the feature vector of the current synthesized image G_img from the end of the encoder path of model B'; then, calculate the average cosine similarity between the feature vector and all historical feature vectors stored in Feat_History_Buffer, and get a similarity penalty value; finally, weight the penalty value with a preset regularization coefficient β (for example, 0.1). The final value of L_task is the sum of the two parts. At the same time, the system stores the feature vector of the current batch in Feat_History_Buffer.

[0069] Further, the calculation of the annotation style consistency loss L_style, which is also executed in each training batch.

[0070] First step: the system respectively applies a standard Sobel edge detection operator to the segmentation annotation G_mask output by generator A' and the pre-calculated annotation style prototype P. The output of this operation is two images representing the edge gradients of G_mask and P respectively.

[0071] Second step: the system calculates the L1 norm distance between the two edge gradient images, that is, the sum of the absolute values of the differences between the corresponding pixels of the two images. The sum is the annotation style consistency loss L_style.

[0072] On the basis of the foregoing detailed description of the system core components and algorithms, this part will further elaborate the calibration method of key parameters in the present application scheme, design considerations and technical discussions constructed to support dependent claims.

[0073] The calibration method of these key parameters will be described in detail below.

[0074] Calibration method of composite loss function weight coefficients λ1 and λ2:

[0075] Parameter definition: λ1 is the weight coefficient of "dynamic task utility loss L_task", and λ2 is the weight coefficient of "annotation style consistency loss L_style". These two coefficients jointly determine the balance between "pursuing task utility" and "maintaining annotation quality" for the generator A'.

[0076] Calibration methodology:

[0077] Magnitude alignment: At the beginning of training, first run the system independently for several iterations without parameter updates, and only observe and record the average magnitude of the two loss values L_task and L_style. For example, if the magnitude of L_task is around 10 -1 and the magnitude of L_style is around 10 -3 , the weights should be set so that the expected magnitude of λ1 * L_task and λ2 * L_style is roughly equal. This is an initial step to ensure gradient stability.

[0078] Grid search and agent evaluation: Based on magnitude alignment, set a two-dimensional grid search space. For example, the search range of λ1 is [0.5, 1.0, 2.0], and the search range of λ2 is [0.01, 0.05, 0.1]. For each (λ1, λ2) value in the grid, perform a shortened version (e.g., 20% of the total training time) of training.

[0079] Optimal value determination: After each shortened version of training is completed, use the generated synthetic data to enhance the training of an independent evaluation model, and record the performance (such as the Dice coefficient) of the evaluation model on the final test set. Select the (λ1, λ2) combination that can achieve the highest performance of the evaluation model as the optimal parameter configuration for this application scenario.

[0080] Through the above calibration method, in a preferred embodiment, the weight coefficients suitable for the colorectal polyp segmentation scenario of the present application can be set as λ1 = 1.0 and λ2 = 0.05.

[0081] Calibration method of internal parameters of course learning scheduling module:

[0082] Parameter definition: including the focus parameter γ in phase two L_task, the regularization coefficient β and feature history buffer size K in phase three L_task, and the performance threshold and entropy threshold for phase judgment.

[0083] Calibration methodology (taking the focus parameter γ as an example):

[0084] Parameter role understanding: First of all, it should be understood that the larger the γ value, the higher the attention of the system to "hard core" samples, but too large γ may lead to unstable training.

[0085] Single variable sensitivity analysis: Fix all other parameters, select a set of candidate values for γ, e.g. [1.0, 1.5, 2.0, 2.5, 3.0].

[0086] Performance curve observation: For each candidate value, run a complete training process. During the training process, not only focus on the final downstream task performance, but also observe the smoothness of the training loss curve and the diversity of the generated samples.

[0087] Trade-off selection: Select a γ value that can improve the downstream performance while not causing severe fluctuations in the training loss and not leading to a significant decrease in the diversity of the generated samples. This value is the optimal trade-off point. For β, K and other threshold parameters, similar single variable sensitivity analysis methods can be used for calibration.

[0088] According to the above methodology, a suitable focus parameter γ value is about 2.0; an exemplary regularization coefficient β can be set to 0.1; and the feature history buffer size K can be set to store the feature data of the last 100 batches.

[0089] To further clarify the design motivation of the specific technical features in the present application and the technical advantages brought by them, the following two key technical points are described in detail.

[0090] In a preferred embodiment of the present application, the generation method of the labeled style prototype P, i.e. the method of pixel-level probability statistics on all real labels and thresholding of the generated probability map, has the following technical advantages compared to other simplified methods (such as direct arithmetic averaging on binary labeled images). First, the present method has higher robustness to noise and abnormal labels that may exist in the data set. Specifically, the simple arithmetic average method is extremely sensitive to outliers. For example, if there are a few bad labels in the data set due to operation errors or inaccurate expert judgment, which are morphologically strange or positionally offset, these bad labels will have an excessive and disproportionate impact on the average result, resulting in the final prototype P being contaminated. Second, the probability map method used in the present application is essentially a voting mechanism. Each pixel value in prob_map represents the "confidence" that the point is identified as "polyp" by all experts. The pixel points in an isolated and accidental "bad label" have very low weight in the voting process, and their influence will be "overwhelmed" by a large number of normal labels. The thresholding step (e.g. threshold = 0.5) that follows is equivalent to a "majority over minority" decision-making process, which can effectively filter out the influence of these noise and outliers. Therefore, this generation method can ensure that the final generated prototype P has higher statistical robustness and credibility.

[0091] In another preferred embodiment of the present application, the curriculum learning scheduling module adopts a "reverse target guiding" strategy in the initial training stage, i.e., in the basic construction period, the calculation method of the dynamic task utility loss is set to guide the generator to generate samples that can improve the performance of the downstream task agent model. The design consideration of this strategy is to avoid instability in the initial training stage. In the initial stage of training, the output of the generator A' is random in structure and content. If it is forced to generate difficult samples at this time, it may cause gradient information confusion, even cause gradient vanishing or explosion, so that the training cannot converge. The "reverse target guiding" strategy encourages the generator A' to generate simple samples that can be easily and correctly segmented by the model B' by establishing a stable positive feedback loop in the initial stage, which provides a stable and clear gradient signal for the parameter update of A', so that it can quickly learn to generate images with reasonable macrostructure. This strategy reduces the task difficulty in the initial stage, builds a reliable training starting point for the entire system, and lays a solid foundation for challenging difficult samples in the subsequent "hard core mining period".

[0092] After completing the detailed description of the core technical solutions, parameter calibration methods, and key design considerations of the present application, this part will further describe the optional implementation paths and application scope of the solutions.

[0093] Several core functional modules of the present application can be implemented through various specific algorithm paths, and these paths all embody the core technical concept of the present application.

[0094] For example, for the calculation of the "annotation style consistency loss L_style", in addition to the method based on the spatial domain edge L1 distance adopted in the foregoing embodiments, there are other feasible calculation methods. One alternative implementation is to measure the annotation style in the frequency domain. This method can include the following steps: performing two-dimensional fast Fourier transform (FFT) on the generated annotation G_mask and the style prototype P, and calculating the distance (e.g., L2 norm distance) between the amplitude spectra of the two to serve as L_style. Another alternative implementation is to calculate based on the structural similarity index (SSIM) of the perception model, in which case L_style can be defined as 1 - SSIM(G_mask, P).

[0095] For example, for the "learning progress measurement of the curriculum learning scheduling module", in addition to the method of evaluating performance indicators on an independent validation set adopted in the foregoing embodiments, there is also an alternative measurement method based on the internal state of the model. One alternative implementation is to monitor the average gradient norm of the network parameters of the downstream task agent model B' during training. When the moving average of this gradient norm continuously falls below a preset stable state threshold for multiple consecutive training rounds, it can also be used as a basis for triggering the curriculum stage conversion.

[0096] Although the medical image segmentation scene of the colorectal cancer polyp is described in detail in the embodiments of the present application, the core technical solution, i.e., the method of dynamically guiding data synthesis by combining task utility, style consistency and curriculum learning, is not limited to this specific application. The underlying technical logic of the solution has good universality.

[0097] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed in a preferred embodiment, it is not intended to limit the present application, and any person skilled in the art can make some changes or modifications to the above-mentioned methods and technical contents without departing from the scope of the technical solution of the present application. Therefore, any simple modification, equivalent change and modification of the above embodiments made according to the technical essence of the present application, which does not deviate from the content of the technical solution of the present application, still belongs to the protection scope of the technical solution of the present application. The present application is limited by the claims and their entire scope and equivalents.

Claims

1. A generative model based visual image data augmentation and synthesis system, characterized in that, The method comprises the following steps: a generation module is configured to generate a synthetic data pair comprising a synthetic image and a synthetic label; a task utility evaluation module is configured to input the synthetic image into a downstream task agent model to obtain a predicted label, and calculate a dynamic task utility loss based on the predicted label and the synthetic label; a label style prototype calculation unit is configured to perform pixel-level probability statistics on a plurality of real labels in a real label set to generate a probability map, and perform thresholding processing on the probability map to generate a label style prototype for use by the style consistency evaluation module; a style consistency evaluation module is configured to calculate a label style consistency loss based on the synthetic label and the label style prototype; a curriculum learning scheduling module is configured to monitor the learning state of the downstream task agent model, and according to the learning state, control the task utility evaluation module to select one of at least two preset training strategies to determine the specific calculation method of the dynamic task utility loss; wherein the curriculum learning scheduling module is configured to: when the learning state of the downstream task agent model is lower than a basic performance threshold, a basic construction period strategy is adopted, wherein the calculation method of the dynamic task utility loss is set to encourage to reduce the task loss of the downstream task agent model; when the learning state of the downstream task agent model is higher than the basic performance threshold, a hard-core mining period strategy is adopted, wherein the calculation method of the dynamic task utility loss is set to weight the samples that cause the downstream task agent model to produce high task loss; and the generation module is configured to update its own network parameters according to a composite loss function composed of the dynamic task utility loss and the label style consistency loss.

2. The visual image data augmentation and synthesis system based on generative models of claim 1, wherein, The style consistency evaluation module is configured to extract the edges of the synthetic label and the label style prototype respectively, and calculate the distance between the edges of the synthetic label and the edges of the label style prototype as the label style consistency loss.

3. The visual image data augmentation and synthesis system based on generative models of claim 1, wherein, The downstream task agent model included in the task utility evaluation module is a pre-trained segmentation model whose network parameters are frozen during the training process of the generation module.

Images