Image enhancement joint optimization method and system based on downstream task performance

Abstract

The embodiment of the application provides an image enhancement joint optimization method and system based on downstream task performance, and relates to the technical field of image processing. The method first acquires a training data set and task evaluation data, inputs sample images in the training data set into an image enhancement model to obtain enhanced images, and inputs the enhanced images into a task model to obtain task output results. Then, a target training mode is determined according to the task evaluation data, and a parameter update strategy is set based on the target training mode. Then, according to the parameter update strategy, task metric information is calculated according to the task output results and task annotation information, and the model parameters of the trained model are updated according to the task metric information. The method can form an evaluation and closed-loop optimization mode based on the performance of the downstream task as the core constraint, and by introducing the task metric information under controllable conditions, a task-driven optimization strategy and a stabilization mechanism are used in the training stage, so that the robustness and stability of the image processing process are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and system for joint optimization of image enhancement based on downstream task performance. Background Technology

[0002] Image enhancement refers to using algorithms or models to generate image outputs that are more suitable for observation, analysis, or machine processing, addressing degradation factors in the input image. For example, it involves denoising, deblurring, super-resolution reconstruction, dehazing and low-light enhancement, and spectral consistency restoration of multispectral or hyperspectral images to address noise, blur, low resolution, low illumination, haze scattering, compression artifacts, motion jitter, and sensor imaging defects, resulting in enhanced images better suited to downstream task requirements.

[0003] To obtain enhanced images, image enhancement models can be constructed using methods such as paired-data supervised learning, unpaired or weakly supervised learning, and perceptual quality optimization. These models are then integrated with downstream task models to form an image task processing system. Taking paired-data supervised learning as an example, the image enhancement model is trained using paired samples of degraded and reference clear images. During training, training losses such as pixel reconstruction loss, structural similarity loss, and frequency domain or edge consistency loss can be employed to emphasize the proximity of the output to the reference at the pixel or structural level, facilitating convergence and evaluation.

[0004] However, because image enhancement is developed relatively independently from downstream task models as part of the preprocessing workflow, the enhanced image is output after the enhancement model is trained and then input into task models such as detection, segmentation, and recognition. Therefore, pixel similarity metrics do not strictly correspond to task performance. The selection of image enhancement as a preprocessing step lacks reproducible and comparable task-driven evaluation benchmarks, leading to a disconnect between enhancement training objectives and task objectives, making it difficult to form a stable task-driven optimization mechanism. Furthermore, task models are sensitive to changes in degradation distribution; using image enhancement as a black-box preprocessing step lacks a unified task-driven evaluation protocol and a stable task-driven enhancement optimization mechanism, resulting in difficulties in end-to-end collaboration and reduced robustness and stability of the image processing process. Summary of the Invention

[0005] In view of this, embodiments of this application provide a joint optimization method and system for image enhancement based on downstream task performance to solve the problems of low robustness and stability in the image processing process.

[0006] According to a first aspect of this application, a joint optimization method for image enhancement based on downstream task performance is provided, the method comprising:

[0007] Acquire training dataset and task evaluation data. The training dataset includes multiple sample images with task annotation information. The task evaluation data includes at least one of task requirement information, task training progress, and task evaluation results.

[0008] The sample images in the training dataset are input into the image enhancement model to obtain the enhanced image generated by the image enhancement model; and the enhanced image is input into the task model to obtain the task output result generated by the task model.

[0009] The target training mode is determined based on the task evaluation data, and the target training mode is either a first mode or a second mode; wherein, in the first mode, the task model is frozen to train the image enhancement model; in the second mode, the image enhancement model and the task model are alternately frozen according to an alternating cycle to perform alternating joint training on the image enhancement model and the task model;

[0010] A parameter update strategy is set based on the target training mode, and the parameter update strategy includes the calculation method of the trained model and task metric information; the trained model includes at least one of the image enhancement model and the task model;

[0011] According to the parameter update strategy, task metric information is calculated based on the task output and the task annotation information, and the model parameters of the trained model are updated based on the task metric information.

[0012] In some embodiments, obtaining the training dataset and task evaluation data includes:

[0013] Acquire raw images and task-annotated data;

[0014] The task requirement information is extracted from the task evaluation data. The task requirement information includes at least one of the following: preprocessing rules, input resolution processing rules, data partitioning list, random seed, evaluation index, and calculation threshold.

[0015] According to the task requirements information, the original image is annotated using the task annotation data to generate a sample image with task annotation information;

[0016] Generate a training dataset based on the sample images;

[0017] The training dataset is divided into a training set, a validation set, and a test set.

[0018] In some embodiments, determining the target training mode based on the task evaluation data includes:

[0019] Perform quantitative index mapping on the task evaluation data to obtain multiple data quantitative indicators;

[0020] The task evaluation value is calculated based on multiple data quantification indicators, and the task evaluation value is the weighted sum of multiple data quantification indicators;

[0021] Query the target pattern range to which the task evaluation value belongs;

[0022] The training mode corresponding to the target mode interval is determined as the target training mode.

[0023] In some embodiments, determining the target training mode based on the task evaluation data includes:

[0024] Obtain a pattern mapping table, which includes the mapping relationship between training modes and pattern mapping items;

[0025] Extract at least one pattern mapping item from the task evaluation data;

[0026] Based on the pattern mapping item, the target training pattern is searched in the pattern mapping table.

[0027] In some embodiments, when the target training mode is the first mode, task metric information is calculated according to the parameter update strategy, based on the task output and the task annotation information, including:

[0028] Freeze the model parameters of the task model;

[0029] Initialize the training parameters of the image enhancement model;

[0030] A first loss function is constructed, which is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to a first set of weights; the first set of weights is set according to the image enhancement evaluation index of the image enhancement model; the image enhancement evaluation index includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error;

[0031] Based on the first loss function, calculate the first total loss according to the task output and the task annotation information;

[0032] The task metric information is calculated based on the first total loss.

[0033] In some embodiments, based on the first loss function, a first total loss is calculated according to the task output and the task annotation information, including:

[0034] Obtain the downstream task type of the task model; the downstream task type is object detection or image segmentation;

[0035] The task loss is calculated based on the task output and the task annotation information; wherein, when the downstream task type is object detection, the task loss is a combination of classification loss, bounding box regression loss and object existence loss; when the downstream task type is image segmentation, the task loss is a combination of pixel-level cross-entropy loss and Dice loss.

[0036] Obtain the first weight set, the regularization loss, and the auxiliary loss;

[0037] The task loss, the regularization loss, and the auxiliary loss are weighted and summed according to the first set of weights to obtain the first total loss.

[0038] In some embodiments, the first weight set includes a first weight, a second weight, and a third weight; the first weight is used to perform a weighted calculation on the task loss; the second weight is used to perform a weighted calculation on the regularization loss; the third weight is used to perform a weighted calculation on the auxiliary loss; the method further includes:

[0039] Set regularization constraints, which include gradient or edge consistency constraints, frequency domain energy constraints, statistical consistency constraints, and reconstruction error auxiliary constraints;

[0040] Based on the regularization constraints, a general enhancement pre-training process is performed on the image enhancement model;

[0041] Under the condition of freezing the task model, the training process of the image enhancement model is fine-tuned in a task-driven manner;

[0042] The first weight is gradually increased, and / or the second weight is gradually decreased, so that the image enhancement model gradually adapts to the task requirements.

[0043] In some embodiments, when the target training mode is the second mode, task metric information is calculated according to the parameter update strategy, based on the task output and the task annotation information, including:

[0044] Obtain an alternating training strategy, wherein the alternating training strategy includes an alternating update strategy;

[0045] Extract the alternation period from the alternation update strategy;

[0046] According to the alternation cycle, the image enhancement model and the task model are alternately frozen to form multiple training sub-stages; the multiple training sub-stages include a first sub-stage and a second sub-stage; in the first sub-stage, the task model is frozen and the image enhancement model is unfrozen; in the second sub-stage, the image enhancement model is frozen and the task model is unfrozen.

[0047] A second loss function is constructed for the training sub-stage. This second loss function is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to a second set of weights. The second set of weights is set based on the current training sub-stage, according to either the image enhancement evaluation metric of the image enhancement model or the task evaluation metric of the task model. The image enhancement evaluation metric includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error. The task evaluation metric includes a task loss based on supervised learning.

[0048] Based on the second loss function, a second total loss is calculated according to the task output and the task annotation information;

[0049] The task metric information is calculated based on the second total loss.

[0050] In some embodiments, the alternating training strategy further includes a phased unfreezing strategy, and the method further includes:

[0051] The learning rates of the image enhancement model and the task model are set; wherein the learning rate of the task model is less than the learning rate of the image enhancement model.

[0052] Based on the phased unfreezing strategy, the model training phase is obtained, which is a first phase and a second phase.

[0053] When the model training phase is the first phase, the backbone feature extraction network of the task model is frozen in order to update the model head parameters of the task model.

[0054] During the second stage of model training, the backbone feature extraction network of the task model is gradually unfrozen.

[0055] According to a second aspect of this application, an image enhancement joint optimization system based on downstream task performance is provided, characterized in that the system comprises:

[0056] The data acquisition module is used to acquire training dataset and task evaluation data. The training dataset includes multiple sample images with task annotation information. The task evaluation data includes at least one of task requirement information, task training progress, and task evaluation results.

[0057] An image processing module is used to input sample images from the training dataset into an image enhancement model to obtain an enhanced image generated by the image enhancement model; and to input the enhanced image into a task model to obtain a task output result generated by the task model.

[0058] The mode determination module is used to determine a target training mode based on the task evaluation data, wherein the target training mode is a first mode or a second mode; wherein, in the first mode, the task model is frozen to train the image enhancement model; in the second mode, the image enhancement model and the task model are alternately frozen according to an alternating cycle to perform alternating joint training on the image enhancement model and the task model;

[0059] The strategy setting module is used to set a parameter update strategy based on the target training mode. The parameter update strategy includes the calculation method of the trained model and the task metric information. The trained model includes at least one of the image enhancement model and the task model.

[0060] The parameter update module is used to calculate task metric information based on the task output and the task annotation information according to the parameter update strategy, and to update the model parameters of the trained model based on the task metric information.

[0061] By employing the above technical solutions, embodiments of this application provide a joint optimization method and system for image enhancement based on downstream task performance. The method first acquires a training dataset and task evaluation data. Sample images from the training dataset are input into an image enhancement model to obtain enhanced images, and the enhanced images are input into a task model to obtain task output results. A target training mode is then determined based on the task evaluation data, and a parameter update strategy is set based on the target training mode. Then, according to the parameter update strategy, task metrics are calculated based on the task output results and task annotation information, and the model parameters of the trained model are updated based on the task metrics. This method can form an evaluation and closed-loop optimization approach based on downstream task performance as the core constraint. By introducing task metrics under controllable conditions, and employing task-driven optimization strategies and stabilization mechanisms during the training phase, the robustness and stability of the image processing process are improved. This method can also establish consistent evaluation criteria and reusable optimization paths between the image enhancement model and the task model, thereby achieving more robust task-side performance under multiple degradation and scenario conditions.

[0062] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1 This is a schematic diagram of the image enhancement joint optimization method based on downstream task performance provided in an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the overall architecture for joint optimization of image enhancement provided in an embodiment of this application;

[0066] Figure 3 This is a schematic diagram of the data processing flow provided in the embodiments of this application;

[0067] Figure 4 This is a schematic diagram of the first mode training enhancement process provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of the second mode training enhancement process provided in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of the image enhancement joint optimization system structure based on downstream task performance provided in an embodiment of this application. Detailed Implementation

[0070] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0071] In this application embodiment, image enhancement is an image processing technique that can generate image output that is more conducive to observation, analysis, or machine processing by addressing degradation factors in the input image based on algorithms or models. In some embodiments, noise, blur, low resolution, low illumination, haze scattering, compression artifacts, motion jitter, sensor imaging defects, etc., of the input image are processed by denoising, deblurring, super-resolution reconstruction, dehazing and low illumination enhancement, and spectral consistency restoration of multispectral or hyperspectral images to obtain an enhanced image more suitable for downstream task requirements.

[0072] Super-resolution reconstruction is used to restore low-resolution images to high-resolution images; denoising is used to suppress random noise and preserve detailed structures; deblurring or de-motion blurring can be used to restore edges and textures; and dehazing and low-light enhancement can be used to improve visibility and contrast.

[0073] The result of image enhancement processing can be called an enhanced image. The generated enhanced image can be combined with its downstream visual tasks to achieve specific applications in fields such as remote sensing, security monitoring, industrial inspection, and maritime surveillance. Downstream visual tasks can include image segmentation, target detection, and target recognition. For example, in remote sensing, it is necessary to enhance the boundaries of small targets and ground features to support mapping and monitoring; in security, it is necessary to improve visible details under nighttime, backlighting, or rain and fog conditions; in industrial inspection, it is necessary to restore defect textures to improve detection rates; and in maritime surveillance, it is necessary to enhance the detectability of targets such as ships or buoys in sea clutter and low-contrast environments.

[0074] As the demand for automated understanding capabilities continues to grow, vision systems can use task metrics such as segmentation, detection, and recognition as the ultimate evaluation criteria. However, in real-world applications, image degradation and environmental complexity can significantly degrade the quality of input images, leading to unstable performance of downstream models. Furthermore, while image enhancement algorithms can improve images at the level of visual perception or pixel similarity, their contribution to the performance of downstream tasks is not always stable, predictable, or reproducible.

[0075] To obtain enhanced images, image enhancement models can be constructed based on paired data supervised learning, unpaired data or weakly supervised learning, and perceptual quality optimization. These image enhancement models can then be spliced ​​and integrated with downstream task models to form an image task processing system.

[0076] In some embodiments, when training image enhancement using supervised learning based on pairwise data, the enhancement network can be trained using paired samples of degraded images and clear reference images, employing methods such as L1 and L2 pixel reconstruction losses, structural similarity losses such as Structural Similarity Index (SSIM) related terms, and frequency domain or edge consistency losses. This paradigm of image enhancement training based on supervised learning using pairwise data emphasizes the proximity of the output to the reference at the pixel / structural level, making it easier to converge and evaluate.

[0077] In some embodiments, when training for image augmentation based on unpaired data or weakly supervised learning, training objectives can be constructed through cycle consistency, adversarial training, domain adaptation, or self-supervised constraints in the absence of strictly paired data, in order to generate clearer and more natural augmentation results in a statistical distribution sense.

[0078] In some embodiments, during image enhancement training aimed at perceptual quality optimization, perceptual loss or adversarial loss, such as feature space-based similarity constraints, can be introduced to improve visual perception, resulting in sharper outputs with more textured details. This achieves a trade-off between subjective visual perception and objective reproducibility metrics.

[0079] In engineering practice, image enhancement processing can be accomplished using an image enhancement model deployed within an enhancement module. This enhancement module, as part of the preprocessing pipeline, is developed relatively independently from the downstream task models. After training, the image enhancement model can output enhanced images, which can then be input into task models used for detection, segmentation, and recognition. Furthermore, when evaluating results, pixel-level or statistical similarity metrics such as Peak Signal-to-Noise Ratio (PSNR), SSIM, Spectral Angle Mapper (SAM), ERGAS, and mean squared error are used as the primary criteria for evaluating the image enhancement effect. It is evident that the image enhancement training paradigm shown in the above embodiments has several shortcomings when addressing the core requirement of whether enhancement truly improves the performance of downstream tasks.

[0080] For example, pixel similarity metrics do not strictly correlate with task performance. In many real-world tasks, downstream models rely on discriminative structural information rather than overall pixel fidelity. Therefore, in practice, we may see high metrics but poor task performance, or mediocre metrics but good task performance. For instance, excessive smoothing might improve PSNR but remove small object textures from the image; while excessive sharpening might improve local contrast but introduce artifacts, leading to false positives. Furthermore, under different sensors and different degradation models, a single pixel metric is insufficient to characterize whether details effective for the task are preserved.

[0081] Furthermore, the selection of enhancements as preprocessing lacks reproducible and comparable task-driven evaluation benchmarks. Even in the same application scenario, using different task model versions, different inference thresholds, and different data partitioning and preprocessing processes can lead to difficulties in reproducing the comparison results of enhancement schemes. The conclusions are prone to fluctuation with changes in evaluation protocols, making it difficult to form transferable selection strategies.

[0082] Furthermore, the disconnect between training objectives and task objectives makes it difficult to establish a stable task-driven optimization mechanism. When attempting to directly train augmentation networks using task loss, problems such as instability, poor convergence, or speculative pseudo-augments are easily encountered. Augmentation networks may learn texture patterns that are beneficial to the task model but not realistic, improving certain metrics in the short term but destroying generalization; or training oscillations may occur due to the non-differentiability of task metrics and large gradient noise, reducing stability and making task-driven augmentation optimization difficult to reliably implement in engineering.

[0083] Downstream vision tasks refer to the model processing steps involved in structured understanding and decision-making output of image content or enhanced image content. For example, task models can be used for image segmentation, object detection, object recognition, or classification. Image segmentation involves assigning a semantic category or distinguishing instances to each pixel for semantic or instance segmentation, outputting masks or contours, and is sensitive to boundaries, texture, and local consistency. Object detection outputs bounding boxes, categories, and confidence scores; it is sensitive to small objects, low-contrast objects, occlusion, and complex backgrounds, and is significantly dependent on detailed texture and local contrast. Object recognition or classification outputs category labels or feature vectors for identification, retrieval, and comparison. Object recognition or classification is sensitive to discriminative texture, shape, and illumination consistency.

[0084] Downstream vision tasks can be performed using task models, which are image processing models built and trained based on specific application scenarios. The training of task models can follow a supervised learning paradigm, using a training dataset containing labeled data such as detection boxes, segmentation masks, and category labels to train the task network. During model training, loss functions such as classification loss, bounding box regression loss, and object presence loss from detection tasks, or task loss functions such as cross-entropy or Dice from segmentation tasks, or classification loss or metric learning loss from image recognition can be used. The main evaluation metrics include Mean Average Precision (mAP), Mean Intersection over Union (mIoU), F1 Score (F1), and Top-1 Accuracy (Top-1).

[0085] In scenarios such as remote sensing, maritime affairs, and security, image degradation and environmental complexity are prominent. For example, small targets occupy only a few pixels, sea clutter introduces pseudostructures, low illumination at night leads to a decrease in signal-to-noise ratio, and camera shake during telephoto operations causes motion blur. These factors directly reduce segmentation boundary accuracy, detection recall, and recognition accuracy, thus creating an urgent need for performance improvements at the task end. However, this need is not necessarily equivalent to improving pixel similarity; therefore, simply relying on enhancement evaluation has inherent biases.

[0086] To address this, the robustness of the task network can be improved through data augmentation, preprocessing, and domain adaptation. For example, random cropping, rotation, color jittering, and noise injection can be applied to the input, or retraining or fine-tuning can be performed for specific scenarios. Furthermore, the image augmentation module is considered a relatively independent input enhancer from the task network; its design and training process often do not directly optimize for the task result, but rather for visual quality or pixel consistency. Therefore, when downstream vision tasks are trained independently without utilizing the impact of augmentation on the task, problems also exist in complex degradation scenarios.

[0087] For example, task models are sensitive to changes in degradation distribution. When the degradation pattern of the deployment scenario is inconsistent with the distribution of the training data, even if the model performs well on standard data, its actual performance may still degrade significantly. Simply relying on task-side augmentation or generalization techniques is insufficient to cover all degradation combinations and sensor differences.

[0088] Using augmentation as a black-box preprocessing method lacks a unified task-driven evaluation protocol. Even by comparing the effects of augmentation A and the detection model combination with augmentation B and the detection model combination, without fixing key conditions such as task model structure or weights, inference threshold, input resolution, non-maximum suppression (NMS) strategy, data partitioning and preprocessing procedures, the conclusions obtained may not be reproducible, and it is difficult to determine whether the performance change comes from the augmentation or from the difference in evaluation configuration.

[0089] Furthermore, the lack of a stable task-driven augmentation optimization mechanism leads to difficulties in end-to-end collaboration. From a system perspective, there is a natural coupling between augmentation and downstream tasks; augmentation alters the statistical properties of the image, and the feature extraction and decision-making of the task network change accordingly. Direct end-to-end joint training can easily result in gradient interference, training instability, and overfitting to a particular task or dataset. If training is completely separate, the augmented output may not be the most useful output for the task.

[0090] In summary, because the pixel similarity metric used on the enhancement side cannot reliably reflect its contribution to the task side, there is a lack of a unified and reproducible task-driven evaluation benchmark for enhancement selection. Furthermore, even if the robustness of the task side can be improved through retraining, it is difficult to systematically utilize the potential of enhancements, and there is a lack of a stable task-driven enhancement optimization mechanism to support the collaborative optimization of both. Therefore, the selection of image processing methods that include image enhancement strategies relies on experience, comparison results are difficult to reproduce, cross-scene transfer costs are high, and task performance improvement is unstable under critical degradation conditions, resulting in low robustness and stability in the image processing process.

[0091] To address the issues of low robustness and stability in image processing, some embodiments of this application provide a joint optimization method for image enhancement based on downstream task performance. This method can establish a task-driven evaluation benchmark and provide a stable task-driven optimization mechanism under controllable conditions, ensuring that the selection and training of enhancement steps are truly aligned with the performance goals of the task, thus laying the foundation for subsequent tasks.

[0092] The method can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application embodiment uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in this application embodiment. Figure 1 As shown, the method includes:

[0093] S101. Obtain the training dataset and task evaluation data.

[0094] When jointly optimizing the image enhancement model and the task model, a training dataset can be obtained first. This training dataset includes multiple sample images with task-specific annotations. These sample images are generated by collecting image data from the business domain and annotating them with task-related information according to actual task requirements. By collecting sample images in batches, a training dataset of a certain scale can be formed.

[0095] Training datasets can be stored in data source devices such as cloud servers or local databases. When a training dataset needs to be retrieved, a training data retrieval request can be generated and sent to the data source device, which can then respond to the request and provide the training dataset to the electronic device.

[0096] While acquiring the training dataset, task evaluation data can also be obtained. This task evaluation data includes at least one of the following: task requirement information, task training progress, and task evaluation results. Task requirement information is predefined, rule-based text data based on the actual needs of image processing tasks in real-world business scenarios. Examples include task proposals, task template files, and task requirement items set by the user in real time. Task training progress is numerical or textual data recorded by the electronic device during model training. For example, training progress can be represented by 10% or initial stage. Training progress can also be represented using metrics such as epochs, iteration steps, and convergence status. Task evaluation results characterize the quality of the final output of the image processing process and can be represented by evaluation parameters calculated by the model, statistical parameters, and information assessed by users or evaluation agencies.

[0097] In some embodiments, when acquiring training datasets and task evaluation data, the original images and task-annotated data can be acquired first, and task requirement information can be extracted from the task evaluation data. This task requirement information includes at least one of preprocessing rules, input resolution processing rules, a data partitioning list, a random seed, an evaluation metric, and a calculated threshold. Then, according to the task requirement information, the original images are annotated using the task-annotated data to generate sample images with task annotation information. Finally, a training dataset is generated based on the sample images, and the training dataset is divided into a training set, a validation set, and a test set according to a set data partitioning ratio.

[0098] For example, such as Figure 2 As shown, to achieve task-driven image enhancement evaluation and training, functional modules such as data acquisition and management, image enhancement, downstream task inference, task indicator evaluation, and training control can be deployed in electronic devices. When acquiring training datasets and task evaluation data, the data acquisition and management module can first acquire the original images and task evaluation data from the data source device. Then, task annotation information is read from the task evaluation data, and data annotation is performed to annotate the original images with task annotation information, forming sample images. Multiple sample images are then combined into a dataset, thus obtaining the training dataset. Finally, the obtained training dataset is partitioned, for example, into a training set, a validation set, and a test set in a 7:2:1 ratio.

[0099] In some embodiments, after obtaining the training dataset, the sample images can also be preprocessed. For example, such as... Figure 3 As shown, remote sensing image L0, L1, and L2 products, as well as task-labeled data, are obtained through data reading. The task-labeled data includes coordinate-class mapping relationships. Data cleaning is then performed, including geometric correction as needed, removal of duplicate and low-quality samples, and normalization, cropping, and reprojection. Next, data sampling and partitioning are performed, dividing the input data into training, validation, and test sets, and saving the random seed and partitioning method for data reproduction.

[0100] S102. Input the sample images from the training dataset into the image enhancement model to obtain the enhanced image generated by the image enhancement model; and input the enhanced image into the task model to obtain the task output result generated by the task model.

[0101] After obtaining the training dataset, image enhancement and task processing can be performed on the sample images in the training dataset according to the normal processing flow of image processing tasks. Therefore, the sample images in the training dataset can be input into the image enhancement model to perform enhancement processing such as denoising, deblurring, super-resolution reconstruction, dehazing and low-light enhancement, and spectral consistency restoration of multispectral or hyperspectral images, thereby obtaining the enhanced image generated by the image enhancement model.

[0102] Image enhancement models can be single networks, cascaded structures (such as denoising followed by super-resolution), or multi-head structures sharing a backbone. During training, loss weights and freezing strategies can be set separately for different sub-modules to support the adaptation of combined enhancements to different tasks.

[0103] After generating the augmented image corresponding to the sample image, the augmented image can be used as input data for the task model. The task model can then perform image processing tasks such as image segmentation, object detection, and object recognition on the augmented image, thereby obtaining the task output results generated by the task model.

[0104] For example, image enhancement models can be image-to-image mapping models based on deep neural networks, including but not limited to: Convolutional Neural Network (CNN) restoration models, Transformer-based restoration models, Generative Adversarial Network (GAN) models, Diffusion models, and Normalizing Flow models; such as SRCNN, DnCNN, SwinIR, DeblurGAN, CycleGAN, and Diffusion models. Task models can employ different types of models based on different task types. Taking image segmentation as an example, image segmentation models can understand the content of an image and perform pixel-level classification, outputting a label mask. Essentially, image segmentation models are discriminative models that classify each pixel in an image, such as U-Net, FCN, and SAM.

[0105] After obtaining the training dataset, the image processing workflow can be followed. First, an image enhancement model can be used to perform one or more enhancement processes on the input sample images, such as super-resolution, denoising, and deblurring, to generate enhanced images. Then, the enhanced images are fed into downstream task models to produce task outputs such as detection boxes, segmentation masks, or category recognition results.

[0106] Image enhancement models and task models can be equipped with evaluation modules, either separately or uniformly. The evaluation module is used to calculate task indicators based on task output and annotations, and generate comparable scores or gain metrics according to preset rules to support the comparison of different enhancement schemes and model selection.

[0107] S103. Determine the target training mode based on the task evaluation data.

[0108] While using the image enhancement model and the task model for image processing, a target training mode can be determined based on task evaluation data. This target training mode can be either a first mode or a second mode. The first mode is used to perform individual training of the models, while the second mode is used to perform joint training of the models. In the first mode, the task model can be frozen to train the image enhancement model. In the second mode, the image enhancement model and the task model are alternately frozen in an alternating cycle to perform alternating joint training of the two models.

[0109] The first mode is suitable for optimizing augmentation mapping under given task model conditions, so that the augmentation output is more conducive to the task model to obtain higher task indicators, while reducing the optimization uncertainty caused by the co-change of augmentation and task network.

[0110] The second mode allows for joint alternating training of the image augmentation model and the task model. Joint alternating training aims to establish a synergistic adaptation relationship between the image augmentation model and the task model. That is, the image augmentation model learns to generate an input distribution that is more conducive to task parsing, while the task model simultaneously learns to adapt to the statistical properties of the augmented output, thereby achieving more consistent overall performance.

[0111] To determine the target training mode, in some embodiments, when determining the target training mode based on task evaluation data, a quantization index mapping can first be performed on the task evaluation data to obtain multiple data quantization indexes, and then a task evaluation value can be calculated based on these multiple data quantization indexes. The task evaluation value is a weighted sum of the multiple data quantization indexes. Then, based on the calculated task evaluation value, the target mode interval to which the task evaluation value belongs is queried, and the training mode corresponding to the target mode interval is determined as the target training mode.

[0112] For example, task evaluation data can include multiple task evaluation items. Each item can evaluate the task output in one dimension, such as whether it meets preprocessing rules or whether the model output is correct. For each task evaluation item, the task evaluation data can be quantized into a data quantification index through quantification. For example, when the processing of the sample image meets the preprocessing rules, the task evaluation item can be quantized as a value of 1, while when the processing of the sample image does not meet the preprocessing rules, the task evaluation item can be quantized as a value of 0.

[0113] After quantifying multiple task evaluation items separately, the resulting data quantification metrics can be weighted and summed to represent them using a single task evaluation value, such as Vte. A pre-built interval query database is then invoked, storing multiple pattern intervals and their corresponding training modes. Therefore, based on this database, the target pattern interval to which the task evaluation value Vte belongs can be found. Specifically, if V1 ≤ Vte ≤ V2, the interval [V1, V2] can be identified as the target pattern interval. Finally, the training mode corresponding to the target pattern interval is obtained, thus determining the target training mode.

[0114] In some embodiments, a pattern mapping table can also be used to determine the target training mode. That is, when determining the target training mode based on task evaluation data, a pattern mapping table can be obtained first, wherein the pattern mapping table includes the mapping relationship between training modes and pattern mapping items. Then, at least one pattern mapping item is extracted from the task evaluation data, and then the target training mode is searched in the pattern mapping table based on the pattern mapping item.

[0115] For example, multiple types of pattern mapping items can be pre-defined based on the task requirements, task training progress, and task evaluation results in the task evaluation data, such as preprocessing items, resolution items, data partitioning items, random seed items, evaluation index items, and calculation threshold items, and a pattern mapping table can be constructed based on multiple pattern mapping items and corresponding suitable training modes.

[0116] After obtaining the task evaluation data, the pattern mapping items contained in the current task evaluation data can be extracted. Then, by referring to the pattern mapping table, the training mode to be used under different combinations of pattern mapping items can be queried, that is, the target training mode can be determined.

[0117] S104. Set parameter update strategy based on target training mode.

[0118] After determining the target training mode, a parameter update strategy can be set based on the target training mode. This parameter update strategy includes the calculation method for the trained model and task metrics. For example, a training control module built into an electronic device can determine the parameter update strategy, i.e., the parameter freeze and update strategy, based on the training mode determined according to task evaluation data (i.e., the first mode or the second mode). This strategy is used to perform forward inference, loss calculation, backpropagation, and parameter updates, and during the verification phase, it calls a fixed evaluation protocol to output verifiable metric results.

[0119] In the defined parameter update strategy, different trained models can be determined based on different target training modes. The trained model includes at least one of an image enhancement model and a task model. When the target training mode is mode one, the trained model is an image enhancement model; when the target training mode is mode two, the trained model can alternate between an image enhancement model and a task model according to an alternating freeze strategy.

[0120] S105. According to the parameter update strategy, calculate the task metric information based on the task output and task annotation information, and update the model parameters of the trained model based on the task metric information.

[0121] After setting the parameter update strategy, the electronic device can perform model training according to the parameter update strategy. That is, according to the parameter update strategy, the task metric information is calculated based on the task output results and task annotation information, and then the model parameters of the trained model are updated based on the task metric information.

[0122] In some embodiments, when the target training mode is the first mode, the model parameters of the task model can be frozen first according to the parameter update strategy, and the training parameters of the image augmentation model can be initialized. Then, a first loss function is constructed. Based on the first loss function, a first total loss is calculated according to the task output and task annotation information, and then task metric information is calculated based on the first total loss.

[0123] The first loss function is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to the first weight set; the first weight set is set according to the image enhancement evaluation index of the image enhancement model; the image enhancement evaluation index includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error.

[0124] For example, such as Figure 4 As shown, in the first mode, the task model can be frozen, and only the image augmentation model can be trained. In the first mode, the downstream task model serves as a fixed task parser and feedback channel. The structure and parameters of the task model remain unchanged during training, and only the parameters of the image augmentation model are updated.

[0125] Therefore, in a training cycle, an electronic device can sample a batch of sample images from the training dataset. I And its corresponding task annotation information Y. Then, the image enhancement model of the image enhancement module. Received sample images I and output enhanced image. I' = E ( I Subsequently, the task model was frozen. right I' Perform reasoning to obtain the task output.Y' = T ( I' ).

[0126] The training control module then... Y and Y' Calculate task loss Loss task ( Y , Y' ), and further calculate the regularization loss related to the enhancement constraints ( Loss reg ) and auxiliary losses ( Loss aux Regularization loss and auxiliary loss can be used to provide general enhancement capabilities or to maintain output stability, respectively. The final total loss function is constructed in a weighted form:

[0127]

[0128] in, The first total loss is represented by α; the first weight is represented by α. This represents the task loss; β represents the second weight. Represents the regularized loss; γ represents the third weight; This indicates an auxiliary loss.

[0129] After calculating the first total loss, the model parameters of the image enhancement model can be updated only according to the parameter update strategy. θ Perform backpropagation and updates, without affecting the model parameters of the task model. Updates are performed. Furthermore, during training, task metrics can be periodically evaluated on the validation set according to a fixed evaluation protocol, and the optimal weights of enhancement modules are saved using the optimal validation metrics as the model selection criterion.

[0130] In some embodiments, in order to calculate the total loss, when performing the calculation of the first total loss based on the first loss function and the task output and task annotation information, the downstream task type of the task model can be obtained first. The downstream task type is either object detection or image segmentation.

[0131] Then, the task loss is calculated based on the task output and task annotation information. Specifically, when the downstream task type is object detection, the task loss is a combination of classification loss, bounding box regression loss, and object existence loss; when the downstream task type is image segmentation, the task loss is a combination of pixel-level cross-entropy loss and Dice loss.

[0132] Then, obtain the first set of weights, the regularization loss, and the auxiliary loss, and perform a weighted summation on the task loss, the regularization loss, and the auxiliary loss according to the first set of weights to obtain the first total loss.

[0133] For example, when calculating the total loss, the task loss can be constructed first. When the downstream task is object detection, the task loss can consist of classification loss, bounding box regression loss, and object presence loss. Classification loss can use cross-entropy or Focal class loss to handle class imbalance; regression loss can use L1 / L2 or IoU-based losses, such as GIoU, DIoU, CIoU, etc.; object presence loss is used to constrain the object prediction of candidate boxes. Under the condition of task model freezing, gradients can be passed to the enhancement network through the sensitive direction of the task network's enhancement output, thereby guiding the enhancement network to learn image transformations that are more conducive to detection accuracy.

[0134] When the downstream task is image segmentation, the task loss can be a combination of pixel-level cross-entropy loss and Dice loss to improve the fitting ability of class boundaries and alleviate class imbalance. In recognition or classification tasks, the task loss can be cross-entropy, or in retrieval applications, metric learning-related loss can be used to improve feature discrimination. The commonality of the first mode across different task types is that the task loss provides the main optimization guidance, while the enhancement regularization loss is used to constrain the stability and rationality of the enhanced output.

[0135] In some embodiments, when the task evaluation metric of ultimate interest, such as mAP, cannot be directly used as a differentiable objective in backpropagation, or its gradient is approximately unstable, a strategy can be adopted that uses differentiable task loss for training and task evaluation metrics for validation. That is, during the training phase, task loss is used... Loss task ( Y , Y' As the primary optimization objective, the verification phase calculates indicators such as mAP and mIoU according to the solidified evaluation protocol, and selects the enhanced model based on the optimal indicators.

[0136] Furthermore, differentiable surrogate terms related to the evaluation metrics can be constructed based on the confidence distribution, positive and negative sample intervals, or alignment errors output by the task model, and used to assist in the training of the augmentation network. In addition, for discrete parameters such as augmentation strength or threshold, grid or Bayesian search can be performed under a fixed evaluation protocol, and the search results can be used as one of the initialization or deployment configurations of the augmentation module.

[0137] In some embodiments, the first set of weights includes a first weight, a second weight, and a third weight; the first weight is used to perform weighted calculation on the task loss; the second weight is used to perform weighted calculation on the regularization loss; and the third weight is used to perform weighted calculation on the auxiliary loss.

[0138] Therefore, regularization constraints can be set during model training. These constraints include gradient or edge consistency constraints, frequency domain energy constraints, statistical consistency constraints, and reconstruction error auxiliary constraints. Based on these regularization constraints, general enhancement pre-training is then performed on the image enhancement model.

[0139] Under the condition of freezing the task model, the training process of the image enhancement model is fine-tuned in a task-driven manner by gradually increasing the first weight and / or gradually decreasing the second weight, so that the image enhancement model gradually adapts to the task requirements.

[0140] For example, to reduce the risk of unstable textures or artifacts generated by augmentation modules under task-driven training, the model training process can support augmentation output stability constraints and staged training. To this end, several regularization constraints can be introduced in the first mode. Regularization constraints can be used to limit the offset between augmentation output and input in terms of content structure, such as using gradient-based or edge-based consistency constraints to maintain geometric and structural stability. Simultaneously, abnormal drift in overall brightness, contrast, or color can be limited to avoid systematic deviations when deployed across scenes. For ringing, block effects, or unrealistic high-frequency textures, frequency domain energy constraints or local statistical consistency constraints can be introduced to suppress abnormal high-frequency peaks and repetitive textures. For training data with a clear reference image, L1 or L2 reconstruction error can be used as an auxiliary term, but not as the sole objective, to avoid pursuing pixel similarity at the expense of effective task information.

[0141] To improve convergence stability, phased training can be performed. First, the enhancement module undergoes general enhancement pre-training to acquire basic denoising, deblurring, or super-resolution capabilities. Then, task-driven fine-tuning is performed with the model frozen, using a stepwise approach. α Or gradually reduce β The weight scheduling allows the model to gradually adapt to task requirements while maintaining stable output.

[0142] In some embodiments, to improve coverage of difficult cases, the sampling probability of low-light, fog, sea clutter or small target samples can be increased, and bucketing indicators can be output during the verification phase, with key subset indicators as supplementary selection criteria.

[0143] like Figure 5 As shown, in some embodiments, when the target training mode is the second mode, an alternating training strategy, including an alternating update strategy, can be obtained first. Then, an alternating period can be extracted from the alternating update strategy, thereby alternatingly freezing the image enhancement model and the task model according to the alternating period to form multiple training sub-stages. These multiple training sub-stages include a first sub-stage and a second sub-stage; in the first sub-stage, the task model is frozen and the image enhancement model is unfrozen; in the second sub-stage, the image enhancement model is frozen and the task model is unfrozen.

[0144] A second loss function is then constructed for the training sub-stage. Corresponding to the first loss function, the second loss function is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to a second set of weights. The second set of weights is based on the current training sub-stage and is set according to the image enhancement evaluation metric of the image enhancement model or the task evaluation metric of the task model. The image enhancement evaluation metric includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error. The task evaluation metric includes the task loss based on supervised learning.

[0145] Then, based on the second loss function, the second total loss is calculated according to the task output and task labeling information, and the task metric information is calculated based on the second total loss.

[0146] For example, to mitigate training instability caused by simultaneous end-to-end updates, in the second mode, an alternating training strategy, including an alternating update strategy, can be acquired. The image enhancement model and the task model are then alternately frozen according to the alternating cycles within this strategy, forming multiple training sub-stages. That is, the model training process can be divided into two alternating sub-stages. The first sub-stage freezes the task model parameters. Only update the image enhancement model parameters θ The computation in the first sub-stage is the same as in the first mode, that is, the optimization signal is passed to the image enhancement model through the task loss, while the enhancement regularization is introduced to suppress artifacts.

[0147] In the second sub-stage, the parameters of the image enhancement model are frozen. θ Only update the task model parameters At this point, the output of the image augmentation model is considered a component of the input distribution of the task model, while the task model minimizes the task loss using standard supervised learning methods, thereby gradually adapting to the statistical characteristics of the augmented input.

[0148] The two sub-stages described above alternate with a preset number of iterations or small cycles. Therefore, during model training, when the image enhancement model update causes significant changes in the input distribution, the number of task model update steps can be appropriately increased, or the image enhancement model can be refined after the task model update to adapt to the new task parsing characteristics.

[0149] In the second mode, a loss function also needs to be constructed based on the task loss, regularization loss, and auxiliary loss to calculate the second total loss. For example, the total loss is constructed as follows:

[0150]

[0151] in, The second total loss is represented by α; the first weight is represented by α. This represents the task loss, i.e., the task-side loss term; β represents the second weight. γ represents the regularization loss, used to enhance output stability; γ represents the third weight. This represents auxiliary loss, used to provide general enhancement capabilities or structural consistency assistance.

[0152] In some embodiments, the training control module can also determine the model parameters θ of the image enhancement model and the model parameters of the task model based on the stage and strategy. The update order and learning rate, i.e., the alternating training strategy, also includes a phased unfreezing strategy. During model training, the learning rates for the image augmentation model and the task model can be set first. The learning rate for the task model is lower than that for the image augmentation model.

[0153] Based on the phased unfreezing strategy, the model training stages are then determined, wherein the model training stages are a first stage and a second stage. The first stage represents the initial stage of model training; the second stage represents the middle and later stages of model training.

[0154] During the first stage of model training, the backbone feature extraction network of the task model is frozen to update the model head parameters. During the second stage of model training, the backbone feature extraction network of the task model is gradually unfrozen.

[0155] For example, the training process can be further stabilized by using a phased unfreezing strategy and different learning rates. In the early stages of training, only the task head or detection head of the task model is updated, while the backbone feature extraction network is frozen to reduce the perturbation of deep representations caused by changes in input distribution. In the later stages of training, the backbone network is gradually unfrozen, allowing the task model to gain more comprehensive adaptability.

[0156] Meanwhile, different learning rates can be set for the enhancement module and the task model. A relatively smaller learning rate is used for the task model to avoid unstable oscillations caused by rapid changes in the input distribution. Furthermore, gradient clipping or stricter regularization weights can be applied to the enhancement module to suppress short-term anomalous texture enhancements. Optionally, exponential moving average weights can be used for validation and derivation on either the enhancement or task model to improve validation stability and inference consistency.

[0157] According to the parameter update strategy, after calculating the task metric information based on the task output and task annotation information, the model parameters of the trained model can be updated based on the task metric information. After multiple iterations of training to complete the parameter update of the trained model, it enters the model validation stage.

[0158] In the second mode, the validation phase also strictly follows the established evaluation protocol. The electronic device can calculate task metrics on the validation set and output bucketed statistics for target size or scene subsets to identify the model's changing trends on key difficult examples.

[0159] Alternatively, a single principal metric such as overall mAP or overall mIoU can be used as the criterion, and key subset metrics can be introduced as constraints based on the overall metric to select the model, thus avoiding overall improvement accompanied by key subset degradation. Furthermore, the training control module can be configured with an early stopping condition, stopping training and rolling back to the validation optimum when the validation metric no longer improves within a certain number of evaluation periods. , θ )combination.

[0160] In some embodiments, inference deployment can also be performed on the validated image enhancement model and the task model. During inference deployment, a serial structure can be adopted in which the image enhancement model outputs an enhanced image and then inputs it into the task model to output the final detection, segmentation, or recognition result.

[0161] For deployment environments with limited resources or significant variations in degradation levels, quality assessment or scene recognition modules can be introduced to determine whether to enable enhancements, select enhancement strength, or select enhancement sub-modules, thereby controlling computational overhead while ensuring output stability.

[0162] By applying the technical solutions of the above embodiments, the image enhancement joint optimization method based on downstream task performance described in the above embodiments can establish a reproducible task-driven evaluation benchmark. That is, the structure and parameters of the downstream task model, such as weights, input resolution, thresholding strategies, and post-processing rules, are pre-determined and fixed in the evaluation process. Different enhancement algorithms or different enhancement parameters are evaluated under the same data partitioning and inference configuration. Because the evaluation conditions remain consistent, differences between enhancement methods can be more directly reflected as differences in task output, reducing randomness and incomparability caused by differences in model version, inference parameters, or data processing, making the evaluation conclusions easier to reproduce and verify.

[0163] Furthermore, the method uses task metrics such as detection, segmentation, and recognition as the primary basis for measuring the effectiveness of the enhancement, and can further introduce a gain metric relative to the unenhanced baseline. Compared to using only pixel-by-pixel error or structural or spectral similarity metrics, this approach better reflects whether the enhanced output retains and strengthens the structural information that contributes to task discrimination, thereby reducing the occurrence of situations where pixel metrics improve but task performance deteriorates, and improving the consistency between the evaluation target and actual task requirements.

[0164] Because the proposed method supports normalization, scenario-based weighting, target-scale binning statistics, and multi-task fusion based on task metrics, it allows for more granular characterization and comparison of the effectiveness of augmentation methods under various environmental conditions such as fog, nighttime, sea clutter, and low signal-to-noise ratio. Furthermore, it can evaluate small, medium, and large targets separately, facilitating the identification of the impact of augmentation strategies on key difficult examples. This structured output is beneficial for creating reusable evaluation reports and strategy libraries, making the selection criteria for augmentation schemes clearer.

[0165] The method also provides a stable task-driven enhancement optimization mechanism. Specifically, it allows freezing the downstream task model, updating only the enhancement module parameters, and using the task loss or its surrogate as the primary optimization signal. During training, the task model can serve as a stable evaluation and feedback channel, reducing the coupling uncertainty caused by simultaneous changes in the enhancement and task networks, making the training process easier to tune and reproduce. Simultaneously, it enables the enhancement module to progressively learn image transformations more beneficial to the task while maintaining a fixed task parser.

[0166] The method also supports joint or alternating training of the image enhancement model and the task model, and introduces mechanisms such as freeze or alternating update strategies, differentiated learning rate settings, gradient clipping, early stopping, and validation set selection to reduce gradient interference and training oscillations. Compared to direct end-to-end simultaneous updates, the training control strategy of this method is more conducive to obtaining a stable convergence process and reduces sensitivity to specific data partitions or random seeds.

[0167] To prevent the enhancement module from generating textures or artifacts that do not conform to imaging rules under task feedback, thereby leading to false detections, missed detections, or segmentation boundary distortion, the method also introduces various consistency constraints and regularization terms, such as brightness or color consistency constraints, frequency domain energy constraints, artifact suppression terms, and content consistency or perceptual consistency constraints. A phased training strategy is adopted, first performing general enhancement pre-training and then task-driven fine-tuning, to improve task performance while maintaining output stability and interpretability. Through the constraint mechanism, the risk of short-term improvements, long-term degradation, or cross-scene failures during task-driven training can be reduced.

[0168] The method is compatible with various enhancement types and their combinations, facilitating expansion and reuse. Specifically, the framework can cover multiple enhancement tasks such as super-resolution, denoising, and deblurring, and supports sequential or switchable combinations. Evaluation and training control remain relatively decoupled from the implementation details of the enhancement modules, allowing for comparative evaluation and retraining without altering the overall evaluation protocol and optimization process when changing the enhancement network structure, degradation model, or data domain. This improves the reusability of the solution across different tasks, devices, and imaging conditions.

[0169] Furthermore, the method can also solidify the evaluation script, model version, and inference parameters as part of the protocol, so that the validation metrics in the training phase are consistent with the actual operating conditions in the inference phase. This is beneficial for conducting control experiments, regression tests, and problem localization during model iteration, and reduces the problem of effective training but deployment failure.

[0170] In some embodiments, as a specific implementation of the image enhancement joint optimization method based on downstream task performance described in the above embodiments, some embodiments of this application also provide an image enhancement joint optimization system based on downstream task performance, such as... Figure 6 As shown, the system includes:

[0171] The data acquisition module is used to acquire training dataset and task evaluation data. The training dataset includes multiple sample images with task annotation information. The task evaluation data includes at least one of task requirement information, task training progress, and task evaluation results.

[0172] An image processing module is used to input sample images from the training dataset into an image enhancement model to obtain an enhanced image generated by the image enhancement model; and to input the enhanced image into a task model to obtain a task output result generated by the task model.

[0173] The mode determination module is used to determine a target training mode based on the task evaluation data, wherein the target training mode is a first mode or a second mode; wherein, in the first mode, the task model is frozen to train the image enhancement model; in the second mode, the image enhancement model and the task model are alternately frozen according to an alternating cycle to perform alternating joint training on the image enhancement model and the task model;

[0174] The strategy setting module is used to set a parameter update strategy based on the target training mode. The parameter update strategy includes the calculation method of the trained model and the task metric information. The trained model includes at least one of the image enhancement model and the task model.

[0175] The parameter update module is used to calculate task metric information based on the task output and the task annotation information according to the parameter update strategy, and to update the model parameters of the trained model based on the task metric information.

[0176] By applying the technical solutions of the above embodiments, the image enhancement joint optimization system based on downstream task performance described in the above embodiments can be implemented in a single-machine or distributed training environment, and the inference end can be deployed on a server or edge device. The system first acquires a training dataset and task evaluation data, then inputs sample images from the training dataset into an image enhancement model to obtain enhanced images, and inputs the enhanced images into a task model to obtain task output results. Next, it determines a target training mode based on the task evaluation data and sets a parameter update strategy based on the target training mode. Then, according to the parameter update strategy, it calculates task metrics based on the task output results and task annotation information, and updates the model parameters of the trained model based on the task metrics. The system can form an evaluation and closed-loop optimization method based on downstream task performance as the core constraint. By introducing task metrics under controllable conditions, it adopts a task-driven optimization strategy and stabilization mechanism during the training phase to improve the robustness and stability of the image processing process. The method can also form a consistent evaluation basis and reusable optimization path between the image enhancement model and the task model, thereby obtaining more robust task-side performance under multiple degradation and multi-scenario conditions.

[0177] It should be noted that other corresponding descriptions of the functional units involved in the image enhancement joint optimization system based on downstream task performance provided in the embodiments of this application can be found in the corresponding descriptions in the image enhancement joint optimization method based on downstream task performance provided in the above embodiments, and will not be repeated here.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A joint optimization method for image enhancement based on downstream task performance, characterized in that, The method includes: Acquire training dataset and task evaluation data. The training dataset includes multiple sample images with task annotation information. The task evaluation data includes at least one of task requirement information, task training progress, and task evaluation results. The sample images in the training dataset are input into the image enhancement model to obtain the enhanced image generated by the image enhancement model; and the enhanced image is input into the task model to obtain the task output result generated by the task model. The target training mode is determined based on the task evaluation data, and the target training mode is either a first mode or a second mode; wherein, in the first mode, the task model is frozen to train the image enhancement model; in the second mode, the image enhancement model and the task model are alternately frozen according to an alternating cycle to perform alternating joint training on the image enhancement model and the task model; A parameter update strategy is set based on the target training mode, and the parameter update strategy includes the calculation method of the trained model and task metric information; the trained model includes at least one of the image enhancement model and the task model; According to the parameter update strategy, task metric information is calculated based on the task output and the task annotation information, and the model parameters of the trained model are updated based on the task metric information.

2. The method according to claim 1, characterized in that, Obtain the training dataset and task evaluation data, including: Acquire raw images and task-annotated data; The task requirement information is extracted from the task evaluation data. The task requirement information includes at least one of the following: preprocessing rules, input resolution processing rules, data partitioning list, random seed, evaluation index, and calculation threshold. According to the task requirements information, the original image is annotated using the task annotation data to generate a sample image with task annotation information; Generate a training dataset based on the sample images; The training dataset is divided into a training set, a validation set, and a test set.

3. The method according to claim 1, characterized in that, Determining the target training mode based on the task evaluation data includes: Perform quantitative index mapping on the task evaluation data to obtain multiple data quantitative indicators; The task evaluation value is calculated based on multiple data quantification indicators, and the task evaluation value is the weighted sum of multiple data quantification indicators; Query the target pattern range to which the task evaluation value belongs; The training mode corresponding to the target mode interval is determined as the target training mode.

4. The method according to claim 1, characterized in that, Determining the target training mode based on the task evaluation data includes: Obtain a pattern mapping table, which includes the mapping relationship between training modes and pattern mapping items; Extract at least one pattern mapping item from the task evaluation data; Based on the pattern mapping item, the target training pattern is searched in the pattern mapping table.

5. The method according to claim 1, characterized in that, When the target training mode is the first mode, according to the parameter update strategy, task metric information is calculated based on the task output and the task annotation information, including: Freeze the model parameters of the task model; Initialize the training parameters of the image enhancement model; A first loss function is constructed, which is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to a first set of weights; the first set of weights is set according to the image enhancement evaluation index of the image enhancement model; the image enhancement evaluation index includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error; Based on the first loss function, calculate the first total loss according to the task output and the task annotation information; The task metric information is calculated based on the first total loss.

6. The method according to claim 5, characterized in that, Based on the first loss function, a first total loss is calculated according to the task output and the task annotation information, including: Obtain the downstream task type of the task model; the downstream task type is object detection or image segmentation; The task loss is calculated based on the task output and the task annotation information; wherein, when the downstream task type is object detection, the task loss is a combination of classification loss, bounding box regression loss and object existence loss; when the downstream task type is image segmentation, the task loss is a combination of pixel-level cross-entropy loss and Dice loss. Obtain the first weight set, the regularization loss, and the auxiliary loss; The task loss, the regularization loss, and the auxiliary loss are weighted and summed according to the first set of weights to obtain the first total loss.

7. The method according to claim 5, characterized in that, The first set of weights includes a first weight, a second weight, and a third weight; the first weight is used to perform a weighted calculation on the task loss; the second weight is used to perform a weighted calculation on the regularization loss. The third weight is used to perform a weighted calculation on the auxiliary loss; the method further includes: Set regularization constraints, which include gradient or edge consistency constraints, frequency domain energy constraints, statistical consistency constraints, and reconstruction error auxiliary constraints; Based on the regularization constraints, a general enhancement pre-training process is performed on the image enhancement model; Under the condition of freezing the task model, the training process of the image enhancement model is fine-tuned in a task-driven manner; The first weight is gradually increased, and / or the second weight is gradually decreased, so that the image enhancement model gradually adapts to the task requirements.

8. The method according to claim 1, characterized in that, When the target training mode is the second mode, according to the parameter update strategy, task metric information is calculated based on the task output and the task annotation information, including: Obtain an alternating training strategy, wherein the alternating training strategy includes an alternating update strategy; Extract the alternation period from the alternation update strategy; According to the alternation cycle, the image enhancement model and the task model are alternately frozen to form multiple training sub-stages; the multiple training sub-stages include a first sub-stage and a second sub-stage; in the first sub-stage, the task model is frozen and the image enhancement model is unfrozen; in the second sub-stage, the image enhancement model is frozen and the task model is unfrozen. A second loss function is constructed for the training sub-stage. This second loss function is used to perform a weighted summation of the task loss, regularization loss, and auxiliary loss according to a second set of weights. The second set of weights is set based on the current training sub-stage, according to either the image enhancement evaluation metric of the image enhancement model or the task evaluation metric of the task model. The image enhancement evaluation metric includes at least one of structural similarity index, peak signal-to-noise ratio, and mean squared error. The task evaluation metric includes a task loss based on supervised learning. Based on the second loss function, a second total loss is calculated according to the task output and the task annotation information; The task metric information is calculated based on the second total loss.

9. The method according to claim 8, characterized in that, The alternating training strategy also includes a phased unfreezing strategy, and the method further includes: The learning rates of the image enhancement model and the task model are set; wherein the learning rate of the task model is less than the learning rate of the image enhancement model; Based on the phased unfreezing strategy, the model training phase is obtained, which is a first phase and a second phase. When the model training phase is the first phase, the backbone feature extraction network of the task model is frozen in order to update the model head parameters of the task model. During the second stage of model training, the backbone feature extraction network of the task model is gradually unfrozen.

10. A joint optimization system for image enhancement based on downstream task performance, characterized in that, The system includes: The data acquisition module is used to acquire training dataset and task evaluation data. The training dataset includes multiple sample images with task annotation information. The task evaluation data includes at least one of task requirement information, task training progress, and task evaluation results. An image processing module is used to input sample images from the training dataset into an image enhancement model to obtain an enhanced image generated by the image enhancement model; and to input the enhanced image into a task model to obtain a task output result generated by the task model. The mode determination module is used to determine a target training mode based on the task evaluation data, wherein the target training mode is a first mode or a second mode; wherein, in the first mode, the task model is frozen to train the image enhancement model; in the second mode, the image enhancement model and the task model are alternately frozen according to an alternating cycle to perform alternating joint training on the image enhancement model and the task model; The strategy setting module is used to set a parameter update strategy based on the target training mode. The parameter update strategy includes the calculation method of the trained model and the task metric information. The trained model includes at least one of the image enhancement model and the task model. The parameter update module is used to calculate task metric information based on the task output and the task annotation information according to the parameter update strategy, and to update the model parameters of the trained model based on the task metric information.

Images