Multi-agent collaborative evolution-based severe weather image restoration method and related device

Abstract

This application provides a method and related equipment for image restoration in severe weather based on multi-agent co-evolution. The method includes: acquiring a degraded image; generating guidance information for restoration through a prompting agent; generating multiple candidate restoration results through a restoration agent; performing multi-dimensional quality evaluation on the candidate results through an evaluation agent to generate a preference signal reflecting relative superiority or inferiority; optimizing the restoration agent based on the preference signal using a reinforcement learning strategy, and storing the interaction data in an experience replay pool to offline optimize the prompting agent and / or the evaluation agent, forming a closed-loop co-evolutionary mechanism of "generation-evaluation-optimization". This application achieves adaptive learning without paired supervision, can stably improve the image restoration quality in real severe weather scenarios, and supports continuous optimization of video streams, and can be applied to fields such as autonomous driving and intelligent monitoring.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a method and related equipment for restoring severe weather images based on multi-agent cooperative evolution. Background Technology

[0002] Images acquired under adverse weather conditions are often affected by factors such as raindrops, fog, and snow, leading to decreased image contrast, loss of detail, and reduced visibility. This severely restricts the performance of visual systems used in autonomous driving, intelligent monitoring, and aerial photography. Therefore, effective restoration of images acquired under adverse weather conditions has significant research and application value.

[0003] Existing image restoration methods primarily rely on supervised learning frameworks, achieving image restoration by constructing a mapping relationship between degraded and clear images. However, real-world weather degradation is complex and continuously changing, making it difficult to accurately model using synthetic data. This results in models trained on synthetic data exhibiting significant limitations in generalization ability in practical applications. Furthermore, existing methods typically employ fixed loss functions or evaluation metrics as optimization objectives. In the absence of real reference images, they struggle to provide stable and reliable learning signals, easily leading to problems such as over-enhancement, detail falsification, or structural distortion.

[0004] Existing relatively similar technical solutions mainly include multi-weather image restoration methods based on unified models (such as WeatherDiff and PromptIR) and restoration methods based on visual language models (such as DA-CLIP). The former achieves joint modeling of multiple weather degradation types by sharing network parameters or introducing multi-task learning mechanisms; the latter utilizes visual language models to perform semantic understanding of images or generate prompts to assist the image restoration process. However, these methods usually rely on fixed evaluation models or predefined reward functions, and their evaluation capabilities are difficult to adapt to the constantly changing degradation distribution in the real world. At the same time, they lack a co-optimization mechanism between generation and evaluation, resulting in biased or unstable learning signals, thus limiting the performance of the models in complex scenarios.

[0005] Therefore, how to provide a method for restoring severe weather images that can adaptively learn and continuously optimize without paired supervision is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The main objective of this application is to propose a method, electronic device, storage medium, and program product for severe weather image restoration based on multi-agent co-evolution, aiming to solve the following problems existing in the prior art: most existing severe weather image restoration methods rely on synthetic data for supervised training, resulting in insufficient generalization ability; fixed optimization targets are unreliable in the absence of reference ground values ​​when learning signals are unreliable, which can easily lead to semantic shifts and structural distortions; and there is a lack of effective learning mechanisms for unlabeled real data, making it difficult to continuously optimize in dynamic environments.

[0007] To achieve the above objectives, one aspect of this application proposes a method for reconstructing severe weather images based on multi-agent cooperative evolution, the method comprising: Obtain the degraded image to be processed; The degraded image is semantically modeled by a prompting agent to generate prompting information to guide the image restoration process; By using a recovery agent, multiple candidate recovery results are generated based on the degraded image and the prompt information; By evaluating the intelligent agent, a multi-dimensional quality assessment is performed on the multiple candidate recovery results, and a preference signal reflecting the relative superiority or inferiority among the candidate recovery results is generated based on the assessment results. Based on the preference signal, a reinforcement learning strategy is used to optimize and update the parameters of the recovery agent, and the data generated during the interaction is stored in the experience replay pool for offline optimization of the prompting agent and / or the evaluation agent, forming a closed-loop co-evolutionary mechanism of "generation-evaluation-optimization".

[0008] In some embodiments, the prompting agent and the evaluation agent are constructed based on a visual language model; the prompting information includes one or more of the following: a description of the degradation type, a description of the degree of degradation, information about the affected area, and the target recovery requirements, and is generated through a chain-like reasoning process.

[0009] In some embodiments, the multidimensional quality assessment includes perceived quality assessment, semantic consistency assessment, and structural preservation assessment; The perception quality assessment integrates at least one no-reference image quality evaluation index with the perception score output by the evaluation agent. The semantic consistency assessment is achieved by comparing the semantic representation similarity between the degraded image and the candidate restoration result under the constraint of the prompt information. The structure preservation assessment is achieved by comparing the geometrical consistency between different candidate recovery results.

[0010] In some embodiments, the preference signal is a reward value generated by a weighted fusion of the evaluation results of perceived quality, semantic consistency and structural preservation, and is obtained by sorting the reward values ​​of multiple candidate recovery results.

[0011] In some embodiments, the weighting coefficients used for weighted fusion are adaptively adjusted by minimizing the difference between the combined reward ranking and the overall preference ranking of the evaluating agent.

[0012] In some embodiments, the step of optimizing and updating the parameters of the recovering agent using a reinforcement learning strategy based on the preference signal specifically includes: The reward value corresponding to the preference signal is standardized to obtain the advantage function; Based on the probability ratio between the current policy and the old policy, a policy optimization objective function is constructed that includes pruning terms and KL divergence constraints to limit the policy update magnitude.

[0013] In some embodiments, storing the data generated during the interaction process in an experience replay pool for offline optimization of the prompting agent and / or the evaluation agent specifically includes: During the online phase, the recovering agent is updated using the immediate preference signals of the current sample; During the offline phase, historical trajectory data is sampled from the experience replay pool, and offline rewards for the prompting agent and the evaluation agent are calculated respectively, and the parameters of the prompting agent and the evaluation agent are updated accordingly. The online and offline updates are executed asynchronously and alternately.

[0014] In some embodiments, the method further includes an adaptive control step based on a validation set: Calculate the change in performance metrics on the validation set; The learning rate of the reinforcement learning strategy is dynamically adjusted based on the changes in the performance metrics. When the performance metric declines for several consecutive stages, a model parameter rollback operation is performed to restore the model to its historical optimal state.

[0015] In some embodiments, the method is further extended to continuous real-world video streaming scenarios: Continuous video frames input in chronological order are treated as an unlabeled data stream; Perform online adaptive updates on consecutive frames of data within the current time period to update the recovery agent; The updated recovery agent is used to process video frames in subsequent time periods, enabling the model to continuously evolve in a dynamic environment.

[0016] In some embodiments, the recovery agent employs a conditional generation model, and the multiple candidate recovery results are generated by performing multiple independent samplings under the same input conditions.

[0017] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0018] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0019] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0020] Compared with the prior art, this application has the following beneficial effects: 1) Stable and reliable learning signals: By constructing a closed-loop collaborative mechanism between the recovery agent, the prompting agent, and the evaluation agent, the generation and evaluation processes are coupled and dynamically updated. An adaptive preference signal is formed without the need for a reference truth value, which effectively reduces optimization bias and improves training stability and result consistency.

[0021] 2) Multi-dimensional quality constraints: A multi-level preference modeling method based on perceptual quality, semantic consistency and structure preservation is introduced to comprehensively evaluate and optimize the image restoration results. This allows the model to improve image clarity while taking into account semantic information and geometric structure, resulting in a more balanced and reliable restoration effect.

[0022] 3) Strong adaptability to real-world scenarios: It does not rely on paired labeled data. It constructs effective learning signals from unlabeled real images through a multi-agent collaborative mechanism, enabling the model to directly adapt to the complex and ever-changing harsh weather conditions in the real world.

[0023] 4) Supports continuous optimization and expansion: Through a collaborative training strategy that combines online and offline methods, the model can be continuously updated and optimized during use, and can be extended to video input scenarios, possessing the ability to continuously evolve for dynamic environments. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the multi-agent co-evolutionary framework in severe weather image restoration in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the principle of the stable co-evolutionary mechanism in the embodiments of this application; Figure 3 This is a schematic diagram of the online continuous adaptive process based on continuous real video stream in the embodiments of this application; Figure 4 This is a flowchart of the severe weather image restoration method based on multi-agent cooperative evolution provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0028] 1) An intelligent agent is an entity that can perceive its environment, make autonomous decisions, and take actions to achieve specific goals. It can be software, hardware, or a system. Compared with traditional artificial intelligence, intelligent agents not only possess the ability to understand, but also emphasize the execution ability to "do things," and can independently complete complex tasks.

[0029] Limitations of existing technologies: Images acquired under adverse weather conditions are often affected by factors such as raindrops, fog, and snow, leading to decreased image contrast, loss of detail, and reduced visibility. To address this issue, existing image restoration methods primarily rely on supervised learning frameworks, constructing a mapping between degraded and clear images to achieve image restoration. However, real-world weather degradation is complex and continuously changing, making accurate modeling with synthetic data difficult. This results in models trained on synthetic data exhibiting significant limitations in generalization ability in practical applications. Furthermore, existing methods typically employ fixed loss functions or evaluation metrics as optimization objectives. Without real reference images, they struggle to provide stable and reliable learning signals, easily leading to over-enhancement, detail falsification, or structural distortion, thus affecting the realism and consistency of the restored results.

[0030] Existing technologies offer methods for multi-weather image restoration based on unified models and restoration methods assisted by visual language models. The former achieves joint modeling of multiple weather degradation types by sharing network parameters or introducing multi-task learning mechanisms; the latter utilizes visual language models to perform semantic understanding of images or generate prompts to assist the image restoration process. Furthermore, some existing solutions introduce reinforcement learning or evaluation model-based optimization strategies to improve restoration performance under no-reference conditions. However, these solutions typically rely on fixed evaluation models or predefined reward functions, whose evaluation capabilities struggle to adapt to the constantly changing degradation distributions in the real world. They also lack a collaborative optimization mechanism between generation and evaluation, leading to biased or unstable learning signals, thus limiting the model's performance in complex scenarios.

[0031] In summary, the existing technical solutions have the following drawbacks: 1) Unreliable learning signals: Existing methods usually rely on fixed loss functions or predefined evaluation metrics as optimization targets. In the absence of real reference images, these fixed signals are difficult to accurately reflect the recovery quality, which can easily lead to biases in the model during the optimization process, resulting in problems such as over-enhancement, artifact introduction, or detail distortion.

[0032] 2) The generation and evaluation processes are disconnected: In existing technologies, image restoration models and quality evaluation mechanisms are usually independent of each other and lack an effective collaborative optimization process. This results in the evaluation results being unable to dynamically adapt to changes in the model's capabilities, thereby limiting the effectiveness and stability of the learned signals.

[0033] 3) Lack of continuous adaptive capability: Existing methods lack a mechanism for continuous learning from unlabeled real data. When faced with the ever-changing weather degradation distribution in the real world, it is difficult to achieve long-term stable performance improvement and adaptive optimization, which affects their application effect in real-world scenarios.

[0034] In view of this, this application provides a method, electronic device, storage medium, and program product for severe weather image restoration based on multi-agent co-evolution. This solution achieves adaptive learning under unsupervised conditions by constructing a closed-loop interaction mechanism among a recovery agent, a prompting agent, and an evaluation agent. Specifically, this application guides the recovery agent to generate multiple candidate results by having the prompting agent generate restoration instructions based on degradation perception. The evaluation agent then comprehensively evaluates these results based on multi-dimensional information such as perception quality, semantic consistency, and structural preservation, forming a preference ranking signal for continuous optimization of the restoration model. Simultaneously, by introducing a training strategy combining online and offline methods, co-evolution among different agents is achieved, thereby gradually improving model performance without the need for real labels. Furthermore, this application supports continuous adaptive optimization in video streaming scenarios, enabling continuous evolution and performance improvement of the model in dynamically changing real-world environments through online learning of continuous frame data. Compared with existing technologies, this application can provide a more stable and reliable learning mechanism under complex and ever-changing harsh weather conditions in the real world. While improving image clarity, it can effectively maintain semantic information and structural consistency, and has the ability to continuously optimize for video streams, thereby significantly improving the image and video restoration effect.

[0035] Specifically, the core innovations of this solution mainly focus on the following technologies: 1) Multi-agent Co-evolution Framework: The embodiments of this application construct a collaborative system consisting of a recovery agent, a prompt agent, and an evaluation agent. Through a closed-loop interaction mechanism of "generation-evaluation-optimization", the prompt generation, image restoration and quality evaluation processes are coupled and dynamically updated. Adaptive learning signals are formed without paired supervision, thereby improving the stability and generalization ability of the model in complex real environments. 2) Multi-level Preference Modeling: This application presents a multi-level evaluation method based on perceptual quality, semantic consistency and structure preservation. By comprehensively scoring and ranking multiple candidate recovery results, it provides a stable and comprehensive basis for optimization under no-reference conditions, thereby improving image clarity while maintaining the consistency of semantic information and geometric structure. 3) Preference-driven Collaborative Optimization: The embodiments of this application adopt a reinforcement learning strategy based on relative preferences and combine an online and offline training mechanism to achieve collaborative optimization among different agents, enabling the model to continuously update using unlabeled real data, thereby gradually improving the image restoration effect.

[0036] The severe weather image restoration method based on multi-agent cooperative evolution provided in this application relates to the fields of artificial intelligence and computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the severe weather image restoration method based on multi-agent cooperative evolution, but is not limited to the above forms.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] like Figure 4 As shown in the figure, this application provides a method for severe weather image restoration based on multi-agent cooperative evolution, which specifically includes the following steps: Step S1: Obtain the degraded image to be processed; Step S2: The degraded image is semantically modeled by the prompting agent to generate prompting information to guide the image restoration process; Step S3: Using a recovery agent, generate multiple candidate recovery results based on the degraded image and the prompt information; Step S4: By evaluating the intelligent agent, perform multi-dimensional quality evaluation on the multiple candidate recovery results, and generate a preference signal reflecting the relative superiority or inferiority relationship between the candidate recovery results based on the evaluation results; Step S5: Based on the preference signal, the parameters of the recovery agent are optimized and updated using a reinforcement learning strategy, and the data generated during the interaction is stored in the experience replay pool for offline optimization of the prompting agent and / or the evaluation agent, forming a closed-loop co-evolution mechanism of "generation-evaluation-optimization".

[0039] The solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific application examples.

[0040] This embodiment proposes a severe weather image restoration method based on multi-agent co-evolution. By constructing a closed-loop interaction mechanism between generation, evaluation, and optimization, it enables prompt generation, image restoration, and result evaluation to work collaboratively within a unified framework, achieving adaptive learning and continuous optimization of the model without the need for paired supervised data. This technical solution continuously generates and updates learning signals through interaction and feedback among multiple agents, thereby gradually improving image restoration performance.

[0041] (1) Multi-agent collaborative generation and interaction and preference-based reinforcement learning co-evolutionary mechanism like Figure 1 As shown in (a), this embodiment constructs a severe weather image restoration system based on multi-agent co-evolution. The system includes a prompt agent (policy denoted as...). ), Restoration Agent (policy represented as ), Evaluation Agent (the evaluation function is represented as...) The system includes an experience replay buffer. These modules form a unified closed-loop structure by sharing interactive trajectory data, enabling prompt generation, image restoration, and quality assessment to operate collaboratively under the same optimization objective.

[0042] The prompting and evaluation agents are built based on a visual language model to establish a mapping relationship between visual and semantic information in images, and to support structured expression based on reasoning processes. The recovery agent employs a conditional generation model, whose input consists of the degraded image and prompt information, and whose output is the restored image. The experience replay pool uses a scalable data cache structure to continuously store interaction data generated during system operation, and supports sampling by time sequence or priority.

[0043] For the input degraded image First, the prompting agent performs the degradation perception and semantic modeling process. The input image is encoded into semantic feature representations through a visual language model. This feature representation contains semantic information about degradation patterns in the image. Based on this semantic representation, and combined with a preset prompt template and a chained reasoning process, editing instructions are generated. The editing instructions consist of several components, including a description of the degradation type, a description of the degree of degradation, information on the affected area, and the target recovery requirements, and are input into the recovery agent in a structured form.

[0044] Recover the agent in the conditional probability distribution The image generation process is then executed. For the same input image, while keeping the input conditions unchanged, a set of candidate restoration results is generated through multiple independent samplings:

[0045] in, Indicates the first One candidate result, Indicates the number of candidate results. Indicates the height of the image. Indicates the width of the image. This indicates the number of channels in the image.

[0046] Each candidate result corresponds to a different generation path, and the difference arises from the randomness introduced during the sampling process. This method forms multiple recovery hypotheses in the solution space, thereby avoiding the local optimum problem caused by a single generation path.

[0047] Each candidate result and its corresponding generation conditions together constitute part of the interaction trajectory. The evaluation agent performs a quality evaluation process on the candidate results, and the evaluation function is expressed as:

[0048] in, Indicates the first A scalar reward signal This indicates that a reward model is used for scoring.

[0049] The input includes the original degraded image, editing instructions, and the restored image. The evaluation process comprises multiple evaluation dimensions. The perceptual quality component fuses a no-reference image quality evaluation metric with a perceptual score generated by a visual language model to measure the overall visual effect of the image. The semantic consistency component evaluates the semantic alignment between the input image and the restored image based on cue conditions. The structure preservation component constrains structural distortion by analyzing the geometric consistency of the restored image. The above evaluation results are then fused to obtain a unified scalar reward signal. .

[0050] Based on the set of evaluation results The candidate results are ranked to form a preference relationship. This preference relationship reflects the relative merits of different candidate results in terms of overall quality and serves as a foundational signal for subsequent strategy optimization. Data generated during each interaction, including the input image, semantic feature representation, prompting reasoning process, candidate recovery results, and evaluation results, are uniformly organized as trajectory data and written into the experience playback pool.

[0051] As the training process progresses, the data in the experience replay pool accumulates, gradually forming a historical dataset covering various degradation patterns and recovery results. This dataset is not only used for sampling in subsequent training processes but also for information sharing between different modules, enabling prompt generation, image restoration, and quality assessment to evolve collaboratively under a unified data distribution.

[0052] In this closed-loop structure, the output of the recovery agent directly affects the evaluation result of the assessment agent. The evaluation result is further fed back to the recovery agent as an optimization signal, prompting the agent to continuously adjust its generation strategy based on historical data. The three agents form a dynamic coupling relationship driven by shared data, enabling the system to gradually optimize the recovery effect through multiple rounds of interaction.

[0053] (2) Online-offline joint reward mechanism based on multi-level preference modeling Based on the aforementioned multi-agent co-evolutionary framework, this embodiment constructs an online-offline joint reward mechanism based on multi-level preference modeling to provide stable and effective optimization signals for the reinforcement learning process under unsupervised conditions. This reward mechanism improves the overall stability and generalization ability of the system by characterizing the image restoration quality from multiple dimensions and combining historical trajectory information to jointly optimize different agents.

[0054] For candidate results of the recovery agent generation First, a multi-level preference reward system is constructed to characterize the restoration quality. The perceptual quality part is modeled by fusing multiple no-reference image quality evaluation metrics with the perceptual score output by the evaluating agent, and is expressed as:

[0055] in, This represents the normalized, no-reference quality index, including MUSIQ, LIQE, and CLIP-IQA, etc. For the corresponding weighting coefficients, To evaluate the perceptual quality score output by the agent, it is used to supplement local artifacts and visual distortions that are difficult to characterize by traditional indicators.

[0056] The semantic consistency component constrains the semantic alignment between the restored image and the input image under cue conditions, and is defined as follows:

[0057] in, This represents the semantic representation of the image and cues jointly encoded by the visual language model. This represents the similarity calculation function obtained based on the evaluation of the agent's reasoning process. This term effectively suppresses semantic shift and structural illusion by comparing the semantic consistency between the input image and the recovered result under cue constraints.

[0058] The structure preservation part is achieved by constraining the geometric consistency between candidate results, which is expressed as:

[0059] in, This represents the depth map generated by the pre-trained depth estimation model. This indicates the position of a specific pixel in an image. This represents the pixel domain. This term suppresses geometric drift that may occur during the restoration process by constraining the depth differences between different candidate results.

[0060] Based on the above three parts, a unified reward function is defined:

[0061] in, , , The weighting coefficients are adaptively adjusted by minimizing the difference between the combined reward ranking and the overall preference ranking of the evaluating agent. This reward function is used to generate preference relationships between candidate outcomes and serves as direct input for optimizing the agent's policy.

[0062] During online optimization, the aforementioned rewards are calculated in real-time for the current input sample and used to drive the update of the recovery agent's policy parameters, enabling the model to quickly adapt to the current data distribution. Simultaneously, an offline reward mechanism based on an experience replay pool is introduced to optimize the prompting agent and the evaluation agent. For the prompting agent, its offline reward is defined as:

[0063] in, This represents historical degraded images in the experience playback pool. This indicates the output of the chained reasoning process generated by the agent. Represents historical reward value. This represents the historical reward weighting coefficient. This represents the cross-modal similarity scoring function computed based on the CLIP model. This reward measures the consistency between the cue reasoning process and the semantics of the image.

[0064] For the evaluation agent, the offline reward is applied to both the perceptual quality and semantic consistency reasoning processes. The corresponding reward forms are:

[0065]

[0066] in, This represents the offline perception quality inference reward for evaluating the agent. This represents the offline semantic consistency reasoning reward for evaluating the agent. This represents the historical recovery results in the experience replay pool. The aforementioned reward achieves continuous optimization of evaluation capabilities by measuring the cross-modal consistency between the agent's reasoning process and historical images.

[0067] Online and offline rewards work synergistically within a unified optimization framework. The online component provides immediate preference signals for the current sample, enabling the recovery agent to update its policy based on the current data distribution. The offline component utilizes historical trajectory data to stabilize and optimize the prompting and evaluation agents, resulting in more accurate prompts and evaluations. This is achieved by introducing historical reward items. and weighting coefficients This achieves a balance between current and historical data, thereby suppressing fluctuations during the training process.

[0068] Through the aforementioned multi-level preference modeling and online-offline joint optimization mechanism, this application constructs a stable reward system under unsupervised conditions, enabling different agents to collaboratively optimize under a unified feedback signal, thereby significantly improving the quality control capability and training stability in the image restoration process.

[0069] (3) Reinforcement learning optimization and stable co-evolution mechanism based on relative preferences Based on the aforementioned closed-loop interaction mechanism, this embodiment uses a reinforcement learning method based on relative preferences to optimize the strategy of the recovery agent, enabling the model to learn an effective image restoration strategy under unpaired supervised data conditions.

[0070] For the candidate result set and their corresponding evaluation results First, the rewards are standardized. By calculating the statistical properties of the candidate results, the original evaluation values ​​are transformed into an advantage function:

[0071] in, This represents the average reward value of the candidate results. Indicates standard deviation, This is a stabilizing term used to avoid the division-by-zero problem. This normalization process makes the rewards between different input samples comparable, thereby improving training stability.

[0072] In the strategy optimization process, a probability ratio is introduced to measure the difference between the current strategy and historical strategies:

[0073] in, Indicates the current strategy. Indicates historical strategy.

[0074] Based on this probability ratio, construct the policy optimization objective function:

[0075] in, This represents the truncation function. This represents the weighting coefficient of the KL regularization term. Indicates the Kullback–Leibler divergence. Indicates a reference strategy.

[0076] The pruning term in the objective function limits the magnitude of policy updates, keeping the probability ratio within a set range to prevent overly rapid policy updates from causing training instability. The KL divergence term constrains the difference between the current policy and the reference policy, ensuring the updated policy remains within a reasonable range. Used to control constraint strength.

[0077] The training process employs an asynchronous online-offline collaborative mechanism. In the online phase, the recovering agent performs a generation process on the input data based on the current policy and updates the policy parameters according to the evaluation results. Each generation process corresponds to a complete interaction trajectory, which is written into the experience replay pool. In the offline phase, historical data is sampled from the experience replay pool to batch train the prompting agent and the evaluation agent, enabling them to learn more stable prompt generation and evaluation policies from diverse historical data.

[0078] Online and offline updates are performed alternately over time, enabling the recovery agent to quickly adapt to the current data distribution, while prompting and evaluating agents maintain stability using historical data. This asynchronous collaborative mechanism effectively mitigates the non-stationarity problem in multi-agent joint optimization.

[0079] like Figure 2 As shown, during long-term reinforcement learning, policy oscillations or reward distribution drift may occur due to the interaction between different modules. To address this, an adaptive control mechanism based on the validation set is introduced. Performance metrics are calculated on the validation data. And define the performance variation:

[0080] in, This indicates the difference in performance indicators. Indicates the first Performance metrics at any given time Indicates the first Performance metrics at any given moment.

[0081] The learning rate is dynamically adjusted based on this change:

[0082] in, This represents the learning rate at the current training time step. Indicates the learning rate growth factor. This represents the learning rate decay factor. This represents the upper bound of the learning rate. This represents the lower bound of the learning rate. This indicates the threshold for performance changes.

[0083] When validation performance degrades across multiple consecutive phases, a model parameter rollback operation is performed to restore the model to its historical best state and reduce the learning rate, ensuring the stability of the training process. This mechanism, together with experience replay training and KL constraints, enables the multi-agent system to remain stable during long-term optimization.

[0084] Through the aforementioned reinforcement learning optimization method based on relative preferences, asynchronous collaborative training mechanism, and verification-guided stable control strategy, this application achieves continuous collaborative evolution among multiple agents, enabling the recovery agent to gradually improve performance under unpaired supervised data conditions and maintain a stable optimization process in complex degenerate environments.

[0085] (4) Online Continuous Adaptive Method Based on Continuous Real Video Stream like Figure 3 As shown, based on the multi-agent co-evolutionary framework, this embodiment further extends to continuous real video stream input scenarios, constructs an online continuous adaptive method driven by data stream, and enables the model to continuously optimize the dynamically changing degradation distribution under unpaired supervision.

[0086] The input data arrives continuously as a video stream, represented as a sequence of images arranged in chronological order. The image sequences are derived from continuously acquired visual data in real-world environments. The degradation type and intensity vary over time, constituting a non-stationary data distribution. The model does not rely on any paired labeled data during processing; it performs optimization solely based on the current input data and the preference signals generated internally by the system.

[0087] During video stream input, the system performs online adaptive updates on consecutive frames from the previous time segment, using them as unlabeled training data to update the policy parameters of the recovery agent. The updated model is then used to process input frames from subsequent time segments, thus achieving seamless integration of training and inference over time. This mechanism enables the model to gradually adapt to degradation patterns in the current environment during actual operation.

[0088] To address the characteristic of degradation distribution changing over time in continuous video scenes, the proposed method employs a data stream-driven parameter update process, enabling the recovery strategy to dynamically respond to changes in the input distribution. Under short-time-series input conditions, the model parameter update process can be triggered with a limited number of frames of data, achieving rapid adaptation. Under long-time-series conditions, continuous online updates and experience playback mechanisms allow the model to gradually optimize its performance over time.

[0089] To verify this adaptive capability, the system divides the continuous video sequence into an adaptation phase and an evaluation phase. The input for the adaptation phase is used as an unlabeled data stream for model updates, while the input for the evaluation phase is used to test the model's generalization performance on unseen data. This approach enables a unified evaluation of the model's online adaptability and cross-temporal distribution generalization ability.

[0090] Under continuous data input conditions at different time scales, the proposed method maintains a stable performance improvement. In short time series, effective adaptation can be achieved with only a few consecutive frames; and it can maintain stable performance even under cross-time or cross-environment conditions, indicating that the multi-agent co-evolutionary mechanism is robust to temporal variations.

[0091] By employing the aforementioned online continuous adaptive method based on continuous real video streams, this application enables the model to achieve long-term optimization capabilities in non-stationary real-world environments, allowing the image restoration system to continuously improve performance during actual deployment and adapt to complex and variable weather degradation conditions.

[0092] (5) Experimental verification This embodiment conducted comparative experiments on multiple real-world severe weather image datasets, and the results show that the proposed technical solution has good effectiveness and practicality. Details are as follows: Quantitative Comparison: This embodiment conducted comparative experiments on multiple real-world severe weather image datasets, covering different degradation types such as rain, snow, and fog. It was compared with various representative methods, including methods trained under supervised training based on synthetic data such as WeatherDiff, WGWS-NET, PromptIR, and DA-CLIP, as well as unsupervised or weakly supervised methods such as CSUD, RSCP2GAN, and DA-RCOT. Quantitative evaluation was performed using no-reference image quality metrics, including CLIP-IQA, LIQE, and MUSIQ, combined with perceptual quality scoring and semantic consistency scoring based on visual language models for comprehensive analysis. Experimental results show that this application achieves best performance in various weather scenarios, maintaining a relatively balanced performance in both perceptual quality and semantic consistency.

[0093] Qualitative comparison: In the visualization comparison experiment of real-world images, this application can better preserve scene structure and semantic information while removing degradation factors such as rain, snow and fog. Compared with the comparison method, it reduces phenomena such as over-enhancement, loss of detail and structural distortion, and shows a more stable restoration effect in complex real-world scenes.

[0094] Validation of Adaptability: In a constructed real-world video stream scenario, this application utilizes unlabeled continuous frames for online adaptive training and then tests it on subsequent unseen data. Experimental results show that the model can achieve performance improvement with short-term continuous data input and maintains relatively stable recovery performance in dynamically changing environments, validating the adaptability of this method in practical applications.

[0095] User Research: To further evaluate the subjective visual quality of the restored results, a comparative experiment based on human perception was designed. Twenty participants were invited to participate in the evaluation. Fifty images were randomly selected from a real-world test set, covering different degradation types such as rain, snow, and fog. For each test image, the restoration results corresponding to each method were shown to the participants in a randomized order without method names to avoid human bias. Participants selected the result they considered to be of higher quality based on visual effect. The proportion of times each method was selected out of the total number of selections was used as a preference rate index for statistical analysis. The experimental results show that the present invention achieved a high preference rate in subjective evaluation, reflecting its good performance in terms of perceived quality.

[0096] In summary, the above experiments demonstrate that this application can achieve stable and effective restoration of severe weather images without paired supervision, and exhibits good practicality in real-world scenarios and under continuous data input conditions.

[0097] (6) Advantages and beneficial effects In summary, compared with the prior art, the technical solution proposed in this application has at least the following advantages: 1) The learning signal is more stable and reliable: This application constructs a closed-loop collaborative mechanism between the recovery agent, the prompting agent and the evaluation agent, so that the generation and evaluation processes are coupled and dynamically updated. An adaptive learning signal is formed without the need to refer to the true value. Compared with methods that rely on fixed loss functions or predefined evaluation indicators, it can effectively reduce optimization bias and improve the stability of the training process and the consistency of the results.

[0098] 2) Achieving multi-dimensional restoration quality constraints: This application introduces a multi-layer preference modeling method based on perceptual quality, semantic consistency and structure preservation to comprehensively evaluate and optimize the image restoration results. This enables the model to improve image clarity while taking into account semantic information and geometric structure. Compared with methods that only focus on a single visual indicator, it can achieve a more balanced and reliable restoration effect.

[0099] 3) Enhanced real-world adaptability: This application does not rely on paired labeled data. It constructs effective learning signals from unlabeled real images through a multi-agent collaborative mechanism, enabling the model to directly adapt to complex and ever-changing harsh weather conditions in the real world. Compared with methods that rely on synthetic data for training, it has better practical application results.

[0100] 4) Support for continuous optimization and application expansion: This application uses a collaborative training strategy that combines online and offline methods to enable the model to be continuously updated and optimized during use. At the same time, this method can be extended to video input scenarios, thereby enhancing its applicability in practical systems.

[0101] In summary, this application achieves stable and effective severe weather image restoration without pairing supervision through multi-agent collaborative evolution and preference-driven optimization mechanisms, and its application performance in real-world scenarios is superior to existing technologies.

[0102] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0103] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0104] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0105] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0106] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0107] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0109] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for reconstructing severe weather images based on multi-agent co-evolution, characterized in that, The method includes the following steps: Obtain the degraded image to be processed; The degraded image is semantically modeled by a prompting agent to generate prompting information to guide the image restoration process; By using a recovery agent, multiple candidate recovery results are generated based on the degraded image and the prompt information; By evaluating the intelligent agent, a multi-dimensional quality assessment is performed on the multiple candidate recovery results, and a preference signal reflecting the relative superiority or inferiority among the candidate recovery results is generated based on the assessment results. Based on the preference signal, a reinforcement learning strategy is used to optimize and update the parameters of the recovery agent, and the data generated during the interaction is stored in the experience replay pool for offline optimization of the prompting agent and / or the evaluation agent, forming a closed-loop co-evolutionary mechanism of "generation-evaluation-optimization". The prompting agent and the evaluation agent are constructed based on a visual language model; the prompting information includes one or more of the following: a description of the degradation type, a description of the degradation degree, information about the affected area, and the target recovery requirements, and is generated through a chain-like reasoning process; The multi-dimensional quality assessment includes perceived quality assessment, semantic consistency assessment, and structural preservation assessment. The perception quality assessment integrates at least one no-reference image quality evaluation index with the perception score output by the evaluation agent. The semantic consistency assessment is achieved by comparing the semantic representation similarity between the degraded image and the candidate restoration result under the constraint of the prompt information. The structural preservation assessment is achieved by comparing the geometrical structural consistency between different candidate recovery results; The optimization and updating of the parameters of the recovering agent based on the preference signal using a reinforcement learning strategy specifically includes: The reward value corresponding to the preference signal is standardized to obtain the advantage function; Based on the probability ratio of the current policy to the old policy, a policy optimization objective function is constructed that includes pruning terms and KL divergence constraints to limit the policy update magnitude. The step of storing the data generated during the interaction process into an experience replay pool for offline optimization of the prompting agent and / or the evaluation agent specifically includes: During the online phase, the recovering agent is updated using the immediate preference signals of the current sample; During the offline phase, historical trajectory data is sampled from the experience replay pool, and offline rewards for the prompting agent and the evaluation agent are calculated respectively, and the parameters of the prompting agent and the evaluation agent are updated accordingly. The online and offline updates are executed asynchronously and alternately. The method also includes an adaptive control step based on a validation set: Calculate the change in performance metrics on the validation set; The learning rate of the reinforcement learning strategy is dynamically adjusted based on the changes in the performance metrics. When the performance metric declines for multiple consecutive stages, a model parameter rollback operation is performed to restore the model to its historical optimal state. It also includes the step of extending the method to continuous real-world video streaming scenarios: Continuous video frames input in chronological order are treated as an unlabeled data stream; Perform online adaptive updates on consecutive frames of data within the current time period to update the recovery agent; The updated recovery agent is used to process video frames in subsequent time periods, enabling the model to continuously evolve in a dynamic environment.

2. The method according to claim 1, characterized in that, The preference signal is a reward value generated by a weighted fusion of the evaluation results of perceived quality, semantic consistency and structural preservation, and is obtained by sorting the reward values ​​of multiple candidate recovery results.

3. The method according to claim 1, characterized in that, The recovery agent employs a conditional generation model, and the multiple candidate recovery results are generated by performing multiple independent samplings under the same input conditions.

4. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 3.

Images