An image editing and generation method and system based on reinforcement learning closed-loop feedback optimization

Abstract

This invention discloses an image editing and generation method and system based on reinforcement learning closed-loop feedback optimization, belonging to the field of image processing technology. The method includes acquiring a reference image, generating structured text editing instructions through an instruction policy network; obtaining a generated image based on the text editing instructions and the reference image; evaluating the generated image, calculating a feedback reward, and using the feedback reward to optimize the instruction policy network through reinforcement learning; the feedback reward includes a target attribute consistency reward and a task discriminative power reward based on a gating threshold. This invention significantly improves the semantic effectiveness and model generalization ability of generated images in complex, long-tail downstream tasks by constructing a complete closed loop of "editing-evaluation-optimization".

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image editing and generation method and system based on reinforcement learning closed-loop feedback optimization, and particularly to an image editing and generation method and system that combines a multimodal large language model, a diffusion generation model and reinforcement learning techniques. Background Technology

[0002] Image editing and generation is a core technology for computer vision and artificial intelligence content creation, and a key cornerstone for providing high-quality data for downstream visual tasks and building smart cities and intelligent regulatory systems.

[0003] However, in real-world visual applications, the performance of downstream models such as object detection and re-identification is highly dependent on the scale and diversity of the training data. Due to privacy protection, annotation costs, and physical limitations, data collected in real-world scenarios often exhibits a severe long-tail distribution. Specifically, this manifests as: data redundancy in common viewpoints and simple scenes; while "tail" samples—such as those for extreme occlusion, small targets, and densely overlapping samples for object detection, and rare viewpoints (side view, extreme upward view) and complex backgrounds (nighttime backlighting, rain, snow, fog, haze, complex sea conditions) for re-identification—are extremely scarce. This imbalance in data distribution directly limits the generalization ability of visual models when facing complex real-world scenes, making them highly susceptible to missed detections, false detections, or incorrect identity matching.

[0004] To alleviate the aforementioned problems of data scarcity and long-tail distribution, image editing and generation-based augmentation techniques have emerged. Traditional data augmentation methods (random erasure, cropping, color dithering) only involve pixel-level geometric or photometric transformations and cannot generate high-level semantic information (smooth viewpoint transitions or complex changes in ambient lighting), thus encountering a bottleneck in improving the performance of downstream tasks. In recent years, text-driven generative augmentation using large-scale diffusion models has become a research hotspot.

[0005] However, existing image editing and generative enhancement methods typically employ an "open-loop" approach. Specifically, these methods mostly rely on predefined heuristics or random sampling to generate prompts, then directly control the generator to synthesize images for downstream use. This open-loop approach has two core drawbacks:

[0006] 1. The "Instruction Gap": Existing image generators, such as ControlNet and StableDiffusion, while possessing powerful image rendering capabilities, lack awareness of the specific requirements of downstream tasks. The generator cannot autonomously determine which anchoring attributes in an image must be strictly preserved and which are non-critical environmental factors that can be freely changed. For example, images edited with random prompts are highly susceptible to attribute manipulation, causing the identity or category labels of downstream tasks to become invalid, thus introducing a large amount of harmful noise data.

[0007] 2. "Imitation Trap": Open-loop prompt generation based on heuristic strategies often tends to mimic the common distribution of source domain data. This means that the generator produces mostly "easy positives" that conform to conventional statistical laws, while it is difficult to actively explore and generate truly challenging "hard negatives" or "corner cases" located near the decision boundary of the downstream model. As a result, the generated high-fidelity images have very low actual effectiveness in broadening the model's discrimination boundary, causing a huge waste of computing power.

[0008] Therefore, how to construct a closed-loop image editing and generation method that can perceive the needs of downstream visual tasks and automatically learn "what should be edited and generated" through specific task performance feedback, so as to maximize the exploration of the diversity of long-tail features while accurately locking key attributes, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] In view of the above problems, this invention proposes an image editing and generation method and system based on reinforcement learning closed-loop feedback optimization, aiming to solve the problems of lack of downstream visual task feedback in existing image editing and generation technologies, resulting in low effectiveness of generated samples and limited generalization of downstream models. Specifically, by constructing a "edit-evaluation-optimization" closed loop, reinforcement learning algorithms are used to optimize the instruction strategy, enabling the generator to proactively generate high-fidelity, diverse, long-tailed samples that can significantly improve the performance of downstream tasks.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide an image editing and generation method based on reinforcement learning closed-loop feedback optimization, the steps of which include: Acquire a reference image and generate structured text editing instructions through an instruction policy network; An image is generated based on the text editing instructions and the reference image; The generated image is evaluated, a feedback reward is calculated, and the instruction policy network is optimized through reinforcement learning using the feedback reward. The feedback rewards include a target attribute consistency reward and a task discrimination reward based on a gating threshold.

[0011] Preferably, structured text editing instructions are generated through an instruction policy network, the steps of which include: Identify the global semantic attributes of the reference image and predict key anchoring attributes and non-key attributes; the global semantic attributes include at least scene background, global illumination, viewpoint and subject pose, and the key anchoring attributes include at least the texture, shape, markers or local imprints of a specific target; Based on the global semantic attributes, the key anchoring attributes, and the non-key attributes, text editing instructions containing attribute preservation constraints and attribute variation descriptions are generated.

[0012] Preferably, generating an image based on the text editing instructions and the reference image includes: The text editing instructions are mapped to adapted conditional embedding vectors using a text alignment adapter; The reference image and the conditional embedding vector are input into the diffusion model to obtain the generated image; The text alignment adapter is jointly optimized with the instruction policy network, and the diffusion model keeps its parameters frozen.

[0013] Preferably, the conditional inputs of the diffusion model include at least one of the following control parameters: diffusion steps, guidance coefficient, random seed strategy, editing intensity, or resolution.

[0014] Preferably, the feedback reward includes a target attribute consistency reward and a task discrimination reward introduced based on a gating threshold, including: The feature cosine similarity between the reference image and the generated image is used as the target attribute consistency reward; expressed as:

[0015] In the formula, Indicates the features of the reference image. Indicates the generated image features, When the consistency reward of the target attribute is greater than the gating threshold, a task discrimination reward is introduced. Specifically: At that time, total reward

[0016] At that time, total reward

[0017] in and The preset weighting coefficients, This is an indicator function; it takes the value 1 when the condition is true, and 0 otherwise.

[0018] Preferably, the gating threshold is adaptively updated based on at least one of the gating pass rate, reward distribution quantile, and validation set recognition performance.

[0019] Preferably, the task discrimination reward is determined in the following manner:

[0020] In the formula, Indicates the generated image features, This represents the negative sample in the current training batch that has the smallest distance to the generated image features. This represents the positive sample in the current training batch that has the largest distance to the generated image features. Represents distance metric, For marginal hyperparameters, Represents the hyperbolic tangent function. This represents a difficulty metric or adversarial loss function specific to downstream tasks.

[0021] Preferably, when optimizing the instruction policy network, a structure-aware entropy regularization term is introduced, the steps of which include: Obtain the cross-attention map of the generated image during the generation process; The sensitivity score of each word in the text editing instruction is calculated based on the cross-attention map, and each word is divided into key anchoring words and variation exploration words based on the sensitivity score. A first entropy coefficient is applied to the key anchoring word, and a second entropy coefficient is applied to the change exploration word, wherein the first entropy coefficient is smaller than the second entropy coefficient; The entropy regularization loss is calculated based on the first entropy coefficient and the second entropy coefficient, and then superimposed on the objective function of the reinforcement learning. The entropy regularization loss is expressed as:

[0022] In the formula, L represents the total length of the generated instruction sequence (i.e., the total number of tokens), t represents the t-th token in the current planning and sampling, and H( The entropy function () represents the information entropy calculation function, which measures the randomness and uncertainty of the policy network output distribution. The larger the entropy value, the stronger the exploratory nature of the model at that step; the smaller the entropy value, the more the model tends to output deterministic results. This represents the state input for the t-th word. The state... It contains multimodal contextual information, specifically including the features of the input reference image and the historical text instruction sequence generated by the first t-1 lexical units; Indicates the instruction policy network in state The action policy distribution is the probability distribution of each word in the prediction vocabulary of the semantic reasoning agent (SRA) being selected in the current step.

[0023] Preferably, the reinforcement learning algorithm includes Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), or variants thereof. The update process is based on the calculated feedback reward, performs relative advantage estimation on multiple sets of structured text editing instructions sampled from the same reference image, and introduces KL divergence constraints or reference policy constraints to limit the distribution shift between the updated instruction policy network and the baseline policy network, thereby improving training stability.

[0024] Preferably, the generated image undergoes quality screening before being added to the training set of the downstream task. The quality screening is based on at least one of the following: attribute consistency reward threshold, image sharpness, target subject area ratio, or generation failure marker.

[0025] Secondly, embodiments of the present invention provide an image editing and generation system based on reinforcement learning closed-loop feedback optimization, used to implement the image editing and generation method based on reinforcement learning closed-loop feedback optimization as described in any of the preceding claims, including: The instruction generation module is used to acquire reference images and generate structured text editing instructions through an instruction policy network; The image generation module is used to generate an image based on the text editing instructions and the reference image; The policy update module is used to evaluate the generated image, calculate the feedback reward, and use the feedback reward to optimize the instruction policy network through reinforcement learning; the feedback reward includes a target attribute consistency reward and introduces a task discrimination reward based on a gating threshold.

[0026] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image editing and generation method based on reinforcement learning closed-loop feedback optimization as described in any of the preceding claims.

[0027] The image editing and generation method and system based on reinforcement learning closed-loop feedback optimization provided in this invention can effectively prevent the instruction policy network from inducing the generation of noisy images that have lost the target subject by introducing a metric alignment gating reward mechanism. At the same time, by combining structure-aware entropy and dynamically adjusting the sampling randomness of different instruction terms through cross-attention maps, it achieves accurate local feature locking and global diversity exploration.

[0028] Compared with existing technologies, the specific beneficial effects include: 1. Closed-Loop Feedback and Task-Oriented Editing Optimization: This invention transforms traditional image data augmentation from a blindly stacking "open-loop generation" process to a task performance-oriented "closed-loop decision-making" process. By introducing real feedback signals from downstream visual tasks (such as object detection and re-identification networks) to evaluate the model, it directly drives and optimizes the instruction generation strategy. This allows the synthesized edited images to specifically fill the long-tail distribution blind spots in the training set, more effectively broadening the model's discrimination boundary in the feature space, thereby significantly improving the accuracy and generalization ability of downstream visual models when facing complex, long-tailed scenes.

[0029] 2. Highly Reliable Reward Design Balancing Security and Effectiveness: The metric-aligned gating reward (MAGR) mechanism proposed in this invention cleverly constructs an implicit curriculum learning process through the dynamic setting of attribute gating thresholds. This prompts the policy network to first learn "attribute preservation," and then, above this safety baseline, to further learn "hardness mining." This mechanism fundamentally overcomes the "reward hacking" problem induced by reinforcement learning in image generation due to the pursuit of feature distance alone, ensuring the semantic usability and task effectiveness of the generated data.

[0030] 3. Fine-grained local control balancing stability and diversity: The Structure-Aware Entropy (SAE) regularization mechanism proposed in this invention successfully overcomes the "stability-diversity trade-off" bottleneck in reinforcement learning for image editing and generation tasks by deeply mining the cross-attention map information within the image generation model. This mechanism can accurately prevent key anchor words (such as the structural outline of the subject, specific clothing texture, vehicle inspection stickers, or ship hull numbers) from being tampered with, while maximizing the generation richness of multiple dimensions such as environmental background, lighting conditions, and target pose, achieving fine-grained decoupling control that is difficult to achieve with traditional global entropy regularization.

[0031] 4. Computationally efficient and easily transferable modular architecture: This invention employs a strategy of freezing the core parameters of large-scale image generation models. This eliminates the need for costly fine-tuning of the massive generation base, requiring only updates to the instruction policy network and text alignment adapter, which have a very small number of parameters. This not only significantly reduces memory usage and time overhead during training but also gives the system architecture strong modular versatility. It can be easily plugged and played into various mainstream generator bases and can be quickly generalized to complex downstream visual tasks such as object detection and multi-object cross-camera re-identification at extremely low cost. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the overall process of the image editing and generation method based on reinforcement learning closed-loop feedback optimization provided in the embodiments of the present invention; Figure 2 A flowchart for hierarchical planning of the semantic reasoning agent provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the calculation logic for the metric alignment gating reward provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure-aware entropy mechanism provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] To address the technical challenges in existing image editing and generation technologies, such as "open-loop" blind generation, easy loss of key attributes, and insufficient exploration of long-tail high-value samples due to the lack of downstream visual task requirement awareness, this invention discloses an image editing and generation method and system based on reinforcement learning closed-loop feedback optimization. This method and system are applicable to scenarios such as intelligent monitoring, security retrieval, cross-camera target tracking, intelligent traffic analysis, and wide-area (e.g., maritime) surveillance. They can also be widely used for image semantic editing and high-fidelity sample synthesis. In particular, they provide high-quality, task-oriented training data to solve the problem of insufficient generalization ability of downstream visual task models (such as target detection and multi-target cross-camera re-identification models in complex scenes) caused by scarce training data and realistic long-tail distribution.

[0036] In one embodiment, the image editing and generation method based on reinforcement learning closed-loop feedback optimization includes the following steps: S1. Obtain a reference image and generate structured text editing instructions through an instruction policy network; S2. Obtain the generated image according to the text editing instructions and the reference image; S3. Evaluate the generated image, calculate the feedback reward, and use the feedback reward to optimize the instruction policy network through reinforcement learning; The feedback rewards include a target attribute consistency reward and a task discrimination reward based on a gating threshold.

[0037] The following section uses the target re-identification (ReID) scenario as an example to further describe the technical solution of the present invention in detail. Figure 1 This is a flowchart illustrating an image editing and generation method optimized based on reinforcement learning closed-loop feedback.

[0038] In some optional implementations, in S1, the instruction policy network uses a semantic reasoning agent based on a pre-trained visual language model (such as Qwen-VL, GPT-4V, or other open-source or commercial multimodal large-scale models) for parameter initialization. Given a reference image, the instruction policy network does not directly generate a simple image description, but instead executes a context-based hierarchical chain-like reasoning process to obtain text editing instructions. In this implementation, as... Figure 2 As shown, the reference image is first converted into Visual Tokens by a visual encoder, and then the semantic reasoning agent performs the following four progressive sub-steps: In the first stage, global perception is performed on the reference image to identify the global semantic attributes of the reference image. The global semantic attributes include at least the scene background, global illumination, viewpoint, and subject pose. For example, for pedestrians, it is "This is a back view, the lighting is dim, and it is in a complex street environment"; for vehicles, it is "This is a rear view, it is in backlight, and the background is a highway".

[0039] The second stage involves identifying and extracting tamper-proof key identity clues and predicting key anchoring attributes. Key anchoring attributes include at least the texture, shape, identifiers, or local markings of a specific target. For example, for a pedestrian, it would be "the target is wearing black sweatpants with white stripes and carrying a red backpack"; for a vehicle, it would be "the target has a silver roof rack, minor scratches on the left door, and a vehicle inspection sticker of a specific color on the upper right corner of the windshield". The third stage involves predicting non-critical attributes. Based on global perception and task requirements, it involves inferring which attributes are non-identity factors that are permissible and need to be changed to increase data diversity. For example, the target background can be changed to snow or rain to increase environmental robustness, the lighting can be changed to strong light, and the perspective should be changed to the side perspective that is scarce in the training set.

[0040] The fourth stage is instruction synthesis: based on the global semantic attributes, the key anchoring attributes, and the non-key attributes, a text editing instruction containing attribute preservation constraints and attribute variation descriptions is generated. For example, the instruction is "Keep the features of the red backpack and black sweatpants / roof rack and vehicle inspection sticker unchanged, replace the background with a snowy environment, and simulate the imaging effect of a side view."

[0041] As a preferred embodiment, the input to the above-mentioned instruction policy network includes a reference image and optional auxiliary information, which includes at least one of the following: task category or identity tag, acquisition device metadata, timestamp, or historical images of similar targets obtained by retrieval.

[0042] The purpose of introducing the aforementioned auxiliary information is to provide the Semantic Reasoning Agent (SRA) with multi-dimensional prior context constraints, thereby more accurately guiding the generation direction of editing instructions. Specifically: Task category or identity label: Provides explicit semantic anchors for the policy network. This enables it to bypass the ambiguity of pure visual understanding during chain-of-action (CoT) reasoning, quickly and accurately locking onto immutable "key anchor attributes" (such as explicitly telling the network that the current target is a "truck" rather than a "car," or a "specific pedestrian with identity ID 001"). This prevents semantic drift during instruction generation and avoids semantic ambiguity in complex contexts, ensuring that the instruction generation process can accurately distinguish between the "inherent attributes of a specific target" and "variable environmental attributes." This achieves efficient decoupling of attributes from the source and improves the category / identity fidelity of the generated image.

[0043] Collect device metadata (such as camera type, focal length, installation height, and angle): Provide physical space and viewpoint references for the policy network, introduce physical space mapping constraints into the generation process, enable the network to perceive visual distortion and perspective relationships, and generate viewpoint transformation instructions that conform to real optical and geometric laws (for example, based on the metadata of "drone camera", the network will tend to generate "top-down view" variation instructions), thereby improving the physical fidelity of long-tail view synthesis, solving the common spatial structure distortion problem in generated images, and making the augmented data more consistent with the distribution of actual engineering scenarios.

[0044] Timestamps provide environmental cues with temporal dimensions, including illumination, weather, and shadow distribution. The auxiliary policy network infers and generates reasonable illumination conditions or weather changes suitable for that time period (e.g., based on the "2 AM" timestamp, the network automatically plans and generates challenging long-tailed interference scenes such as "low-light nighttime," "vehicle headlight backlight," or "road glare"). By translating temporal priors into specific rendering instructions, long-tailed samples for specific time periods can be synthesized in a targeted manner, significantly enhancing the robustness of downstream models to all-weather environmental changes.

[0045] Historical images of similar targets retrieved from the database serve as multi-view few-shot visual cues, providing hidden surface details missing from the reference images during the generation process. When the reference image is in a single viewpoint or lacks key features due to occlusion, the historical images, acting as an "external knowledge base," can supplement the details of the target from other viewpoints (such as side lettering on a vehicle or a backpack on a pedestrian's back). This effectively suppresses the "hallucination" of the diffusion model when generating enhanced images with large-angle viewpoint changes, ensuring strict consistency of features across viewpoints.

[0046] As a preferred embodiment, the above text editing instructions are generated using a structured template, which includes at least: a constraint section, a change section, and an operation section; wherein, the constraint section is used to explicitly constrain the key anchoring attributes from being tampered with or lost during image editing and generation.

[0047] In this embodiment, this structured reasoning process ensures that the generated editing instructions have both high semantic diversity (effectively exploring long-tail distribution) and strict adherence to the identity consistency constraints required for the target re-identification task.

[0048] In some optional implementations, in S2, the generated image is synthesized based on the frozen generator. In some implementations, to ensure high fidelity of the generated image and reduce training costs, a pre-trained diffusion image generation model (such as a Stable Diffusion 3 or Flux model based on the MMDiT architecture) is used as the generator. And its parameters are completely frozen during training.

[0049] Since the instructions involved in ReID tasks typically contain dense attribute descriptions, directly inputting them into a general generator may lead to comprehension biases. Therefore, this embodiment introduces a lightweight text alignment adapter (TAR) to map text editing instructions to conditional embedding vectors that reflect generator preferences. Optionally, TAR uses a small multilayer perceptron (MLP) or Transformer layer.

[0050] Further, the reference image and the conditional embedding vector are input into the diffusion model to obtain the generated image; the generation process expression is:

[0051] in, Indicates text editing commands. TAR Indicates a text alignment adapter. Indicates a reference image. This indicates a generator.

[0052] In the subsequent reinforcement learning process, the text alignment adapter and the instruction policy network are jointly optimized, and the diffusion model keeps its parameters frozen.

[0053] Preferably, the conditional inputs of the diffusion model include at least one of the following control parameters: guiding coefficient, random seed strategy, editing intensity, diffusion steps, or resolution.

[0054] In traditional open-loop image editing techniques, the aforementioned parameters are typically set manually based on experience as static hyperparameters, making it difficult to adapt to the complex and ever-changing demands of downstream tasks. However, in the closed-loop reinforcement learning architecture of this invention, the aforementioned generation control parameters serve as an extended action space or dynamic adaptive condition for the policy instruction network (SRA). Their specific roles and the resulting synergistic technical effects are as follows: 1. The dynamic control effect of the guidance scale: The guidance scale determines the degree to which the generation process obeys the structured text editing instructions.

[0055] In this application, the guidance coefficient and the Measurement Alignment Gated Reward (MAGR) mechanism work in deep collaboration. When the downstream feedback of "specific anchor attribute subset consistency reward" is lower than the gating threshold (i.e., attribute drift occurs), the system can adaptively reduce the guidance coefficient; when it is necessary to increase the "task discriminative reward" to mine more difficult long-tail samples, the system can adaptively increase the guidance coefficient. The dynamic introduction of this parameter avoids the image oversaturation distortion caused by a fixed high guidance coefficient, as well as the failure of mutation instructions caused by a low guidance coefficient, thus achieving an optimal dynamic balance between text control and image naturalness.

[0056] 2. Decoupling control of Editing / Denoising Strength: In the image-to-image editing mode based on the reference image, it determines the proportion of initial noise injected into the reference image, that is, it determines the degree to which the features of the original image are destroyed and reconstructed.

[0057] This invention utilizes this parameter to decouple "editing intensity" from "attribute locking." Combined with a Structure-Aware Entropy (SAE) regularization term, for text instructions involving extreme scene changes (such as severe occlusion or inclement weather), the system can assign a high editing intensity to ensure the environment is fully rewritten; simultaneously, relying on SAE, it applies a very small entropy coefficient to "key anchor words," forcibly locking the generation trajectory of the target subject. This contradictory unity of "high editing intensity + strong local constraint" is impossible to achieve with traditional manual parameter tuning, greatly expanding the generation boundary of difficult samples.

[0058] 3. The diversity decoupling effect of the Random Seed Strategy: This strategy determines the initial Gaussian noise distribution at the starting point of diffusion denoising. This invention employs a controlled strategy combining "anchored seeds" and "mutated seeds".

[0059] In the advantage estimation process of reinforcement learning (such as multi-sample sampling in the GRPO algorithm), by using different "mutation seeds" to sample the same text instruction in batches, the system can accurately separate the "reward gain brought by the instruction policy" from the "reward fluctuations caused by pure noise randomness". This makes the gradient update of the policy network smoother and more stable, effectively preventing the non-convergence problem of RL algorithms in high-dimensional continuous action spaces such as image generation.

[0060] 4. The role of Diffusion Steps in balancing computational power and quality: The number of diffusion steps determines the number of iterations for denoising in the forward inference of the diffusion model.

[0061] By incorporating feedback from downstream tasks, the system can adaptively truncate the number of diffusion steps. For simple photometric transformation instructions, fewer diffusion steps are sufficient, significantly accelerating the iteration efficiency of reinforcement learning. However, for generating complex structures (such as small targets or complex occlusion relationships in object detection tasks), more diffusion steps are allocated to ensure the physical plausibility of semantic details, achieving intelligent coordination between computational overhead and generation quality.

[0062] 5. Multi-scale perception of resolution: used to control the spatial pixel dimensions of the generated image.

[0063] When serving downstream tasks that are highly sensitive to scale, such as object detection, the resolution parameter serves as an important control variable, enabling the generation of long-tail data containing either very small or very large objects. By introducing multi-resolution conditional inputs into reinforcement learning training, the generator's cross-scale semantic rendering capability can be effectively improved, thereby enhancing the robust recall rate of downstream detection models for multi-scale objects.

[0064] In summary, this invention does not simply list or call existing diffusion model parameters, but rather deeply reconstructs and links the above parameter group with the closed-loop feedback network (MAGR, SAE), thereby overcoming the shortcomings of existing technologies that rely on manual trial and error and cannot dynamically adapt to task difficulty.

[0065] In one alternative implementation, an alignment-gated reward is innovatively proposed to guide the instruction policy network to generate difficult samples that are both identity-preserving and discriminative. This reward is used to optimize the instruction policy network through reinforcement learning. In some implementations, this reward includes a target attribute consistency reward and introduces a task discriminative reward based on a gating threshold.

[0066] For example, refer to Figure 3 First, a frozen ReID feature extractor (such as ResNet50-IBN) independent of downstream tasks is used to extract features from the reference image. and generating image features .

[0067] The target attribute consistency reward, i.e., the identity maintenance reward, is calculated using the following formula:

[0068] In the formula, Indicates the features of the reference image. Indicates the generated image features, Simultaneously, calculate the reward for the discriminative ability of the measurement task. In this embodiment, a hard sample memory database is maintained to store the most difficult-to-distinguish negative samples in the current training batch. and positive samples ; The goal is to maximize the triplet distance margin when the generated sample is used as a positive sample, that is, to encourage the generated sample to be close to the center of positive samples and far away from negative samples in the feature space; the expression is:

[0069] In the formula, Indicates the generated image features, This represents the negative sample in the current training batch that has the smallest distance to the generated image features. This represents the positive sample in the current training batch that has the largest distance to the generated image features. Represents distance metric, For marginal hyperparameters, Represents the hyperbolic tangent function. This represents a difficulty metric or adversarial loss function specific to downstream tasks.

[0070] For example, when the downstream task is "target re-identification (ReID)," then... Typically, this is constructed based on triplet feature distance. The difficulty of re-identification lies in the large intra-class discrepancies (the same target doesn't look alike) and the small inter-class discrepancies (different targets look alike). When the downstream task is "object detection," then... It is usually constructed based on the detector's localization regression loss or intersection-over-union (IoU) loss. The difficulty in detection lies in the inaccuracy of bounding boxes caused by occlusion and small objects.

[0071] Finally, a gating mechanism is applied, that is, a gating threshold is set. In this embodiment, the gating threshold is a fixed threshold, preferably 0.5; or it is adaptively updated based on at least one of the gating pass rate, reward distribution quantiles, and validation set recognition performance. Furthermore, like The total reward At this point, only the loss of identity is punished, forcing the strategy to first learn to "draw the target object correctly".

[0072] like The total reward At this point, task rewards are activated to encourage strategies to generate more challenging samples; among them, and The preset weighting coefficients, This is an indicator function; it takes the value 1 when the condition is true, and 0 otherwise.

[0073] To further optimize the above technical solution and prevent strategy collapse (i.e., repeatedly generating identical instructions), this embodiment introduces a structure-aware entropy regularization term. However, a globally uniform entropy coefficient can lead to the following problems: increasing entropy for identity words (such as "red") can cause identity changes; not increasing entropy for background words can lead to insufficient diversity. Therefore, this embodiment proposes a structure-aware entropy mechanism, referring to... Figure 4 The specific steps are as follows: During the process of generating an image, the cross-attention map between each word in the text instruction and the image pixel is extracted; Spatial pooling is performed on the attention map of each word to obtain the sensitivity score of that word. High sensitivity means that the word corresponds to a salient region in the image (usually the main body region of a human body, ship, or vehicle). Based on the aforementioned sensitivity scores, each lexical unit is divided into key anchoring lexical units and variation exploration lexical units, and entropy coefficients are dynamically allocated. For example, a quantile threshold is set. ; for The token (identity key) is assigned a very small first entropy coefficient. This sharpens the output distribution and locks in identity features; for The token (environmental change term) is assigned a larger second entropy coefficient. This flattens out the output distribution and encourages diverse exploration.

[0074] Then, the entropy regularization loss is calculated based on the first entropy coefficient and the second entropy coefficient, and the entropy regularization loss is added to the objective function of the reinforcement learning. The global weight of the structure-aware entropy regularization term is dynamically adjusted during the training phase: the global weight is increased in the early stages of training to improve policy exploration capability, and decreased in the later stages of training to improve generation stability and target identity preservation capability.

[0075] In this embodiment, the entropy regularization loss is expressed as:

[0076] In the formula, L represents the total length of the generated instruction sequence (i.e., the total number of tokens), t represents the t-th token in the current planning and sampling, and H( The entropy function () represents the information entropy calculation function, which measures the randomness and uncertainty of the policy network output distribution. The larger the entropy value, the stronger the exploratory nature of the model at that step; the smaller the entropy value, the more the model tends to output deterministic results. This represents the state input for the t-th word. The state... It contains multimodal contextual information, specifically including the features of the input reference image and the historical text instruction sequence generated by the first t-1 lexical units; Indicates the instruction policy network in state The action policy distribution is the probability distribution of each word in the prediction vocabulary of the semantic reasoning agent (SRA) being selected in the current step.

[0077] This represents the dynamic entropy weight coefficient corresponding to the t-th word. The value of is dynamically allocated based on the sensitivity score of the cross-attention map obtained in the preceding steps. The specific mapping rule is as follows: when the t-th word is identified as a "key anchor word", The value is taken as the first entropy coefficient (using a smaller value to suppress randomness and lock in key attributes); when the t-th lexical is identified as a "change exploration lexical", The value is taken as the second entropy coefficient (using a larger value to amplify randomness and encourage the generation of diverse long-tail environments); the first entropy coefficient is strictly smaller than the second entropy coefficient.

[0078] Furthermore, reinforcement learning algorithms may include Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), or variations thereof. In some implementations, compared to traditional PPO algorithms, GRPO algorithms, by sampling multiple sets of instructions from the same reference target image and calculating the relative advantage within each set, can significantly reduce memory overhead and are more suitable for fine-tuning large generative models. For example, for the same reference image, the instruction-policy network samples and generates... Group different instructions Calculate the reward for each set of instructions. And calculate the relative advantage within the group:

[0079] in, Represents the mean function, It is the standard deviation function. It is a minimum value to prevent the denominator from being 0.

[0080] By utilizing this advantage function, combined with the aforementioned entropy regularization term and KL divergence constraint, the parameters of the instruction policy network and text alignment adapter are updated using gradient descent to limit the distribution offset between the updated instruction policy network and the baseline policy network, thereby improving training stability.

[0081] Through continuous iterative training, the instruction policy network gradually learns to plan enhancement strategies that both preserve identity and maximize the training benefits of the ReID model for different target images. This results in excellent performance on test sets (such as Market-1501 and MSMT17).

[0082] In one alternative implementation, the generated images undergo quality screening before being added to the training set of downstream tasks. The quality screening is based on at least one of the following: attribute consistency reward threshold, image sharpness, target subject area ratio, or generation failure marker.

[0083] Simultaneously, the generated images are used to train or fine-tune downstream task models, and are mixed with real images at a preset ratio to construct training batches; the preset ratio is a fixed ratio or a ratio that is dynamically adjusted according to the training rounds.

[0084] To verify the effectiveness of the method described in this embodiment, the generated enhanced target images were used to train the downstream re-identification model. Comparative verification experiments were conducted on mainstream target re-identification datasets (such as Market-1501). The experiments uniformly used the CLIP-ReID architecture as the evaluator and baseline model. Specific comparative experimental data are as follows: Baseline 1 (baseline model, no data augmentation): Trained using only the original Market-1501 training set. Testing yielded a mean average precision (mAP) of 88.2% and a Rank-1 accuracy of 94.5%.

[0085] Basis 2 (Comparison method: traditional "open-loop" random generation augmentation): Random prompts are used in conjunction with existing diffusion models to generate augmented data for the training set. Due to the lack of feedback from downstream tasks, a large amount of "key attribute drift" occurs during the generation process, introducing noisy data. Tests show that the model's mAP is 88.5%, and the Rank-1 accuracy is 94.8% (almost no significant improvement compared to the baseline).

[0086] Experimental data demonstrate that joint training with task-oriented, high-value long-tail data generated using this method significantly improves the model's mAP to 90.6% and Rank-1 accuracy to 96.2%, outperforming existing augmentation methods that employ "open-loop" random generation.

[0087] Addressing the Long-Tail Distribution Challenge: Thanks to the "mutation planning" capability of the SRA agent, this method can proactively generate scarce, difficult samples in the training set. Experimental tests show that the model improves the recognition accuracy by 6.3% when facing "Side View" targets and significantly improves the recognition accuracy by 6.7% when facing "Occluded" targets, perfectly verifying the technical superiority of the closed-loop mechanism of this invention in solving the problem of omissions in the recognition of complex, long-tail scenes.

[0088] Based on the same inventive concept, embodiments of the present invention provide an image editing and generation system based on reinforcement learning closed-loop feedback optimization. This system can be deployed as a standardized software toolkit in electronic devices containing high-performance computing units (such as GPUs / TPUs). Its architecture logically includes: The instruction generation module is used to acquire reference images and generate structured text editing instructions through an instruction policy network. In some implementations, a semantic reasoning agent (SRA) based on a multimodal large language model is configured to run the instruction policy network. This network is responsible for receiving the reference image to be edited and autonomously planning and outputting structured text editing instructions that balance "preservation of key anchoring attributes" and "variation of non-key attributes" through hierarchical chain-like reasoning. This includes: first, performing global semantic analysis on the reference image (including scene background, global illumination, viewpoint, and subject pose); then identifying key anchoring attributes that must be retained in the image (including subject outline, unique texture, landmarks, or local markings, such as a pedestrian's backpack, a vehicle's inspection sticker, or a ship's hull number); and finally planning the set of non-key attributes that can be modified, thereby generating structured text editing instructions that balance attribute preservation and environmental variation.

[0089] An image generation module is used to generate an image based on the text editing instructions and the reference image. In some implementations, it includes a lightweight text alignment adapter and a pre-trained, parameter-frozen diffusion generation model base. This base is responsible for mapping the structured text editing instructions into conditional embedding vectors, which, along with the reference image, are input into the diffusion model. The denoising process of the diffusion model is then used to synthesize the edited generated image, thereby efficiently rendering and synthesizing a high-fidelity generated image while ensuring low computational overhead. In this embodiment, the core parameters of the diffusion model are kept frozen throughout the training and inference phases. This not only fully utilizes its powerful pre-trained image rendering prior knowledge but also significantly reduces training computational overhead and effectively avoids the catastrophic forgetting phenomenon of the generator.

[0090] A policy update module is used to evaluate the generated image, calculate a feedback reward, and optimize the instruction policy network through reinforcement learning using the feedback reward. The feedback reward includes a target attribute consistency reward and a task discrimination reward based on a gating threshold. In some implementations, this module specifically includes a reward evaluation unit and an optimization unit. The reward evaluation unit is configured to run downstream task evaluation models, such as object detection models or object re-identification networks, to evaluate the generated images. It calculates the metric alignment gating reward (MAGR) signal through a hard sample mining algorithm and forms a closed-loop feedback. This gating mechanism can effectively prevent the "reward hacking" problem, in which the policy network guides the generator to synthesize noisy images that completely lose the key attributes of the subject in order to obtain a high exploration score.

[0091] The optimized unit configuration is used to perform reinforcement learning algorithms and structure-aware entropy (SAE) regularization calculations. That is, it receives the feedback reward signal, combines it with cross-attention map information, and dynamically and finely updates the policy parameters of the instruction generation module, thereby maximizing the exploration of the diversity of long-tail high-value samples while locking onto the target subject.

[0092] The principle by which the above module solves the problem in this embodiment is consistent with the steps in the aforementioned image editing and generation method based on reinforcement learning closed-loop feedback optimization. Therefore, the implementation details of this system can be found in the implementation of the aforementioned method, and the repeated parts will not be described again.

[0093] Furthermore, based on this system architecture, this embodiment provides a specific application implementation scheme in real intelligent transportation or security detection tasks, and the specific process is as follows: 1) Generation of instruction strategies for detection tasks: In this embodiment, in addition to the reference image, the input to the Semantic Reasoning Agent (SRA) also includes the original bounding box and category label. The SRA still uses Chain-of-Track (CoT) reasoning: Identify key anchoring attributes: Identify the target outline, core structure, and category features (such as "physical edge of a car" or "torso of a pedestrian") within the bounding box as unalterable anchoring attributes to ensure that the original bounding box labels remain valid after generation.

[0094] Long-tail mutation exploration planning: Targeting common pain points in object detection, SRA focuses on outputting mutation instructions related to environment and occlusion, such as "generate tree branches to block the view in front of the vehicle", "convert scene lighting to nighttime backlight" or "add dense fog weather".

[0095] 2) Spatial controlled freeze generator synthesis: The structured text editing instructions described above are input into the text alignment adapter and fed together with the reference image into the diffusion model with frozen parameters. During the denoising generation process, spatial control conditions are introduced through the text alignment adapter (such as introducing a controllable network ControlNet with the edge map or depth map of the original target as constraints) to ensure that the editing operation only changes non-critical environment and texture, while strictly maintaining the spatial position and bounding box of the target, thereby achieving lossless inheritance of detection labels.

[0096] 3) Measurement Alignment Gated Reward (MAGR) Feedback for Object Detection Specialization: The generated image is fed into a pre-trained object detection network, which serves as the evaluator, to calculate the MAGR reward specifically for the detection task: 1. Consistency reward for a specific subset of anchored attributes ( ): Extract the confidence level of the detection network's category prediction for the original bounding box region in the generated image. The higher the confidence level, the better the core category features of the target are preserved, and no semantic drift has occurred.

[0097] 2. Reward for task judgment ( ): Extract the localization loss of the detection network for the target (such as Bounding Box Regression Loss or IoU Loss). The larger the localization loss, the more successfully the generated occlusion or camouflage has deceived the detector, and the sample belongs to the high-value "hard sample".

[0098] 3. Gating mechanism execution: Only when the class prediction confidence ( Only when the value exceeds the set attribute gating threshold (i.e., ensuring the target remains in its original category) will the high localization loss be applied. This reward is included in the overall feedback. This mechanism perfectly prevents the generator from directly "erasing" the target to cheat and claim rewards for high positioning losses.

[0099] 4. Optimization of SAE strategy for bounding box awareness: When updating the SRA using the GRPO algorithm, the Structure-Aware Entropy (SAE) mechanism precisely controls the lexical units based on the cross-attention map: for the "key anchoring lexical units" corresponding to the target's main structure and category, a very small entropy coefficient is applied to lock their form; for the "change exploration lexical units" describing "occlusions", "rainy or snowy weather" or "shadows", a large entropy coefficient is applied to encourage the policy network to explore all sorts of detection interference terms, thereby continuously improving the robustness of the target detector in extreme environments in the closed loop.

[0100] In some other implementations, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image editing and generation method based on reinforcement learning closed-loop feedback optimization as described in any of the preceding claims.

[0101] Alternatively, an electronic device may be provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method described in Embodiment 1. The electronic device may be a high-performance server, workstation, or cloud computing node, typically equipped with a high-performance graphics processing unit (GPU) such as an NVIDIA A100 or H100 to support inference and training of large models.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0103] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image editing and generation method based on reinforcement learning closed-loop feedback optimization, characterized in that, include: Acquire a reference image and generate structured text editing instructions through an instruction policy network; An image is generated based on the text editing instructions and the reference image; The generated image is evaluated, a feedback reward is calculated, and the feedback reward is used to optimize the instruction policy network through reinforcement learning; The feedback rewards include a target attribute consistency reward and a task discrimination reward based on a gating threshold.

2. The method as described in claim 1, characterized in that, The steps for generating structured text editing instructions through an instruction policy network include: Identify global semantic attributes of reference images and predict key anchoring attributes and non-key attributes; Based on the global semantic attributes, the key anchoring attributes, and the non-key attributes, text editing instructions containing attribute preservation constraints and attribute variation descriptions are generated.

3. The method as described in claim 1, characterized in that, The generated image is obtained based on the text editing instructions and the reference image, including: The text editing instructions are mapped to adapted conditional embedding vectors using a text alignment adapter; The reference image and the conditional embedding vector are input into the diffusion model to obtain the generated image; The text alignment adapter is jointly optimized with the instruction policy network, and the diffusion model keeps its parameters frozen.

4. The method as described in claim 3, characterized in that, The conditional inputs of the diffusion model include at least one of the following control parameters: diffusion steps, guidance coefficient, random seed strategy, editing intensity, or resolution.

5. The method as described in claim 1, characterized in that, The feedback reward includes a target attribute consistency reward and a task discrimination reward based on a gating threshold, including: The feature cosine similarity between the reference image and the generated image is used as the target attribute consistency reward. When the target attribute consistency reward is greater than the gating threshold, a task discrimination reward is introduced.

6. The method as described in claim 1, characterized in that, The gating threshold is adaptively updated based on at least one of the gating pass rate, reward distribution quantile, and validation set recognition performance.

7. The method as described in claim 1, characterized in that, The task discrimination reward is determined in the following manner: In the formula, Indicates the generated image features, This represents the negative sample in the current training batch that has the smallest distance to the generated image features. This represents the positive sample in the current training batch that has the largest distance to the generated image features. Represents distance metric, For marginal hyperparameters, Represents the hyperbolic tangent function. This represents a difficulty metric or adversarial loss function specific to downstream tasks.

8. The method as described in claim 1, characterized in that, When optimizing the instruction policy network, a structure-aware entropy regularization term is introduced, and the steps include: Obtain the cross-attention map of the generated image during the generation process; The sensitivity score of each word in the text editing instruction is calculated based on the cross-attention map, and each word is divided into key anchoring words and variation exploration words based on the sensitivity score. A first entropy coefficient is applied to the key anchoring word, and a second entropy coefficient is applied to the change exploration word, wherein the first entropy coefficient is smaller than the second entropy coefficient; The entropy regularization loss is calculated based on the first entropy coefficient and the second entropy coefficient, and then the entropy regularization loss is added to the objective function of the reinforcement learning.

9. An image editing and generation system based on reinforcement learning closed-loop feedback optimization, characterized in that, An image editing and generation method based on reinforcement learning closed-loop feedback optimization as described in any one of claims 1-8, comprising: The instruction generation module is used to acquire reference images and generate structured text editing instructions through an instruction policy network; The image generation module is used to generate an image based on the text editing instructions and the reference image; The policy update module is used to evaluate the generated image, calculate the feedback reward, and use the feedback reward to optimize the instruction policy network through reinforcement learning; the feedback reward includes a target attribute consistency reward and introduces a task discrimination reward based on a gating threshold.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image editing and generation method based on reinforcement learning closed-loop feedback optimization as described in any one of claims 1-8.

Images