Multi-modal condition driven closed loop sensing and generation optimization method and device and medium

Abstract

The invention discloses a multi-modal condition-driven closed-loop sensing and generation optimization method and device and a medium, which are applied to an automatic driving long-tail scene, firstly, multi-modal conditions such as text description, a semantic map and a 3D layout are uniformly represented, a causal inference network is introduced, a causal structure between scene elements is explicitly inferred from multi-modal input, and a multi-modal condition-driven closed-loop sensing and generation optimization model is obtained. Generating structured causal embedding to realize causal perception enhancement of generation conditions; in a diffusion generation stage, a causal consistency mechanism is deeply fused into a condition control and denoising process, and a diffusion model is guided to effectively inhibit generation of unreasonable or common sense violating scenes in a generation process through an anti-fact condition constructed by a causal inference network, so that semantic reasonability, spatial consistency and dynamic credibility of long-tail data are remarkably improved. According to the method, the problem that long-tail scene data is deficient is solved, causal constraints are introduced to a generation source, and the reliability, robustness and cross-domain generalization ability of an automatic driving system in an extreme scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, generative models, artificial intelligence, and autonomous driving, and particularly to a multimodal condition-driven perception and generation closed-loop optimization method, apparatus, and medium. Background Technology

[0002] Generative models, with diffusion models as a typical example, have opened up new technical paths for addressing autonomous driving perception tasks in dynamic and complex environments. These models possess the ability to efficiently model complex data distributions, synthesizing highly realistic visual data and diverse traffic scenes, thus providing more comprehensive and covering training data support for perception models. They are particularly suitable for key tasks such as long-tail scene generation and high-fidelity simulation. In recent years, multi-source perception and controllable generation methods for autonomous driving based on diffusion models have gradually gained attention from academia and industry, and related research has achieved phased results. Leveraging the implicit representation learning and probability-driven generation mechanisms in diffusion models, research has initially achieved joint modeling of multimodal data such as LiDAR point clouds, visual images, semantic maps, and text descriptions, thereby promoting the synthesis and application of dynamic scenes. However, one of the core challenges of autonomous driving technology is dealing with the complexity and diversity of the real world, especially the long-tail problem of low-frequency but high-risk edge scenarios and the cross-scenario generalization problem of the adaptability of autonomous driving perception models in different environments. Therefore, how to construct a controllable long-tail scene generation driven by multimodal conditional coupling to solve the domain adaptation problem of perception models caused by the scarcity of difficult samples is a key scientific problem that this project needs to solve. This invention controls the generation of long-tail scenes through multimodal conditions and designs a perception-generation closed-loop optimization framework to further enhance the cross-scene generalization ability of perception models under difficult samples, which is a task of great practical significance. Summary of the Invention

[0003] Purpose of the invention: To provide a multimodal condition-driven closed-loop optimization method, device, and medium for perception and generation, addressing the problem of weak model generalization ability caused by the scarcity of long-tail data in autonomous driving scenarios. By designing a multimodal condition-driven closed-loop optimization framework for perception and generation, this invention investigates how to optimize the perception and generation process using multimodal condition control under conditions of data scarcity and uneven distribution, and achieves synergistic optimization of the perception and generation models through closed-loop iteration, ultimately enhancing the cross-scenario generalization ability of autonomous driving systems.

[0004] Technical Solution: The multimodal condition-driven perception and generation closed-loop optimization method described in this invention is applied to long-tail scenarios in autonomous driving and includes the following steps:

[0005] (1) The multimodal conditions are represented in a unified manner and fused into a unified condition vector;

[0006] (2) Introduce the scene causal inference network (CIN) and embed scene causal graph information into the multimodal conditional control to construct the causal perception conditional control space as the input of the generative model;

[0007] (3) Construct a diffusion noise scheduler CANS with causal consistency constraints, and introduce causal modulation noise parameters into the denoising network to realize the dynamic alignment of the diffusion process with the causal structure of the scene.

[0008] (4) During the diffusion generation process, a causal consistency noise scheduling and condition control mechanism is introduced to generate long-tail scenes that meet the multimodal conditions and causal structure constraints in a targeted manner.

[0009] (5) Perform feature difference and semantic consistency analysis on the generated results, introduce a semantic consistency discrimination mechanism, combine causal graph and text conditions to perform consistency verification and semantic completion of the generated scene, construct automatically labeled long-tail scene training samples, and use them for data augmentation and updating of the perception model;

[0010] (6) Test the model on cross-scenario datasets, and optimize the generated model through difference detection, problem localization and feedback generation to achieve dynamic causal calibration of the generation process.

[0011] Further, step (1) includes the following steps:

[0012] (11) Input the multi-view images into the pre-trained Segmentation Model (SAM) to obtain the semantic segmentation mask, and then obtain the semantic encoding features of n views through an encoder.

[0013] (12) The predefined BEV map semantic segmentation and 3D bounding box are used as 3D layout conditions. They are projected onto the two-dimensional image plane through camera parameters, and the 3D layout encoding features are obtained through an encoder. ;

[0014] (13) Use a pre-trained CLIP model to obtain text encoding features. ;

[0015] (14) All multimodal conditional controls are mapped to a d-dimensional feature space, and the image-related features are connected to obtain the conditional controls. .

[0016] Furthermore, step (2) includes the following steps:

[0017] (21) Extracting entity sets based on entity detection and semantic masking Inferring an initial causal relationship candidate set based on textual description information and 3D layout conditions. Using learnable scenario-based causal inference networks (CIN) in , Output the weighted edge set to obtain the causal graph. ,Will Mapped to a fixed-dimensional vector ;

[0018] (22) Image conditional control through attention mechanism Text conditional control and causal embedding The fusion process yields the final conditional control c.

[0019] Furthermore, step (3) includes the following steps:

[0020] (31) Construct a diffusion noise scheduler CANS with causal consistency constraints, and use the obtained causal embedding vector Through a noise weight mapping network Noise modulation coefficient for generating causal perception Used to dynamically adjust noise weights during the diffusion process. , ,in , These are the noise scheduling parameters adjusted for causal consistency.

[0021] (32) In the UNet denoising function, introduce causal adjustment noise parameters. , And combined with the dynamic condition gating module to generate time-related weights The image conditions, text conditions, and causal graph conditions are respectively used to construct the final time-step conditional control vector. This conditional vector is used as a control signal in the denoising network.

[0022] Furthermore, step (4) includes the following steps:

[0023] (41) Input a set of multi-view images x, which first pass through an encoder network Compress the image into a low-dimensional latent space representation. The causal consistency noise scheduler was used during the diffusion process. , Constructing the forward diffusion distribution of causal perception: The noise map is obtained after the diffusion process. ;

[0024] (42) To ensure that the scheduling process of forward diffusion noise is consistent with the structure of the scene causal graph, a causal embedding vector is used. Construct causal consistency constraints , is represented as: ,in To remove the conditional vector of causal embedding;

[0025] (43) Noise map After passing through the UNet denoising network and decoder module And combined with causal adjustment of noise parameters Perform reverse denoising to obtain the generated multi-view image. Integrating conditional denoising functions in the UNet model Allowed to accept various input formats The controlled synthesis process is performed under the influence of the objective function, which is expressed as: .

[0026] Furthermore, step (5) includes the following steps:

[0027] (51) Feature difference analysis, input generation domain With the target domain The difference in feature distribution of the intermediate layer features in the perception model is expressed as follows: ,in These are the characteristic mean and covariance matrix, respectively;

[0028] (52) Semantic difference analysis, category distribution of input perception model output Construct the KL divergence loss term for long-tail perception: The weighting coefficient Weighted by inverse frequency, it is expressed as: ;

[0029] (53) To prevent the generative model from generating pseudo-samples that are inconsistent with the semantics of the scene, a semantic consistency discriminator is introduced. Used to determine multi-view images Does it meet the text conditions? With scene causal graph The high-level semantic priors are obtained through reasoning, and the semantic consistency loss is calculated based on these priors. ,in Indicates from Extracted semantic graph, This represents the structured semantic prior graph inferred from the text and the causal graph. The overall objective function is expressed as: ,in , , It's a hyperparameter. It is a regularization term;

[0030] (54) Based on the obtained multi-view images Combined with predictive semantic graph With semantic prior graph The generated samples undergo consistency verification and semantic completion. When the generated samples meet the semantic consistency constraints, the system utilizes... and The fusion result automatically assigns pixel-level semantic labels and structured relationship annotations to the generated long-tail scene samples, forming an automatically annotated long-tail scene training sample set. ,in This is the final semantic annotation result. It is the set of structured causal relationship annotations explicitly recovered from the generated samples;

[0031] (55) The long-tail scene sample set obtained by automatic annotation Compared with real labeled datasets Mix them to build an enhanced training set. The enhanced training set is then used to train or fine-tune the perception model to obtain updated perception model parameters.

[0032] Furthermore, step (6) includes the following steps:

[0033] (61) Construct a difference sensitivity matrix for difference localization: ,in It is a semantic segmentation mask;

[0034] (62) The condition parameters of the generated model are corrected, and the adjustment amount is: ,in, The learning rate is denoted as .

[0035] The present invention discloses a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the multimodal condition-driven perception and generation closed-loop optimization method described above.

[0036] An electronic device according to the present invention includes a memory and a processor, wherein:

[0037] Memory is used to store computer programs that can run on a processor;

[0038] A processor, configured to, while running the computer program, execute the steps of the multimodal condition-driven perception and generation closed-loop optimization method as described above.

[0039] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: 1. Representation and Fusion of Multimodal Conditions: This invention unifies and deeply fuses multimodal condition signals such as text description, semantic map, and 3D layout, and introduces causal embedding vectors to explicitly model the causal dependencies between conditions, providing structured and interpretable control signals for the generative model; 2. Generation of Long-Tail Scenes Controlled by Multimodal Conditions: This invention precisely controls the physical rationality and semantic logic of the generated content by fusing multimodal condition signals and applying causal consistency constraints to the diffusion process, thereby efficiently generating high-quality, real-world causal data for challenging scenes; 3. Perception-Generation Closed-Loop Optimization and Cross-Scene Testing: This invention mixes the generated long-tail scene data with real data to train the next-generation perception model, and uses the trained perception model to perform cross-scene testing feedback on the generated data; this feedback not only evaluates the generation quality but also corrects the deviations in causal embedding and consistency constraints, thereby guiding the parameter optimization of the generative model; through this closed-loop iterative mechanism, the co-evolution of the perception and generation models is achieved, significantly improving the causal robustness and reliability of the system in extreme scenarios, thus overcoming the constraint of long-tail scene data scarcity on model performance. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the characterization and fusion of multimodal conditions;

[0041] Figure 2 It is a framework diagram for generating long-tailed scenes under multimodal condition control. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings.

[0043] like Figure 1 , Figure 2This invention proposes a multimodal condition-driven closed-loop optimization method for perception and generation. First, it unifies the representation of multimodal conditions such as text description, semantic map, and 3D layout, and introduces a Causal Inference Network (CIN) to explicitly infer the causal structure between scene elements from the multimodal input, generating structured causal embeddings and enhancing the causal perception of the generation conditions. Second, in the diffusion generation stage, a causal consistency mechanism is deeply integrated into the condition control and denoising process. Through counterfactual conditions constructed by the CIN, the diffusion model is guided to strictly follow the physical and semantic causal laws of the real world during the generation process, effectively suppressing unreasonable or common-sense-violating scene generation and significantly improving the semantic rationality, spatial consistency, and dynamic credibility of long-tail data. Then, the generated high-quality long-tail scenes are mixed with real data to train the next-generation perception model. Simultaneously, the perception model evaluates the causal rationality of the generation results in cross-scene testing, and the feedback signal is used to jointly optimize the CIN and the generation model, forming a collaborative iterative mechanism of "causal modeling—controllable generation—closed-loop verification". Ultimately, the closed-loop optimization framework not only solves the problem of data scarcity in long-tail scenarios but also introduces causal constraints at the generation source, significantly improving the reliability, robustness, and cross-domain generalization ability of the autonomous driving system in extreme scenarios. The specific implementation process is as follows:

[0044] Step 1: As Figure 1 As shown, multimodal conditions are uniformly represented and fused into a unified condition vector.

[0045] Multi-view images are input into a pre-trained semantic segmentation model (SAM) to obtain semantic segmentation masks, which are then passed through an encoder to obtain semantic encoded features for n views. The predefined BEV map semantic segmentation and 3D bounding boxes are used as 3D layout conditions. These conditions are projected onto a 2D image plane using camera parameters, and the 3D layout encoding features are obtained through an encoder. Text encoding features are obtained using a pre-trained CLIP model. All multimodal conditional controls are mapped to a d-dimensional feature space, and the image-related features are concatenated to obtain the conditional controls. .

[0046] Step 2: Introduce the Scene Causal Inference Network (CIN) to embed scene causal graph information into the multimodal conditional control, construct the causal perception conditional control space, and use it as the input to the generative model.

[0047] Entity set extraction based on entity detection and semantic masking Inferring an initial causal relationship candidate set based on textual description information and 3D layout conditions. Using Learnable Contextual Causal Inference Networks (CIN) in , Output the weighted edge set to obtain the causal graph. ,Will Mapped to a fixed-dimensional vector Introducing CIN provides causal priors to the conditional vectors, enabling the model to understand long-tail scenarios at the "event-entity-relationship" level. This helps generate reasonable causal structures, such as rear-end collisions, occlusions, and sudden crossings.

[0048] Image conditional control through attention mechanism Text conditional control and causal embedding The fusion process yields the final conditional control c.

[0049] Step 3: Construct a causal consistency constraint-based diffusion noise scheduler (CANS) and introduce causal modulation noise parameters into the denoising network to achieve dynamic alignment of the diffusion process with the causal structure of the scene.

[0050] Embedding causality into vectors Through a noise weight mapping network Noise modulation coefficient for generating causal perception Used to dynamically adjust noise weights during the diffusion process. , ,in , These are the noise scheduling parameters adjusted for causal consistency. The role of CANS is to maintain the consistency of scene generation with the causal structure during the diffuse noise injection stage and to suppress scene combinations that do not conform to causal relationships.

[0051] In the UNet denoising function, a causal adjustment noise parameter is introduced. , And combined with the dynamic condition gating module to generate time-related weights The image conditions, text conditions, and causal graph conditions are respectively used to construct the final time-step conditional control vector. This conditional vector will be used as a control signal in the denoising network below.

[0052] Step 4: As Figure 2 As shown, during the diffusion generation process, a causal consistency noise scheduling and condition control mechanism is introduced to generate long-tailed scenes that meet multimodal conditions and causal structure constraints in a targeted manner.

[0053] Input a set of multi-view images x, which are first processed by an encoder network. Compress the image into a low-dimensional latent space representation. The causal consistency noise scheduler was used during the diffusion process. , Constructing the forward diffusion distribution of causal perception: The noise map is obtained after the diffusion process. .

[0054] To ensure that the scheduling process of forward diffusion noise remains consistent with the structure of the scene's causal graph, this invention is based on causal embedding vectors. Construct causal consistency constraints , is represented as: ,in The conditional vector to remove causal embeddings.

[0055] noise map After passing through the UNet denoising network and decoder module And combined with causal adjustment of noise parameters Perform reverse denoising to obtain the generated multi-view image. Integrating conditional denoising functions in the UNet model Allowed to accept various input formats The controlled synthesis process is performed under the influence of the objective function, which is expressed as: .

[0056] Step 5: Perform feature difference and semantic consistency analysis on the generated results, introduce a semantic consistency discrimination mechanism, combine causal graph and text conditions to perform consistency verification and semantic completion of the generated scene, construct automatically labeled long-tail scene training samples, and use them for data augmentation and updating of the perception model.

[0057] Feature difference analysis: Input generation domain With the target domain The difference in feature distribution of the intermediate layer features in the perception model is expressed as follows: ,in These are the characteristic mean and covariance matrix, respectively.

[0058] Semantic difference analysis: Category distribution of input-aware model output Construct the KL divergence loss term for long-tail perception: The weighting coefficient Weighted by inverse frequency, it is expressed as: .

[0059] To prevent the generative model from generating pseudo-samples that are inconsistent with the semantics of the scene, a semantic consistency discriminator is introduced. Used to determine the generated image Does it meet the text conditions? With scene causal graph The high-level semantic priors are obtained through reasoning, and the semantic consistency loss is calculated based on these priors. ,in Indicates from Extracted semantic graph, This represents the structured semantic prior graph inferred from the text and the causal graph. The overall objective function is expressed as: ,in , , It's a hyperparameter. It is a regularization term.

[0060] Based on generating multi-view images And combined with the obtained predictive semantic graph With semantic prior graph The generated samples undergo consistency verification and semantic completion. When the generated samples meet the semantic consistency constraints, the system utilizes... and The fusion result automatically assigns pixel-level semantic labels and structured relationship annotations to the generated long-tail scene samples, forming an automatically annotated long-tail scene training sample set. ,in This is the final semantic annotation result. It is the set of structured causal relationship annotations explicitly recovered from the generated samples.

[0061] The long-tail scene sample set obtained by automatic annotation Compared with real labeled datasets Mix them to build an enhanced training set. The enhanced training set is then used to train or fine-tune the perception model, resulting in updated perception model parameters. The updated perception model is then used in cross-scene testing to evaluate the semantic rationality and causal consistency of generated samples. The feedback results are used for subsequent difference detection and generation model optimization, providing a foundation for the perception-generation closed-loop iteration.

[0062] Step 6: Test the model on a cross-scenario dataset, and optimize the generated model through difference detection, problem localization and feedback generation to achieve dynamic causal calibration of the generation process.

[0063] Construct a difference sensitivity matrix for difference localization: ,in It is a semantic segmentation mask.

[0064] The conditional parameters of the generated model are adjusted by the following amount: ,in The learning rate is denoted as .

[0065] Finally, the model was tested on a cross-scene dataset. Through feature statistical moment matching and semantic distribution reweighting, an effective framework was provided for the closed-loop optimization of perception-generation.

[0066] The present invention also provides a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the multimodal condition-driven perception and generation closed-loop optimization method as described above.

[0067] The present invention also provides an electronic device, including a memory and a processor, wherein: the memory is used to store a computer program that can run on the processor; the processor is used to execute, when running the computer program, the steps of the multimodal condition-driven perception and generation closed-loop optimization method as described above.

[0068] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal condition-driven closed-loop optimization method for perception and generation, applied to long-tail scenarios of autonomous driving, characterized in that, Includes the following steps: (1) The multimodal conditions are represented in a unified manner and fused into a unified condition vector; (2) Introduce the scene causal inference network (CIN) and embed scene causal graph information into the multimodal conditional control to construct the causal perception conditional control space as the input of the generative model; (3) Construct a diffusion noise scheduler CANS with causal consistency constraints, and introduce causal modulation noise parameters into the denoising network to realize the dynamic alignment of the diffusion process with the causal structure of the scene. (4) During the diffusion generation process, a causal consistency noise scheduling and condition control mechanism is introduced to generate long-tail scenes that meet the multimodal conditions and causal structure constraints in a targeted manner. (5) Perform feature difference and semantic consistency analysis on the generated results, introduce a semantic consistency discrimination mechanism, combine causal graph and text conditions to perform consistency verification and semantic completion of the generated scene, construct automatically labeled long-tail scene training samples, and use them for data augmentation and updating of the perception model; (6) Test the model on cross-scenario datasets, and optimize the generated model through difference detection, problem localization and feedback generation to achieve dynamic causal calibration of the generation process.

2. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (1) includes the following steps: (11) Input the multi-view images into the pre-trained Segmentation Model (SAM) to obtain the semantic segmentation mask, and then obtain the semantic encoding features of n views through an encoder. (12) The predefined BEV map semantic segmentation and 3D bounding box are used as 3D layout conditions. They are projected onto the two-dimensional image plane through camera parameters, and the 3D layout encoding features are obtained through an encoder. ; (13) Use a pre-trained CLIP model to obtain text encoding features. ; (14) All multimodal conditional controls are mapped to a d-dimensional feature space, and the image-related features are connected to obtain the conditional controls. .

3. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (2) includes the following steps: (21) Extracting entity sets based on entity detection and semantic masking Inferring an initial causal relationship candidate set based on textual description information and 3D layout conditions. Using learnable scenario-based causal inference networks (CIN) in , Output the weighted edge set to obtain the causal graph. ,Will Mapped to a fixed-dimensional vector ; (22) Image conditional control through attention mechanism Text conditional control and causal embedding The fusion process yields the final conditional control c.

4. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (3) includes the following steps: (31) Construct a diffusion noise scheduler CANS with causal consistency constraints, and use the obtained causal embedding vector Through a noise weight mapping network Noise modulation coefficient for generating causal perception Used to dynamically adjust noise weights during the diffusion process. , ,in , These are the noise scheduling parameters adjusted for causal consistency. (32) In the UNet denoising function, introduce causal adjustment noise parameters. , And combined with the dynamic condition gating module to generate time-related weights The image conditions, text conditions, and causal graph conditions are respectively used to construct the final time-step conditional control vector. This conditional vector is used as a control signal in the denoising network.

5. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (4) includes the following steps: (41) Input a set of multi-view images x, which first pass through an encoder network Compress the image into a low-dimensional latent space representation. The causal consistency noise scheduler was used during the diffusion process. , Constructing the forward diffusion distribution of causal perception: The noise map is obtained after the diffusion process. ; (42) To ensure that the scheduling process of forward diffusion noise is consistent with the structure of the scene causal graph, a causal embedding vector is used. Construct causal consistency constraints , is represented as: ,in To remove the conditional vector of causal embedding; (43) Noise map After passing through the UNet denoising network and decoder module And combined with causal adjustment of noise parameters Perform reverse denoising to obtain the generated multi-view image. Integrating conditional denoising functions in the UNet model Allowed to accept various input formats The controlled synthesis process is performed under the influence of the objective function, which is expressed as: .

6. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (5) includes the following steps: (51) Feature difference analysis, input generation domain With the target domain The difference in feature distribution of the intermediate layer features in the perception model is expressed as follows: ,in These are the characteristic mean and covariance matrix, respectively; (52) Semantic difference analysis, category distribution of input perception model output Construct the KL divergence loss term for long-tail perception: The weighting coefficient Weighted by inverse frequency, it is expressed as: ; (53) To prevent the generative model from generating pseudo-samples that are inconsistent with the semantics of the scene, a semantic consistency discriminator is introduced. Used to determine multi-view images Does it meet the text conditions? With scene causal graph The high-level semantic priors are obtained through reasoning, and the semantic consistency loss is calculated based on these priors. ,in Indicates from Extracted semantic graph, This represents the structured semantic prior graph inferred from the text and the causal graph. The overall objective function is expressed as: ,in , , It's a hyperparameter. It is a regularization term; (54) Based on the obtained multi-view images Combined with predictive semantic graph With semantic prior graph The generated samples undergo consistency verification and semantic completion. When the generated samples meet the semantic consistency constraints, the system utilizes... and The fusion result automatically assigns pixel-level semantic labels and structured relationship annotations to the generated long-tail scene samples, forming an automatically annotated long-tail scene training sample set. ,in This is the final semantic annotation result. It is the set of structured causal relationship annotations explicitly recovered from the generated samples; (55) The long-tail scene sample set obtained by automatic annotation Compared with real labeled datasets Mix them to build an enhanced training set. The enhanced training set is then used to train or fine-tune the perception model to obtain updated perception model parameters.

7. The multimodal condition-driven perception and generation closed-loop optimization method according to claim 1, characterized in that, Step (6) includes the following steps: (61) Construct a difference sensitivity matrix for difference localization: ,in It is a semantic segmentation mask; (62) The condition parameters of the generated model are corrected, and the adjustment amount is: ,in, The learning rate is denoted as .

8. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by at least one processor, implements the steps of the multimodal condition-driven perception and generation closed-loop optimization method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, Includes memory and processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to, while running the computer program, perform the steps of the multimodal condition-driven perception and generation closed-loop optimization method as described in any one of claims 1 to 7.

Images