Multi-modal painting and calligraphy generation method based on multi-dimensional verifiable reinforcement learning

Abstract

The invention discloses a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning, and the method comprises the steps: receiving multi-modal input which comprises a text, voice or image prompt, and extracting a semantic vector through a multi-modal encoder; in a painting and calligraphy generation process, a Step-aware-GRPO step-by-step reward mechanism is adopted to carry out reward evaluation and feedback on a generated intermediate step result, and the problem of reward sparsity is relieved; constructing a multi-dimensional verifiable reward signal based on RLVR, wherein the coverage dimensions comprise semantic consistency, aesthetic quality, character readability, target attribute matching and artistic style integrating degree; an SARRB-MoE mechanism is adopted, appropriate reward dimensions are dynamically selected and fused according to different generation stages and task types, and self-adaptive multi-reward optimization is achieved; and performing reinforcement learning training on the generative model based on the reward feedback until a generation result reaches a preset standard on the multi-dimensional evaluation index. The training efficiency, the generation quality and the artistic style integrating degree of the multi-modal painting and calligraphy generation model can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal generation technology, in particular to a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning. BACKGROUND

[0002] As an important branch of AI Generated Content (AIGC), calligraphy and painting generation has wide application value in digital art creation, cultural heritage, intelligent design, and human-computer interaction. Traditional generation methods are usually based on single-modal input (such as text prompts) and rely on pre-trained diffusion models or generative adversarial networks (GAN) for training. These methods can generate works of a certain quality under a single modality, but in complex multi-modal scenarios (such as combining text, speech, and image prompts), single-modal input often fails to fully express semantic intent, limiting the accuracy and artistic quality of the generated results.

[0003] With the development of multi-modal learning, multi-modal calligraphy and painting generation has gradually become an effective way of artistic creation. By integrating text, speech, and image information, models can achieve better results in semantic understanding and artistic expression. However, in practical applications, how to effectively integrate different modal information, how to dynamically adjust the weights between modalities, and how to balance the consistency of artistic style are still difficult points in current technology.

[0004] The inventors have found that the related art has at least the following technical problems:

[0005] Most current multi-modal generation methods use simple concatenation, weighted averaging, or deep neural networks (such as convolutional neural networks, transformers) for modality fusion. These methods usually rely on manually designed rules or fixed parameters, lack adaptive adjustment mechanisms, and are difficult to cope with dynamic changes in modality weights under different task types or prompt conditions. In addition, such methods have obvious shortcomings in reward design:

[0006] On the one hand, most existing methods only provide a single-dimensional reward signal (such as CLIP score or aesthetic score) after generation is complete, and the reward is sparse, making it difficult to provide effective guidance for the intermediate generation process;

[0007] On the other hand, a single-dimensional reward cannot fully reflect the multi-faceted quality of calligraphy and painting generation works, ignoring key factors such as semantic consistency, text readability, target attribute matching, and artistic style fit, which can easily cause overfitting of the model on a certain indicator.

[0008] In recent years, reinforcement learning (RL) has been introduced into the task of generating images from text, to optimize the generation model through reward signals. However, existing reinforcement learning methods still face the following shortcomings in multi-modal calligraphy and painting generation: first, most methods lack a step-by-step reward mechanism, making it difficult to address the training instability problem caused by sparse rewards; second, the construction and utilization of multi-dimensional reward signals are insufficient, and reward fusion mostly stays at the level of simple weighting, failing to adaptively adjust the focus of different generation stages; third, there is a lack of effective expert feedback and verifiable mechanisms, resulting in deficiencies in the consistency of style and artisticity of the generated results. SUMMARY

[0009] The purpose of the present application is to address the shortcomings of existing multi-modal generation methods in terms of reward sparsity, single reward, and insufficient fusion, and to propose a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning.

[0010] Some embodiments of the present application provide a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning, comprising the following steps: S1, receiving multi-modal input, including text, speech, or image prompts, and extracting semantic vectors through a multi-modal encoder; S2, during the calligraphy and painting generation process, using a Step-aware-GRPO step-by-step reward mechanism to reward and evaluate and provide feedback on the intermediate step results of generation, to alleviate the problem of reward sparsity; S3, constructing a multi-dimensional verifiable reward signal based on RLVR, covering dimensions including semantic consistency, aesthetic quality, text readability, target attribute matching, and artistic style fit; S4, using a SARRB-MoE mechanism to dynamically select and fuse appropriate reward dimensions according to different generation stages and task types, to achieve adaptive multi-reward optimization; S5, based on the reward feedback described above, reinforcing learning training of the generation model until the generated results meet the preset standards in terms of multi-dimensional evaluation indicators.

[0011] The specific implementation of the present application is as follows: a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning, characterized in that the method first extracts features from text, speech, and image inputs to form a multi-modal semantic feature set; then, based on Step-aware-GRPO, provides step-by-step rewards for intermediate results during the generation process to avoid reward sparsity; next, designs multi-dimensional verifiable rewards using the RLVR approach, including semantic consistency, aesthetic quality, text readability, target attribute matching, and artistic style fit; subsequently, through the SARRB-MoE reward routing mechanism, dynamically selects the optimal reward dimension for fusion according to the generation stage; finally, uses the GRPO strategy optimization method to train the generation model, and continuously iterates under the joint guidance of multi-dimensional reward signals until the generated results meet the preset requirements for semantics, artistry, and style, to achieve high-quality multi-modal calligraphy and painting generation.

[0012] The method realizes efficient, stable and artistic style matching calligraphy and painting generation by introducing a step-by-step reward mechanism in the generation process, constructing a multi-dimensional verifiable reward signal, and combining an expert mixed dynamic routing mechanism. Traditional multi-modal generation methods usually rely on fixed weights or a single indicator when dealing with modal fusion and reward design. However, the present application can dynamically adjust the reward dimensions and modal weights through reinforcement learning technology, overcoming this deficiency.

[0013] The present application learns the optimal generation strategy from multi-modal input and verifiable reward signal to improve the quality and stability of calligraphy and painting generation. Specifically, first, the input from text, speech or image prompts is feature extracted to form a multi-modal semantic vector; then, a Step-aware-GRPO step-by-step reward mechanism is introduced in each step of calligraphy and painting generation to evaluate the intermediate results and alleviate the reward sparsity problem; next, an RLVR-based multi-dimensional reward system is constructed to cover semantic consistency, aesthetic quality, text readability, target attribute matching and artistic style matching, and the corresponding scores are calculated through a verifiable model; subsequently, an expert mixed mechanism based on step-by-step reward routing (SARRB-MoE) is used to emphasize semantics and layout in the early stage of generation, highlight attribute and text readability in the middle stage, and focus on aesthetics and style performance in the later stage, realizing dynamic fusion of reward signals; finally, the generation model is iteratively updated using the GRPO strategy optimization method until the generated work meets the preset standards in multiple dimensions.

[0014] The advantages of the present application are: combining the step-by-step reward mechanism to solve the reward sparsity problem; breaking through the limitations of single reward through multi-dimensional verifiable reward signal to realize comprehensive optimization of generated works in terms of semantics, aesthetics, text and style; the expert mixed mechanism based on step-by-step reward routing realizes dynamic adaptation of reward signals, effectively improving the stability and artistic expression of the generation results. Compared with existing multi-modal generation methods, the present application no longer relies on fixed reward weighting rules, but dynamically adjusts the reward and modal weights through intelligent learning, thereby improving the adaptability and generalization ability of the system.

[0015] Compared with the prior art, the present application has the following beneficial technical effects and significant technical progress:

[0016] 1) The present application proposes a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning, which introduces a step-by-step reward mechanism and a dynamic reward routing strategy, and can adaptively adjust the weights of each modal input and reward dimension at different generation stages, thereby significantly improving the stability of the calligraphy and painting generation process and the overall quality of the generated works.

[0017] 2) The application constructs a multi-dimensional verifiable reward system covering semantic consistency, aesthetic quality, text readability, target attribute matching, and artistic style fit, breaking through the limitations of single reward signal, and can comprehensively evaluate the multi-aspect characteristics of generated works, effectively avoiding overfitting of the model on a single indicator.

[0018] 3) The application combines the expert mixed mechanism (SARRB-MoE), emphasizes semantics and layout in the early stage, highlights attributes and text readability in the middle stage, and focuses on aesthetics and style performance in the later stage, realizes dynamic adaptive fusion of reward signals, and improves the artistic and style consistency of the generated results.

[0019] 4) The application only needs limited strategy optimization and multi-dimensional reward injection in the generation model, reduces the dependence on large-scale labeled data and artificial rule design, and significantly improves the robustness and generalization ability of the system under complex tasks and diversified input conditions. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 An exemplary flowchart of a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning is provided for some embodiments of the application.

[0021] Figure 2 A specific operation diagram of Embodiment One of the application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme of the embodiments of the application will be described clearly and completely in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0023] In this paper, the following terms are used:

[0024] Step-aware-GRPO, in Chinese, step-aware group relative policy optimization;

[0025] OCR, full name Optical Character Recognition, in Chinese, Optical Character Recognition.

[0026] The present application aims to solve the deficiencies of the prior art in multi-modal calligraphy and painting generation methods, especially the challenges in sparse rewards, single rewards, and insufficient reward fusion. To this end, the present application proposes a multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning, which combines a step-by-step reward mechanism, a multi-dimensional reward signal construction, and an expert hybrid routing mechanism to improve the quality, stability, and artistic expression of calligraphy and painting generation. First, the present application extracts features from input data from different modalities (such as text prompts, voice descriptions, and reference images) to generate corresponding multi-modal semantic feature sets. These feature sets serve as input data for subsequent generation and optimization processes. Next, during the calligraphy and painting generation process, a Step-aware-GRPO step-by-step reward mechanism is used. At each intermediate step (such as low-resolution sketches, stroke outlines, or local layouts) of the generation process, the generated results are evaluated for rewards. This mechanism alleviates the sparsity problem of only giving rewards at the final result by injecting reward signals at multiple stages of generation, allowing the model to learn and optimize more stably. Then, an RLVR-based multi-dimensional verifiable reward system is constructed. This reward system is automatically calculated by a verifiable model, which includes the following components: a cross-modal alignment model to calculate the semantic consistency score between the generated calligraphy and painting and the input semantics; an aesthetic evaluation model to calculate the aesthetic quality score of the work; an OCR module to calculate the readability score of the text in calligraphy works; a target detection and attribute discrimination module to calculate the matching degree of target elements and attributes in paintings; and a style discrimination model to calculate the artistic style compatibility score of calligraphy or painting. Subsequently, a step-by-step reward routing-based expert hybrid mechanism (SARRB-MoE) is introduced. At different stages of generation, the system dynamically selects appropriate reward dimensions for fusion: in the early stages of generation, the system prioritizes routing semantic consistency and layout-related rewards; in the middle stages of generation, the system dynamically fuses attribute matching and text readability rewards; and in the later stages of generation, the system focuses on combining aesthetic and style-related rewards. This mechanism uses a gating network and a soft routing strategy to achieve weighted combination between different reward experts, thereby achieving adaptive fusion of reward signals. Finally, a GRPO-based policy optimization method is used. The system maps the above multi-dimensional reward signals to a weighted gradient target, updates and optimizes the generation strategy. In each training round, the model iteratively optimizes under the guidance of multi-dimensional rewards, gradually improving the comprehensive quality of the generated works in terms of semantic consistency, aesthetic expression, text readability, and style compatibility until the generated results meet the preset standards. Through the above steps, the method of the present application can effectively alleviate the problem of sparse rewards, avoid overfitting caused by single rewards, and dynamically adapt the rewards at different stages, ultimately significantly improving the accuracy, stability, and artistic expression of multi-modal calligraphy and painting generation.

[0027] Refer to the accompanying Figure 1The multi-modal calligraphy and painting generation method based on multi-dimensional verifiable reinforcement learning specifically comprises the following steps:

[0028] S1, receiving multi-modal input including text, voice or image prompts, and extracting semantic vectors through a multi-modal encoder;

[0029] S2, in the calligraphy and painting generation process, a Step-aware-GRPO step-by-step reward mechanism is used to reward and evaluate and feedback the intermediate step results generated, to alleviate the reward sparsity problem;

[0030] S201, extracting intermediate features or low-resolution results at each generated sampling step;

[0031] S202, calculating a reward signal for the intermediate results and returning to the policy optimization process;

[0032] S203, realizing step-by-step reinforcement generation training through distributed policy gradient updating.

[0033] S3, constructing a multi-dimensional verifiable reward signal based on RLVR, covering semantic consistency, aesthetic quality, text readability, target attribute matching and artistic style matching degree, etc.

[0034] S301, calculating a semantic consistency score based on a text-image alignment model;

[0035] S302, calculating an aesthetic quality score of the generated results based on an aesthetic evaluation model;

[0036] S303, calculating a text readability score in calligraphy and painting works based on an optical character recognition (OCR) module;

[0037] S304, calculating a target attribute matching score of the generated works based on a target detection and classification module;

[0038] S305, calculating a style matching degree score of calligraphy and painting generation based on an artistic style discrimination model.

[0039] S4, using an SARRB-MoE (Step-aware Reward Routing Mixture-of-Experts) mechanism to dynamically select and fuse appropriate reward dimensions according to different generation stages and task types, to realize adaptive multi-reward optimization;

[0040] S401, preferentially selecting semantic consistency and layout-related rewards in the initial generation stage;

[0041] S402, dynamically routing attribute matching and text readability rewards in the middle generation stage;

[0042] S403, generating a late-stage fusion aesthetic, details and artistic style related rewards;

[0043] S404, weighting combination between different expert rewards through a gating network or a soft routing mechanism;

[0044] S405, weighting initialization of reward dimensions based on semantic features or task types of input prompts;

[0045] S406, during the training process, dynamically adjusting the weights of each reward dimension according to different task types or input features to achieve adaptive routing.

[0046] S5, reinforcement learning training of the generation model based on the above reward feedback until the generation result reaches the preset standard in multiple-dimensional evaluation indicators.

[0047] S501, updating the generation strategy by using the GRPO strategy optimization method;

[0048] S502, jointly mapping the multi-dimensional reward signal into a weighted gradient target, and simultaneously optimizing the semantic consistency, aesthetic quality, text readability, attribute matching and artistic style matching degree indicators in the policy gradient update to improve the training stability and convergence efficiency.

[0049] The present application will be further described below in conjunction with specific examples and drawings.

[0050] Example 1

[0051] Reference Figure 2The embodiment is explained for a multi-modal calligraphy and painting generation task. First, a multi-modal input dataset containing text prompts, voice descriptions, and reference images is selected, which contains both regular text-image pairs and samples with calligraphy and painting style annotations. For different modal data, data preprocessing and feature extraction are performed respectively, for example, using a BERT model or a large language model to extract text semantic features, using a speech recognition and semantic encoding model to extract voice features, and using a convolutional neural network (CNN) or a vision transformer (Vision Transformer) to extract reference image features. Then, the Step-aware-GRPO step-by-step reward mechanism is used to optimize the generation process: the system rewards the generated results at the initial sketch, intermediate outline, and local detail stages to alleviate the sparsity problem caused by traditional methods that only provide rewards at the final result. Next, a multi-dimensional verifiable reward system is constructed based on the RLVR idea, including semantic consistency score, aesthetic quality score, text readability score, target attribute matching score, and calligraphy and painting style matching score. This system is automatically calculated by a cross-modal alignment model, an aesthetic evaluation model, an OCR module, a target detection model, and a style discrimination network, thereby ensuring the verifiability and comprehensiveness of the reward signal. Subsequently, the SARRB-MoE (Step-by-Step Reward Routing-based Expert Hybrid Mechanism) is introduced, which prioritizes semantic and layout rewards at the early stage of generation, dynamically integrates attribute matching and text readability rewards at the middle stage, and focuses on aesthetic and artistic style rewards at the later stage. This mechanism adaptively selects the appropriate reward dimension through a gating network or soft routing method, effectively improving the flexibility of reward fusion and the consistency of the generated results. On this basis, the GRPO strategy optimization method is used to train the generation model, which maps the above multi-dimensional reward signals to a weighted gradient target, and realizes the joint update of multi-dimensional rewards in the strategy optimization process. The system iterates continuously in each training round, gradually optimizing the generated results in terms of semantic consistency, aesthetic performance, text readability, and artistic style matching. Finally, when the multi-dimensional indicators of the generated works reach the preset threshold or the expert satisfaction standard, the training process is terminated. Through the method of the embodiment, high-quality calligraphy and painting generation can be achieved under the guidance of multi-modal input, taking into account semantic accuracy and artistic expressiveness, and realizing iterative performance improvement under the joint action of expert feedback and multi-dimensional reward signals, ultimately achieving high-precision, stability, and style matching generation effect.

[0052] The above only describes the preferred embodiments of the present application, and certain modifications can be made within the scope defined by the claims of the present application, but all will fall within the protection scope of the present application.

Claims

1. A multimodal calligraphy and painting generation method based on multidimensional verifiable reinforcement learning, characterized in that, Includes the following steps: S1. Receive multimodal input, including text, voice, or image prompts, and extract semantic vectors through a multimodal encoder; S2. In the process of generating calligraphy and paintings, a Step-aware-GRPO step-by-step reward mechanism is adopted to evaluate and provide feedback on the results of intermediate steps, thereby alleviating the problem of reward sparsity. S3. Construct a multi-dimensional verifiable reward signal based on RLVR, covering dimensions including semantic consistency, aesthetic quality, text readability, target attribute matching, and artistic style fit. S4. The SARRB-MoE mechanism is adopted to dynamically select and integrate appropriate reward dimensions according to different generation stages and task types to achieve adaptive multi-reward optimization. S5. Based on the above reward feedback, perform reinforcement learning training on the generative model until the generated results reach the preset standards in multi-dimensional evaluation indicators.

2. The method according to claim 1, characterized in that, In step S2, the specific steps are as follows: S201. Extract intermediate features or low-resolution results at each sampling step generated; S202. Calculate the reward signal for the intermediate result and return it to the strategy optimization process; S203. Through distributed policy gradient updates, progressively reinforced generative training is achieved.

3. The method according to claim 1, characterized in that, In step S3, the specific steps are as follows: S301. Calculate semantic consistency score based on text-image alignment model; S302. Calculate the aesthetic quality score of the generated results based on the aesthetic evaluation model; S303. Calculate the readability score of characters in calligraphy and painting works based on the optical character recognition module; S304. Calculate the target attribute matching score of the generated work based on the target detection and classification module; S305. Calculate the style fit score of calligraphy and painting based on the art style discrimination model.

4. The method according to claim 1, characterized in that, In step S4, the specific steps are as follows: S401. Prioritize semantic consistency and layout-related rewards in the initial generation stage; S403, Dynamic routing attribute matching and text readability rewards during the mid-generation phase; S403, Incorporating aesthetic, detail, and art style-related rewards in the later stages of generation; S404. Weighted combination of different expert rewards is performed through gating networks or soft routing mechanisms. S405. Initialize the reward dimension with weights based on the semantic features or task type of the input prompt; S406. During training, the weights of each reward dimension are dynamically adjusted according to different task types or input features to achieve adaptive routing.

5. The method according to claim 1, characterized in that, In step S5, the specific steps are as follows: S501. Update the generation strategy using the GRPO strategy optimization method; S502: The multi-dimensional reward signals are jointly mapped to a weighted gradient objective, and semantic consistency, aesthetic quality, text readability, attribute matching and artistic style fit are optimized simultaneously in the policy gradient update to improve training stability and convergence efficiency.

Images