Copywriting generation method and device based on large model, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a copywriting generation method and device based on a large model, equipment and a medium. The method uses a reinforcement learning algorithm to adjust and train a copywriting generation base model constituted by a large language model, so that the trained base model can generate more reliable and high-quality copywriting. When the reinforcement learning algorithm is used to fine-tune the base model, the base model is first trained for the first time based on a first reward function of a rule-driven type, then a second reward function is determined by using a reward model, the first reward function and the second reward function are combined to obtain a target reward function, and the base model is trained for the second time based on the target reward function, so that fine-grained optimization of the base model is realized, the trained base model can generate copywriting meeting the expectation, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for generating copy based on a large model. Background Technology

[0002] The copy generation capabilities of LLM (Large Language Model) have already demonstrated enormous potential in the marketing field. However, further optimization of training methods is still needed to ensure that the model generates marketing copy that meets style requirements and delivers high-quality content. The traditional approach is to perform SFT (Supervised Fine-Tuning) on ​​the LLM, which involves training it with a large amount of domain data. However, simple SFT often fails to adequately align the model output with human preferences.

[0003] Therefore, how to more effectively adjust and train the LLM so that the trained LLM can generate copy that better meets user expectations and has high-quality content is a technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, device and medium for generating copy based on a large model, in order to solve the problem that copy automatically generated based on a large model in the prior art cannot well meet user expectations.

[0005] A first aspect of this application provides a copy generation method based on a large model, comprising:

[0006] In response to receiving the instruction to generate copy, the base model is obtained, which is the first pre-trained large language model;

[0007] Determine the first reward function, which is a rule-driven reward function. The rule is determined based on at least one of the following: copy content elements, copy style constraints, copy violation rules, and copy generation process constraints.

[0008] The base model is fine-tuned and trained using a reinforcement learning algorithm based on the first reward function to obtain the base model after the first training.

[0009] Invoke at least one reward model to determine the second reward function;

[0010] The first reward function and the second reward function are weighted and combined to obtain the target reward function. The reinforcement learning algorithm is used to fine-tune the base model after the first training based on the target reward function to obtain the base model after the second training.

[0011] Based on the instructions to generate copy, the base model trained a second time is used to generate copy.

[0012] A second aspect of this application provides a copywriting generation apparatus based on a large model, comprising:

[0013] The acquisition module is configured to acquire the base model in response to receiving a text generation instruction. The base model is the first pre-trained large language model.

[0014] The determination module is configured to determine the first reward function, which is a rule-driven reward function. The rule is determined based on at least one of the following: copy content elements, copy style constraints, copy violation rules, and copy generation process constraints.

[0015] The training module is configured to fine-tune the base model using a reinforcement learning algorithm based on the first reward function to obtain the base model after the first training.

[0016] The determination module is also configured to invoke at least one reward model to determine a second reward function;

[0017] The training module is also configured to weight and combine the first reward function and the second reward function to obtain the target reward function, and use a reinforcement learning algorithm to fine-tune the base model after the first training based on the target reward function to obtain the base model after the second training.

[0018] The generation module is configured to generate text using the second-trained pedestal model based on the text generation instructions.

[0019] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0021] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment uses a reinforcement learning algorithm to adjust and train a copywriting generation base model composed of a large language model, so that the trained base model can generate more reliable and high-quality copywriting; wherein, when using the reinforcement learning algorithm to fine-tune the base model, the base model is first trained based on the first reward function of the rule-driven class, then the second reward function is determined using the reward model, and the first reward function and the second reward function are weighted and combined to obtain the target reward function, and then the base model is trained a second time based on the target reward function, thereby realizing fine-grained optimization of the base model, so that the trained base model can generate copywriting that meets expectations and improves the user experience. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a copy generation method based on a large model provided in an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating the method for determining the first reward function provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart illustrating the method for determining a rule subtask reward function based on each rule, as provided in the embodiments of this application.

[0026] Figure 4 This is a flowchart illustrating the method for determining a second reward function by invoking at least one reward model, as provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating another method for determining a second reward function by invoking at least one reward model, as provided in an embodiment of this application.

[0028] Figure 6 This is a flowchart illustrating the method for periodically statistically generating the difference between the quality level and the expected quality level of a document, as provided in an embodiment of this application.

[0029] Figure 7 This is a flowchart illustrating the method for fine-tuning a base model or a base model after its first training using a reinforcement learning algorithm, as provided in this application embodiment.

[0030] Figure 8 This is a schematic diagram of a copywriting generation device based on a large model provided in an embodiment of this application.

[0031] Figure 9 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for generating text based on a large model according to embodiments of this application.

[0034] As mentioned above, the traditional method for fine-tuning LLM training is to perform SFT on the LLM, which involves training with a large amount of domain data. However, SFT alone often fails to adequately align the model output with human preferences.

[0035] In related technologies, reinforcement learning algorithms can be used to fine-tune and train an LLM (Limited Management Model) to better meet user expectations. In some examples, marketing copy generated by the LLM can be used as training data to fine-tune the LLM using reinforcement learning algorithms, and then the trained LLM can be used to generate marketing copy that better meets user expectations.

[0036] However, using reinforcement learning algorithms to fine-tune and train LLM, and then using the trained LLM for marketing copy generation, still faces the following challenges:

[0037] 1) Copywriting quality is difficult to quantify. The quality of marketing copy depends on many factors, such as whether the writing style matches the brand's tone, whether the format is standardized, whether the content covers the product's selling points, and whether it can effectively attract the audience's attention. These objectives are difficult to quantify directly using a single loss function or rule, which means that even after supervised fine-tuning, the model's output may still deviate from marketing needs. For example, the model may output off-topic content, inconsistent style, or a lack of appealing statements.

[0038] 2) How to optimize the design of the reward function. Balancing different reward items and avoiding reward-driven imbalances in the model is a key technical challenge in reward function design.

[0039] 3) Most current generative models directly output results, lacking constraints on intermediate thinking. This can lead to models missing key information or having flawed logic. How to guide models to plan before expressing themselves when generating copy, and how to assign rewards to the rationality of the thinking process and the quality of the output content respectively, are technical problems that need to be solved.

[0040] 4) How to improve the training efficiency and stability of LLM reinforcement learning.

[0041] 5) How to achieve integration and compatibility with existing copywriting generation and review processes.

[0042] In view of this, this application provides a copywriting generation method based on a large model. This method uses reinforcement learning algorithms to adjust and train a copywriting generation base model composed of a large language model, so that the trained base model can generate more reliable and high-quality copywriting. Specifically, when fine-tuning the base model using reinforcement learning algorithms, the base model is first trained based on a first reward function driven by rules. Then, a second reward function is determined using the reward model, and the first and second reward functions are weighted and combined to obtain a target reward function. The base model is then trained a second time based on the target reward function, thereby achieving fine-grained optimization of the base model. This allows the trained base model to generate copywriting that meets expectations, improving the user experience.

[0043] Figure 1 This is a flowchart illustrating a copy generation method based on a large model provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0044] In step S101, in response to receiving the instruction to generate text, the base model is obtained.

[0045] Among them, the base model is the first pre-trained large language model.

[0046] In step S102, a first reward function is determined, which is a rule-driven reward function.

[0047] The rules are determined based on at least one of the following: content elements of the copywriting, style constraints of the copywriting, rules on copywriting violations, and constraints on the copywriting generation process.

[0048] In step S103, the base model is fine-tuned and trained using a reinforcement learning algorithm based on the first reward function to obtain the base model after the first training.

[0049] In step S104, at least one reward model is invoked to determine the second reward function.

[0050] In step S105, the first reward function and the second reward function are weighted and combined to obtain the target reward function. The reinforcement learning algorithm is used to fine-tune the base model after the first training based on the target reward function to obtain the base model after the second training.

[0051] In step S106, the copy is generated using the base model after the second training, based on the copy generation instruction.

[0052] In some embodiments of this application, the method can be executed by a terminal device or a server to automatically generate copy based on user-input instructions. In one example, the generated copy can be Chinese marketing copy or other types of copy.

[0053] In some embodiments of this application, after receiving a user's instruction to generate copy, a base model can be obtained. This base model can be a first pre-trained LLM model. In one example, the pre-trained LLM model can be an LLM model with Chinese understanding and generation capabilities. If the LLM model is a general model, it can also be initially fine-tuned. For example, a certain scale of Chinese marketing copy data can be used to perform SFT on the base model, so that the pre-trained LLM model after initial fine-tuning can initially learn the corpus style and basic patterns of marketing copy, such as how to write relevant copy given a product or keywords. The Chinese marketing copy data may include product descriptions and corresponding advertising copy, marketing headlines, short copy examples, etc.

[0054] Furthermore, if the base model itself already has a certain instruction alignment capability, the SFT step can be skipped, and the learning phase can proceed directly.

[0055] In some embodiments of this application, a first reward function for the base model can be determined, which may be a rule-driven reward function. This rule may be determined based on at least one of the following: text content elements, text style constraints, text violation rules, and text generation process constraints.

[0056] The base model can be fine-tuned using a reinforcement learning algorithm based on this first reward function to obtain the base model after the first training. At this point, the base model after the first training can meet the user's basic requirements for the copywriting rules.

[0057] In some embodiments of this application, at least one reward model can be invoked to determine a second reward function, which is a model-driven reward function. The target reward function can be obtained by weighted combination of the first and second reward functions.

[0058] In some implementations, the first reward function and the second reward function can be weighted and summed, then normalized or truncated to obtain the target reward function. The weights can be assigned according to actual needs, with positive rewards given to maximizing positive indicators and negative rewards given to minimizing them, thereby guiding the model towards a multi-objective optimal direction. Furthermore, the weight values ​​can be adjusted in real time based on the model's training progress.

[0059] After determining the target reward function, a reinforcement learning algorithm can be used again to fine-tune the base model trained the first time, based on this target reward function, to obtain the base model trained the second time. At this point, the base model trained the second time can further meet users' quality requirements for the copywriting. These quality requirements include, but are not limited to, the fluency of the copywriting, the uniqueness of the copywriting's creativity, and the effectiveness of the copywriting's marketing.

[0060] Finally, the base model, trained a second time, can be used to generate copy based on the user's input command to generate copy.

[0061] According to the technical solution provided in the embodiments of this application, a copywriting generation base model composed of a large language model is adjusted and trained by using a reinforcement learning algorithm, so that the trained base model can generate more reliable and high-quality copywriting. Specifically, when fine-tuning the base model using the reinforcement learning algorithm, the base model is first trained based on the first reward function of the rule-driven class. Then, the second reward function is determined using the reward model, and the first reward function and the second reward function are weighted and combined to obtain the target reward function. The base model is then trained a second time based on the target reward function, thereby achieving fine-grained optimization of the base model. This enables the trained base model to generate copywriting that meets expectations and improves the user experience.

[0062] Figure 2 This is a flowchart illustrating the method for determining the first reward function provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0063] In step S201, the rules for copy content elements and the rules for copy style constraints are determined based on the copy generation instruction, and the rules for copy violation and the rules for copy generation process constraints are obtained.

[0064] In step S202, a rule subtask reward function is determined based on each rule.

[0065] In step S203, the reward functions of each rule subtask are weighted and combined to obtain the first reward function.

[0066] In some embodiments of this application, when determining the first reward function, the text content element rules and text style constraint rules can be determined first based on the text generation instruction, and then a rule sub-task reward function can be determined for each rule. Finally, the first reward function can be obtained by weighted combination of the reward functions of each rule sub-task.

[0067] One method for weighted combining the reward functions of each rule subtask is to assign weights to each reward function, then sum them up by weights and normalize the sum to obtain the first reward function. The weights of the reward functions of each rule subtask can be set based on empirical values, or an initial weight can be set and then dynamically adjusted during the first fine-tuning training of the base model using reinforcement learning.

[0068] Figure 3 This is a flowchart illustrating the method for determining a reward function for a rule subtask based on each rule, as provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0069] In step S301, the reward function for the first rule subtask is determined based on the text content element rules.

[0070] The first rule subtask reward function rewards generated text that includes content elements and penalizes generated text that omits content elements.

[0071] In step S302, the reward function for the second rule subtask is determined based on the copywriting style constraint rules.

[0072] The second rule subtask reward function rewards generated copy that conforms to the copywriting style and penalizes generated copy that does not conform to the copywriting style; the copywriting style is determined by at least one of the copywriting format, copywriting style, and copywriting language style.

[0073] In step S303, the reward function for the third rule subtask is determined based on the text violation rules.

[0074] The third rule subtask reward function rewards generated copy that conforms to the copywriting violation rules and penalizes generated copy that does not conform to the copywriting violation rules.

[0075] In step S304, the reward function for the fourth rule subtask is determined based on the constraints of the copywriting generation process.

[0076] The fourth rule subtask reward function rewards generated copy that meets the constraints of the copy generation process, and also rewards generated copy that does not meet the constraints of the copy generation process. The constraints of the copy generation process include that the copy is generated in the format of thinking process first and then output content.

[0077] In some embodiments of this application, determining a rule subtask reward function based on each rule can be achieved by determining a first rule subtask reward function based on the text content element rule. The first rule subtask reward function rewards generated text that includes content elements and penalizes generated text that omits content elements.

[0078] In one example, content elements may include a specified product name, key selling points, and keywords. If the reward function of the first rule subtask determines that the generated copy includes the specified product name and key selling points, a reward will be given; otherwise, a penalty will be imposed.

[0079] In other embodiments of this application, determining a rule subtask reward function based on each rule can be achieved by determining a second rule subtask reward function based on copywriting style constraint rules. The second rule subtask reward function rewards generated copywriting that conforms to the copywriting style and penalizes generated copywriting that does not conform to the copywriting style; the copywriting style is determined by at least one of the following: copywriting format, copywriting style, and copywriting language style.

[0080] In one example, copywriting style can include copywriting format such as output length, copywriting style, and copywriting language style such as whether it includes specified sentence patterns. For example, if the reward function of the second rule subtask determines that the output length of the copywriting is within a preset range, or if the copywriting ends with the call-to-action phrase "Experience it now!", then a reward can be given; otherwise, a penalty can be given.

[0081] In some embodiments of this application, determining a rule subtask reward function based on each rule can also involve determining a third rule subtask reward function based on the copywriting violation rule. The third rule subtask reward function rewards generated copy that conforms to the copywriting violation rule and penalizes generated copy that does not conform to the copywriting violation rule.

[0082] In one example, the reward function for the third rule subtask can detect the generated copy based on a list of sensitive words and preset violation rules to determine whether it contains content that violates regulations or disparages competitors. If not, a reward can be given; otherwise, a penalty will be imposed.

[0083] In some further embodiments of this application, determining a rule subtask reward function based on each rule can also involve determining a fourth rule subtask reward function based on the constraints of the copywriting generation process. The fourth rule subtask reward function rewards generated copy that satisfies the constraints of the copywriting generation process and rewards generated copy that does not satisfy the constraints. The constraints of the copywriting generation process include that the copy is generated according to a format of thinking process followed by output content.

[0084] To accurately evaluate the LLM model, it is possible to set the LLM model to explicitly go through a thinking process when generating text. In this way, the thinking process of the model and the overall generated text can be evaluated, improving the accuracy of the evaluation.

[0085] That is to say, it is possible to set the LLM to separate the thinking process from the generated text during the text generation process. In one example, it can be stipulated that the format of the LLM answer contains two parts: the <think (thinking process)> part and the <answer (generated text)>. Among them, <think>The tags can list the internal conceptualization process or reasoning chain of an LLM. <answer>The text within the tags is the final generated text.

[0086] For example, when the model receives a generation task, it first... <think>List the key points, structure, or creative ideas in the paragraph, and then... <answer>Each paragraph provides a complete and coherent marketing copy. This format makes the intermediate reasoning of the model explicit, facilitating separate evaluation.

[0087] Based on the constraints of this generation process, the reward function of the fourth rule subtask can be used to detect the content during the copy generation process. If the reward function of the fourth rule subtask is determined... <think>If some of the reasoning is reasonable—for example, if it enumerates enough product selling points or mentions the target user's pain points—then it can be determined that it meets the generation process constraints and is rewarded. Furthermore, the more comprehensive the information covered and the clearer the structure of the reasoning process, the higher the reward. Conversely, if the reasoning process does not meet the generation process constraints, a penalty can be imposed.

[0088] The above rules can be directly calculated based on pre-specified criteria, and have clear interpretability. By using the first reward function corresponding to these rules, and then using reinforcement learning algorithms to fine-tune the base model based on the first reward function, we can first obtain a base model after the first training that conforms to the rules that must be followed in the copywriting generation process. This provides a foundation for further training of the base model based on the copywriting quality level.

[0089] Figure 4 This is a flowchart illustrating a method for determining a second reward function by invoking at least one reward model, as provided in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0090] In step S401, at least one copywriting evaluation index is obtained, and each copywriting evaluation index is used to evaluate a quality level of the generated copywriting.

[0091] The quality level includes at least the fluency of the generated copy, the uniqueness of the creative concept, and the marketing effectiveness.

[0092] In step S402, the reward model corresponding to each evaluation indicator is determined.

[0093] The reward model is either a pre-trained reward model or a second pre-trained large language model, and the second pre-trained large language model may be the same as or different from the first pre-trained large language model.

[0094] In step S403, the reward value of each reward model is determined to be the reward function of the model subtask.

[0095] In step S404, the reward functions of each model subtask are weighted and combined to obtain the second reward function.

[0096] In some embodiments of this application, in determining the second reward function, at least one copywriting evaluation index can be obtained first, with each index used to evaluate a quality level of the generated copywriting. Then, the reward model corresponding to each evaluation index is determined. Next, the reward value of each reward model is determined as a model sub-task reward function. Finally, the second reward function is obtained by weighted combination of the reward functions of each model sub-task.

[0097] The reward model can be a pre-trained reward model. For example, data on marketing copy and human ratings of it in dimensions such as creativity, writing style, and persuasiveness can be collected beforehand to train a small model to predict the overall score of the copy. During reinforcement learning, the model output is input into this rating model to obtain a score as a reward value.

[0098] On the other hand, reward models can also be pre-trained with LLMs. For example, LLMs can be used as evaluators to achieve zero-shot scoring. For instance, an evaluation prompt can be constructed, and the generated text along with the requirements can be input into the LLM to provide an evaluation or judgment on whether the expectations have been met. By leveraging the strong model knowledge of LLMs and their simulation of human preferences, high-quality feedback signals can also be obtained.

[0099] Different evaluation models can be trained or prompted for different dimensions. For example, one model focuses on checking the fluency of the copywriting language, while another focuses on the effectiveness of marketing. Scores are given for each model, and then weighted and summed to form the final model-based reward, i.e., the second reward function.

[0100] It is important to avoid introducing bias when training the reward model and to ensure that the evaluation model and the generative model remain independent.

[0101] Figure 5 This is a flowchart illustrating another method for determining a second reward function by invoking at least one reward model, as provided in an embodiment of this application. Figure 5 Steps S501 to S504 in the illustrated embodiment are Figure 4 Steps S401 to S404 in the illustrated embodiment are basically the same and will not be repeated here. Figure 5 As shown, the method also includes the following steps:

[0102] In step S505, when the reinforcement learning algorithm is used to fine-tune the base model after the first training based on the target reward function, in response to the determination that the base model after the first training has not converged, the difference between the quality level of the generated copy and the expected quality level is periodically statistically analyzed.

[0103] In step S506, the weights of each reward model are updated based on the difference.

[0104] In step S507, the reward functions of each model subtask are weighted and combined based on the updated weights to obtain the second reward function.

[0105] In some embodiments of this application, when fine-tuning the base model after the first training using a reinforcement learning algorithm based on the target reward function, if the base model after the first training fails to converge, the difference between the generated copywriting quality level and the expected quality level can be periodically calculated, and the weights of each reward model can be updated based on this difference. Finally, the reward functions of each model subtask are weighted and combined based on the updated weights to obtain the second reward function.

[0106] Figure 6 This is a flowchart illustrating the method for periodically statistically generating the difference between the document quality level and the expected quality level, as provided in an embodiment of this application. Figure 6 As shown, the method also includes the following steps:

[0107] In step S601, the verification dataset is periodically acquired.

[0108] The validation dataset includes different types of historical generated text instructions and their corresponding historical texts.

[0109] In step S602, the verification text is generated based on the historical text generation instructions using the base model after the first training that did not converge during fine-tuning training.

[0110] In step S603, the difference between the quality level of the verified copy and the quality level of the historical copy is statistically verified to obtain the difference between the quality level of the generated copy and the expected quality level.

[0111] In some embodiments of this application, periodically calculating the difference between the quality level of the generated copy and the expected quality level can be achieved by periodically acquiring a verification dataset, wherein the verification dataset includes different types of historical copy generation instructions and corresponding historical copy. Then, the base model after the first training iteration (which did not converge during fine-tuning training) is used to generate verification copy based on the historical copy generation instructions. Finally, the difference between the quality level of the verification copy and the quality level of the historical copy is calculated to obtain the difference between the quality level of the generated copy and the expected quality level.

[0112] In other words, the reward model can be iteratively optimized during the fine-tuning training of the base model. In one example, the model's performance on a validation set (containing some unseen marketing copywriting tasks) can be periodically evaluated, including the achievement of rule metrics, auxiliary model scoring, and, if necessary, user subjective evaluations. If the model still has significant weaknesses, such as failing to meet the standards for a certain style, the reward weights can be adjusted or new training samples can be added for further fine-tuning. When the model reaches the expected thresholds on all metrics and the training reward stabilizes, convergence can be determined, and training can be terminated.

[0113] In some implementations, rejection sampling and supervised fine-tuning can be combined. For example, after fine-tuning the pedestal model using reinforcement learning algorithms to obtain the second trained pedestal model, high-quality samples generated by the model can be collected and subjected to supervised fine-tuning again to consolidate the model's performance. Similarly, for copywriting generation, excellent copywriting samples produced by the model can be collected, reviewed, and added to the training to continuously improve the model's performance.

[0114] Figure 7 This is a flowchart illustrating a method for fine-tuning a base model or a base model after its first training using a reinforcement learning algorithm, as provided in an embodiment of this application. Figure 7 As shown, the method also includes the following steps:

[0115] In step S701, the currently available resources are obtained.

[0116] In step S702, in response to determining that the currently available resources are greater than or equal to a preset resource threshold, the pedestal model or the pedestal model after the first training is fine-tuned using the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm.

[0117] In step S703, in response to determining that the currently available resources are less than a preset resource threshold, the base model or the base model after the first training is fine-tuned using the Reinforced Style Optimization Algorithm (RLOO).

[0118] In some embodiments of this application, when fine-tuning the pedestal model or the pedestal model after its first training using reinforcement learning algorithms, the currently available resources can be obtained first, and then it can be determined whether the currently available resources are greater than or equal to a preset resource threshold. If so, the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm can be used to fine-tune the pedestal model or the pedestal model after its first training. Conversely, if the currently available resources are less than the preset resource threshold, the Reinforcement Style Optimization (RLOO) algorithm can be used to fine-tune the pedestal model or the pedestal model after its first training.

[0119] In other words, when resources are sufficient, PPO, GRPO, and their variants can be selected for fine-tuning the pedestal model. Conversely, when resources are limited, algorithms such as RLOO can be chosen for fine-tuning the pedestal model. Furthermore, KL divergence (Kullback-Leibler Divergence, relative entropy) constraints can be introduced to limit the difference between the fine-tuned pedestal model and the initial pedestal model to a certain range, thus preventing the model generation from deviating from human language style.

[0120] Using the technical solution provided in this application, the final base model after the second fine-tuning training can output marketing copy that meets the requirements for a given Chinese input, such as product descriptions and marketing keywords. In this process, the model is driven by a finely designed reward function, learning strategies to follow rules and pursue copy quality, thereby overcoming the shortcomings of the original model in generating marketing copy. The technical solution provided in this application is optimized for Chinese content and marketing scenarios, integrating a composite reward of rules and model evaluation, as well as advanced and efficient reinforcement learning algorithms, thus achieving significant performance improvements at a lower cost.

[0121] The following are some typical application scenarios of the technical solutions provided in the embodiments of this application.

[0122] E-commerce product advertising generation: On online retail platforms, thousands of products require promotional copywriting. The model provided in this application embodiment can be used to automatically generate attractive advertising slogans or product descriptions based on each product's title, selling points, and parameter descriptions.

[0123] For example, for a newly launched coffee product, after understanding its flavor characteristics and target consumers, the model first lists selling points such as "rich and invigorating taste," "produced in a well-known region," and "limited-time promotional offers" during the thinking process, and then outputs an enticing advertising slogan such as: <answer> Start your day with a refreshing drink: premium Yirgacheffe beans, rich and aromatic, to invigorate your mind. Order now and enjoy a limited-time discount, giving you a more energetic start than your boss!< / answer> The entire process requires no manual intervention, and the generated copy not only highlights the product but is also persuasive and meets the platform's requirements for word count and content.

[0124] Social media marketing copywriting: Brands need to frequently publish marketing content on social media platforms such as WeChat and Weibo, such as new product promotion articles and holiday promotional tweets. The model provided in this application embodiment can generate creative copywriting that matches the brand's tone based on given themes or event information.

[0125] For example, for Valentine's Day promotional campaigns, given a theme and promotional information, the model can output tweets that are romantic yet maintain the brand's positioning, ensuring the copy includes essential information such as campaign details and participation methods. By incorporating rewards for style and format, the model can control the tone of the text, making the copy both impactful and consistent with the brand. Social media platforms have strict limits on content compliance and word count; these factors are considered by the model before generating copy, driven by rewards, reducing the risk of content failing review.

[0126] Advertising creative copywriting generation service: Advertising agencies or copywriting teams can package this model into an AI (Artificial Intelligence) copywriting assistant service.

[0127] A typical usage scenario involves users providing keywords (brand name, product features, target users, etc.) and requirements (lively style, suspenseful style, etc.). The model immediately generates multiple versions of copywriting with different wording for users to choose from. Because the model is optimized through reinforcement learning, it understands how to follow different style requirements (for example, the evaluation model's preference can be adjusted for "lively" or "suspenseful" styles in the rewards), resulting in diverse but relevant outputs. This accelerates the creative iteration process, allowing copywriters to draw inspiration from the model's suggestions or adopt them directly, greatly improving work efficiency.

[0128] Simultaneously, the service allows users to define custom rules (such as prohibiting the appearance of competitor names), which are injected into the model's reward system in real time to ensure that the generated results meet the user's customized needs. This scenario demonstrates the flexibility and practical value of the technical solution provided in this application embodiment in industrial applications.

[0129] Personalized marketing email generation: When sending customized marketing emails to different customer groups, businesses can use this model to generate corresponding wording and content based on user profiles.

[0130] For example, the selling points and tone emphasized will differ depending on whether the customer is a VIP (Very Important Person), a regular user, or a potential user.

[0131] Traditionally, copywriters need to create multiple templates. However, using the model provided in this application's embodiment, email copy can be automatically generated tailored to individual users, while ensuring all emails adhere to a consistent brand voice and format. The model's thought process can list key points for the specific user group (e.g., VIPs emphasizing a sense of exclusivity, regular users highlighting discounts), and the output is the complete email body. This not only saves manpower but also enables truly large-scale personalized marketing content production.

[0132] Multimodal Content Marketing: The technical solutions provided in this application can also be extended to content scenarios combining images and text. For example, given a product image or promotional poster, the model provided in this application can be combined with a visual model to first analyze the image and extract key information, and then generate matching text descriptions or copy. After specialized training, the model can incorporate image cues into the thought process, such as recognizing the scene of the product in the image, high-end elements, etc. <think>Record some of this information and then produce corresponding copy.

[0133] In typical applications, e-commerce platforms can automatically generate product descriptions highlighting key selling points from a vast number of product images; the tourism industry can generate promotional text based on scenic photos, and so on. Since the core of the technical solution provided in this application lies in the optimization of text generation, combining it with an image processing module can achieve multimodal marketing content generation, further broadening its application scope.

[0134] The above scenarios are merely representative examples. The technical solutions provided in this application are applicable to almost all fields requiring the batch generation of high-quality copy, including but not limited to: news headline writing, film and television promotional slogans, and biopharmaceutical product promotion copy. Whether in content creation platforms, digital marketing companies, or internal marketing departments, the technical solutions provided in this application can be deployed and used to help automate the generation of high-quality Chinese copy. By flexibly setting reward rules, it can also adapt to the specific needs of different industries and markets, making it a truly versatile and scalable AI copy generation solution.

[0135] The following are some examples of automatically generating text using the technical solutions provided in the embodiments of this application.

[0136] Example 1: Reinforcement Learning Optimization of E-commerce Category Marketing Copy

[0137] This embodiment uses the generation of product marketing copy on an e-commerce platform as a scenario to demonstrate how to apply the technical solution provided in this application embodiment to fine-tune and optimize LLM.

[0138] Step 1: Training Data Preparation and Base Model Fine-tuning. First, collect marketing text data for several product categories on e-commerce platforms, including product titles, descriptions, selling points, and manually written advertising copy, such as short product descriptions and promotional slogans. When building the dataset, basic product information can be used as input, and the manually written copy as the target output. In one example, the open-source Chinese base model Qwen2.5-7B can be selected as the initial model. Use the above data to fine-tune Qwen2.5-7B using SFT, enabling it to learn the ability to generate corresponding copy from product information. After fine-tuning, the model can output basically acceptable copy, but creativity and consistency need further improvement.

[0139] Step 2: Design the output format to separate the thought process from the content. Before fine-tuning the reinforcement learning, you can modify the interactive prompts to make the model follow a specific output format: <think> …< / think> The paragraph contains the model's internal thinking about the input product, such as listing the product's main selling points, target consumer group, and proposed tone and style; <answer> …< / answer> Paragraphs are the marketing copy that is ultimately presented to users.

[0140] Taking a smartwatch as an example, when the model receives product parameters, it expects that... <think>Some listings include things like "Selling Point 1: Health Monitoring Function, Selling Point 2: Long Battery Life, Target Users: Fitness Enthusiasts, Style: Inspirational and Uplifting," and then... <answer>A portion of the generated ad copy integrates these key points. This format requires teaching through the addition of several example dialogues to the training samples, with the format explicitly specified in system prompts. The model will be repeatedly reminded to adhere to this format during reinforcement learning.

[0141] Step 3: Implement the reward function. Based on the requirements of this scenario, a composite reward function can be implemented, including:

[0142] 1) Rule Check - Selling Point Coverage. Extract the list of key selling points provided by the product and match it with the model output. If... <answer>If the copy covers all the main selling points, a +1 reward is given; if any one is omitted, a negative reward is given based on the number of omissions. For example, if a phone has three main selling points: "triple camera," "fast charging," and "high refresh rate screen," but the copy does not mention "fast charging," then this reward is -1.

[0143] 2) Rule Check - Format and Tone. Check <answer>Does the paragraph meet the platform's requirements, such as being no more than 50 characters long, having punctuation at the end of sentences, and matching the overall tone to the expected style (this can be roughly judged by the presence of exclamation marks, etc.)? Each item that meets the requirements will earn +0.5 points; otherwise, -0.5 points. <think>Some parts must be hidden from the end user; this can be achieved by identifying them within the environment. <think>Tags ensure that only the final result is displayed. <answer>Content will be severely penalized if the model is found to violate the format (e.g., a reward of -2).

[0144] 3) Rule Check - Prohibited Words. A list of prohibited words in e-commerce copywriting can be created (e.g., superlative terms like "optimal" or "national-level," which are legally prohibited). Scan. <answer>If any forbidden word is found, a huge penalty of -5 is immediately imposed, and the round is marked as terminated, thus ensuring that the model tries its best to avoid touching the red line.

[0145] 4) Model Scoring - Copywriting Quality Scoring. A small evaluation model, RM1, can be trained using a previously collected batch of product copywriting and user feedback (such as click-through rates and highly-liked copywriting). The scoring range is 0-10, measuring the attractiveness and persuasiveness of the copywriting. Each time the model generates... <answer>Then, RM1 scores it and maps it to a reward value in the range of -1 to +1. In one example, a score of 8 or above can be set to +1, a score below 5 to -1, and the rest to a linear mapping.

[0146] 5) Model Scoring - Style Match. Simultaneously, another evaluation model, RM2, can be trained, or a large evaluation model can be directly provided, such as GPT-4, to determine whether the output copy matches the preferences of the product's target users. In one example, copy for fitness products can be assessed using an additional model to determine whether it is dynamic and encouraging. RM2 can output yes or no; a yes output can be scored as +0.5, otherwise 0. This sub-reward ensures the copy style closely matches the intended audience.

[0147] 6) Rewards for the thinking process. To incentivize the correct application of the model. <think>Some parts can be formulated if <think>The paragraph lists more than N selling points, and these selling points are later... <answer>If it is mentioned in most of the texts, then an extra reward of +1 will be given; otherwise, if <think>and <answer>If the content is significantly inconsistent (e.g., listed in the thought process but not used in the output), then -1 is applied. Here, N is a positive integer and can be dynamically set based on the amount of product information. This encourages the model to take the thought process seriously and avoids superficiality.

[0148] All the above sub-rewards are calculated in real time and accumulated to form the total reward R_total. For example, for a certain output, if the copy covers all selling points (+1), has an appropriate length and an exclamation mark (+0.5), contains no prohibited words (+0), has an RM1 score of around 0.8 (+0.8), an RM2 judgment of Yes (+0.5), and demonstrates sufficient thought process (+1), then the total reward is approximately +3.8 points; conversely, if selling points are omitted (-1), prohibited words are present (-5), or the quality score is low (-0.7), then the total reward may be negative, and the model will quickly eliminate this type of strategy.

[0149] The values ​​of each reward can be adjusted according to actual needs; there are no restrictions here.

[0150] Step 4: Reinforcement Learning Fine-Tuning. Fine-tuning can be performed using the GRPO algorithm within the open-source EasyR1 framework. In one example, the base model fine-tuned in Step 1 can be used as the initial policy. Load and prepare a reference model with the same structure but frozen parameters. Used to calculate KL penalties.

[0151] During training, for each product input, the policy model samples, for example, k=4 different items. <think> + <answer>Candidates. Calculate the above rewards R for each candidate. i Then calculate the average reward. As a baseline, i is a positive integer. For each sample, according to Adjust strategy: If If the value is positive, the probability of the sample being generated by the policy increases; otherwise, its probability decreases. Gradient approximation can be achieved by weighting the logarithmic probability using the policy gradient formula. To obtain it. Additionally, a KL constraint can be added. This ensures that the new strategy does not deviate excessively from the original model. For example, a coefficient can be set. Control KL within a certain threshold. After each batch (processing step) completes the update, let... The parameters take a small step in the direction of increasing rewards. After tens of thousands of training steps, the model gradually learns the output strategy that makes the reward function score higher.

[0152] During training, the average reward trend can be monitored. Training is stopped when the average reward stabilizes on the validation set and approaches the theoretical maximum score. The resulting policy model is the optimized copywriting generation model.

[0153] Step 5: Result Validation. Select a set of product information that was not used in training, have the model generate copy, and compare it with the original SFT model output and human copy.

[0154] Experiments show that the copy output by the technical solution provided in this application fully covers the product's selling points, is fluent and persuasive, contains no prohibited words, and fully complies with formatting requirements. In contrast, the SFT model sometimes omits details or uses bland language.

[0155] In blind human testing, marketing team members generally agreed that the copywriting quality of this solution model was close to that of human copywriters, with some even considering it superior to some ordinary human-written copy. This demonstrates that reinforcement learning has indeed enabled the model to capture the essence of human preferences.

[0156] In quantitative evaluation, the model output closely approximates the reference text in coverage metrics such as Rouge-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence, a natural language task evaluation metric) and BLEU (Bilingual Evaluation Understudy, a bilingual evaluation substitution metric), which is consistent with the encouragement to cover selling points during training. In terms of user satisfaction, it is significantly higher than the model without RL optimization.

[0157] Example 2: Copywriting generation with multiple style requirements

[0158] This example demonstrates how the model can be trained using this solution under various style requirements, allowing for flexible adjustment of the copy's tone according to the needs of different brands or events. Suppose there is a copywriting generation service that needs to output two drastically different advertising copy styles: a "humorous style" and a "formal style" for different clients.

[0159] Step 1: Collect training materials in multiple styles. Prepare two sets of sample texts: one set leans towards a humorous and witty style, requiring the use of internet slang and metaphors; the other set leans towards a formal and professional style, requiring standardized vocabulary and a serious tone. Each set contains sample texts on several themes.

[0160] Step 2: Introduce style directives and format the output. You can add a field to the model input or specify the desired style type via system prompts. For example, add "[Style: Humor]" or "[Style: Formal]" to the prompt (0 prompt words). The output format remains the same. <think> / <answer>Separation, but requires <think>Stylistic elements are taken into consideration.

[0161] Step 3: Style-Specific Reward Function Extension. Building upon Example 1, a style-matching reward can be added: two discriminator models are trained using pre-prepared style samples to determine whether a given text belongs to either a humorous or formal style. If the style of the text output by the model matches the input requirements, the corresponding discriminator gives a high score (reward +1); otherwise, the reward -1. Furthermore, checks for specific styles are added to the rule checks. For example, a humorous style expects the presence of emojis or colloquialisms, so these features are detected; a formal style avoids interjections, so the presence of colloquial fillers such as "oh" or "ba" is detected. Any style mismatch is immediately penalized.

[0162] Step 4: Adjusting the Reinforcement Learning Training Process. During training, style requirements can be randomly specified in the input, allowing the model to optimize alternately across different styles. The reward function evaluates the output in real time for the desired style. This is equivalent to having the model learn two policies simultaneously, but switching between them under conditional control with uniform parameters. The GRPO algorithm itself remains unchanged, but it's crucial to carefully control the proportion of samples from different styles during training to prevent the model from favoring one style. In practice, this can be achieved by adding an element that encourages diversity to the reward or by employing a multi-policy fusion method to ensure balance.

[0163] Step 5: Testing and Results. After training, the model was asked to generate both humorous and formal versions of the same text on the same topic. The results showed that the model could flexibly switch writing styles: the humorous version was witty and full of jokes, while the formal version used precise language and emphasized credibility. This indicates that the model has successfully learned style as a controllable dimension. When changing brands or scenarios, only a small stage of fine-tuning with new style samples is needed to adapt to new requirements, demonstrating the adaptability of the technical solution provided in this application embodiment in generating diverse text.

[0164] As illustrated by the above embodiments, the technical solutions provided in this application can be modified and extended in various ways according to actual application needs. For example, as long as a suitable reward function is defined, the technical solutions provided in this application can also be used for the generation and optimization of English marketing copy or multilingual copy; or they can be applied to conversational marketing assistant scenarios, allowing the model to learn to better interact with users and recommend products through reinforcement learning. All these variations are within the scope of this application and will not deviate from its spirit and protection scope.

[0165] The technical solution provided in this application embodiment can comprehensively improve the quality of generated copy. Due to the use of composite reward optimization, the fine-tuned model significantly outperforms the model without this method in terms of content accuracy, style consistency, and creativity. Guided by the reward function, the model learns to follow essential marketing elements (such as including selling points and calls to action) and avoid low-quality content. Experimental results show that the model fine-tuned by RL is more favored by human reviewers than the model with only SFT, and its generated results show significant improvements in usefulness and effectiveness scores. In particular, it achieves higher scores in both automatic evaluation metrics (such as Rouge and BLEU) and human evaluation in advertising copy scenarios.

[0166] Meanwhile, the generated copy meets regulatory compliance requirements. Thanks to the embedded rule rewards, the model output strictly adheres to pre-set business rules and legal norms. For example, it avoids prohibited words and does not deviate from the brand's style guidelines, ensuring the usability and security of the copy in real-world scenarios. In contrast, traditional models often require rule filtering or even manual review after inference to remove non-compliant content, while the model trained by this solution inherently has a built-in compliance bias, significantly reducing post-processing costs.

[0167] Furthermore, the copywriting model's thinking ability has been enhanced. Through reward constraints on the "thinking process," the model has learned to scrutinize copywriting ideas more rigorously, resulting in output text with clear structure, sound logic, and less likelihood of inconsistencies or omissions of key information. This improvement in intrinsic thinking ability also brings benefits to generalization performance: when faced with new products or marketing themes, the model can organize copywriting content in a structured way, rather than simply relying on clichés seen during training. This solves the problem of scattered content or incomplete coverage that sometimes occurred with previous models.

[0168] Furthermore, the training efficiency and stability of the base model for generating copy have been improved. By employing optimization algorithms such as GRPO / RLOO, training resource overhead has been reduced, enabling fine-tuning of large models using RL even under relatively limited resources. While maintaining performance, training time has been shortened, and GPU memory usage has been reduced by more than half. In addition, the multi-reward design plays a role in reward shaping, preventing the model from going to extremes (such as overly catering to a certain metric) under a single objective. The mutual constraints among the various metrics make training converge more smoothly.

[0169] Furthermore, this method is flexible, controllable, and scalable. Because the reward function design is modular, users can adjust or expand the indicator weights according to specific application needs, achieving customized generation optimization. For example, a reward item targeting brand tone can be easily added, or a localized style evaluation indicator can be added before launching in a new market. This method has high controllability; the model is not like a black box that is difficult to intervene in. Instead, the model behavior can be "trained" by adjusting the rewards. In addition, this solution does not depend on a specific model and is applicable to various pre-trained LLMs; it can be transferred as long as the corresponding computing power is provided. This means that whether it's e-commerce product descriptions, social media copywriting, or marketing content generation in other languages, the technical solution provided in this application embodiment can be used for optimization, demonstrating good scalability.

[0170] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0171] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0172] Figure 8 This is a schematic diagram of a copywriting generation device based on a large model provided in an embodiment of this application. Figure 8 As shown, the device includes:

[0173] The acquisition module 801 is configured to acquire the base model in response to receiving the text generation instruction. The base model is the first pre-trained large language model.

[0174] The determination module 802 is configured to determine a first reward function, which is a rule-driven reward function. The rule is determined based on at least one of the following: copywriting content elements, copywriting style constraints, copywriting violation rules, and copywriting generation process constraints.

[0175] Training module 803 is configured to use a reinforcement learning algorithm to fine-tune the base model based on the first reward function to obtain the base model after the first training.

[0176] The determination module 802 is also configured to invoke at least one reward model to determine a second reward function.

[0177] The training module 803 is also configured to weight and combine the first reward function and the second reward function to obtain the target reward function, and use a reinforcement learning algorithm to fine-tune the base model after the first training based on the target reward function to obtain the base model after the second training.

[0178] The generation module 804 is configured to generate text using the second-trained pedestal model based on the text generation instructions.

[0179] According to the technical solution provided in the embodiments of this application, a copywriting generation base model composed of a large language model is adjusted and trained by using a reinforcement learning algorithm, so that the trained base model can generate more reliable and high-quality copywriting. Specifically, when fine-tuning the base model using the reinforcement learning algorithm, the base model is first trained based on the first reward function of the rule-driven class. Then, the second reward function is determined using the reward model, and the first reward function and the second reward function are weighted and combined to obtain the target reward function. The base model is then trained a second time based on the target reward function, thereby achieving fine-grained optimization of the base model. This enables the trained base model to generate copywriting that meets expectations and improves the user experience.

[0180] In some implementations, determining the first reward function includes: determining the copy content element rules and copy style constraint rules based on the copy generation instruction, and obtaining the copy violation rules and copy generation process constraint rules; determining a rule sub-task reward function based on each rule; and weighting and combining the reward functions of each rule sub-task to obtain the first reward function.

[0181] In some implementations, a reward function for each rule subtask is determined, including: determining a first rule subtask reward function based on content element rules, which rewards generated copy that includes content elements and penalizes generated copy that omits content elements; and determining a second rule subtask reward function based on copy style constraint rules, which rewards generated copy that conforms to the copy style and penalizes generated copy that does not conform to the copy style. The copy style is defined by the copy's format, register, and language style. At least one of the following is determined: A third rule subtask reward function is determined based on the copywriting violation rules; the third rule subtask reward function rewards generated copy that conforms to the copywriting violation rules and penalizes generated copy that does not conform to the copywriting violation rules; a fourth rule subtask reward function is determined based on the copywriting generation process constraint rules; the fourth rule subtask reward function rewards generated copy that satisfies the copywriting generation process constraint rules and rewards generated copy that does not satisfy the copywriting generation process constraint rules; wherein, the copywriting generation process constraint rules include that the copy is generated according to the format of thinking process first and then output content.

[0182] In some implementations, determining the second reward function by invoking at least one reward model includes: obtaining at least one copywriting evaluation index, each of which is used to evaluate a quality level of the generated copy, the quality level including at least the fluency of the language, the uniqueness of the creativity, and the marketing effectiveness of the generated copy; determining the reward model corresponding to each evaluation index, the reward model being either a pre-trained reward model or a second pre-trained large language model, the second pre-trained large language model being the same as or different from the first pre-trained large language model; determining the reward value of each reward model as a model subtask reward function; and weightedly combining the reward functions of each model subtask to obtain the second reward function.

[0183] In some implementations, the method further includes: when fine-tuning the base model after the first training using a reinforcement learning algorithm based on the target reward function, in response to determining that the base model after the first training has not converged, periodically calculating the difference between the copywriting quality level and the expected quality level; updating the weights of each reward model based on the difference; and weighting and combining the reward functions of each model subtask based on the updated weights to obtain a second reward function.

[0184] In some implementations, the difference between the quality level of the generated copy and the expected quality level is periodically statistically analyzed, including: periodically acquiring a verification dataset, which includes different types of historical copy generation instructions and corresponding historical copy; using the base model after the first training that did not converge during fine-tuning training to generate verification copy based on the historical copy generation instructions; and statistically analyzing the difference between the quality level of the verification copy and the quality level of the historical copy to obtain the difference between the quality level of the generated copy and the expected quality level.

[0185] In some implementations, a reinforcement learning algorithm is used to fine-tune the pedestal model or the pedestal model after its first training, including: acquiring currently available resources; in response to determining that the currently available resources are greater than or equal to a preset resource threshold, fine-tuning the pedestal model or the pedestal model after its first training using the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm; or in response to determining that the currently available resources are less than a preset resource threshold, fine-tuning the pedestal model or the pedestal model after its first training using the Reinforcement Style Optimization (RLOO) algorithm.

[0186] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0187] Figure 9 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 9 of this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, it implements the steps in the various method embodiments described above. Alternatively, when the processor 901 executes the computer program 903, it implements the functions of each module / unit in the various device embodiments described above.

[0188] Electronic device 9 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 9 may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or different components.

[0189] The processor 901 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0190] The memory 902 can be an internal storage unit of the electronic device 9, such as a hard disk or RAM of the electronic device 9. The memory 902 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 9. The memory 902 can also include both internal and external storage units of the electronic device 9. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0191] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0192] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.< / think> < / answer> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / think> < / think> < / answer> < / answer> < / answer> < / think> < / think> < / answer> < / answer> < / answer> < / think> < / think> < / think> < / answer> < / think> < / answer> < / think>

Claims

1. A copywriting generation method based on a large model, characterized in that, include: In response to receiving a text generation instruction, a base model is obtained, wherein the base model is a first pre-trained large language model; Determine a first reward function, which is a rule-driven reward function, wherein the rule is determined based on at least one of the following: copywriting content elements, copywriting style constraints, copywriting violation rules, and copywriting generation process constraints. The base model is fine-tuned and trained using a reinforcement learning algorithm based on the first reward function to obtain the base model after the first training. Invoke at least one reward model to determine the second reward function; The first reward function and the second reward function are weighted and combined to obtain the target reward function. The base model after the first training is fine-tuned and trained using a reinforcement learning algorithm based on the target reward function to obtain the base model after the second training. Based on the text generation instruction, the base model after the second training is used to generate text; The determination of the first reward function includes: determining content element rules and style constraint rules for the copywriting based on the copywriting generation instruction, and obtaining copywriting violation rules and copywriting generation process constraint rules; determining a first rule sub-task reward function based on the content element rules, wherein the first rule sub-task reward function rewards generated copywriting that includes the content elements and penalizes generated copywriting that omits the content elements; determining a second rule sub-task reward function based on the style constraint rules, wherein the second rule sub-task reward function rewards generated copywriting that conforms to the style and penalizes generated copywriting that does not conform to the style; wherein the style is determined by the copywriting format, the copywriting style type, and the copywriting... At least one of the following language styles is determined: a third rule sub-task reward function is determined based on copywriting violation rules, wherein the third rule sub-task reward function rewards generated copywriting that conforms to the copywriting violation rules and penalizes generated copywriting that does not conform to the copywriting violation rules; a fourth rule sub-task reward function is determined based on copywriting generation process constraint rules, wherein the fourth rule sub-task reward function rewards generated copywriting that satisfies the copywriting generation process constraint rules and rewards generated copywriting that does not satisfy the copywriting generation process constraint rules; wherein the copywriting generation process constraint rules include that the copywriting is generated according to the format of thinking process first and then output content; the first reward function is obtained by weighted combination of the reward functions of each rule sub-task.

2. The method according to claim 1, characterized in that, Invoking at least one reward model to determine the second reward function includes: Obtain at least one copywriting evaluation indicator, each of which is used to evaluate a quality level of the generated copywriting, the quality level including at least the language fluency, creative uniqueness and marketing effectiveness of the generated copywriting; Determine the reward model corresponding to each evaluation indicator. The reward model is either a pre-trained reward model or a second pre-trained large language model. The second pre-trained large language model may be the same as or different from the first pre-trained large language model. The reward value for each reward model is determined to be the reward function of the model's subtask. The second reward function is obtained by weighting and combining the reward functions of each model subtask.

3. The method according to claim 2, characterized in that, The method further includes: When using a reinforcement learning algorithm to fine-tune the base model after the first training based on the target reward function, in response to determining that the base model after the first training has not converged, the difference between the quality level of the generated copy and the expected quality level is periodically statistically analyzed. The weights of each reward model are updated based on the aforementioned differences; The second reward function is obtained by weighting and combining the reward functions of each model subtask based on the updated weights.

4. The method according to claim 3, characterized in that, The periodic statistical analysis of the difference between the generated copy quality level and the expected quality level includes: Periodically acquire a verification dataset, which includes different types of historical generated text instructions and corresponding historical texts; The verification text is generated based on the historical text generation instructions using the base model after the first training that did not converge during fine-tuning training. The difference between the quality level of the verified copy and the quality level of the historical copy is statistically analyzed to obtain the difference between the quality level of the generated copy and the expected quality level.

5. The method according to any one of claims 1 to 4, characterized in that, Fine-tuning the pedestal model or the pedestal model after its initial training using reinforcement learning algorithms includes: Get currently available resources; In response to determining that the currently available resources are greater than or equal to a preset resource threshold, the pedestal model or the pedestal model after the first training is fine-tuned using the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm; or In response to the determination that the current available resources are less than a preset resource threshold, the Reinforced Style Optimization (RLOO) algorithm is used to fine-tune the pedestal model or the pedestal model after the first training.

6. A copywriting generation device based on a large model, characterized in that, include: The acquisition module is configured to acquire a base model in response to receiving a text generation instruction, wherein the base model is a first pre-trained large language model; The determination module is configured to determine a first reward function, which is a rule-driven reward function. The rule is determined based on at least one of the following: copywriting content elements, copywriting style constraints, copywriting violation rules, and copywriting generation process constraints. Determining the first reward function includes: determining copywriting content element rules and copywriting style constraint rules based on the copywriting generation instruction, and obtaining copywriting violation rules and copywriting generation process constraint rules; determining a first rule sub-task reward function based on the copywriting content element rules, whereby the first rule sub-task reward function rewards generated copywriting that includes the content elements and penalizes generated copywriting that omits the content elements; and determining a second rule sub-task reward function based on the copywriting style constraint rules, whereby the second rule sub-task reward function rewards generated copywriting that conforms to the copywriting style and penalizes generated copywriting that does not conform to the copywriting style. The generated copy is penalized based on its style; wherein the style is determined by at least one of the following: copy format, copy style, and copy language style; a third rule sub-task reward function is determined based on copy violation rules, which rewards generated copy that conforms to the copy violation rules and penalizes generated copy that does not conform to the copy violation rules; a fourth rule sub-task reward function is determined based on copy generation process constraint rules, which rewards generated copy that satisfies the copy generation process constraint rules and rewards generated copy that does not satisfy the copy generation process constraint rules; wherein the copy generation process constraint rules include that the copy is generated according to the format of thinking process first and then output content; the first reward function is obtained by weighted combination of the reward functions of each rule sub-task; The training module is configured to fine-tune the base model using a reinforcement learning algorithm based on the first reward function to obtain the base model after the first training. The determining module is also configured to invoke at least one reward model to determine a second reward function; The training module is further configured to weight and combine the first reward function and the second reward function to obtain a target reward function, and use a reinforcement learning algorithm to fine-tune the base model after the first training based on the target reward function to obtain the base model after the second training. The generation module is configured to generate text using the second-trained pedestal model based on the text generation instructions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images