Marketing copy automatic generation method and device based on reinforcement learning and storage medium

Abstract

The application provides a marketing copy automatic generation method and device based on reinforcement learning and a storage medium. The method comprises the following steps: performing semantic matching retrieval on public copy data to obtain a candidate copy; inputting a slot rewriting instruction into a pre-generated language model to generate a first marketing copy; performing supervised fine-tuning training on a preset basic language model to obtain a first training model; inputting new user product information and promotion requirements into the first training model to generate a second marketing copy, scoring the second marketing copy and generating evaluation data; constructing a partial order training sample according to the evaluation data, taking the partial order training sample as a reward signal, performing reinforcement learning training on the first training model to obtain a second training model; and calling the second training model in a copy generation system and outputting a target marketing copy based on user product information and promotion requirements. The application can realize batch generation of marketing copies with high compliance and consistent constraints.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular relates to a marketing copy automatic generation method and device based on reinforcement learning and a storage medium. BACKGROUND

[0002] With the rapid development of e-commerce, short videos and social platforms, the demand for product marketing copy by enterprises has shown explosive growth. Compliant, accurate and creative marketing copy can significantly improve product exposure and conversion rates, so various automatic copy generation technologies based on natural language processing have attracted widespread attention. Large language models (LLM) have become one of the core technology routes for current automated content production due to their general text generation and context understanding capabilities.

[0003] The mainstream solution usually pre-trains LLM with public large-scale corpus, and then generates marketing text through a small number of examples (Few-Shot) or prompt engineering. To improve the performance in vertical fields, some research introduces supervised fine-tuning (SFT) or reinforcement learning based on human feedback (RLHF) to retrain the model to inject specific industry knowledge and writing style. Another method integrates product attribute information into the generation process through keyword insertion, template replacement or retrieval augmented generation, etc. to improve the correspondence between the text and the product.

[0004] The existing solutions still have the following problems: 1. Low rule adaptation: existing LLM lacks explicit modeling of advertising law, platform banned words and industry compliance requirements, and the generated content is easy to touch the regulatory red line. 2. Insufficient coupling of professional information: structured information such as product parameters and domain terminology cannot be fully and accurately integrated into the generated copy, resulting in factual distortion or missing selling points. 3. Poor consistency of multiple constraints: when users simultaneously propose multiple requirements such as style, length and selling points, the model has difficulty in considering all constraints in a single generation, and the output stability is insufficient. 4. Incomplete feedback loop: the traditional SFT training data is limited in quantity and quality, and the RLHF stage relies on artificial dialogue scoring, lacking an expandable and interpretable evaluation rule library, which limits the effectiveness of reinforcement learning. SUMMARY

[0005] Therefore, the embodiments of the present application provide a marketing copy automatic generation method and device based on reinforcement learning and a storage medium to solve the problems of low compliance adaptation and difficulty in simultaneously integrating product information and multiple constraints in the prior art.

[0006] In a first aspect, the embodiment of the present application provides a marketing copy automatic generation method based on reinforcement learning, comprising: obtaining original input containing product information, marketing points and public copy data, performing semantic matching retrieval on the public copy data to obtain candidate copy related to the product information; constructing slot rewriting instructions containing the candidate copy, the product information and the marketing points, inputting the slot rewriting instructions into a pre-generated language model, causing the pre-generated language model to fill in corresponding product information and marketing points at preset slot positions to generate a first marketing copy, and arranging the candidate copy, the product information and the first marketing copy into a supervised fine-tuning training sample; performing supervised fine-tuning training on a preset basic language model by using the supervised fine-tuning training sample to obtain a first training model; inputting new user product information and promotion requirements into the first training model to generate at least one second marketing copy, inputting the second marketing copy into a regularized evaluation system to score the second marketing copy and generate evaluation data; constructing partial order training samples according to the evaluation data, taking the partial order training samples as reward signals, performing reinforcement learning training on the first training model to obtain a second training model; and calling the second training model in a copy generation system to output target marketing copy based on user product information and promotion requirements.

[0007] In a second aspect, the embodiment of the present application provides a marketing copy automatic generation device based on reinforcement learning, comprising: an acquisition module configured to obtain original input containing product information, marketing points and public copy data, perform semantic matching retrieval on the public copy data to obtain candidate copy related to the product information; a generation module configured to construct slot rewriting instructions containing the candidate copy, the product information and the marketing points, input the slot rewriting instructions into a pre-generated language model, cause the pre-generated language model to fill in corresponding product information and marketing points at preset slot positions to generate a first marketing copy, and arrange the candidate copy, the product information and the first marketing copy into a supervised fine-tuning training sample; a first training module configured to perform supervised fine-tuning training on a preset basic language model by using the supervised fine-tuning training sample to obtain a first training model; an evaluation module configured to input new user product information and promotion requirements into the first training model to generate at least one second marketing copy, input the second marketing copy into a regularized evaluation system to score the second marketing copy and generate evaluation data; a second training module configured to construct partial order training samples according to the evaluation data, take the partial order training samples as reward signals, perform reinforcement learning training on the first training model to obtain a second training model; and an output module configured to call the second training model in a copy generation system to output target marketing copy based on user product information and promotion requirements.

[0008] In a third aspect, the embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above method.

[0010] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0011] By obtaining original input containing product information, marketing points and public copywriting data, performing semantic matching retrieval on the public copywriting data, and obtaining candidate copywriting related to the product information, a slotting rewriting instruction containing the candidate copywriting, the product information and the marketing points is constructed, the slotting rewriting instruction is input into a pre-generated language model, the pre-generated language model fills in the corresponding product information and marketing points at a preset slot position, a first marketing copywriting is generated, and the candidate copywriting, the product information and the first marketing copywriting are arranged into a supervised fine-tuning training sample; the supervised fine-tuning training sample is used to perform supervised fine-tuning training on a preset basic language model, and a first training model is obtained; new user product information and promotion requirements are input into the first training model, at least one second marketing copywriting is generated, the second marketing copywriting is input into a regularized evaluation system, the second marketing copywriting is scored and evaluation data is generated; the evaluation data is used to construct a partial order training sample, the partial order training sample is used as a reward signal to perform reinforcement learning training on the first training model, and a second training model is obtained; the second training model is called in a copywriting generation system, and target marketing copywriting is output based on the product information and the promotion requirements of the user. The present application can realize batch generation of marketing copywriting with high compliance and consistent constraints. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 is a flowchart of the marketing copywriting automatic generation method based on reinforcement learning provided by the embodiments of the present application;

[0014] Figure 2 is a structural schematic diagram of the marketing copywriting automatic generation device based on reinforcement learning provided by the embodiments of the present application;

[0015] Figure 3 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0017] The product marketing copy not only contains more communication skills, such as using suspense or exaggeration to attract readers, but also has more special regulations according to the industry regulations and advertising law. In addition, the marketing copy also involves more professional terms in the field, and the information of the parameters and characteristics of the product itself. However, the existing text generation large model lacks understanding of the related field and special regulations. When the input requirements are too much, the existing text generation large model is difficult to completely follow the relevant rules, and even produces fact illusion, not faithful to the product information.

[0018] In view of the defects of the existing solutions, the technical solution of the present application proposes the following solution ideas:

[0019] 1. For the target product and its promotion needs proposed by the user, the marketing expert rewrites and reconstructs the original content based on the publicly available text data on the Internet, combined with the open source large language model. In the rewriting process, the product information, product selling points and related marketing considerations of the user are integrated to generate creative marketing copy content. Based on the above generated creative copy data, a small sample training template for supervised fine-tuning is constructed, micro-tuning training data is generated, and a small-scale language model is supervised and fine-tuned, so as to optimize the output effect of the model in the specific marketing scene.

[0020] 2. Apply the fine-tuned language model to new user product needs, and refer to the specification requirements of the promotion platform by the marketing expert to evaluate the quality of the marketing copy automatically generated by the model, and form structured evaluation information. Use the open source large language model to summarize the above evaluation information, combine user needs and platform standards, establish a unified rule-based evaluation system, and build a systematic evaluation rule library.

[0021] 3. Further integrate the initial creative copy data and the small model generation results and their evaluation information, generate the partial order training data required for reinforcement learning through the open source large language model, form a high-quality feedback data set for training. Based on the above partial order training data, the fine-tuned language model is trained through reinforcement learning to obtain a creative marketing copy generation model with stable output quality and platform adaptation ability.

[0022] The marketing copy generated by the technical solution of the application is more in line with user needs and platform specifications; reduces the artificial cost of creative copywriting and batch generates marketing copies with diversity.

[0023] The content of the technical solution of the application will be described in detail below in combination with the drawings and specific embodiments.

[0024] Figure 1 is a flowchart of the marketing copy automatic generation method based on reinforcement learning provided by the embodiments of the application. As shown in Figure 1 The marketing copy automatic generation method based on reinforcement learning can specifically include:

[0025] S101, obtaining original input containing product information, marketing points and public copy data, performing semantic matching retrieval on the public copy data to obtain candidate copies related to the product information;

[0026] S102, constructing a slot rewriting instruction containing the candidate copy, the product information and the marketing points, inputting the slot rewriting instruction into a pre-generated language model, causing the pre-generated language model to fill in the corresponding product information and marketing points at the preset slot position, generating a first marketing copy, and arranging the candidate copy, the product information and the first marketing copy into a supervised fine-tuning training sample;

[0027] S103, performing supervised fine-tuning training on the preset basic language model using the supervised fine-tuning training sample to obtain a first training model;

[0028] S104, inputting new user product information and promotion requirements into the first training model to generate at least one second marketing copy, inputting the second marketing copy into a regularized evaluation system, scoring the second marketing copy and generating evaluation data;

[0029] S105, constructing a partial order training sample according to the evaluation data, taking the partial order training sample as a reward signal, performing reinforcement learning training on the first training model to obtain a second training model;

[0030] S106, calling the second training model in the copy generation system and outputting a target marketing copy based on the product information and promotion requirements of the user.

[0031] In some embodiments, the semantic matching retrieval is performed on the public copy data to obtain the candidate copy related to the product information, including:

[0032] The product information and the public copy data are respectively input into a semantic encoding network to obtain corresponding vector representations;

[0033] The vector representations of the public copy data are calculated and sorted based on a preset similarity measure;

[0034] According to a preset number or threshold, the highest similarity public document is selected from the ranking results as a candidate document.

[0035] Specifically, in the embodiment of semantic matching retrieval of public document data, the system as a whole follows the three-stage process of "text vectorization representation - similarity calculation ranking - high correlation candidate screening". This process belongs to the front-end data preparation link of the marketing copy automatic generation chain, and its output result directly determines the subsequent rewriting quality and compliance degree.

[0036] First, the key technical features and implementation process are summarized. The system splices the fields of function description, selling point keywords and target audience description in the target product information into a continuous text; at the same time, the public document data set to be retrieved is read and the same text standardization processing is performed on each document. All texts are uniformly input into the semantic coding network to obtain fixed-length vector representation; then the preset similarity measure (cosine similarity is used in this example) is used to calculate the correlation between product vectors and document vectors and generate ranking results; finally, according to the preset number of candidates and the minimum correlation threshold, high correlation documents are extracted from the ranking list as candidate materials.

[0037] The innovative technical features and important term concepts involved in the embodiment are explained below. The semantic coding network is a text double-tower model of a shared weight Transformer architecture, which is trained by a contrastive learning method, so that text pairs with similar semantics are closer in vector space. Vector representation refers to the fixed-length floating-point vector obtained by mapping the text through the semantic coding network, and 768-dimensional representation is used in this embodiment to balance expression ability and retrieval efficiency. Similarity measure is used to measure the closeness between text vectors, and the cosine similarity value range is 0 to 1, the larger the value, the higher the semantic correlation. The public document data set refers to the public text materials from social media, e-commerce evaluation and industry case library, which are used for matching retrieval after desensitization and segment denoising processing.

[0038] In the implementation process, the system first calls a semantic coding network for the product text and each piece of public text respectively to obtain product vector Vp and text vector set {Vi}. The network is pre-trained on a training set containing millions of industry texts, and the encoder parameters are frozen during the training phase to ensure consistency during inference. Then the system uses batch matrix multiplication to calculate the cosine similarity between Vp and {Vi} in parallel to generate a relevance list. To speed up the sorting and subsequent online calls, the relevance list along with the text vector will be written into an approximate nearest neighbor index based on the HNSW structure, facilitating the reuse of candidate texts when new product requests arrive. The system sets the upper limit of the candidate number to 40 and the minimum relevance threshold to 0.70; traverse the sorting results, if cos(Vp, Vi) ≥ 0.70, add the text corresponding to Vi to the candidate set, until the size of the set reaches 40 or the sorting list is exhausted. The selected text along with the similarity score, text source identifier and timestamp are written into the material buffer for direct reading by the slot rewriting module. To ensure the accuracy of retrieval in a multilingual scenario, the semantic coding network also adds a cross-language alignment loss during the training phase, so that Chinese and English texts can be compared in the same embedding space.

[0039] Through the above embodiments, the system can locate the original material that is highly related to the target product and has a good compliance basis in millions of public texts within milliseconds, significantly reducing the semantic drift risk in the rewriting stage and improving the response efficiency and accuracy of the overall generation link.

[0040] In some embodiments, constructing a slot rewriting instruction including candidate texts, product information and marketing points, comprising:

[0041] Performing text structure analysis on the candidate text to determine at least one replaceable semantic segment;

[0042] Writing a slot marker corresponding to the product information or marketing point at each replaceable semantic segment to obtain a rewriting template with placeholders;

[0043] Setting a filling rule for each slot marker, which specifies the mapping relationship between the slot marker and the product information field or the marketing point field;

[0044] Assembling the rewriting template and the filling rule into a unified instruction format to generate a slot rewriting instruction.

[0045] Specifically, before slot rewriting the candidate text, the system needs to complete four consecutive steps of "segment identification-slot labeling-rule binding-instruction packaging" to convert the original text into a rewriting instruction that can be directly parsed by the generation model.

[0046] Firstly, the key technical features are summarized. The system receives the candidate text output by the semantic matching module and performs text structure analysis on the text. The system uses syntax dependency, named entity recognition, and rhetoric pattern detection to identify semantic segments such as title area, product description area, function appeal area, emotional rendering area, and action call area. Insert placeholder-style slot markers inside each semantic segment determined to be replaceable. The slot markers correspond one-to-one with product information fields or marketing key point fields. Then set filling rules for each slot marker, including field mapping relationship, optional synonym set, length limit, and style constraint, to guide the language model to complete content filling in the decoding stage. Finally, assemble the rewritten template with placeholders and corresponding rules into a unified instruction format to provide input for subsequent language model rewriting.

[0047] In this embodiment, replaceable semantic segments refer to text sections in the candidate text that carry product attributes or marketing sentiments. By replacing the key components in these segments, different products can be quickly adapted. Slot markers are symbolic placeholder representations that prompt the language model to replace them with specified content when generating; typical styles include 〈PROD_NAME〉 or 〈BENEFIT_1〉. Filling rules are constraint sets defined at the slot marker level, which clearly define the mapping between markers and product information fields (such as name, specifications, core efficacy) or marketing key point fields (such as activity discounts, brand tone), and require the output to comply with character length, tone, and platform restrictions. The unified instruction format refers to packaging the rewritten template text, slot-field mapping table, and generation style parameters into a parseable input sequence, ensuring that the model can obtain all constraints at once.

[0048] The specific implementation process is as follows. The system first performs sentence and paragraph segmentation on the candidate text, locates replaceable semantic segments in combination with dependency trees and domain keyword dictionaries, for example, detecting "XX essence" as a product name placeholder position and "more transparent" as an efficacy description placeholder position in the sentence "XX essence makes the skin more transparent". The system then inserts slot markers such as 〈PROD_NAME〉 and 〈BENEFIT〉 at these positions and preserves the original sentence structure, generating a rewritten template with placeholders. The system then reads the product information table and marketing key point table, binds "product name" field for 〈PROD_NAME〉 and "lightening efficacy" field for 〈BENEFIT〉, and limits the output corresponding to 〈BENEFIT〉 to use positive emotional adjectives and be no longer than ten Chinese characters in length. If the text needs to meet the platform's length limit for the title, add a character upper limit constraint in the 〈TITLE〉 slot rule. The system places the rewritten template text at the beginning of the instruction, places the slot-field mapping and various rules at the end of the instruction, connects them with separators to form the overall input, and finally generates a slot-based rewriting instruction.

[0049] Through the above examples, the system enables the explicit embedding of product information and marketing points in a slot-rule manner while maintaining the overall rhetorical structure of the candidate text, providing a clear and executable filling guide for subsequent language model rewriting, thereby significantly improving the correspondence between the content and the product information in the rewriting stage and reducing compliance risks.

[0050] In some embodiments, the slot rewriting instruction is input into a pre-generated language model, which fills in the corresponding product information and marketing points at the preset slot position to generate a first marketing text, including:

[0051] Text encoding is performed on the slot rewriting instruction to obtain a semantic vector containing slot markers and mapping relationships;

[0052] When a pre-generated language model is called for conditional text generation, the slot markers are replaced in real time during the decoding stage according to the mapping relationship, and the corresponding product information or marketing points are written into the generated sequence;

[0053] At the end of generation, the first marketing text containing complete product information and marketing points is output, with identification metadata corresponding to the style of the text attached.

[0054] Specifically, after the completion of the slot rewriting stage, the system inputs the generated rewriting instruction as a whole text into a pre-generated language model. The model first performs text encoding on the entire instruction, i.e., it maps the character sequence to a semantic vector using its internal embedding layer and multi-layer Transformer self-attention structure. The slot marker itself is embedded as a unique placeholder vector, while the "slot-field mapping" explicitly indicated at the end of the instruction is parsed as a set of key-value pairs and also converted into vector form. The model establishes a soft alignment between the placeholder vector and the corresponding field vector during the encoding stage, so that the correct filling value can be retrieved during subsequent decoding.

[0055] Further, the generation process adopts a conditional text generation strategy: the model takes the existing context in the rewriting template as the decoding starting point, and triggers a dynamic replacement mechanism at each slot marker position. The dynamic replacement mechanism reads the mapping relationship through a lookup table, writes the pre-bound product information field or marketing point field into the current generation sequence, and performs post-processing according to the filling rules, such as applying synonym replacement or sentiment color transformation, to meet the character length and tone requirements. If the slot marker is located in the title or call-to-action segment, the model will adjust the sentence compactness and punctuation usage according to the register parameters in the style instruction. Throughout the generation process, temperature control and penalty repetition strategies are used to ensure that the output is both diverse and avoids information redundancy.

[0056] Further, when all the tags are replaced and the sentence generation termination condition triggers, the system outputs the complete first marketing copy with the attached metadata. The metadata includes the copy style label, generation timestamp, version number of the mapping rule used, and the source field of each slot filling, facilitating subsequent quality review and tracking. The copy and metadata are written to the cache area for the supervised fine-tuning training sample construction module to read directly.

[0057] Through the process design of the embodiment, the pre-generated language model accurately implants product information and marketing points while ensuring sentence coherence, realizes efficient analysis of the slot rewriting instruction and content filling, and provides clear structure and consistent semantics for subsequent supervised fine-tuning of high-quality training samples.

[0058] In some embodiments, the pre-set basic language model is supervised fine-tuned using the supervised fine-tuning training sample to obtain a first training model, comprising:

[0059] The supervised fine-tuning training sample is formatted according to a unified prompt template and label specification to generate a data set conforming to the input-output pair;

[0060] Quality screening is performed on the data set to eliminate samples that do not meet the semantic completeness threshold or the slot filling completeness rate threshold;

[0061] The screened data is divided into a training set and a validation set, and the pre-set basic language model is parameter updated and trained based on the training set;

[0062] The validation set is used to monitor the loss value in real time during the training process and terminate the training according to the early stopping condition to obtain the first training model.

[0063] Specifically, before the supervised fine-tuning stage is executed, the system has obtained the original sample containing the slot rewriting instruction, the candidate copy, and the first marketing copy. This stage first formats the above-mentioned sample according to a unified template, then completes double quality screening, and finally fine-tunes the pre-set basic language model with the required data to obtain the first training model that can output in the marketing scenario.

[0064] In some examples, in order to ensure that the model can accurately analyze the context and perform slot constraints, the system defines a "prompt template" for each sample. The template consists of three parts: the first part is the key-value pair description of the product information and marketing key points field; the second part is the slot rewriting instruction text; and the third part is the generated result placeholder identifier "〈GEN〉". At the same time, a "label specification" is attached to the sample, which includes style identifier, slot mapping version number, source channel identifier and timestamp, so that the model can perceive the language domain and data age during training. The formatting process is completed by a pipeline script, which reads the fields in turn, concatenates the key-value pairs according to the pre-set separator, and adds the "〈END〉" marker at the end of the text to ensure that the model can distinguish the input and output boundaries in a single forward process.

[0065] Further, the quality screening is divided into two levels of semantic completeness detection and slot filling completeness rate detection. Semantic completeness detection uses a double-tower retrieval model to calculate the relevance score of the generated text and the source instruction. If the score is lower than the threshold value 0.85, it is determined that the sample has content missing and is removed. Slot filling completeness rate detection confirms whether each slot in the generated text is replaced by legal text one by one through regular matching and rule library. If there is a situation that the slot is not replaced or still contains disabled words after replacement, the sample is also removed. After double screening, the coverage rate of the remaining samples is maintained at about 78% of the total amount of original samples, ensuring that the training signal has diversity and reliability.

[0066] In some examples, the qualified samples are randomly divided into a training set and a validation set in a ratio of nine to one. The training set is input into a pre-set basic language model, which has a parameter amount of about seven billion and has been pre-trained on general corpus. In order to reduce the memory occupation and quickly adapt to vertical scenes, the system inserts two layers of attention weight based on LoRA low-rank adaptation in the model, freezes the original parameters, and only trains the LoRA weight and the output layer. The optimizer selects AdamW, the initial learning rate is 3e-5, and the learning rate is warmed up in the first ten percent of steps using linear decay scheduling. The training batch size is limited to sixteen due to memory constraints, and gradient accumulation is used to expand the effective batch size to one hundred and twenty-eight to improve the stability of gradient estimation.

[0067] In some examples, during the training process, after completing one thousand gradient updates, the average cross-entropy loss and BLEU, ROUGE indicators are calculated on the validation set, and the "slot constraint compliance rate" is recorded, which is the proportion of all slots in the validation set samples that are correctly filled. The system sets an early stopping strategy: if the validation loss decreases by less than 0.01 for three consecutive times and the slot constraint compliance rate does not significantly improve, the training is terminated and the model snapshot with the best validation indicators is rolled back, which is the first training model. The model weights, optimizer state, vocabulary and LoRA configuration are archived together, and the training set version number and screening threshold are recorded in the metadata to ensure that subsequent experiments can be repeated.

[0068] Through the above-mentioned embodiments, the preset basic language model learns the slot filling rules and marketing register characteristics explicitly while maintaining the advantages of grammar and fluency, so that the generated results can accurately embed product information and comply with platform compliance requirements, providing a high semantic consistency and strong constraint compliance foundation model for the subsequent partial order reinforcement learning stage.

[0069] In some embodiments, the second marketing script is input into a regularized evaluation system, and the second marketing script is scored and evaluation data is generated, including:

[0070] An evaluation rule library containing compliance rules, content quality rules, and user preference weight templates is established;

[0071] Text analysis is performed on the second marketing script to generate corresponding feature representations;

[0072] Based on the evaluation rule library, the feature representations are matched with the rules, and the compliance dimension score, the content quality dimension score, and the user preference dimension score are calculated respectively;

[0073] According to the user preference weight template, the dimension scores are weighted and fused to obtain the comprehensive score of the second marketing script;

[0074] The structured evaluation data containing the second marketing script, the dimension scores, the comprehensive score, and the trigger rule identifier is generated.

[0075] Specifically, before inputting the second marketing script into the regularized evaluation system, the system first pre-constructs a three-layer evaluation rule library of "compliance rules-content quality rules-user preference templates". Among them, the compliance rules focus on the mandatory restrictions such as advertising law banned words, medical claims, and numerical exaggeration; the content quality rules cover textological indicators such as language fluency, selling point coverage, and sentence diversity; and the user preference templates record the relative weights of the target audience on the dimensions of emotional tone, length, and style label. The above rules are stored in the form of callable regular expressions, keyword Trie trees, or interpretable machine learning classifiers, and are accompanied by deduction or addition parameters in the rule nodes for subsequent scoring calculation.

[0076] Further, after the system receives the second marketing script, first, the text analysis component performs sentence segmentation, part-of-speech tagging, and named entity recognition, decomposes the script into title fragments, main body fragments, and call-to-action fragments, and extracts features such as keywords, sentiment polarity, and numerical quantifiers, and finally constructs a multi-dimensional feature representation vector. Subsequently, the rule matching engine iterates through the compliance rule nodes: if a violation term or exaggerated description is detected, the rule identifier is recorded and the compliance score is deducted according to the node setting; if no risk rule is triggered, the full score is retained according to the baseline. The content quality rule part calls the pre-trained language quality evaluation model to calculate the syntax completeness rate, repetition rate, and selling point coverage rate, and then maps them to the percentage score according to the weight. The user preference score is evaluated according to the weight proportion in the template, the emotional tone, the length of the script, and the style label of the script, for example, if the user prefers a lively tone and the script is marked as formal, the corresponding score will be deducted in this dimension.

[0077] Further, after the three-dimensional score calculation is completed, the scoring fusion module reads the user preference weight template corresponding to the current scene, and generates a comprehensive score by weighted average of the compliance score, the content quality score, and the user preference score. For example, the compliance dimension weight can be fixed at 0.5 to ensure the bottom line, and the content quality and user preference dimension weights are 0.3 and 0.2 respectively. The weight template can be dynamically adjusted to adapt to situations such as holiday promotions or new product releases. After the comprehensive score is generated, the system encapsulates the second marketing script, the dimension scores, the comprehensive score, and the triggered rule ID into a structured JSON, and stores it in the evaluation data warehouse, providing a data source for subsequent partial order sample construction.

[0078] Through the process of this embodiment, the system can automatically and interpretably quantify the performance of each marketing script in terms of compliance, text quality, and personalized adaptation, and output in a unified structured format, so that the subsequent reinforcement learning stage can directly reference these high-credibility scores as partial order labels, thereby effectively improving the model's ability to respond to the linkage between compliance and user preferences.

[0079] In some embodiments, partial order training samples are constructed according to the evaluation data, the partial order training samples are used as reward signals, reinforcement learning training is performed on the first training model to obtain a second training model, including:

[0080] Sample selection is performed on the evaluation data, and at least two candidate scripts are extracted from the second marketing script according to a preset sampling strategy;

[0081] The comprehensive scores of the candidate scripts are compared, the script with a higher score is determined as the preferred script, and the script with a lower score is determined as the secondary selected script, and a partial order label is generated according to the comparison result;

[0082] The preferred text, the second-choice text, the partial order label, and the input context corresponding to the same input context are packaged as a partial order training sample;

[0083] The first training model is used as a policy network, and the consistency degree of the generated result and the partial order label is used as an immediate reward to build a reinforcement learning environment using the partial order training sample;

[0084] A reinforcement learning algorithm based on policy optimization is used to update the parameters of the first training model, and the distribution shift in the updating process is constrained to obtain the second training model.

[0085] Specifically, in the embodiment of performing partial order reinforcement learning based on evaluation data, the system first selects samples from the generated second marketing text and its structured evaluation data, then constructs partial order training entries in the "preferred-second choice" manner, and then builds a reinforcement learning environment with the first training model as the policy network, guides the model learning to output the ranking preference through the reward signal, and finally obtains the second training model.

[0086] In order to ensure the representativeness of the samples, the system uses a hierarchical sampling strategy in the sample selection stage: first, divide all evaluation data into three layers of high, medium and low according to the comprehensive score interval, and then randomly select a number of samples in each layer to form a candidate set. For the same input context (i.e. the same product information and marketing points), the system extracts at least two texts with different scores and a difference of not less than ten points in the candidate set as comparison objects. The system compares the comprehensive scores of the two texts, marks the higher score as the preferred text, marks the lower score as the second-choice text, and generates a partial order label "1" indicating that the preferred text is better than the second-choice text. If the difference between the two texts is too small, the pair is discarded to avoid reward signal noise. The system then encapsulates the input context, the preferred text, the second-choice text, and the partial order label into a single partial order training sample, and writes it into the partial order dataset in JSONL format; Each sample also has a score version number and a rule base snapshot ID to ensure that the subsequent training results can be traced.

[0087] In constructing the reinforcement learning environment, the system takes the first training model as the policy network and fixes the evaluation rule base as the external reward model. Whenever the policy network generates two candidate scripts based on a certain input context, the system calls the rule base to calculate the comprehensive score and determines the pros and cons accordingly; if the model generates a ranking consistent with the partial order label, a positive reward is given, otherwise a negative reward is given. The reward function is designed as r = a • (consistency score) - β • KL, where the consistency score takes 1 when consistent and -1 when inconsistent; the KL term measures the divergence of the current policy distribution from the distribution at the initialization of the first training model, and β is a regularization coefficient to limit excessive deviation from the original language fluency. The system uses the PPO algorithm for policy optimization, each round of update contains multiple partial order samples, and uses small batch advantage estimation gradient and performs weight update; at the same time, gradient clipping and learning rate linear decay are enabled to prevent unstable training. After policy update, the consistency rate and average KL divergence are evaluated on the reserved validation partial order set immediately, and if the consistency rate does not improve or the KL divergence exceeds the threshold for three consecutive rounds, the training is terminated in advance and the best model snapshot is rolled back.

[0088] Through the above process, the partial order training samples provide interpretable relative preference signals to the model, and the policy optimization algorithm guides the model to adjust the generation probability distribution under the dual action of reward driving and distribution constraint, so that it tends to output marketing scripts with higher comprehensive scores and maintains language quality without degradation. The implementation results show that the second training model after the reinforcement learning phase improves the average comprehensive score by about fifteen points in offline evaluation, and the click-through rate and compliance pass rate are significantly higher than those of the first training model in online A / B testing.

[0089] The embodiment uses the multi-stage collaborative optimization method of "fine-tuning + rule-based scoring + partial order reinforcement learning" to construct fine-tuning templates and introduce a rule-based evaluation system, so that the small model has higher generation consistency and adaptability in specific marketing scenarios; reinforcement learning using partial order training data can effectively optimize the model's output ranking ability, so as to preferentially output higher quality candidates when generating multiple script candidates; the multi-round data construction and human-machine collaborative evaluation process in the embodiment enhances the controllability and interpretability of the training data, effectively avoiding the data noise and overfitting problems that easily occur in traditional model fine-tuning; the model training process is compatible with mainstream open source language model architectures, and has strong system scalability and platform deployment friendliness.

[0090] The following is an apparatus embodiment of the present application, which can be used to perform the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0091] Figure 2 is a structural schematic diagram of the marketing script automatic generation device based on reinforcement learning provided by the embodiment of the present application. As shown in Figure 2As shown, the marketing copy automatic generation device based on reinforcement learning comprises:

[0092] The acquisition module 201 is configured to acquire original input containing product information, marketing points, and public copy data, perform semantic matching retrieval on the public copy data, and obtain candidate copy related to the product information;

[0093] The generation module 202 is configured to construct a slot rewriting instruction containing the candidate copy, the product information, and the marketing points, input the slot rewriting instruction into a pre-generated language model, make the pre-generated language model fill in the corresponding product information and marketing points at a preset slot position, generate a first marketing copy, and arrange the candidate copy, the product information, and the first marketing copy into a supervised fine-tuning training sample;

[0094] The first training module 203 is configured to perform supervised fine-tuning training on a preset basic language model using the supervised fine-tuning training sample to obtain a first training model;

[0095] The evaluation module 204 is configured to input new user product information and promotion requirements into the first training model, generate at least one second marketing copy, input the second marketing copy into a regularized evaluation system, score the second marketing copy, and generate evaluation data;

[0096] The second training module 205 is configured to construct a partial order training sample according to the evaluation data, use the partial order training sample as a reward signal, perform reinforcement learning training on the first training model, and obtain a second training model;

[0097] The output module 206 is configured to call the second training model in a copy generation system and output a target marketing copy based on user product information and promotion requirements.

[0098] In some embodiments, Figure 2 The acquisition module 201 of the device inputs the product information and the public copy data into a semantic encoding network respectively to obtain corresponding vector representations, performs similarity calculation and sorting on the vector representations of the public copy data based on a preset similarity measure, and selects the public copy with the highest similarity from the sorting result as the candidate copy according to a preset number or threshold.

[0099] In some embodiments, Figure 2 The generation module 202 of the device performs text structure analysis on the candidate copy to determine at least one replaceable semantic segment, writes a slot marker corresponding to the product information or the marketing point at each replaceable semantic segment to obtain a rewriting template with placeholders, sets a filling rule for each slot marker, and the filling rule specifies the mapping relationship between the slot marker and the product information field or the marketing point field, assembles the rewriting template and the filling rule into a unified instruction format to generate the slot rewriting instruction.

[0100] In some embodiments, Figure 2 The generation module 202 performs text encoding on the slot rewriting instruction to obtain a semantic vector containing slot markers and mapping relationships; when calling a pre-generated language model for conditional text generation, the slot markers are replaced in real time at the decoding stage according to the mapping relationships, and the corresponding product information or marketing points are written into the generation sequence; at the end of generation, a first marketing copy containing complete product information and marketing points is output, and identification metadata corresponding to the copy style is attached.

[0101] In some embodiments, Figure 2 The first training module 203 formats the supervised fine-tuning training samples according to a unified prompt template and label specification to generate a data set conforming to the input-output pairs; performs quality screening on the data set to eliminate samples that do not meet the semantic completeness threshold or the slot filling completeness rate threshold; divides the screened data into a training set and a validation set, and updates the parameters of the pre-set basic language model based on the training set; in the training process, the validation set is used to monitor the loss value in real time, and the training is terminated according to the early stopping condition to obtain the first training model.

[0102] In some embodiments, Figure 2 The evaluation module 204 establishes an evaluation rule library containing compliance rules, content quality rules and user preference weight templates; performs text analysis on the second marketing copy to generate corresponding feature representations; performs rule matching on the feature representations based on the evaluation rule library to calculate compliance dimension scores, content quality dimension scores and user preference dimension scores, respectively; weights and fuses the dimension scores according to the user preference weight template to obtain a comprehensive score of the second marketing copy; generates structured evaluation data containing the second marketing copy, the dimension scores, the comprehensive score and the trigger rule identifier.

[0103] In some embodiments, Figure 2 The second training module 205 selects samples from the evaluation data, extracts at least two candidate copies from the second marketing copy according to a pre-set sampling strategy; compares the comprehensive scores of the candidate copies, determines the copy with a higher score as the preferred copy and the copy with a lower score as the secondary copy, and generates a partial order label according to the comparison result; packs the preferred copy, the secondary copy, the partial order label and the input context corresponding to the same input context into a partial order training sample; constructs a reinforcement learning environment using the partial order training sample, takes the first training model as the strategy network, and takes the consistency degree of the strategy network in the generation result and the partial order label as the instant reward; updates the parameters of the first training model using a reinforcement learning algorithm based on strategy optimization, and constrains the distribution deviation in the update process to obtain the second training model.

[0104] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0105] Figure 3 is a structural schematic diagram of an electronic device 3 provided by the embodiments of the present application. As shown in the figure, the electronic device 3 of the embodiments includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The processor 301 implements the steps in each of the above method embodiments when executing the computer program 303. Alternatively, the processor 301 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 303. Figure 3

[0106] By way of example, the computer program 303 can be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 303 in the electronic device 3.

[0107] The electronic device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device 3 can include but is not limited to the processor 301 and the memory 302. Those skilled in the art can understand that the electronic device 3 can include more or fewer components, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like. Figure 3 The electronic device 3 is only an example and does not constitute a limitation on the electronic device 3, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like.

[0108] The processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0109] ​The memory 302 can be an internal storage unit of the electronic device 3, for example, a hard disk or a memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 3. Further, the memory 302 can include both the internal storage unit and the external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0111] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0112] Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0113] In the embodiments of the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include multiple sub-modules or units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0114] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0115] In addition, each functional unit in the various embodiments of the present application can be integrated into one processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0116] If the integrated module / unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the flow of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0117] The above examples are only used to illustrate the technical solutions of the present application, but not limit the same; although the technical solutions of the present application are described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features thereof can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for automatically generating marketing copy based on reinforcement learning, characterized in that, include: Obtain raw input containing product information, marketing points, and publicly available copy data; perform semantic matching retrieval on the publicly available copy data to obtain candidate copy related to the product information; Construct a slotted rewriting instruction containing the candidate copy, the product information, and the marketing points; input the slotted rewriting instruction into a pre-generated language model; allow the pre-generated language model to fill in the corresponding product information and marketing points in preset slot positions; generate a first marketing copy; and organize the candidate copy, the product information, and the first marketing copy into a supervised fine-tuning training sample. The first training model is obtained by performing supervised fine-tuning training on the preset basic language model using the supervised fine-tuning training samples. Input new user product information and promotion requirements into the first training model to generate at least one second marketing copy. Input the second marketing copy into the rule-based evaluation system to score the second marketing copy and generate evaluation data. Based on the evaluation data, a partial order training sample is constructed. The partial order training sample is used as a reward signal to perform reinforcement learning training on the first training model to obtain the second training model. The second training model is invoked in the copywriting generation system to output target marketing copy based on the user's product information and promotion requirements.

2. The method according to claim 1, characterized in that, The step of performing semantic matching retrieval on the publicly available text data to obtain candidate text related to the product information includes: The product information and the publicly available text data are respectively input into a semantic coding network to obtain corresponding vector representations; The vector representations of the publicly available copywriting data are calculated and sorted based on a preset similarity metric. Based on a preset number or threshold, the publicly available text with the highest similarity is selected from the sorting results as the candidate text.

3. The method according to claim 1, characterized in that, The construction includes slotted rewriting instructions for the candidate copy, the product information, and the marketing points, including: The candidate texts are analyzed to determine at least one replaceable semantic fragment. Write a slot marker corresponding to the product information or the marketing point at each of the replaceable semantic fragments to obtain a rewrite template with placeholders; Set a fill rule for each of the slot marks, the fill rule specifying the mapping relationship between the slot mark and the product information field or marketing point field; The rewrite template and the filling rules are assembled into a unified instruction format to generate the slotted rewrite instruction.

4. The method according to claim 1, characterized in that, The step of inputting the slotted rewriting instructions into the pre-generated language model, causing the pre-generated language model to fill in the corresponding product information and marketing points in the preset slot positions, and generating the first marketing copy includes: The slotted rewrite instruction is text encoded to obtain a semantic vector containing slot markers and mapping relationships; When calling the pre-generated language model to generate conditional text, the slot marker is replaced in real time during the decoding stage according to the mapping relationship, and the corresponding product information or marketing points are written into the generation sequence. At the end of the generation process, the output is a first marketing copy containing complete product information and key marketing points, along with metadata tags corresponding to the copy style.

5. The method according to claim 1, characterized in that, The step of performing supervised fine-tuning training on a preset base language model using the supervised fine-tuning training samples to obtain a first training model includes: The supervised fine-tuning training samples are formatted according to a unified prompt template and label specification to generate a dataset that conforms to input-output pairs; Perform quality screening on the dataset to remove samples that do not meet the semantic integrity threshold or slot fill completeness threshold; The selected data is divided into a training set and a validation set, and the preset basic language model is trained by updating the parameters based on the training set. During training, the loss value is monitored in real time using the validation set, and training is terminated according to the early stopping condition to obtain the first training model.

6. The method according to claim 1, characterized in that, The step of inputting the second marketing copy into the rule-based evaluation system, scoring the second marketing copy, and generating evaluation data includes: Establish an evaluation rule library that includes compliance rules, content quality rules, and user preference weight templates; Perform text parsing on the second marketing copy to generate corresponding feature representations; Based on the evaluation rule base, rule matching is performed on the feature representation to calculate compliance dimension score, content quality dimension score and user preference dimension score respectively; The scores of each dimension are weighted and fused according to the user preference weight template to obtain the comprehensive score of the second marketing copy. Generate structured evaluation data that includes the second marketing copy, scores for each dimension, overall score, and trigger rule identifiers.

7. The method according to claim 1, characterized in that, The step of constructing a partially ordered training sample based on the evaluation data, using the partially ordered training sample as a reward signal, and performing reinforcement learning training on the first training model to obtain a second training model includes: The evaluation data is sampled by selecting at least two candidate copywritings from the second marketing copywriting according to a preset sampling strategy. Compare the overall scores of the candidate copy, determine the copy with the higher score as the preferred copy, and the copy with the lower score as the second-best copy, and generate partial order labels based on the comparison results; The preferred copy, the second-best copy, the partial order label, and the input context corresponding to the same input context are packaged into a partial order training sample; A reinforcement learning environment is constructed using the partially ordered training samples, with the first training model as the policy network, and the consistency between the policy network's generated results and the partially ordered labels is used as the immediate reward. The first training model is updated with parameters using a policy optimization-based reinforcement learning algorithm, and the distribution shift during the update process is constrained to obtain the second training model.

8. A device for automatically generating marketing copy based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire raw input containing product information, marketing points and publicly available copy data, perform semantic matching retrieval on the publicly available copy data, and obtain candidate copy related to the product information; The generation module is used to construct slotted rewriting instructions containing the candidate copy, the product information, and the marketing points, input the slotted rewriting instructions into the pre-generated language model, and fill the pre-generated language model with the corresponding product information and marketing points in the preset slot positions to generate the first marketing copy, and organize the candidate copy, the product information, and the first marketing copy into supervised fine-tuning training samples. The first training module is used to perform supervised fine-tuning training on a preset basic language model using the supervised fine-tuning training samples to obtain a first training model. The evaluation module is used to input new user product information and promotion requirements into the first training model, generate at least one second marketing copy, input the second marketing copy into the rule-based evaluation system, score the second marketing copy and generate evaluation data; The second training module is used to construct a partial order training sample based on the evaluation data, and use the partial order training sample as a reward signal to perform reinforcement learning training on the first training model to obtain the second training model. The output module is used to call the second training model in the copywriting generation system and output target marketing copy based on the user's product information and promotion requirements.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images