Content generation method and device, equipment, storage medium and product

Abstract

The invention relates to a content generation method and device, equipment, a storage medium and a product. The method comprises the steps that a problem is obtained and input to a target model, the target model is obtained through continuous optimization of a reward signal output by an automatic evaluation system, and the reward signal is used for optimizing and adjusting at least one part of model parameters in the target model; the automatic evaluation system is used for evaluating a plurality of evaluation indexes of the sampling data to obtain a reward signal corresponding to each evaluation index; the sampling data is user data sampled from a database; and generating a corresponding answer based on the question through the target model. Therefore, the quality of the searched answers can be obviously optimized, the answers better fit the preference of the user (human) while the correctness is ensured, the complex commercial application requirements can be met, the robustness is high, and the whole evaluation process is traceable and high in reliability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a content generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, large-scale language models have emerged, which can directly generate answers based on user queries. Especially in intelligent search scenarios, artificial intelligence-based search methods can provide users with high-quality answers.

[0003] Currently, intelligent search primarily relies on manual review or evaluation based on a single subjective metric to assess the quality of answers. This lacks objectivity and fails to meet the needs of complex commercial applications.

[0004] Therefore, there is an urgent need for a content generation solution that can optimize the quality of search results and meet the needs of complex business applications. Summary of the Invention

[0005] Therefore, it is necessary to provide a content generation method, apparatus, computer device, computer-readable storage medium, and computer program product that can optimize the quality of search results and meet the needs of complex business applications, in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a content generation method, the method comprising:

[0007] The problem is identified and input into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database.

[0008] The target model generates a corresponding answer based on the question.

[0009] In one embodiment, before obtaining the problem and inputting it into the target model, the method further includes:

[0010] User data is sampled from a database, and a training dataset is constructed based on the user data. The user data includes at least one of the following: questions, search results, time information, and historical search data.

[0011] The content generation model is trained based on the training dataset to obtain an initial model;

[0012] A portion of the training dataset is selected and automatically evaluated by an automated evaluation system, and then manually reviewed to obtain the automatic evaluation results and the manual review results.

[0013] Based on the automatic evaluation results and / or the manual review results, a reward signal is determined for each evaluation indicator, wherein the reward signal includes a score and a reason;

[0014] The overall reward is obtained by weighted aggregation based on the weight and score of each reward signal.

[0015] The initial model is optimized based on the overall reward to obtain multiple candidate models;

[0016] The target model is selected from the multiple candidate models.

[0017] In one embodiment, training the constructed content generation model based on the training dataset to obtain an initial model includes:

[0018] Based on the training dataset, the content generation model is trained using the group relative strategy optimization GRPO algorithm to obtain the initial model.

[0019] In one embodiment, the automated evaluation system includes two hierarchical structures: content evaluation of model responses and end-to-end evaluation. The content evaluation of model responses is used to evaluate responses from at least one of the following dimensions: answer errors, presentation format, language style, and security risk control. The end-to-end evaluation is used to evaluate responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy.

[0020] Here, the bottom line refers to the constraints that the answer must meet, and the optimization refers to the positive incentives that the answer must meet the requirements.

[0021] In one embodiment, after obtaining the automatic evaluation results and the manual review results, the method further includes:

[0022] The evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results are compared with the review scores corresponding to the two dimensions of each aspect in the manual review results to determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the percentage of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension.

[0023] When the consistency ratio is lower than a preset value, the automated evaluation system is iteratively optimized using a preset dataset of difficult cases until the consistency ratio of the automated evaluation results output by the automated evaluation system is not lower than the preset value.

[0024] In one embodiment, before weighted aggregation to obtain the overall reward based on the weights and scores corresponding to each reward signal, the method further includes:

[0025] The weights of each reward signal are determined based on its importance and its impact score on model performance; the impact score of each reward signal on model performance is set manually.

[0026] In one embodiment, selecting the target model from the plurality of candidate models includes:

[0027] The performance of each candidate model is evaluated to obtain the corresponding performance evaluation results;

[0028] Based on the performance evaluation results, the target model was selected.

[0029] In one embodiment, the evaluation of the model performance of each candidate model to obtain the corresponding performance evaluation result includes:

[0030] The same question is input into the candidate model, and multiple corresponding answers are output.

[0031] The multiple answers were sorted manually.

[0032] Based on the ranking of the answers, the performance evaluation results of the candidate models are determined.

[0033] In one embodiment, the method further includes:

[0034] User data is sampled from the database, and a verification dataset is constructed based on the user data;

[0035] The answers generated by the target model are validated using the validation dataset to determine whether the answers match user preferences.

[0036] If the answer does not match the user's preferences, the weights corresponding to each reward signal will be adjusted.

[0037] Secondly, this application also provides a content generation apparatus, the apparatus comprising:

[0038] An input module is used to acquire questions and input them into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. The reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system is used to evaluate the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each evaluation indicator. The sampled data is user data sampled from a database.

[0039] The result output module is used to generate a corresponding answer based on the question using the target model.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0041] The problem is identified and input into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database.

[0042] The target model generates a corresponding answer based on the question.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0044] The problem is identified and input into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database.

[0045] The target model generates a corresponding answer based on the question.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] The problem is identified and input into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database.

[0048] The target model generates a corresponding answer based on the question.

[0049] The aforementioned content generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire a question and input it into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least a portion of the model parameters in the target model. The automated evaluation system evaluates sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database. The target model generates a corresponding answer based on the question. This significantly optimizes the quality of search answers, meets complex business application needs, exhibits strong robustness, and ensures the entire evaluation process is traceable and aligns with user preferences. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a diagram illustrating the application environment of a content generation method in one embodiment.

[0052] Figure 2 This is a flowchart illustrating a content generation method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the architecture of an automated evaluation system according to an embodiment of this application;

[0054] Figure 4 This is a flowchart illustrating the content generation method in another embodiment;

[0055] Figure 5 This is a schematic diagram of the model optimization process in one embodiment of this application;

[0056] Figure 6This is a timing diagram of the content generation method in the first embodiment of this application;

[0057] Figure 7 This is a timing diagram of the content generation method in the second embodiment of this application;

[0058] Figure 8 This is a structural block diagram of a content generation device in one embodiment;

[0059] Figure 9 This is a structural block diagram of the content generation apparatus in another embodiment;

[0060] Figure 10 This is a structural block diagram of the content generation device in yet another embodiment;

[0061] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] To facilitate understanding of the technical solutions described in the various embodiments of this application, some terms that may appear in the various embodiments of this application will be explained first.

[0064] For example, the term "Large Language Model (LLM)" used in this application is one of the most groundbreaking technologies in the field of artificial intelligence in recent years. Trained on massive amounts of text and possessing hundreds of billions of parameters, this deep learning model can understand and generate human language, demonstrating powerful general-purpose capabilities in tasks such as dialogue, writing, and reasoning. It marks a significant leap in Natural Language Processing (NLP) from traditional task-specific models to general-purpose language understanding and generation capabilities. NLP is a technology in artificial intelligence that enables computers to understand, generate, and process human language. Its development has evolved from rule-based to statistical methods to deep learning, and it is widely used in scenarios such as search, translation, and chatbots.

[0065] For example, the term "Reward Model (RM)" used in this application refers to a neural network in reinforcement learning used to learn and predict human preferences. It scores artificial intelligence (AI) behavior by comparing data rather than absolute ratings, and has become a core component of large language models aligning with human values. Specifically, the role of the reward model is to score the quality of the text output by the large language model based on prompts during the training phase, outputting a reward value. This reward value characterizes whether the text output by the large language model conforms to user preferences. A higher reward value indicates that the text conforms more closely to user preferences and has higher quality; a lower reward value indicates that the text deviates significantly from user preferences and has lower quality. By using the reward value output by the reward model as a reward signal for reinforcement learning to optimize the large language model, the model is optimized towards increasing the reward value, thus training a large language model capable of generating text that conforms to user preferences.

[0066] For example, the term "end-to-end evaluation" used in this application is an evaluation method that emphasizes testing or analysis of the entire process from input to output. Compared with traditional modular evaluation, end-to-end evaluation can more realistically reflect system behavior, and is especially suitable for highly integrated AI models or complex distributed systems.

[0067] For example, the term "Group Relative Policy Optimization (GRPO)" used in this application refers to a model reinforcement algorithm that generates optimization signals for the model through intra-group relative comparisons. The GRPO algorithm aims to improve the efficiency and stability of reinforcement learning. Its main feature is that it optimizes the policy model through intra-group relative rewards, rather than relying on a traditional critic model. This method significantly reduces memory usage and computational cost during training while maintaining the stability and efficiency of policy updates. In simple terms, for the same input prompt, the current model generates multiple outputs in parallel (e.g., 4–64 responses), forming a "response group." Then, a reward model or rule (such as answer correctness) is used to score each response. The rewards for all responses within the group are normalized (subtracting the mean and dividing by the standard deviation) to obtain the relative advantage value for each response. Finally, the policy gradient is updated based on this relative advantage value, while introducing a Kullback-Leibler Divergence (KL) penalty term to prevent the output from deviating too far from the initial model, thus ensuring stability.

[0068] For example, the term "reward signal" used in this application refers to a reward value for an evaluation metric output by an automated evaluation system. This value typically includes a score and a reason, and is used to update and train the text generation strategy of a large language model to optimize the content generation model. After multiple rounds of training, the content generation model can be optimized towards a gradually increasing target reward average, resulting in a target model that can generate text that conforms to user preferences.

[0069] For example, the term "automated evaluation system" used in this application can be understood as a white-box reward system, which uses software tools, evaluators and other technical means to efficiently and objectively evaluate the output content of the content generation model (target model) and feed back reward signals based on the evaluation results, thereby guiding the behavior of the content generation model and achieving the goal of continuously optimizing the content generation model.

[0070] With the widespread deployment of large-scale language models in open-domain scenarios such as business, customer service, and content creation, more and more answers are generated directly by these models. This significantly increases the pressure on platforms to filter information and control quality, making the reliability, interpretability, and alignment with human preferences of their output key challenges. In objective fact-based questions, bottom-line issues are particularly prominent: common manifestations include numerical and date errors, treating outdated information as the latest conclusions, failure to provide authoritative sources or inconsistencies between evidence and conclusion, and answering with overconfidence even when evidence is insufficient. These errors directly damage the platform's credibility and compliance in search and community Q&A scenarios, leading to complaints and legal risks. Therefore, higher demands are placed on the factual verifiability, traceability, and calibration of answers. However, purely manual quality inspection suffers from high unit sample costs, extended review times, and susceptibility to shifting standards, making it difficult to match the pace of deployment and compliance requirements. Automated evaluation can help platforms quickly identify problematic answers and stabilize output quality, thereby alleviating the practical bottlenecks of "difficulty in scaling manual review and inconsistent deployment control."

[0071] It should be understood that existing related technologies rely solely on a single subjective score or preference-based reward model, which has high signal-noise ratio and weak interpretability. It is difficult to pinpoint "what is wrong and why it is wrong", and it is also difficult to provide directional evidence with statistical certainty. This results in a trial-and-error-based and dispersed state of resource investment, which can easily lead to problems such as "the total score improves but the factual value deteriorates" and "offline improvement but online degradation".

[0072] To address the problems existing in the relevant technologies, this application aims to provide a content generation method that integrates the evaluation layer and the publishing layer into a unified mechanism. Specifically, in the evaluation layer, a multi-dimensional rather than singular metric stack is adopted to combine key indicators such as accuracy, calibration, robustness, fairness (user preference) / toxicity (such as compliance) and efficiency in the same task context, thereby improving the objectivity of the answers and enabling them to meet the needs of complex business applications.

[0073] The content generation method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. A target model is deployed on server 104. A question is obtained through terminal 102 and input into the target model. The target model is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain the reward signal corresponding to each indicator. The sampled data is user data sampled from a database. The target model generates corresponding answers based on the questions. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0074] In one exemplary embodiment, such as Figure 2 As shown, a content generation method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S202. Wherein:

[0075] Step S201: Obtain the problem and input it into the target model.

[0076] In this embodiment, the target model is pre-deployed on the server. Alternatively, the target model can also be in the form of an application. When the target model is in the form of an application, it can be installed on the terminal side and run in both offline and online states.

[0077] The questions can be in various forms, such as text, voice, a combination of text and images, and video. By entering the questions into the input box on the terminal application interface, the questions can be input into the target model.

[0078] It should be understood that the core feature of large language models lies in their massive scale of parameters, which enables them not only to perform conventional tasks such as translation, summarization, and question answering, but also to exhibit capabilities that previous models did not possess, such as zero-shot reasoning and stepwise thinking.

[0079] In this embodiment, the target model can be an optimized large language model (e.g., a model obtained by improving and optimizing large language models such as GPT series, LLaMA series, and Qwen series).

[0080] For example, interaction with the target model can be conducted through dialogue or by entering questions in the input box (search box) of the application interface. During the interaction, system prompts can be set to define rules, background information, or global instructions for the target model's behavior or output format. By inputting specific control instructions or information into the target model, the output content of the target model can be structurally constrained and standardized, guiding the target model to respond to subsequent queries in a specific manner.

[0081] As an optional example, the target model in this embodiment is continuously optimized by the reward signal output by the automated evaluation system. The reward signal is used to optimize and adjust at least some of the model parameters (model behavior) in the target model. The automated evaluation system is used to evaluate the sampled data using multiple evaluation indicators to obtain the reward signal corresponding to each evaluation indicator. The sampled data is user data sampled from the database.

[0082] In this embodiment, multiple evaluation metrics for the answers can be set according to the actual application scenario, and each evaluation metric corresponds to a reward signal. The reward signal can be used to guide the learning of the target model (providing immediate feedback on the behavior of the target model to guide its subsequent behavior), thereby significantly improving the real-time learning and dynamic adaptation capabilities of the target model.

[0083] For example, Figure 3 This is a schematic diagram of the architecture of an automated evaluation system according to one embodiment of this application, as shown below. Figure 3 As shown, the automated evaluation system in this embodiment includes two hierarchical structures: content evaluation of model responses and end-to-end evaluation. Content evaluation assesses responses from at least one of the following dimensions: answer errors, presentation and / or formatting, language style, and security risk control. End-to-end evaluation assesses responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy. It should be noted that the bottom line refers to the constraints that the response must meet, while improvement refers to the requirements that positively incentivize the response to achieve.

[0084] The automated evaluation system in this embodiment constitutes a white-box, interpretable reward system. It can provide transparent and traceable scoring criteria for each subdivided evaluation dimension, ensuring the interpretability and controllability of each iteration in the target model (training and / or optimization process). Whether it is content evaluation of the model's answer or end-to-end evaluation, the evaluation involves two parts: the first part is rule-based evaluation, and the second part is evaluation based on the large language model.

[0085] Rule-based evaluation primarily targets baseline indicators with clear and objective judgment standards, such as display format, content length, and the appearance of specific keywords. Therefore, the automated evaluation system employs deterministic rule algorithms for efficient and low-cost evaluation. This method offers fast response times, stable results, and completely transparent judgment logic.

[0086] Among these, the evaluation based on large language models primarily targets indicators requiring deep semantic understanding and complex logical reasoning (such as factual errors, demand fulfillment, and logical consistency). Therefore, the automated evaluation system employs a powerful "judgment model," guided by structured instruction templates to score specific dimensions and force it to output detailed reasons for deductions. This ensures that while the evaluation process relies on a large model, its decision-making process remains auditable and interpretable.

[0087] It should be understood that end-to-end evaluation can be broken down into multiple stages. By defining different evaluation benchmarks in the configuration file, the reward mechanism of the automated evaluation system can be seamlessly applied to key upstream nodes of the answer generation task, thus enabling its application to the entire process from a single answer generation stage to complex tasks.

[0088] For example, during the answer generation phase, the automated evaluation system focuses on the quality of the final generated text content. Based on the configuration, the automated evaluation system activates all baseline and optimization metrics related to answer content, format, style, and security, providing direct optimization signals for the target model to generate high-quality summaries or answers.

[0089] For example, during the requirements understanding phase, the automated evaluation system assesses whether the target model's understanding of the user's original query intent is accurate and comprehensive.

[0090] For example, during the supply quality phase, that is, before the model generates answers, an automated evaluation system assesses the quality, relevance, and completeness of intermediate information (e.g., search results, API calls) retrieved, integrated, or planned by the target model.

[0091] For example, in the external verification accuracy phase, the automated evaluation system assesses whether the content generated by the target model can be accurately verified by external knowledge sources.

[0092] It should be noted that the aforementioned multi-stage evaluation capabilities enable end-to-end, multi-dimensional diagnosis and optimization of the target model search process, achieving a systematic capability improvement from local components to the global process.

[0093] Among them, the bottom-line dimension focuses more on content and factual errors (e.g., core or partial content contradicts known facts (factual error), content has internal logical contradictions, inappropriate responses to questions that should not be answered (refusal to answer questions), outdated information, and generated text and images that are completely unrelated), presentation and formatting errors (e.g., garbled characters, non-standard language mix, text length that is excessively long or short, and formatting errors in structured content such as lists), supply and demand mismatch errors (e.g., the model's answer completely deviates from the user's intent, fails to provide any effective information, or the provided supply content is completely unrelated to the user's needs), and security risk control (e.g., identifying and punishing unsafe content to ensure the compliance of the model's output), etc.

[0094] It should be understood that the bottom-line dimensions focus on imposing strict, black-and-white "hard constraints" on the target model's output, aiming to identify and punish all fundamental errors that would seriously harm the user experience and are objectively verifiable. These evaluation metrics constitute the minimum guarantee of the target model's output quality, effectively preventing the target model from outputting fundamentally flawed or harmful content during training.

[0095] Among these, the optimization dimensions focus more on content performance, such as content richness and organization (e.g., the richness of effective statements in the answer, the quality and quantity of components (such as images and lists), the overall aesthetic appeal of the answer's format, and the rationality of the information point organization structure), demand satisfaction (assessing whether the target model fully understands and satisfies the user's secondary or implicit needs, going beyond the literal meaning of the question), and supply quality and diversity (encouraging the model to achieve diversity within the information set when providing relevant information and avoiding conflicts between different information points), etc. For the optimization dimensions, positive incentives can be used to guide the target model to explore more deeply, in order to generate higher-quality, richer content that better meets the user's deeper or potential needs.

[0096] It should be understood that these metrics for optimization dimensions are used to reward outstanding model performance and are key to stimulating the capabilities of the target model and raising its performance ceiling.

[0097] For example, regarding incorrect answers, the bottom-line dimensions may include: core content errors or illusions, partial content errors or illusions, refusal to answer, logical contradictions in the content, exceeding the time limit, irrelevance of text and images, etc.; the optimization dimensions may include: answer richness (such as the amount of information contained, the level of detail, and the comprehensive coverage of the topic), the proportion of irrelevant content, semantic redundancy, the quality of images and / or components, etc.

[0098] Furthermore, potential biases and errors during model training can be amplified, leading to factual deviations and distortions, resulting in core content errors or illusions.

[0099] Among them, "refusal to answer" means that the model's response is similar to: "Sorry, no relevant information was found", "Cannot answer", "It is suggested that you use other search methods", etc.

[0100] Among these, logical contradictions are more common in scenarios with strong logical coherence. For example, in a math problem, the question might state that the length of line segment AB is 3, but in subsequent reasoning or search results, the length of line segment AB is found to be 2.5. This results in a logical contradiction.

[0101] Among them, "exceeding timeliness" refers to the fact that the content generation model, due to its static nature, is unable to respond to the latest information when faced with high timeliness demands (policy updates, market dynamics, etc.).

[0102] Among these, the irrelevance of text and image content generally occurs in scenarios where answers are based on images, such as when the generated answer is unrelated to the content of the image.

[0103] For example, regarding presentation and / or formatting, bottom-line dimensions may include: garbled characters, repetition, mixed languages, excessively long or short length, excessively long lists, etc.; optimization dimensions may include: short answer format, answer primacy, component richness, reasonable component triggering, etc.

[0104] For example, regarding language style, the baseline dimension could include: the distinguishability of the answers generated by other language models (such as Gemini); the optimization dimension can be freely defined according to actual needs.

[0105] For example, security risk control mainly involves the bottom-line dimension, such as whether it involves sensitive content, illegal content, etc.

[0106] For example, regarding understanding requirements, the bottom-line dimension may include irrelevant answers (match between the answer and the question); the optimization dimension may include the satisfaction of some secondary requirements.

[0107] For example, regarding the quality of supply, the bottom line dimension may include: no supply, completely irrelevant, etc.; the optimization dimension may include: set-wise diversity (generally refers to the performance or ability of a model or algorithm in processing a set of data in a specific task or benchmark test), demand satisfaction, supply conflict, etc.

[0108] For example, external validation accuracy mainly involves the bottom-line dimension, such as the external validation accuracy of objective knowledge.

[0109] It should be understood that the automated evaluation system in this embodiment provides a tiered reward architecture, which clearly divides the evaluation of multiple aspects into two dimensions: baseline and improvement, thereby ensuring that while pursuing more excellent content, the baseline of content quality and security can be maintained.

[0110] In this embodiment, by adopting the sequential logic of "first meeting the bottom line, then seeking optimization", the target model is ensured to strictly adhere to the basic requirements of factual accuracy and safety compliance while optimizing performance, thus guaranteeing the stability and reliability of the model iteration process.

[0111] Step S202: Generate the corresponding answer based on the question using the target model.

[0112] In this embodiment, the target model can output the answer to the question, which may include the reasoning chain and multiple possible answers.

[0113] Optionally, when multiple possible answers are involved, they can be prioritized according to their matching degree, making it easier for users to choose the most suitable answer.

[0114] It should be understood that the target model in this embodiment is a continuously optimized dynamic model. As the user data in the database continues to grow, the target model can be continuously optimized through the reward signals output by the automated evaluation system, thereby improving the quality of the responses output by the target model and ensuring that the responses are correct while better aligning with user (human) preferences.

[0115] In the above content generation method, a question is obtained and input into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation metrics to obtain a reward signal corresponding to each metric. The sampled data consists of user data sampled from a database. The target model then generates corresponding answers based on the question. This significantly optimizes the quality of search results, meets the needs of complex business applications, demonstrates strong robustness, and ensures the entire evaluation process is traceable and aligns with user preferences.

[0116] In another exemplary embodiment, such as Figure 4 As shown, a content generation method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S401 to S409. Wherein:

[0117] Step S401: Sample user data from the database and construct a training dataset based on the user data.

[0118] In this embodiment, the database can be a local database on a server or a cloud database, storing accumulated user data. This user data includes at least one of the following: questions, search results, time information, and historical search data. Historical search data can be records of searches performed through browsers or various applications, or records of searches performed using manual search tools.

[0119] Optionally, user data can be preprocessed to obtain data in a uniform format. For example, it can be integrated in the form of "question-answer-time-historical search data" to facilitate the subsequent construction of a training dataset.

[0120] Step S402: Train the constructed content generation model based on the training dataset to obtain the initial model.

[0121] In this embodiment, by constructing a training dataset, model training can be effectively performed even in the absence of sufficient historical data, thereby improving the readability of the model, accelerating the convergence process, and enhancing the final performance.

[0122] For example, the content generation model is trained based on the training dataset and the GRPO algorithm optimized by the population relative strategy to obtain the initial model.

[0123] As an alternative example, during training, multiple different answers are sampled for the same question using the old strategy, and each answer is scored according to a reward model or rule. Finally, the absolute reward is converted into a relative advantage, and the answer is adjusted based on this relative advantage. This training process is repeated until the initial model is obtained, provided the number of iterations or convergence conditions are met.

[0124] Step S403: Select a portion of data from the training dataset and perform automatic evaluation and manual review respectively through an automated evaluation system to obtain automatic evaluation results and manual review results.

[0125] In this embodiment, to ensure the accuracy of the automatic evaluation results, the same portion of data will be manually reviewed. That is, a manual evaluation will be conducted on multiple aspects and two dimensions to obtain corresponding scores and reasons. Optionally, the manual review process can also mimic the automatic evaluation process, dividing the overall evaluation process into multiple stages (corresponding to multiple evaluation nodes, such as the demand understanding stage, supply quality stage, answer generation stage, etc.), and then manually scoring each stage and noting the reasons.

[0126] In this embodiment, through a modular, multi-stage evaluation design, the framework of the automated evaluation system can serve as a fundamental capability, flexibly adapting to AI tasks with different architectures without requiring restructuring of the core system. This feature provides a strong engineering foundation for supporting diverse business scenarios and accelerating technology iteration.

[0127] Optionally, the evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results are compared with the review scores corresponding to the two dimensions of each aspect in the manual review results to determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the proportion of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension. When the consistency ratio is lower than the preset value, the automatic evaluation system is iteratively optimized using a preset dataset of difficult cases until the consistency ratio of the automatic evaluation results output by the automatic evaluation system is not lower than the preset value.

[0128] The "difficult case dataset" refers to cases (questions and answers) where the consistency of the automated assessment results is lower than a preset value. By setting difficult cases, the performance of the automated assessment system can be continuously optimized to maintain consistency with the results of human review.

[0129] As an alternative example, human review can supervise the accuracy of automated evaluation systems. For instance, it can be set that the automated evaluation results and the human review results maintain a consistency ratio of no less than 90%. When bottom-line dimensions are involved, a veto power can be set, meaning that if the automated evaluation results show inconsistencies in the bottom-line dimensions, the automated evaluation result is considered erroneous.

[0130] This embodiment establishes a threshold for the automated evaluation system to go live, namely, the automated evaluation system needs to continuously approach human preferences. By monitoring the consistency ratio, the reward structure in the automated evaluation system is significantly improved in the baseline and optimization dimensions, as well as in answer generation and end-to-end multi-level coding.

[0131] It should be understood that this embodiment, through mandatory quantization standards and a systematic alignment process (such as increasing the rejection rate of R1 from 70% to 95%), ensures that the automatic evaluation results are highly consistent with human expert judgment (human review). This guarantees from the source that the reward signal used for reinforcement learning has a high signal-to-noise ratio, thereby significantly improving the efficiency and final effect of model alignment.

[0132] It should be understood that once the evaluation components of the automated evaluation system pass the entry threshold, large-scale, automated evaluation can be carried out, thereby significantly reducing the reliance on expensive and inconsistent manual quality inspections, and providing cost-effective and consistent quality assurance for the rapid and stable iteration of the target model.

[0133] Step S404: Based on the automatic evaluation results and / or manual review results, determine the reward signal corresponding to each evaluation indicator. The reward signal includes a score and a reason.

[0134] In this embodiment, multiple reward signals output by the automated evaluation system can be uniformly preprocessed and calibrated to ensure comparability in terms of numerical scale, confidence interval, and stability. Then, each reward signal is scored, and the reason for the score is noted.

[0135] This embodiment can accurately pinpoint performance bottlenecks in complex systems (such as retrieval enhancement systems) and distinguish whether the root cause of the problem stems from input comprehension bias, insufficient intermediate information quality, or defects in the final generation stage. This capability transforms optimization from passively correcting the final output to proactively and fundamentally improving the internal components of the system, significantly enhancing the overall system's optimization efficiency and final performance.

[0136] Step S405: Based on the weight and score corresponding to each reward signal, the overall reward is obtained by weighted aggregation.

[0137] Optionally, before executing step S405, the weights corresponding to each reward signal are determined based on the importance of each reward signal and the impact score of each reward signal on the model performance; the impact score of the reward signal on the model performance is set manually.

[0138] Step S406: Optimize the initial model based on the overall reward to obtain multiple candidate models.

[0139] For example, Figure 5 This is a schematic diagram of the model optimization process in one embodiment of this application, such as... Figure 5As shown, online data (or data from a server database) is divided into two branches. One branch constructs a training dataset for the content generation model, which is then used to enhance the model using the GRPO algorithm, resulting in an initial model. The other branch constructs a training dataset for both manual review and automated evaluation, yielding manual review results and automated evaluation results. Furthermore, the manual review results can be used to verify the consistency of the automated evaluation results output by the automated evaluation system, ensuring the accuracy of the automated evaluation system. The automated evaluation system can output reward signals corresponding to multiple evaluation metrics based on the automated evaluation results and / or manual review results, thereby optimizing the initial model using these reward signals to obtain multiple candidate models.

[0140] It should be understood that in this embodiment, multiple content generation models can be trained and optimized simultaneously, thereby obtaining multiple candidate models, which facilitates the subsequent selection of the model with the best performance from these candidate models as the target model.

[0141] Step S407: Select the target model from multiple candidate models.

[0142] In this embodiment, the performance of each candidate model is evaluated to obtain the corresponding performance evaluation results; based on the performance evaluation results, the target model is selected.

[0143] For example, the same question is input into the candidate model, and multiple corresponding answers are output; the multiple answers are sorted manually; and the performance evaluation result of the candidate model is determined based on the sorting of the answers.

[0144] Step S408: Obtain the problem and input it into the target model.

[0145] In this embodiment, the target model obtained in step S407 is deployed on the server, and the terminal communicates with the server. The terminal can interact with the server through the human-computer interaction operation of the application interface.

[0146] The input questions can take various forms, including text, voice, a combination of text and images, and video. Taking a terminal as an example, the question can be entered into the input box on the terminal application interface, thereby sending the question to the target model on the server side.

[0147] In this embodiment, end users can interact with the target model through dialogue or by entering questions in the input box (search box) of the application interface. During the interaction, system prompts can be set to define rules, background information, or global instructions for the target model's behavior or output format. By inputting specific control instructions or information into the target model, the output content of the target model is structurally constrained and standardized, guiding the target model to respond to subsequent queries in a specific manner.

[0148] For example, see Figure 5 As shown, the automated evaluation system in this embodiment can provide transparent and traceable scoring criteria for each subdivided evaluation dimension, ensuring the interpretability and controllability of each iteration in the target model (training process and / or optimization process).

[0149] For example, if the target model's output includes a long reasoning chain, then when evaluating this answer, the reasoning chain can be divided into multiple stages (or multiple evaluation nodes can be set). Each stage or evaluation node can be automatically evaluated from two dimensions to obtain a score and the reasons. This makes the automatic evaluation results traceable and interpretable, providing a direct basis for optimization during the optimization of the target model.

[0150] For example, see Figure 3 As shown, both content evaluation of the model's response and end-to-end evaluation involve two parts: rule-based evaluation and large language model-based evaluation. Rule-based evaluation primarily targets baseline indicators with clear and objective judgment standards, such as presentation format, content length, and the appearance of specific keywords. Therefore, automated evaluation systems can employ deterministic rule algorithms for efficient and low-cost evaluation. This method offers fast response times, stable results, and completely transparent judgment logic.

[0151] It should be understood that evaluations based on large language models primarily target metrics requiring deep semantic understanding and complex logical reasoning (such as factual errors, demand fulfillment, and logical consistency). Therefore, automated evaluation systems can guide the evaluation model to score specific dimensions using structured instruction templates and force it to output detailed reasons for deductions. This ensures that while the evaluation process relies on a large model, its decision-making process remains auditable and interpretable.

[0152] As an example, score ranges of 0, 0.5, and 1 can be set for each evaluation indicator. The score and the reason for scoring each evaluation indicator constitute the reward signal for that evaluation indicator.

[0153] It should be understood that end-to-end evaluation can be broken down into multiple stages. By defining different evaluation benchmarks in the configuration file, the reward mechanism of the automated evaluation system can be seamlessly applied to key upstream nodes of the answer generation task, thus enabling its application to the entire process from a single answer generation stage to complex tasks.

[0154] For example, in the answer generation phase, the automated evaluation system focuses on the quality of the final generated text content. Based on its configuration, the system activates all baseline and optimization metrics related to answer content, format, style, and security, providing direct optimization signals for the target model to generate high-quality summaries or answers. In the demand understanding phase, the system assesses whether the target model accurately and comprehensively understands the user's original query intent. In the supply quality phase, i.e., before the model generates an answer, the system evaluates the quality, relevance, and completeness of the intermediate information retrieved, integrated, or planned by the target model (e.g., search results, API calls). In the external verification accuracy phase, the system assesses whether the content generated by the target model can be accurately verified by external knowledge sources.

[0155] The aforementioned multi-stage evaluation capabilities enable end-to-end, multi-dimensional diagnosis and optimization of the target model search process, achieving a systematic capability improvement from local components to the global process.

[0156] For example, see Figure 3 As shown, the bottom-line dimension focuses more on content and factual errors (e.g., core or partial content contradicts known facts, content has internal logical contradictions, inappropriate responses to questions that should not be answered, outdated information, and generated text and images that are completely unrelated), presentation and formatting errors (e.g., garbled characters, non-standard language mix, text that is excessively long or short, and formatting errors in structured content such as lists), supply and demand mismatch errors (e.g., the model's answer is completely deviating from the user's intent, failing to provide any effective information (no supply), or the provided supply content is completely unrelated to the user's needs), and security risk control (e.g., identifying and penalizing unsafe content to ensure the compliance of the model's output), etc.

[0157] In this embodiment, the bottom-line dimension focuses on imposing strict constraints on the target model's output, aiming to identify and penalize all fundamental errors that would seriously harm the user experience and are objectively verifiable. These evaluation metrics constitute the minimum guarantee of the target model's output quality, effectively preventing the target model from outputting fundamentally flawed or harmful content during training.

[0158] For example, see Figure 3As shown, the optimization dimensions focus more on content performance, such as content richness and organization (e.g., the richness of effective statements in the answer, the quality and quantity of components (such as images and lists), the overall aesthetic appeal of the answer's format, and the rationality of the information point organization structure), demand satisfaction (assessing whether the target model fully understands and satisfies the user's secondary or implicit needs, going beyond the literal meaning of the question), and supply quality and diversity (encouraging the model to achieve diversity within the information set when providing relevant information and avoiding conflicts between different information points), etc. For the optimization dimensions, positive incentives can be used to guide the target model to explore more deeply, in order to generate higher-quality, richer content that better meets the user's deeper or potential needs.

[0159] In this embodiment, these metrics of the optimization dimension are used to reward the model's outstanding performance, which is key to stimulating the target model's capabilities and raising the upper limit of the target model's performance.

[0160] For example, regarding incorrect answers, the bottom-line dimensions may include: core content errors or illusions, partial content errors or illusions, refusal to answer, logical contradictions in the content, exceeding the time limit, irrelevance of text and images, etc.; the optimization dimensions may include: answer richness (such as the amount of information contained, the level of detail, and the comprehensive coverage of the topic), the proportion of irrelevant content, semantic redundancy, the quality of images and / or components, etc.

[0161] Furthermore, potential biases and errors during model training can be amplified, leading to factual deviations and distortions, resulting in core content errors or illusions.

[0162] Among these, "refusal to answer" refers to a situation where the model fails to generate a relevant answer based on the input question, and no task prompts are provided. Content logical contradictions occur more frequently in scenarios with strong logical connections. For example, in a math problem, the question might state that the length of line segment AB is 3, but in subsequent reasoning or the answer, the length of line segment AB is given as 2.5, resulting in a logical contradiction. "Outdated" refers to a situation where the content generation model, due to its static nature, cannot provide the latest information when faced with highly time-sensitive needs (policy updates, market dynamics, etc.). "Irrelevant text and image content" generally occurs in image-based answering scenarios, such as when the generated answer is unrelated to the image content.

[0163] For example, regarding presentation and / or formatting, baseline dimensions may include: garbled characters, repetition, language mixing, excessively long or short length, excessively long lists, etc.; optimization dimensions may include: short answer format, answer primacy, component richness, reasonable component triggering, etc. Regarding language style, baseline dimensions may include: distinguishability from answers generated by other language models; optimization dimensions can be freely defined according to actual needs. Regarding security risk control, it mainly involves baseline dimensions, such as whether sensitive or illegal content is involved. Regarding understanding requirements, baseline dimensions may include irrelevant answers (i.e., the matching between the answer and the question); optimization dimensions may include the satisfaction of some secondary requirements. Regarding supply quality, baseline dimensions may include: no supply, completely irrelevant, etc.; optimization dimensions may include: set-wise diversity (generally referring to the performance or ability of a model or algorithm in processing a set of data in a specific task or benchmark test), requirement satisfaction, supply conflict, etc. Regarding external verification accuracy, it mainly involves baseline dimensions, such as the external verification accuracy of objective knowledge.

[0164] It should be understood that in this embodiment, the evaluation of multiple aspects is clearly divided into two dimensions: baseline and improvement, through an automated evaluation system. This ensures that while pursuing more excellent content, the baseline of content quality and security can be maintained.

[0165] The method in this embodiment ensures that the target model optimizes performance while strictly adhering to basic requirements such as factual accuracy and safety compliance, thus guaranteeing the stability and reliability of the model iteration process.

[0166] Step S409: Generate the corresponding answer based on the question using the target model.

[0167] In this embodiment, the target model can output the answer to the question, which may include the reasoning chain and multiple possible answers.

[0168] Optionally, when multiple possible answers are involved, they can be prioritized according to their matching degree, making it easier for users to choose the most suitable answer.

[0169] It should be understood that the target model in this embodiment is a continuously optimized dynamic model. As the user data in the database continues to grow, the target model can be continuously optimized through the reward signals output by the automated evaluation system, thereby improving the quality of the responses output by the target model and ensuring that the responses are correct while better aligning with user (human) preferences.

[0170] It should be noted that the user preferences in this embodiment can be selected based on the preferences of most users or on the preferences of a single user. For example, when dealing with a new user, since there is limited user data associated with them, the preferences of most users can be used to optimize the target model. When dealing with an existing user, since there is sufficient user data associated with them, the target model has formed a preference memory for that existing user through multiple searches, and therefore, preferences can be tailored for that existing user.

[0171] It should be understood that during the training and / or optimization of the target model, human or systematic preference signals (such as correctness, efficiency, style, etc.) can be introduced to enable the target model to learn to distinguish between good and bad outputs and optimize its generation behavior.

[0172] As another example, user data can be sampled from a database and a validation dataset can be built based on the user data. The answers generated by the target model are then validated using the validation dataset to determine whether the answers match user preferences. If the answers do not match user preferences, the weights corresponding to each reward signal are adjusted.

[0173] In this embodiment, the target model's answers are validated using a validation dataset to determine whether the answers deviate from user preferences. Based on the results, an adjustment to the reward signal weights (similar to a correction mechanism) is initiated. For example, for questions requiring high real-time performance, the target model outputs answers ordered chronologically, providing the user with the most recently published information. However, users tend to prefer viewing information with high numbers of shares and comments. Therefore, the model can reduce the weight of time-based metrics during answer search, lowering the real-time requirement and making the final output more aligned with user preferences.

[0174] For example, after the target model is trained, the consistency ratio between human review results and automatic evaluation results can be used as an outer-loop validation metric. Specifically, on the unseen reserve set (unused training dataset), pairwise preference discrimination is performed on the outputs of the target model and the control (or different version models) group, and the consistency between human preferences and the internal reward ranking of the system is statistically analyzed. If the outer-loop validation shows a validity deviation of "rewards continuously increasing while human win rate does not increase", a correction mechanism is immediately triggered. Combining the marginal contribution of the outer-loop metric, the relative weights of multiple reward signals are adjusted upward or downward in a directional manner to suppress a single path that may be over-amplified.

[0175] This embodiment combines the automated rewards within the automated evaluation system with external human preference verification. When a discrepancy is detected between the automated reward score and the human preference evaluation result, the automated evaluation system initiates a correction mechanism (adjusting the weights of each reward signal).

[0176] This embodiment can also inject objective executable signals, subjective preference signals, safety compliance signals, and cost robust signals into the strategy optimization process without relying on large-scale manual annotation. This not only improves the sample efficiency and engineering scalability of alignment training, but also significantly reduces the human cost of manual evaluation and iteration.

[0177] It should be understood that by combining automated rewards and preference verification to form a dual verification mechanism, the behavior of the target model that only maximizes automated reward scores without truly improving output quality can be promptly detected and corrected. This solves the key problem of "offline metrics improving but online experience declining," ensuring that the optimization direction remains consistent with real human preferences. Consequently, the automated evaluation system can continuously approximate human preferences and effectively suppress distortions such as reward manipulation, data drift, and jailbreak breaches, achieving higher consistency and lower resource consumption.

[0178] In this embodiment, the hierarchical architecture of the automated evaluation system can independently quantify different quality attributes output by the target model, avoiding the loss of dimensional information caused by a single scalar reward value.

[0179] Furthermore, the automated evaluation system's framework is end-to-end traceable, ensuring that any unexpected changes in model performance can be traced back to specific reward weights or constraint configurations. In the event of problems, it enables rapid diagnosis and targeted correction or version rollback, providing technical assurance for safe and robust model deployment and iteration in commercial environments. It provides auditable evidence for each iteration of the target model (such as rule hit records and detailed inference chains for evaluating the model), giving the optimization process clear attribution capabilities and enabling targeted model correction.

[0180] In summary, this embodiment obtains user data by sampling from a database and constructs a training dataset based on the user data; it then trains the constructed content generation model on the training dataset to obtain an initial model; a portion of the data from the training dataset is selected for automatic evaluation by an automated evaluation system and for manual review, yielding automatic evaluation results and manual review results; based on the automatic evaluation results and / or manual review results, a reward signal corresponding to each evaluation indicator is determined, the reward signal including a score and a reason; based on the weights and scores corresponding to each reward signal, a weighted aggregation is performed to obtain an overall reward; the initial model is optimized based on the overall reward to obtain multiple candidate models; and a target model is selected from the multiple candidate models. Thus, by combining the reward signal generated from the automatic evaluation results and / or the manual review results, the content generation model can be continuously optimized, resulting in a target model with better model performance, significantly improving the quality of answers searched by the target model, meeting complex commercial application needs, exhibiting strong robustness, and ensuring the entire evaluation process is traceable and better aligns with user preferences.

[0181] In one embodiment, taking the above content generation method implemented through interaction between server 104 and terminal 102 as an example, that is, in Figure 1 This explanation is based on the application environment shown; see [link / reference]. Figure 6 The timing diagram of the content generation method in the first embodiment shown includes:

[0182] Step S601: The terminal displays the application display interface of the target model.

[0183] At least one dialog box is displayed on the application's display interface.

[0184] In step S602, the terminal responds to the question entered in the dialog box.

[0185] The questions can take various forms, including text, voice, a combination of text and images, video, and audio.

[0186] In step S603, the terminal sends the problem to the server.

[0187] In step S604, the server searches for the question using the target model and generates an answer.

[0188] In step S605, the server sends the response to the terminal.

[0189] Step S606: The terminal displays the answer.

[0190] In this embodiment, the terminal displays the application interface of the target model. In response to a question entered in the dialog box, the terminal sends the question to the server. The server uses the target model to perform reasoning and searching on the question, outputs the reasoning process and answer, and then sends the answer back to the terminal, which finally displays the answer. This allows for accurate analysis and searching of the input question in a chat-like question-and-answer format, providing feedback on the completed reasoning process and answer to the user terminal. This makes the entire analysis process more aligned with human thinking habits, helping people solve more complex problems and providing high-quality, human-friendly answers.

[0191] In another embodiment, taking the above content generation method implemented through interaction between server 104 and terminal 102 as an example, that is, in Figure 1 This explanation is based on the application environment shown; see [link / reference]. Figure 7 The timing diagram of the content generation method in the second embodiment shown includes:

[0192] Step S701: The terminal displays the application display interface of the target model.

[0193] At least one dialog box is displayed on the application's display interface.

[0194] In step S702, the terminal responds to the question entered in the dialog box.

[0195] The questions can take various forms, including text, voice, a combination of text and images, video, and audio.

[0196] In step S703, the terminal sends the problem to the server.

[0197] In step S704, the server searches for the question using the target model and generates multiple answers.

[0198] In step S705, the server sends multiple responses to the terminal.

[0199] In step S706, the terminal displays multiple answers in a simplified form according to a preset priority order.

[0200] In step S707, the terminal responds to a touch operation on any answer by displaying the complete answer.

[0201] In this embodiment, the terminal displays the application interface of the target model. Upon receiving a question entered in the dialog box, the terminal sends the question to the server. The server uses the target model to perform reasoning and searching for the question, outputs the reasoning process and answer, and then sends multiple answers to the terminal. Finally, the terminal displays these multiple answers in a thumbnail format according to a preset priority order. In response to a touch operation on any answer, the complete answer is displayed. This allows for the rapid generation of multiple answers in a chat-style question-and-answer format, prioritizing them and providing feedback to the user for easy selection.

[0202] Optionally, the terminal can also record the user's selection of multiple answers and form user preference data. This allows the terminal to adjust the priority of answers based on user preferences in the next response, ensuring that the final answer meets the user's expectations and improving the user experience.

[0203] For example, suppose a user enters the question "Which city is most suitable for travel in May?" into the terminal application interface. The target model analyzes the user's historical search data and ranks the tourist cities and their corresponding reasons for recommendation. For instance, for user A, who prefers mountain climbing (inferred from the user's search data on outdoor mountaineering equipment), cities with famous mountains and rivers are ranked ahead of coastal cities. For user B, who prefers swimming and diving (based on the user's saved diving and swimming notes, as well as the user's historical questions about coastal cities), cities with coastlines are ranked first.

[0204] As another possible example, the target model can also be deployed on the terminal side, which would make it suitable for problem searching in offline mode.

[0205] It should be understood that regardless of the location and form of the target model deployment, it can be integrated with an automated evaluation system. Furthermore, the functionality of the automated evaluation system can be integrated into the target model, enabling continuous optimization based on real user feedback during the process of searching for relevant results.

[0206] It should be understood that, in addition to continuously optimizing the target model, the database storing user data can also be continuously updated. This allows for the construction of new training datasets based on the updated database, enabling synchronous optimization of the automated evaluation system and ensuring that reward signals better reflect the latest user needs.

[0207] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0208] Based on the same inventive concept, this application also provides a content generation apparatus for implementing the content generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more content generation apparatus embodiments provided below can be found in the limitations of the content generation method described above, and will not be repeated here.

[0209] In one exemplary embodiment, such as Figure 8 As shown, a content generation apparatus is provided, including: an input module 801 and a result output module 802, wherein:

[0210] The input module 801 is used to acquire the problem and input it into the target model. The target model is continuously optimized by the reward signal output by the automated evaluation system. The reward signal is used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system is used to evaluate the sampled data for multiple evaluation indicators to obtain the reward signal corresponding to each evaluation indicator. The sampled data is user data sampled from the database.

[0211] The result output module 802 is used to generate the corresponding answer based on the question using the target model.

[0212] This embodiment acquires a question and inputs it into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation metrics to obtain a reward signal corresponding to each metric. The sampled data consists of user data sampled from a database. The target model generates corresponding answers based on the question. This significantly optimizes the quality of search results, meets the needs of complex business applications, demonstrates strong robustness, and ensures that the entire evaluation process is traceable and aligns with user preferences.

[0213] In another exemplary embodiment, such as Figure 9 As shown, a content generation apparatus is provided, in Figure 8 Based on the device shown, it may also include:

[0214] The training dataset construction module 803 is used to sample user data from the database and construct a training dataset based on the user data. The user data includes at least one of the following: questions, search results, time information, and historical search data.

[0215] Training module 804 is used to train the constructed content generation model based on the training dataset to obtain the initial model;

[0216] The evaluation module 805 is used to select a portion of the training dataset for automatic evaluation by an automated evaluation system and for manual review, so as to obtain the automatic evaluation results and the manual review results.

[0217] The reward signal generation module 806 is used to determine the reward signal corresponding to each evaluation indicator based on the automatic evaluation results and / or manual review results. The reward signal includes a score and a reason.

[0218] The weighted aggregation module 807 is used to obtain the overall reward by weighted aggregation based on the weight and score corresponding to each reward signal;

[0219] Optimization module 808 is used to optimize the initial model based on the overall reward to obtain multiple candidate models;

[0220] The selection module 809 is used to select the target model from multiple candidate models.

[0221] For example, the training module 804 is specifically used to train the constructed content generation model based on the training dataset and by using the population relative strategy to optimize the GRPO algorithm to obtain an initial model.

[0222] For example, the automated evaluation system includes two hierarchical structures: content evaluation of model responses and end-to-end evaluation. The content evaluation of model responses is used to evaluate responses from at least one of the following dimensions: answer errors, presentation format, language style, and security risk control, in terms of both baseline and improvement dimensions. The end-to-end evaluation is used to evaluate responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy, in terms of both baseline and improvement dimensions. Here, the baseline refers to the constraints that the response must meet, and the improvement refers to the requirements that the response should meet with positive incentives.

[0223] For example, the above-mentioned device may further include: a consistency verification module 810, used to compare the evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results with the review scores corresponding to the two dimensions of each aspect in the manual review results, and determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the proportion of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension. When the consistency ratio is lower than a preset value, the automatic evaluation system is iteratively optimized through a preset difficult case dataset until the consistency ratio of the automatic evaluation results output by the automatic evaluation system is not lower than the preset value.

[0224] For example, the above apparatus may further include: a weight determination module 811, used to determine the weight corresponding to each reward signal based on the importance of each reward signal and the influence score of each reward signal on the model performance; the influence score of the reward signal on the model performance is set manually.

[0225] For example, module 809 is specifically used to evaluate the model performance of each candidate model and obtain the corresponding performance evaluation results; based on the performance evaluation results, the target model is selected.

[0226] For example, module 809 is specifically used to input the same question into the candidate model and output multiple corresponding answers; sort the multiple answers manually; and determine the performance evaluation result of the candidate model based on the sorting of the answers.

[0227] In yet another exemplary embodiment, such as Figure 10 As shown, a content generation apparatus is provided, in Figure 8 or Figure 9 Based on the device shown, it may also include:

[0228] The validation dataset construction module 812 is used to sample user data from the database and construct a validation dataset based on the user data.

[0229] The preference verification module 813 is used to perform preference verification on the answers generated by the target model using a verification dataset to determine whether the answers conform to user preferences.

[0230] The weight adjustment module 814 is used to adjust the weights of each reward signal if the answer does not meet the user's preferences.

[0231] Each module in the aforementioned content generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0232] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores user data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a content generation method.

[0233] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a content generation method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0234] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0235] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0236] The process involves obtaining questions and inputting them into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation metrics to obtain the reward signal corresponding to each metric. The sampled data consists of user data sampled from a database. The target model then generates corresponding answers based on the questions.

[0237] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0238] User data is sampled from a database, and a training dataset is constructed based on this user data. The user data includes at least one of the following: questions, search results, time information, and historical search data. The content generation model is trained on the training dataset to obtain an initial model. A portion of the training dataset is selected for automatic evaluation by an automated evaluation system and for manual review, yielding automatic evaluation results and manual review results. Based on the automatic evaluation results and / or manual review results, a reward signal is determined for each evaluation indicator, including a score and a reason. The overall reward is obtained by weighted aggregation based on the weights and scores of each reward signal. The initial model is optimized based on the overall reward to obtain multiple candidate models. The target model is selected from the multiple candidate models.

[0239] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0240] The initial model is obtained by training the content generation model based on the training dataset and using the group relative strategy to optimize the GRPO algorithm.

[0241] In one embodiment, the automated evaluation system includes two hierarchical structures: content evaluation of model responses and end-to-end evaluation. The content evaluation of model responses assesses responses from at least one of the following dimensions: answer errors, presentation format, language style, and security risk control. The end-to-end evaluation assesses responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy. Here, the bottom line refers to the constraints that the response must meet, and the optimization refers to the requirements that the response should meet.

[0242] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0243] After obtaining the automatic evaluation results and the manual review results, the evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results are compared with the review scores corresponding to the two dimensions of each aspect in the manual review results to determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the percentage of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension. When the consistency ratio is lower than the preset value, the automatic evaluation system is iteratively optimized using a preset dataset of difficult cases until the consistency ratio of the automatic evaluation results output by the automatic evaluation system is not lower than the preset value.

[0244] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0245] Before the overall reward is obtained by weighted aggregation based on the weights and scores of each reward signal, the weights of each reward signal are determined according to their importance and their impact scores on model performance; the impact scores of the reward signals on model performance are set manually.

[0246] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0247] The performance of each candidate model is evaluated to obtain the corresponding performance evaluation results; based on the performance evaluation results, the target model is selected.

[0248] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0249] The same question is input into the candidate model, and multiple corresponding answers are output. The multiple answers are sorted manually. Based on the sorting of the answers, the performance evaluation results of the candidate model are determined.

[0250] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0251] User data is sampled from the database, and a validation dataset is constructed based on the user data. The responses generated by the target model are validated using the validation dataset to determine whether the responses conform to user preferences. If the responses do not conform to user preferences, the weights corresponding to each reward signal are adjusted.

[0252] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0253] The process involves obtaining questions and inputting them into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation metrics to obtain the reward signal corresponding to each metric. The sampled data consists of user data sampled from a database. The target model then generates corresponding answers based on the questions.

[0254] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0255] User data is sampled from a database, and a training dataset is constructed based on this user data. The user data includes at least one of the following: questions, search results, time information, and historical search data. The content generation model is trained on the training dataset to obtain an initial model. A portion of the training dataset is selected for automatic evaluation by an automated evaluation system and for manual review, yielding automatic evaluation results and manual review results. Based on the automatic evaluation results and / or manual review results, a reward signal is determined for each evaluation indicator, including a score and a reason. The overall reward is obtained by weighted aggregation based on the weights and scores of each reward signal. The initial model is optimized based on the overall reward to obtain multiple candidate models. The target model is selected from the multiple candidate models.

[0256] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0257] The initial model is obtained by training the content generation model based on the training dataset and using the group relative strategy to optimize the GRPO algorithm.

[0258] In one embodiment, the automated evaluation system includes two hierarchical structures: content evaluation of model responses and end-to-end evaluation. The content evaluation of model responses assesses responses from at least one of the following dimensions: answer errors, presentation format, language style, and security risk control. The end-to-end evaluation assesses responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy. Here, the bottom line refers to the constraints that the response must meet, and the optimization refers to the requirements that the response should meet.

[0259] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0260] After obtaining the automatic evaluation results and the manual review results, the evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results are compared with the review scores corresponding to the two dimensions of each aspect in the manual review results to determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the percentage of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension. When the consistency ratio is lower than the preset value, the automatic evaluation system is iteratively optimized using a preset dataset of difficult cases until the consistency ratio of the automatic evaluation results output by the automatic evaluation system is not lower than the preset value.

[0261] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0262] Before the overall reward is obtained by weighted aggregation based on the weights and scores of each reward signal, the weights of each reward signal are determined according to their importance and their impact scores on model performance; the impact scores of the reward signals on model performance are set manually.

[0263] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0264] The performance of each candidate model is evaluated to obtain the corresponding performance evaluation results; based on the performance evaluation results, the target model is selected.

[0265] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0266] The same question is input into the candidate model, and multiple corresponding answers are output. The multiple answers are sorted manually. Based on the sorting of the answers, the performance evaluation results of the candidate model are determined.

[0267] In one embodiment, when a computer program is executed by a processor, it performs the following steps:

[0268] User data is sampled from the database, and a validation dataset is constructed based on the user data. The responses generated by the target model are validated using the validation dataset to determine whether the responses conform to user preferences. If the responses do not conform to user preferences, the weights corresponding to each reward signal are adjusted.

[0269] In this embodiment, a question is acquired and input into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation metrics to obtain a reward signal corresponding to each metric. The sampled data consists of user data sampled from a database. The target model generates corresponding answers based on the question. This significantly optimizes the quality of search results, meets the needs of complex business applications, demonstrates strong robustness, and ensures that the entire evaluation process is traceable and aligns with user preferences.

[0270] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the method steps of the various embodiments described above.

[0271] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0272] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0273] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0274] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A content generation method, characterized in that, The method includes: The problem is identified and input into a target model, which is continuously optimized using reward signals output by an automated evaluation system. These reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system evaluates the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each indicator. The sampled data consists of user data sampled from a database. The target model generates a corresponding answer based on the question.

2. The method according to claim 1, characterized in that, Before acquiring the problem and inputting it into the target model, the method further includes: User data is sampled from a database, and a training dataset is constructed based on the user data. The user data includes at least one of the following: questions, search results, time information, and historical search data. The content generation model is trained based on the training dataset to obtain an initial model; A portion of the training dataset is selected and automatically evaluated by an automated evaluation system, and then manually reviewed to obtain the automatic evaluation results and the manual review results. Based on the automatic evaluation results and / or the manual review results, a reward signal is determined for each evaluation indicator, wherein the reward signal includes a score and a reason; The overall reward is obtained by weighted aggregation based on the weight and score of each reward signal. The initial model is optimized based on the overall reward to obtain multiple candidate models; The target model is selected from the multiple candidate models.

3. The method according to claim 2, characterized in that, The process of training the constructed content generation model based on the training dataset to obtain an initial model includes: Based on the training dataset, the content generation model is trained using the group relative strategy optimization GRPO algorithm to obtain the initial model.

4. The method according to claim 2, characterized in that, The automated evaluation system comprises two hierarchical structures: content evaluation of model responses and end-to-end evaluation. The content evaluation of model responses assesses responses from at least one of the following dimensions: answer errors, presentation format, language style, and security risk control, using both baseline and improvement dimensions. The end-to-end evaluation assesses responses from at least one of the following dimensions: demand understanding, supply quality, and external verification accuracy, using both baseline and improvement dimensions. Here, the bottom line refers to the constraints that the answer must meet, and the optimization refers to the positive incentives that the answer must meet the requirements.

5. The method according to claim 4, characterized in that, After obtaining the automatic evaluation results and the manual review results, the method further includes: The evaluation scores corresponding to the two dimensions of each aspect in the automatic evaluation results are compared with the review scores corresponding to the two dimensions of each aspect in the manual review results to determine the consistency ratio of the automatic evaluation results. The consistency ratio refers to the percentage of automatic evaluation results and manual review results with the same score for the same aspect and the same dimension. When the consistency ratio is lower than a preset value, the automated evaluation system is iteratively optimized using a preset dataset of difficult cases until the consistency ratio of the automated evaluation results output by the automated evaluation system is not lower than the preset value.

6. The method according to claim 2, characterized in that, Before obtaining the overall reward by weighted aggregation based on the weights and scores corresponding to each reward signal, the method further includes: The weights of each reward signal are determined based on its importance and its impact score on model performance; the impact score of each reward signal on model performance is set manually.

7. The method according to claim 2, characterized in that, The step of selecting the target model from the plurality of candidate models includes: The performance of each candidate model is evaluated to obtain the corresponding performance evaluation results; Based on the performance evaluation results, the target model was selected.

8. The method according to claim 7, characterized in that, The evaluation of the model performance of each candidate model to obtain the corresponding performance evaluation results includes: The same question is input into the candidate model, and multiple corresponding answers are output. The multiple answers were sorted manually. Based on the ranking of the answers, the performance evaluation results of the candidate models are determined.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: User data is sampled from the database, and a verification dataset is constructed based on the user data; The answers generated by the target model are validated using the validation dataset to determine whether the answers match user preferences. If the answer does not match the user's preferences, the weights corresponding to each reward signal will be adjusted.

10. A content generation apparatus, characterized in that, The device includes: An input module is used to acquire questions and input them into a target model. The target model is continuously optimized using reward signals output by an automated evaluation system. The reward signals are used to optimize and adjust at least some of the model parameters in the target model. The automated evaluation system is used to evaluate the sampled data using multiple evaluation indicators to obtain a reward signal corresponding to each evaluation indicator. The sampled data is user data sampled from a database. The result output module is used to generate a corresponding answer based on the question using the target model.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images