Method for creating comparison task and electronic equipment
Through the multi-agent collaboration framework and task graph optimization, A/B experimental tasks are automatically processed, solving the high cost problem caused by manual configuration on the business side and realizing efficient and intelligent comparison task creation.
Patent Information
- Application Number
- CN202510497201.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-09-12
AI Technical Summary
The existing A/B experiment service platform requires manual configuration operations by the business side, which takes up a lot of time and manpower costs, resulting in inefficiency in creating comparison tasks.
A multi-agent collaborative framework is adopted, including process management agent, configuration analysis agent, indicator analysis agent and parameter integration agent. The AI generative model is used to automatically process and compare task requirements, and the task graph is used to optimize the execution sequence to avoid dead loops.
It realizes the intelligence and automation of comparison tasks, reduces time and labor costs, improves creation efficiency, and ensures the correctness of output quality and execution sequence.
Smart Images
Figure CN120631467A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information processing technology, and in particular to a method and electronic device for creating a comparison task. Background Art
[0002] A comparative experiment (also known as A / B experiment, A / B test, etc.) is a commonly used comparative experiment method used to evaluate the effectiveness of two or more strategies, designs or products. In an A / B experiment, participants are randomly assigned to different groups, each group receives a different treatment or intervention, and then the results between different groups are compared to evaluate which strategy, design or product is more effective, thereby optimizing the product or service. A / B experiments emphasize testing of users with similar attributes grouped in the same time dimension. For example, in software testing, an A / B experiment is a method of comparing two schemes or versions of a web page or application to determine which scheme or version performs better. It involves randomly displaying two variants (A and B) to users and using statistical analysis to determine which scheme or version is more effective in achieving predefined goals, such as improving click-through rate, conversion rate or any other key performance indicators.
[0003] Many products require A / B experimentation. For example, a certain group company includes multiple software product lines, each of which includes multiple software products. Each product may involve A / B experimentation during development and version updates. Therefore, the demand for A / B experimentation is high and relatively high-frequency. This is why platforms specializing in A / B experimentation services have emerged. These A / B experimentation service platforms provide businesses with a one-stop experimentation process, including a visual interface, scientific data analysis, and automated experiment reporting. Through scientific experimental methods and real user behavior, they verify the optimal solution, thereby driving business growth.
[0004] The emergence of this A / B experiment service platform eliminates the need for each business party to build their own experimental process, thus providing convenience for the business party. However, the business party still needs to complete some configuration operations on the platform to create a specific test instance. For example, the business party needs to clarify the objectives of the experiment, propose reasonable experimental hypotheses, and clarify the null hypothesis and alternative hypothesis; the business party also needs to determine the audience of the experiment and determine which users will enter the experimental diversion and become the denominator of the indicator; in addition, the business party needs to select indicators to measure the hypothesis. Indicators are important criteria for measuring the effectiveness of the experiment. Only by selecting appropriate core indicators and auxiliary indicators can the results of the experiment be accurately described and verified, etc. The execution of the above configuration operations still requires a lot of time and manpower costs for the business party. Summary of the Invention
[0005] This application provides a method and electronic device for creating a comparison task, which can realize the intelligent and automated construction of the comparison task.
[0006] This application provides the following solutions:
[0007] A method for creating a comparison task, the method being applied to a comparison service system, the system including multiple artificial intelligence (AI) agents, the multiple AI agents being designed by modeling and role-dividing a process for creating a comparison task, and including at least a process management agent; the method comprising:
[0008] Obtain comparison requirement information submitted by users;
[0009] The process management agent is called according to the comparison requirement information, so that the process management agent determines the execution order of other AI agents and the tool information required to be called by other AI agents during the execution process by analyzing the comparison requirement information, so that the other AI agents can obtain relevant contextual knowledge by calling corresponding tools;
[0010] Calling the other AI agents in the execution order, wherein, when calling the called AI agent, determining input prompt information for the called AI agent based on the comparison requirement information, information about the tool required to be called by the called AI agent, and output information of the upstream AI agent, so as to determine basic configuration information and recommended indicator information corresponding to the comparison requirement through the other AI agent, and integrate them into an input parameter set for the task creation interface;
[0011] The task creation interface is called according to the input parameter set to create a corresponding comparison task.
[0012] Among them, the other AI agents include configuration analysis agents, indicator analysis agents, and parameter integration agents;
[0013] The configuration analysis agent is used to determine basic configuration information corresponding to the comparison requirements, wherein the basic configuration information includes: task objectives, audience information, comparison group design, traffic distribution ratio, bucketing logic, and / or hypothetical content;
[0014] The indicator analysis agent is used to determine recommended indicators, wherein the indicators include core indicators, driving indicators and / or guardrail indicator information;
[0015] The parameter integration agent is used to integrate the basic configuration information and the indicators of the recommended configuration into an input parameter set of the task creation interface.
[0016] The tools required to be called by the configuration analysis agent include: a tool for querying user information, a tool for querying spatial information context, and / or a tool for searching for comparison task information recently created by the user.
[0017] The tools that the indicator analysis agent needs to call include: a tool for querying indicator context information configured in the comparison task created by the user, and / or a tool for querying optional indicator information.
[0018] The indicator analysis agent is also used to explain the reasons, significance, calculation methods and applicable scenarios of the recommended indicators.
[0019] Among them, also include:
[0020] A task graph is provided to the process management agent. The task graph is a directed acyclic graph generated by summarizing the process of creating the comparison task, which includes a list of AI agents participating in the current task and information on accessible execution paths between different AI agents, so that the process management agent can determine the execution order of other AI agents based on the task graph.
[0021] Among them, the multiple AI agents are implemented using a reasoning-action model, so that the multiple AI agents observe the environment and extract key details in the process of receiving and responding to information, and use the extracted information to guide their own thinking and subsequent actions, so as to achieve autonomous learning of the multiple AI agents.
[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the aforementioned methods.
[0023] An electronic device, comprising:
[0024] one or more processors; and
[0025] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the aforementioned methods.
[0026] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions are capable of implementing the steps of any of the aforementioned methods when executed by a processor in an electronic device.
[0027] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0028] Through the embodiment of the present application, multiple AI agents can be designed in advance by modeling and dividing roles for the process of creating A / B experiments and other comparison tasks, which may include a process management agent. After receiving the comparison requirement information submitted by the user, the process management agent can be first called according to the comparison requirement information, so that the process management agent can determine the execution order of other AI agents and the tool information that other AI agents need to call during the execution process by analyzing the comparison requirement information. Afterwards, other AI agents can be called according to the execution order determined by the process management agent, wherein, when calling a specific called AI agent, the input prompt information of the called AI agent can be determined based on the comparison requirement information input by the user, the tool information that the called AI agent needs to call, and the output information of the upstream AI agent, so as to determine the basic configuration information and recommended indicator information corresponding to the comparison requirement through the other AI agents, and integrate them into the input parameter set of the task creation interface. Finally, the task creation interface can be called according to the input parameter set output by the AI agent to create the corresponding comparison task. In the above implementation method, the process is designed around the A / B comparison task, a multi-agent collaboration framework is built, and pre-designed tools are used to help AI agents obtain the required contextual knowledge, thereby improving the ability of AI generative models to solve complex problems and promoting the intelligent and automated development of comparison task construction.
[0029] In a preferred implementation, a directed acyclic task graph can be generated by summarizing the process of creating a comparison task, which may include a list of AI agents participating in the current task and information on the execution paths accessible between different AI agents. The task graph can be provided to the process management agent so that the agent can determine the execution order of other AI agents based on the task graph, thereby improving efficiency and avoiding sequence errors or "dead loops".
[0030] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0032] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;
[0033] Figure 2 is a flow chart of the method provided in an embodiment of the present application;
[0034] Figure 3 is a schematic diagram of an intelligent agent provided in an embodiment of the present application;
[0035] Figure 4 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0036] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.
[0037] In an embodiment of the present application, in order to reduce the time and manpower cost of creating comparative tasks such as A / B experiments, the ability of AI (Artificial Intelligence) generative models can be used to realize the creation of automated comparative experimental tasks. Among them, the so-called AI generative model usually refers to a deep learning model. This AI generative model is large in scale and can store and process a large amount of information, thereby achieving higher performance in various tasks. For example, the "Wenshengwen" type AI generative model is an artificial intelligence model based on deep learning technology. It can complete a variety of natural language processing tasks such as text generation, knowledge question answering, reasoning calculation, etc. by performing unsupervised or self-supervised learning on massive text data. Therefore, it can rely on the powerful natural language text understanding and logical thinking ability of the pre-trained AI generative model to help realize the creation of automated comparative tasks. It should be noted that the embodiment of the present application does not limit the number of parameters contained in the AI generative model, and the goal is to meet actual needs.
[0038] However, as mentioned in the technical background section, creating a comparison task is often complex. It requires clarifying the comparison objective, formulating reasonable experimental hypotheses, determining the target audience, and determining which users will be included in the experimental flow. Furthermore, the business side must select metrics to measure the hypotheses, among other things. Simply using a single AI generative model to accomplish these complex tasks is challenging, even for a powerful model, and the quality of the output information is likely to be low.
[0039] In response to the above situation, an embodiment of the present application provides a multi-agent solution based on an AI generative model. By modeling and dividing the roles of the process of manually creating comparison tasks, multiple agents can be designed. For example, it can include AI agents with multiple roles such as process managers, experimental analysts, indicator analysts, parameter integrators, etc. The process management agent arranges the execution order of other AI agents to achieve effective execution of complex tasks.
[0040] The so-called "AI agent" refers to an entity or program with the ability to perceive, make decisions, and act. It can perceive information about its environment, reason and make decisions based on its own goals and strategies, and take corresponding actions to influence the environment or interact with other agents. AI generative models can often be the core component of AI agents, and AI agents are the "carrier" of AI generative models. Through a combination of software and hardware, they transform the capabilities of AI generative models into specific functions and services. In short, AI generative models empower AI agents, and AI agents provide data for AI generative models.
[0041] A multi-agent system is a collection of AI agents that interact and collaborate to accomplish complex tasks that would be difficult for a single agent to accomplish. Each agent in a multi-agent system has its own goals, strategies, and behaviors. They interact through communication, negotiation, and collaboration to solve problems or optimize the overall system.
[0042] Among them, since the information required to be provided to the AI agent in the specific execution process of the task may be relatively large, and the length of the input information supported by the AI agent is usually limited, therefore, in order to enable the AI agent to obtain enough information or data when the input information length is limited, the embodiment of the present application can also design some tools so that the AI agent can obtain relevant data through such tools in the process of executing the task, without having to directly input these data in the form of prompt word information. As for which tools the specific AI agent needs to call, it can also be determined by the process management agent. That is to say, the process management agent can arrange the execution order of other agents, and determine which tools other agents need to call according to the specific experimental requirements to obtain relevant data. Subsequently, other agents can be called in this order. Each time another agent is called, the input prompt information of the called AI agent can be determined by comparing the requirement information, the tool information required to be called by the called AI agent, and the output information of the upstream AI agent.
[0043] In terms of specific implementation details, since the process management agent needs to choose which other agents to use and the execution order of other agents based on specific experimental requirements, in order to improve the processing efficiency of the process management agent and the quality of the output information and avoid the occurrence of "dead loops", it is also possible to generate a task graph in advance by summarizing the process of manually creating comparative experimental tasks. The task graph can be a directed acyclic graph, which can express the list of AI agents participating in the current task and the execution order information between different AI agents, so that the process management agent can complete the execution order arrangement of other agents according to the task graph. With the task graph as a reference, the processing efficiency of the process management agent can be improved. At the same time, since it is a directed acyclic graph, logical errors in the execution order of other agents or the occurrence of "dead loops" can be avoided.
[0044] This approach, in which the process management agent orchestrates the execution order and required tools of other AI agents based on the task graph, decouples AI agent design from task graph design. This means that when designing an AI agent, one only needs to focus on implementing the subtasks, while the task graph can be designed separately based on specific requirements, reducing design complexity. Furthermore, new AI agents can be added at any time, enabling flexible expansion of system functionality.
[0045] Alternatively, specific AI agents can be designed based on the ReAct model. ReAct is a framework that combines reasoning and action. Using an AI generative model, the AI agent generates reasoning steps to clarify the current task state and goal. Based on the reasoning results, the agent then selects appropriate actions to execute the task, performing reasoning, observation, and action through a chain of thought processes. This allows the AI agent to not only receive and respond to information but also observe the environment to extract key details. These observations guide its thinking and subsequent actions. Finally, these key details extracted from the environment can be stored in memory as valuable information for future reference, making each AI agent an active learner in the system. "Actions" here refer to the specific actions or behaviors an AI agent takes based on its own decisions and environmental information. These actions can range from generating responses in a dialogue system to querying information within a knowledge graph. Action selection is typically based on the AI agent's strategy and its perception of the environment.
[0046] Specifically from the perspective of system architecture, see Figure 1, the embodiment of the present application can provide the above-mentioned automated and intelligent comparison task creation function in the comparison service platform. Users (for example, developers of specific pages, applications, etc.) can submit specific comparison requirement information through the comparison service platform. It can be a requirement document or other forms of requirement description information. The specific requirement information can be expressed in text form. Then, the main program can input the specific comparison requirement information as part of the prompt information into the process management agent. The prompt information can also include information such as a specific task description. In an optional manner, a directed acyclic task graph can also be provided to the process management agent. Then, the process management agent can determine the specific other AI agents to be used, the execution order, and which tools the specific AI agent needs to call by performing natural language understanding and other processing on the comparison requirement information. Then, the main program can call other AI agents in sequence based on the information output by the process management agent. When each AI agent is called, the input prompt information of the AI agent can be determined based on the comparison requirement information submitted by the user, the tool information required to be called by the current AI agent, and the output information of the upstream AI agent, and then input it into the AI agent and initiate the call. Finally, through the cooperation of multiple other AI agents, the basic experimental configuration information and recommended indicator information corresponding to the comparative experiment requirements can be obtained, and integrated into the input parameter set of the task creation interface. Afterwards, the main program can call the task creation interface according to the input parameter set to create the corresponding comparative task. After creating the comparative task, the user can view the specific creation results through the interface provided by the comparative service platform. The user can also edit on this basis, for example, modify some indicators, etc. After confirmation, the comparative task can be formally created. Then, the comparative service platform can complete the specific comparative task by executing the comparative task and give the comparative results.
[0047] The specific implementation scheme provided in the embodiments of this application is introduced in detail below.
[0048] First, the embodiment of the present application provides a method for creating a comparison task. As described above, the method is applied to a comparison service system. In the system, multiple artificial intelligence (AI) agents can be pre-designed. The multiple AI agents can be designed by modeling the process of creating the comparison task and dividing the roles, including at least a process management agent. On this basis, see Figure 2 , the method may specifically include:
[0049] S201: Obtain comparison requirement information submitted by the user.
[0050] Users can submit comparison requirements through the interface provided by the specific experimental platform. This interface can take the form of a conversational interface, with the user on one side and an intelligent robot on the other. Users can submit requirements by uploading a requirements document in the conversational interface, or they can describe their requirements through a conversation. Specific comparison test requirements can be described in natural language, including text.
[0051] S202: Call the process management agent according to the comparison requirement information, so that the process management agent determines the execution order of other AI agents and the tool information required to be called by other AI agents during the execution process by analyzing the comparison requirement information, so that the other AI agents can obtain relevant contextual knowledge by calling corresponding tools.
[0052] After receiving the comparison requirement information, the main program can fill the comparison requirement information into the pre-set prompt information template, generate prompt information, and input it into the process management agent. Among them, different AI agents can correspond to different prompt information templates, but the specific formats can be similar, which can include role definition information, task description information, output format information, etc. In addition, the prompt information template corresponding to the process management agent can also include description information of a variety of pre-defined tools, etc. After receiving the prompt information, the process management agent can understand the comparison requirement information and infer the other AI agents that need to be called, the execution order of other AI agents, and which tools other AI agents need to call during the execution process.
[0053] In addition, in an optional manner, a task graph can also be provided to the process management agent. The specific task graph can be a directed acyclic graph, which is generated by summarizing the process of manually creating comparison tasks. It can include a list of AI agents participating in the current task and information on the execution paths that can be reached between different AI agents, so that the process management agent can determine the execution order of other AI agents based on the task graph. For example, in a specific example, the AI agents include four AI agents, namely process management, configuration analysis, indicator analysis, and parameter integration. The specific task graph can be as follows: Figure 3As shown in the figure, the process manager corresponds to the process management agent, the configuration analyst corresponds to the configuration analysis agent, the indicator analyst corresponds to the indicator analysis agent, and the experiment editor corresponds to the parameter integration agent. Because this task graph is generated by summarizing existing experience, standard operating procedures can be encoded into the prompt information of the process management agent through the task graph. This can give the process management agent domain expertise equivalent to that of human professionals, helping it to more efficiently produce higher-quality output results and avoid illogical execution sequences or infinite loops.
[0054] In specific implementation, the task graph can represent the task content in a format such as JSON (JavaScript Object Notation). For example, a task instance can be identified by the task ID. For each task, the entry can be used to specify which agent to start from, and the roles can be used to specify the list of agents participating in the task, and so on.
[0055] S203: Call the other AI agents in the execution order, wherein, when calling the called AI agent, determine the input prompt information of the called AI agent based on the comparison requirement information, the tool information required to be called by the called AI agent, and the output information of the upstream AI agent, so as to determine the basic configuration information and recommended indicator information corresponding to the comparison requirement through the other AI agents, and integrate them into the input parameter set of the task creation interface.
[0056] After the process management agent outputs the execution order of other agents and the tools that need to be called, the main program can call other agents in sequence according to the execution order in the output results. Among them, when calling each AI agent, the input prompt information of the AI agent can be determined based on the comparison requirement information input by the user, the tool information that the AI agent needs to call, and the output information of the upstream AI agent. After receiving the prompt information, each AI agent will iteratively execute the specified subtask. After completion, based on the results of the current subtask, all actions performed by the AI agent will be summarized and passed to the downstream AI agent. In this way, different AI agents can interact and share information, and have collective intelligence.
[0057] It should be noted here that when generating prompt information for each AI agent, a pre-generated prompt template can be used. In an embodiment of the present application, each AI agent can be designed based on the ReAct model to reason, observe, and act through a chain of thoughts. The agent not only receives and responds to information, but also observes the environment to extract key details. These observations guide their thinking and subsequent actions. Finally, the important information extracted from the environment is stored in memory for future reference, making each agent an active learner in the system.
[0058] Actions are standardized behaviors that AI agents can perform, and their properties include:
[0059] Prefix: Use the character's prefix to establish the character's background.
[0060] AI Generative Model Agent: Each Action contains an AI generative model agent that can be called through methods to enrich the action details using the input context expressed in natural language prompts. In addition, the Action class can call tools to obtain contextual information and provide useful information results to the model.
[0061] Standardized output schema: This is used to extract structured data. By defining the expected output schema, the results of the AI generative model can be parsed into structured output.
[0062] Instruction content: Structured data extracted from the output of an action using a standardized output schema. This information is encapsulated as a message and eventually synchronized to the environment.
[0063] Retry mechanism: By defining the number of attempts and waiting time, you can support retrying of actions and enhance system robustness.
[0064] In the case of designing each AI agent based on the ReAct model, the specific prompt information template can be as follows:
[0065] """
[0066] Answer the following questions in the given format.
[0067] You can use the following tools:
[0068] {tools}
[0069] The answer should follow the following format:
[0070] ---
[0071] Question: Questions that need to be answered
[0072] Thought: What do I need to do to answer this?
[0073] Action: One of the tool names in {tool_names} must be in {tools}
[0074] Action Input: Select the input required by the tool
[0075] Observation: Select the results returned by the tool
[0076] ...(This thought / action / action input / observation can be repeated N times)
[0077] Thought: I now know the final answer
[0078] Final Answer: The final answer to the original input question
[0079] ---
[0080] Start answering now, and remember to follow the specified format to reason step by step before giving your final answer.
[0081] Question:{input}
[0082] """
[0083] """
[0084] Answer the following questions in the given format.
[0085] You can use the following tools:
[0086] {tools}
[0087] The answer should follow the following format:
[0088] ---
[0089] Question: Questions that need to be answered
[0090] Thought: What do I need to do to answer this?
[0091] Action: One of the tool names in {tool_names}, must be in {tools}
[0092] Action Input: Select the input required by the tool
[0093] Observation: Select the results returned by the tool
[0094] ...(This thought / action / action input / observation can be repeated N times)
[0095] Thought: I now know the final answer
[0096] FinalAnswer: The final answer to the original input question
[0097] ---
[0098] Start answering now, and remember to follow the specified format to reason step by step before giving your final answer.
[0099] Question:{input}
[0100] """
[0101] There are many specific implementation methods for how to divide roles and design corresponding AI agents. For example, in one specific implementation method, in addition to the process management agent, other AI agents can be respectively: configuration analysis agent, indicator analysis agent and parameter integration agent. The configuration analysis agent can be used to determine the basic configuration information corresponding to the comparison requirements. The basic configuration information may include: task objectives, audiences, control group design, traffic distribution ratio, bucketing logic, and / or hypothetical content. The indicator analysis agent can be specifically used to determine the recommended indicators. The indicators may include core indicators, driving indicators and / or guardrail indicator information. The parameter integration agent is used to integrate the basic configuration information and the recommended configuration indicators into the input parameter set of the task creation interface.
[0102] Under different AI agent design methods, the specific tools provided may also be different. For example, under the design method in the above example, the specific configuration analysis agent needs to determine the specific experimental configuration information. In the process of determining this information, it may need to use user information, context information of the business space where the experiment is located, information about the comparison tasks recently created by the user, and so on. Therefore, the tools that the configuration analysis agent needs to call may include: tools for querying user information, tools for querying the context of spatial information, and / or tools for searching for comparison task information recently created by the user. Specifically, the process management agent can select some or all of the above tools from the above tools based on the actual comparison requirement information as the tools that the configuration analysis agent needs to call.
[0103] Specifically, when calling the configuration analysis agent, the prompt information can include comparison requirements, information about the tools that the configuration analysis agent needs to call, and other information output by the process management agent. Furthermore, the prompt information template can also include information about the agent's role settings and task descriptions. For example, the following example can be used:
[0104] “Role setting:
[0105] You are a professional AB experiment design assistant, focusing on helping users automatically create AB experiments.
[0106] Task Description:
[0107] Your task is to generate a scientifically sound experimental configuration based on the objectives, background information, and constraints provided by the user.
[0108] The experimental configuration includes:
[0109] Design of experimental and control groups.
[0110] Traffic distribution ratio (such as 50% / 50% or 10% / 90%)
[0111] Bucketing logic (random bucketing or stratified bucketing)
[0112] Experimental cycle recommendations
[0113] Experiment audience suggestions
[0114] Experimental objectives and hypotheses."
[0115] After receiving specific prompt information, the configuration analysis agent can analyze the basic configuration information suitable for the current experimental needs.
[0116] Since the indicator analysis agent needs to determine the recommended indicators, it usually needs to use the indicator context information in historical experiments, information on optional indicators, etc. when determining the indicators. Therefore, the tools that need to be called by the indicator analysis agent may include tools for querying the indicator context information configured in the comparative experiment task created by the user, and / or tools for querying optional indicator information, or other tools may also be included, which can be specifically determined by the process management agent after reasoning.
[0117] Specifically, when calling the above-mentioned indicator analysis agent, the user demand information and the information of the tool that the indicator analysis agent needs to call output by the process management agent can be expressed in the prompt information. In addition, it can also include other information output by the process management agent and other information output by the configuration analysis agent, etc. In specific implementation, the specifically recommended indicators can include multiple categories. For example, the core indicators of experimental observations, driving indicators, guardrail indicator information, etc. can be recommended. In a preferred implementation method, the indicator analysis agent can also output information such as the reasons for recommending specific indicators, calculation methods, applicable scenarios, etc. At this time, in order to enable the indicator analysis agent to understand the meaning of various indicators, explanatory information for various indicators can be included in the prompt information template. In addition, the prompt information template can also include information about the role setting and task description of the agent. For example, the following examples can be given:
[0118] “Role setting:
[0119] You are a professional AB experiment design assistant, focusing on providing users with scientific and reasonable observation indicator recommendations
[0120] Task Description:
[0121] Your task is to recommend appropriate core indicators, driving indicators, and guardrail indicators based on the user's experimental goals, business background, and existing data, and explain the meaning, calculation method, and applicable scenarios of each indicator.
[0122] Indicator definition:
[0123] Core Metrics: Also known as success metrics or North Star metrics, they reflect the ultimate relationship of the business.
[0124] Driver indicators: Also known as signpost indicators, proxy indicators, indirect or predictive indicators. Driver indicators reflect a mental causal model of what can make the business more successful, that is, an assumption about the driving force of success, rather than what success itself looks like.
[0125] Guardrail indicators: These protect the program from violations and are of two types: indicators that protect the business and indicators that assess the reliability and internal validity of experimental results.
[0126] After the configuration analysis agent outputs the basic configuration information and the indicator analysis agent outputs the recommended indicators, the parameter integration agent can integrate them into the input parameter set of the task creation interface according to the interface definition of creating the experiment.
[0127] It should be noted that, in the actual implementation, the tools for querying experimental information may be shared by the configuration analysis agent and the indicator analysis agent. For example,
[0128] Tools for querying experimental information can include:
[0129] Query the details of the three most recently created experiments in the space, including the experiment selection indicators.
[0130] Query the details of the three experiments created by the user recently, including the experimental selection indicators
[0131] Query experimental information of the same keyword
[0132] Among them, the first two tools can not only be used to query the experiment details, but also to query the selected indicators of the experiment.
[0133] In addition, it can also include tools specifically for querying indicator information, such as:
[0134] Query the business list of the indicator market
[0135] Indicator templates that can be used in the query space
[0136] You can query the selectable indicator information based on filtering conditions such as keywords and business lists.
[0137] S204: calling the task creation interface according to the input parameter set to create a corresponding comparison task.
[0138] After obtaining the input parameter set, the main program can call the task creation interface according to the input parameter set to create a corresponding comparison task. After the comparison task is created, the details of the specific comparison task created can be displayed to the user. Among them, the comparison task created above can be a version to be confirmed, and the user can edit it on this basis, including modifying some basic information, indicator information, etc. For example, based on the comparison task created by AI, a new control group can be added through manual operation, or new offline indicators can be added, etc. The interface can provide users with corresponding operation options. After the user edits or confirms that it is correct, the formal task creation can be completed.
[0139] In summary, through the embodiments of the present application, multiple AI agents can be designed in advance by modeling and dividing roles for the process of creating comparison tasks such as A / B experiments, which may include a process management agent. After receiving the comparison requirement information submitted by the user, the process management agent can first be called according to the comparison requirement information, so that the process management agent can determine the execution order of other AI agents and the tool information that other AI agents need to call during the execution process by analyzing the comparison requirement information. Afterwards, other AI agents can be called according to the execution order determined by the process management agent, wherein, when calling a specific called AI agent, the input prompt information of the called AI agent can be determined based on the comparison requirement information input by the user, the tool information that the called AI agent needs to call, and the output information of the upstream AI agent, so as to determine the basic configuration information and recommended indicator information corresponding to the comparison requirement through the other AI agents, and integrate them into the input parameter set of the task creation interface. Finally, the task creation interface can be called according to the input parameter set output by the AI agent to create the corresponding comparison task. In the above implementation method, the process is designed around the A / B comparison task, a multi-agent collaboration framework is built, and pre-designed tools are used to help AI agents obtain the required contextual knowledge, thereby improving the ability of AI generative models to solve complex problems and promoting the intelligent and automated development of comparison task construction.
[0140] In a preferred implementation, a directed acyclic task graph can be generated by summarizing the process of manually creating comparison tasks, which may include a list of AI agents participating in the current task and information on the execution paths accessible between different AI agents. The task graph can be provided to the process management agent so that the agent can determine the execution order of other AI agents based on the task graph, thereby improving efficiency and avoiding sequence errors or "dead loops".
[0141] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).
[0142] Corresponding to the aforementioned method embodiment, an embodiment of the present application further provides a device for creating a comparison task, which is applied to a comparison service system. The system includes multiple artificial intelligence (AI) agents, which are designed by modeling and dividing roles of the process for creating the comparison task, and include at least a process management agent. The device includes:
[0143] A demand acquisition unit, used to acquire comparison demand information submitted by users;
[0144] A first calling unit is configured to call the process management agent according to the comparison requirement information, so that the process management agent determines the execution order of other AI agents and tool information required to be called by other AI agents during execution by analyzing the comparison requirement information, so that the other AI agents can obtain relevant contextual knowledge by calling corresponding tools;
[0145] a second calling unit, configured to call the other AI agents in the execution order, wherein, when calling the called AI agent, input prompt information for the called AI agent is determined based on the comparison requirement information, information about the tool to be called by the called AI agent, and output information of the upstream AI agent, so as to determine, through the other AI agent, basic configuration information and recommended indicator information corresponding to the comparison requirement, and integrate them into an input parameter set for the task creation interface;
[0146] The task creation unit is configured to call the task creation interface according to the input parameter set to create a corresponding comparison task.
[0147] Among them, the other AI agents include configuration analysis agents, indicator analysis agents, and parameter integration agents;
[0148] The configuration analysis agent is used to determine basic configuration information corresponding to the comparison requirements, wherein the basic configuration information includes: task objectives, audience information, comparison group design, traffic distribution ratio, bucketing logic, and / or hypothetical content;
[0149] The indicator analysis agent is used to determine recommended indicators, wherein the indicators include core indicators, driving indicators and / or guardrail indicator information;
[0150] The parameter integration agent is used to integrate the basic configuration information and the indicators of the recommended configuration into an input parameter set of the task creation interface.
[0151] Specifically, the tools required to be called by the configuration analysis agent include: a tool for querying user information, a tool for querying spatial information context, and / or a tool for searching for comparison task information recently created by the user.
[0152] The tools that the indicator analysis agent needs to call include: a tool for querying indicator context information configured in the comparison task created by the user, and / or a tool for querying optional indicator information.
[0153] The indicator analysis agent is also used to explain the reasons, significance, calculation methods and applicable scenarios of the recommended indicators.
[0154] In addition, the device may further include:
[0155] A task graph providing unit is used to provide a task graph to the process management agent. The task graph is a directed acyclic graph, which is generated by summarizing the process of creating a comparison task. The task graph includes a list of AI agents participating in the current task and information on accessible execution paths between different AI agents, so that the process management agent can determine the execution order of other AI agents based on the task graph.
[0156] Among them, the multiple AI agents are implemented using a reasoning-action model, so that the multiple AI agents observe the environment and extract key details in the process of receiving and responding to information, and use the extracted information to guide their own thinking and subsequent actions, so as to achieve autonomous learning of the multiple AI agents.
[0157] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.
[0158] And an electronic device comprising:
[0159] one or more processors; and
[0160] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.
[0161] A computer program product includes a computer program / computer executable instructions, which implement the steps of the method described in the above method embodiment when executed by a processor in an electronic device.
[0162] in, Figure 4 The electronic device architecture is shown as an example, and may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.
[0163] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.
[0164] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) for controlling the low-level operations of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and a comparison task creation processing system 425, etc. can also be stored. The above-mentioned comparison task creation processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.
[0165] The input / output interface 413 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.
[0166] The network interface 414 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).
[0167] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 ).
[0168] It should be noted that although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.
[0169] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.
[0170] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.
[0171] The above describes in detail the method and electronic device for creating a comparison task provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is intended only to help understand the method and core concept of this application. At the same time, those skilled in the art will appreciate that the specific implementation methods and scope of application may vary based on the concept of this application. In summary, the contents of this specification should not be construed as limiting this application.
Claims
1. A method for creating a comparison task, characterized in that: The method is applied to a comparison service system, wherein the system includes multiple artificial intelligence (AI) agents, which are designed by modeling and role-dividing the process of creating comparison tasks, and include at least a process management agent. The method includes: Obtain comparison requirement information submitted by users; The process management agent is called according to the comparison requirement information, so that the process management agent determines the execution order of other AI agents and the tool information required to be called by other AI agents during the execution process by analyzing the comparison requirement information, so that the other AI agents can obtain relevant contextual knowledge by calling corresponding tools; Calling the other AI agents in the execution order, wherein, when calling the called AI agent, determining input prompt information for the called AI agent based on the comparison requirement information, information about the tool required to be called by the called AI agent, and output information of the upstream AI agent, so as to determine basic configuration information and recommended indicator information corresponding to the comparison requirement through the other AI agent, and integrate them into an input parameter set for the task creation interface; The task creation interface is called according to the input parameter set to create a corresponding comparison task.
2. The method according to claim 1, characterized in that The other AI agents include configuration analysis agents, indicator analysis agents, and parameter integration agents; The configuration analysis agent is used to determine basic configuration information corresponding to the comparison requirements, wherein the basic configuration information includes: task objectives, audience information, comparison group design, traffic distribution ratio, bucketing logic, and / or hypothetical content; The indicator analysis agent is used to determine recommended indicators, wherein the indicators include core indicators, driving indicators and / or guardrail indicator information; The parameter integration agent is used to integrate the basic configuration information and the indicators of the recommended configuration into an input parameter set of the task creation interface.
3. The method according to claim 2, characterized in that The tools required to be called by the configuration analysis agent include: a tool for querying user information, a tool for querying spatial information context, and / or a tool for searching for comparison task information recently created by the user.
4. The method according to claim 2, characterized in that The tools that the indicator analysis agent needs to call include: a tool for querying indicator context information configured in the comparison task created by the user, and / or a tool for querying optional indicator information.
5. The method according to claim 2, characterized in that The indicator analysis agent is also used to explain the reasons, significance, calculation methods and applicable scenarios of the recommended indicators.
6. The method according to claim 1, characterized in that Also includes: A task graph is provided to the process management agent. The task graph is a directed acyclic graph generated by summarizing the process of creating the comparison task, which includes a list of AI agents participating in the current task and information on accessible execution paths between different AI agents, so that the process management agent can determine the execution order of other AI agents based on the task graph.
7. The method according to claim 1, characterized in that The multiple AI agents are implemented using a reasoning-action model, so that the multiple AI agents observe the environment and extract key details in the process of receiving and responding to information, and use the extracted information to guide their own thinking and subsequent actions, so as to achieve autonomous learning of the multiple AI agents.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
9. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 7.
10. A computer program product comprising a computer program / computer executable instructions, characterized in that When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 7 are implemented.