Multi-agent marketing plan scheme effect evaluation feedback method and system

Abstract

The application discloses a multi-agent marketing planning scheme effect evaluation feedback method and system, relates to the technical field of marketing planning, and comprises the following steps: collecting historical marketing case data, extracting entities and the relationship between entities, and constructing a knowledge graph with entities as nodes and the relationship between entities as directed edges; meanwhile, based on the historical marketing case data, a triple is constructed, a general embedding encoder is trained and fine-tuned, and a domain embedding encoder is obtained; in the knowledge graph, a feature space is constructed with directed edges as the axis, each step of decision-making of the LLM generated planning scheme is mapped to a coordinate point in the feature space and connected to form a generation path, each component of the scheme can be traced back to specific historical cases and knowledge graph relationships, transparent decision-making basis is provided for the review and adjustment of planning personnel, the black box problem in the LLM generation process is effectively solved, and the trust degree of users for the generation result is improved.

Description

Technical Field

[0001] This invention relates to the field of marketing planning technology, specifically to a method and system for evaluating and providing feedback on the effectiveness of marketing planning schemes based on multi-agent systems. Background Technology

[0002] In the marketing planning industry, current work models generally rely on manual data collection and analysis. Specifically, planners manually download raw data from multiple data platforms, manually organize it into market insight reports, audience analysis reports, and competitor analysis reports, and then output planning proposals based on these. After project execution, post-project reviews also rely on manual summarization of campaign data and actual results, comparing them with expected goals to write a review report. Furthermore, public opinion monitoring is typically done manually, periodically retrieving and filtering relevant information. All of these workflows are manually driven, with data fragmented across different scenarios, lacking a systematic mechanism for the accumulation of digital assets.

[0003] Existing technologies typically suffer from the following drawbacks:

[0004] Low efficiency of manual data integration: The data required for each stage of planning (market insight, audience analysis, competitor analysis) is scattered across multiple external platforms. It needs to be manually exported, cleaned, and summarized one by one, which is cumbersome and prone to human error, making it difficult to support high-frequency, large-scale planning needs.

[0005] Historical project experience is difficult to accumulate and reuse: Project debriefing reports are stored in the form of unstructured documents and lack a unified data model and indexing mechanism. As a result, valuable experience such as planning strategies, audience characteristics, and campaign effects in historical cases cannot be retrieved and reused in subsequent projects, forming knowledge silos.

[0006] AI-assisted creative output lacks business context: When generating planning ideas, existing general-purpose language models cannot perceive the company's historical project data, the client's brand tone, and the domain knowledge accumulated in the industry, resulting in output content that is highly generalized but lacks customization and is difficult to apply directly to actual business scenarios.

[0007] The lack of collaborative mechanisms in the flow of data across multiple scenarios: Data flow between the three major scenarios of planning, review, and public opinion relies on manual transmission, lacking structured contextual transmission and knowledge feedback mechanisms, which limits the overall efficiency of business operations.

[0008] The marketing industry requires a certain level of professional expertise. In the traditional model, it is limited by personnel. The uneven ability of personnel and the turnover of personnel will cause huge differences in output standards, which will greatly increase the difficulty of providing services, which is the core service capability of the marketing industry.

[0009] The capabilities and results are all accumulated by individuals. For enterprises, there is no way to accumulate and summarize reusable methodologies or past service data assets, which is not conducive to enterprises achieving long-term services. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for evaluating and providing feedback on the effectiveness of multi-agent marketing planning schemes, so as to solve at least one of the above-mentioned deficiencies in the prior art.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a method for evaluating and providing feedback on the effectiveness of multi-agent marketing planning schemes, comprising the following steps:

[0012] S1. Collect historical marketing case data, extract entities and relationships between entities, and construct a knowledge graph with entities as nodes and relationships between entities as directed edges; at the same time, construct triples based on historical marketing case data, train and fine-tune the general embedding encoder to obtain the domain embedding encoder.

[0013] S2. Receive task parameters input by the user, call multi-source data through the data access layer and perform standardized processing to generate business data objects;

[0014] S3. Use multiple insight agents to perform parallel analysis on the business data object to obtain insight conclusions and corresponding confidence levels for each dimension, and then weight and fuse the insight conclusions based on the confidence levels.

[0015] S4. Dynamically construct a prompt word template based on the task parameters;

[0016] S5. Construct a retrieval key based on the task parameters, encode the retrieval key into a query vector using the domain embedding encoder, perform vector retrieval in the pre-built generalized vector library, and simultaneously perform path retrieval in the knowledge graph. Merge and fine-rank the two retrieval results, and perform context compression on the finely ranked candidate cases to obtain the compressed context.

[0017] S6. Assemble the compressed context, the weighted fusion insight conclusion, and the prompt word template into an enhanced prompt word, and input it into a large language model to generate multiple candidate planning schemes;

[0018] S7. Map the generation process of the multiple candidate planning schemes to a feature space constructed with the directed edges of the knowledge graph as axes to form a generation path for tracing each candidate scheme; predict the effect of each candidate scheme according to the pre-trained effect prediction model or similar case retrieval of the knowledge graph to obtain the prediction data of each preset core effect index, obtain the comprehensive score of prediction effect by weighted average, and filter and recommend to users according to the comprehensive score of prediction effect.

[0019] Furthermore, the method also includes updating the knowledge graph, including the following steps:

[0020] After the candidate planning scheme is actually implemented, the actual data of each core effect indicator is collected, the deviation between the predicted data of each core effect indicator and the actual data of the corresponding core effect indicator is calculated, and an error matrix is ​​generated.

[0021] Write the entities and relationships of the new case into the knowledge graph, and store the error matrix of the case as a node meta-attribute;

[0022] For dimensions with an absolute error value less than a first preset threshold, increase the confidence of the corresponding relational path for that dimension; for dimensions with an absolute error value greater than a second preset threshold, decrease the confidence of the corresponding relational path for that dimension, so that the adjustment of edge weights is accurate to the dimension level.

[0023] If the similarity of the strategy features of a new case and an existing historical case exceeds a preset threshold, and the error matrices of the two cases show the same deviation in the corresponding dimension and the difference in deviation exceeds a preset difference threshold, then the confidence level of the historical case in that dimension will be lowered.

[0024] If a new case has a similar strategy to an existing historical case but the error matrix shows an inverse deviation, then the generation of environmental condition constraints is triggered and written into the knowledge graph as an additional retrieval condition.

[0025] Cases with absolute error values ​​less than the third preset threshold are taken as positive samples, and cases with absolute error values ​​greater than the fourth preset threshold in any dimension are taken as hard negative samples. The incremental fine-tuning training set of the domain embedding encoder is collected and generated.

[0026] When the accumulated incremental samples in the incremental fine-tuning training set reach a preset size, the domain embedding encoder is incrementally fine-tuned using the incremental fine-tuning training set.

[0027] Furthermore, S3 includes the following steps:

[0028] After the scheduler identifies that there is no data dependency between multiple insight agents, the multiple insight agents are marked as parallel nodes and synchronously triggered. Each insight agent reads the corresponding business data object. The multiple insight agents include at least a market insight agent, a crowd analysis agent, and a competitor analysis agent.

[0029] The three indicators of data completeness, data source quality rating, and consistency of multiple sampling model outputs for each insight agent are weighted and calculated to obtain the insight analysis conclusion vector and confidence score of the corresponding insight agent.

[0030] During weighted fusion, the confidence scores of each insight agent are used as weights to perform weighted fusion of the insight analysis conclusion vectors. The weight of the insight analysis conclusion vectors in subsequent prompt word assembly is allocated according to the weight ratio of the confidence scores. When the confidence score of a certain insight agent is detected to be lower than the preset minimum score threshold, the corresponding insight analysis conclusion vector is marked as a low confidence input. During weighted fusion, the weight of the insight analysis conclusion vector is reduced or a supplementary data request for the corresponding business data object is triggered.

[0031] Furthermore, S4 includes the following steps:

[0032] Based on the industry category obtained from the task parameters, the corresponding industry-specific basic framework is matched from the preset prompt word template library;

[0033] Based on the brand tone tags obtained from the task parameters, differentiated preset creative style constraints and preset prohibited word rules are injected into the corresponding industry-specific basic framework to form a prompt word base with strategic emphasis slots;

[0034] The strategy emphasis slots of the prompt word base are dynamically filled according to the task type obtained from the task parameters, resulting in a strategy emphasis module as a complete prompt word structure.

[0035] Furthermore, the vector retrieval, path retrieval, and fusion and ranking of the two retrieval results in S5 include the following steps:

[0036] Vector retrieval: Historical marketing case data is cleaned into structured case samples, and corresponding case sample metadata is extracted. The case samples are encoded into historical case vectors using the domain embedding encoder. The historical case vectors are associated with the corresponding case sample metadata and collected to generate a generalized vector library. The search key is encoded into a query vector using the domain embedding encoder. Cosine similarity calculation is performed in the generalized vector library to recall the top N candidate cases with the highest similarity, where N is a preset positive integer.

[0037] Path retrieval: Starting with the brand tone tag and industry category in the task parameters, perform multi-hop path query in the knowledge graph according to the preset path paradigm, and filter by effect rating, retaining cases with a rating of the preset first rating threshold and above.

[0038] The two search results are merged and finely ranked: The two search results are merged using a reciprocal ranking fusion algorithm, and the fusion score formula is as follows:

[0039]

[0040] Where R is the set of retrieval methods, Let be the rank number of document d in the r-th retrieval method, and k be a smoothing constant;

[0041] After fusion, a cross-encoder re-ranking model is used for fine scoring. The top K cases with the highest scores are taken as the fine ranking results, and low-scoring items are filtered out by a relevance threshold, where K is a preset positive integer.

[0042] Furthermore, the context compression includes the following steps:

[0043] The LongLLMLingua compression framework is used, and a small language model is used as a surrogate model to calculate the conditional perplexity of each tag in the ranked candidate case text.

[0044] The tags are pruned from low to high according to the conditional perplexity, retaining tags with perplexity higher than a preset threshold; a progressive strategy from coarse to fine is adopted during the compression process, first performing coarse-grained pruning according to a multiple of the preset compression ratio, and then performing fine pruning at the tag level to achieve the target compression ratio.

[0045] The compressed context length is one-quarter to one-third of the original target length, and semantic consistency checks ensure that no semantic breaks occur.

[0046] Furthermore, the feature space in S7 is constructed through the following steps:

[0047] With the set of directed edges in a knowledge graph Each directed edge serves as a coordinate axis in the feature space. For each feature dimension, a preset mapping function maps the attribute of each directed edge to a scalar with a value between 0 and 1 as the coordinate value on the corresponding directed edge.

[0048] In each decoding step of the large language model generating candidate planning schemes, a mapping relationship is established between the key decision points output by the model and the corresponding directed edges in the knowledge graph, and the directed edges and coordinate values ​​corresponding to this step in the feature space are recorded.

[0049] Connect all coordinate values ​​of the same candidate scheme in the order of generation to form the generation path of that scheme;

[0050] When outputting the planning scheme, the generated path map is also output. Each coordinate value on the generated path map is associated with a corresponding historical case identifier and effect rating.

[0051] Furthermore, the effect prediction model in S7 is pre-trained through the following steps:

[0052] Structured features of the solutions are extracted from historical marketing case data. These structured features include strategy type, budget size, and channel mix. The structured features are then encoded into a first feature vector.

[0053] The graph embedding representation of the corresponding node of the historical case in the knowledge graph is generated by the graph embedding method and used as the second feature vector.

[0054] The first feature vector and the second feature vector are concatenated as the model input. The vector composed of actual data of each core performance indicator in historical marketing cases is used as the supervision label. The LightGBM or XGBoost model is trained by multi-output regression to obtain the performance prediction model. This model is used to output the prediction data of each core performance indicator based on the input candidate planning scheme. The prediction data of each core performance indicator are weighted and summed to obtain the comprehensive score of the prediction effect.

[0055] A multi-agent marketing planning scheme effectiveness evaluation and feedback system includes:

[0056] The data access layer calls multi-source data through the open interface of the dispatch platform, and outputs standardized business data objects after field mapping, anomaly filtering and semantic annotation.

[0057] The asset accumulation layer includes a relational database, a cache database, a vector database, and a graph database. The graph database stores a knowledge graph composed of entities and directed edges.

[0058] The model layer contains a large language model and an embedded encoder.

[0059] The Agent layer includes a market insight agent, a crowd analysis agent, a competitor analysis agent, and a planning generation agent. The execution order of each agent is arranged based on a directed acyclic graph. Among them, the market insight agent, the crowd analysis agent, and the competitor analysis agent execute in parallel when there is no data dependency, and the planning generation agent is activated after the output of the first three is ready.

[0060] The application layer receives planning task parameters input by the user and presents candidate planning schemes and a review report;

[0061] The review system collects actual execution effect data, and after manual confirmation, the review conclusions are fed back to the vector database and the knowledge graph.

[0062] Compared with existing technologies, the multi-agent marketing planning scheme effect evaluation and feedback method and system provided by this invention constructs a feature space in the knowledge graph with directed edges as axes. Each step of the LLM generating scheme is mapped to coordinate points in the feature space and connected to form a generation path. This makes each component of the scheme traceable to specific historical cases and knowledge graph relationships, providing transparent decision-making basis for planners' review and adjustment. It effectively solves the black box problem of the LLM generation process and enhances users' trust in the generated results.

[0063] By introducing a context compression step after RAG retrieval, calculating the importance score of each tag in the prompt words through a proxy model, prioritizing the pruning of tags with low information content, and combining a coarse-to-fine progressive compression strategy, the context length is compressed to one-quarter to one-third of the original length while retaining key information. This effectively alleviates the problems of information overload and positional bias caused by long contexts and reduces inference latency.

[0064] By comparing the actual implementation effect of the solution with the predicted effect in each dimension to generate an error matrix, and automatically updating the weights and effect ratings of the directed edges in the knowledge graph based on the error matrix, the knowledge graph can continuously optimize its retrieval priority according to the actual implementation feedback, forming a closed-loop update, effectively preventing outdated experience from continuously misleading subsequent planning, and realizing the continuous appreciation of knowledge assets. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0066] Figure 1 This is a flowchart illustrating the method steps provided in an embodiment of the present invention;

[0067] Figure 2 This is a system structure block diagram provided for an embodiment of the present invention. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0069] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0070] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0071] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0072] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0074] The embodiments described herein can be described with reference to plan views and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations can be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to those shown in the drawings, but include modifications to configurations formed based on manufacturing processes. Therefore, the areas illustrated in the drawings are schematic in nature, and the shapes of the areas shown in the figures illustrate specific shapes of areas of an element, but are not intended to be limiting.

[0075] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art.

[0076] Please see Figure 1 The method for evaluating and providing feedback on the effectiveness of multi-agent marketing plans includes the following steps:

[0077] S1. Construct a knowledge graph and fine-tune the pre-trained prediction model of the embedded encoder.

[0078] Collect and preprocess historical marketing case data:

[0079] First, the system reads all historical marketing planning task data from the enterprise relational database (PostgreSQL), including task input parameters (brand, industry, target audience, budget, and distribution channels), planning scheme content (creative theme, communication strategy, and execution pace), execution performance data (exposure, click-through rate, conversion rate, and return on investment), and debriefing conclusions (highlights, reasons for deviations, and optimization suggestions). Simultaneously, it extracts text from unstructured planning scheme documents and debriefing reports using a document parsing service, and supplements external performance data through the Paicha platform's open interface.

[0080] The collected raw data underwent text preprocessing, including word segmentation, stop word removal, and special character filtering of unstructured text. Subsequently, the unstructured planning schemes and debriefing reports were extracted into structured fields using the Qwen3.6 model according to a predefined schema, with the output format limited to JSON. Based on actual performance data, a comprehensive performance score for each case was calculated (a weighted sum of four metrics: exposure achievement rate, click achievement rate, conversion achievement rate, and return on investment achievement rate), mapped to a five-level performance rating system (S / A / B / C / D) according to the score range. Finally, a structured historical case sample library was obtained and stored in PostgreSQL.

[0081] Building a knowledge graph:

[0082] Based on the obtained structured historical case sample library, the Qwen3.6 model is used to identify entities for each case. The extracted entity types include: brand, industry category, audience tags, strategy type (new product launch / brand refresh / event marketing), communication channel, creative theme keywords, and performance rating. The prompt is designed as a structured JSON output format, requiring the model to return a list of entities according to a specified schema. Semantic relationships between entities are then identified, with relationship types including "brand_industry", "case_application_strategy type", "case_achievement_performance rating", "brand_competitor_brand", and "strategy_prerequisite_strategy". Each relationship record is represented as a directed edge. Where h is the head entity identifier, r is the relation type, and t is the tail entity identifier. Entities are written as nodes and relations as directed edges into a graph database (Neo4j). Nodes store entity attributes, and directed edges store relation attributes (relation type, confidence level, timestamp, performance rating). Finally, high-frequency query path patterns (such as "brand → affiliation → industry → application → case → achievement → performance rating") are pre-calculated and cached to obtain a marketing case knowledge graph.

[0083] Fine-tuning the encoder yields a neighborhood embedding encoder:

[0084] The system extracts task description text, brand tags, industry categories, strategy keywords, solution summaries, and performance rating tags from historical case samples to construct training triples: the anchor is the task description summary of a case; the positive sample is another case related to the anchor's business (same industry or brand tone) and with a performance rating of A or above; and the negative sample is a case unrelated to the anchor's business (different industry and brand tone) or with a performance rating of C or below. The triple construction follows a progressive difficulty principle—first constructing simple negative samples (completely unrelated), then constructing difficult negative samples (partially similar but with significant performance differences). Triplet Loss is used, with the training objective being to minimize the distance between the anchor and positive samples in the embedding space and maximize the distance between the anchor and negative samples. The encoder can use a general embedding model, such as the text-embedding-v3 model, and perform efficient fine-tuning using a low-rank adaptation method, updating only a small number of parameters; alternatively, the encoder can use the BGE (BAAI General Embedding) series, Conan-Embedding series, etc.

[0085] Pre-trained performance prediction model:

[0086] The structured features of the solutions are extracted from historical case samples, including strategy type, budget size and channel combination. These structured features are encoded into a first feature vector. A graph embedding representation of the corresponding node of the historical case in the knowledge graph is generated using a graph embedding method (Node2Vec or GraphSAGE) as a second feature vector. The first feature vector and the second feature vector are concatenated and used as the model input.

[0087] Using vectors composed of actual data from the core performance metrics of historical marketing cases as supervisory labels, the LightGBM or XGBoost model is trained using multi-output regression to obtain the performance prediction model.

[0088] The root mean square error (RMSE) and grading accuracy of the predicted and actual values ​​of each core performance indicator are calculated on the test set. When the amount of historical case data is insufficient (e.g., less than 5000 records), the performance prediction model is not pre-trained; instead, an analogy reasoning scheme based on similar cases from a knowledge graph is used. During the online reasoning phase, the performance prediction model outputs predicted data for each core performance indicator based on the input candidate planning scheme. The predicted data for each core performance indicator are then weighted and summed to obtain a comprehensive score for the predicted performance.

[0089] S2. Receive planning tasks, call multi-source data, and generate standardized business data objects.

[0090] Users input planning task parameters on the front end, including brand name, industry, target audience profile, budget range, task type (new product launch / brand refresh / event marketing), and special requirements (tone preference, prohibited words). The parameters are received and verified through the application layer application interface and then passed to the scheduler.

[0091] The scheduler parses the task parameters, queries the knowledge graph for the corresponding brand node using the brand name as the key, and obtains meta-information such as the brand's industry affiliation and historical case distribution. Then, it constructs a directed acyclic graph execution plan, initializes the task context object (including the task's unique identifier, parameter snapshot, and timestamp), and writes the task instance to the PostgreSQL task table, marking its status as "initialized".

[0092] After task initialization, the data endpoints of the Paicha platform are called in parallel through standardized application programming interfaces to configure the access parameters of each data source and collect external multi-source data, including market trend data (industry size, growth rate, popularity trend), target audience profile data (demographic attributes, interests and preferences, consumption behavior), and competitor advertising data (competitor brands, advertising channels, advertising volume, creative content).

[0093] In one embodiment, a public opinion monitoring system is also included, which continuously collects public opinion data from multiple sources as auxiliary environmental variables. The collected raw data undergoes an extraction-transformation-loading pipeline: fields returned by each data source are mapped to a unified target field according to a predefined business data object schema; anomalies such as missing fields, out-of-range values, and duplicate records are detected, and default values ​​are removed or filled according to preset rules; semantic tags (the business domain to which the field belongs, and the data source quality rating) are added to the standardized data fields. The processing flow of each data source can be executed in parallel, ultimately yielding standardized business data objects (corresponding to market insights, user profiles, and competitor analysis), which are stored in a cache for consumption by the corresponding intelligent agents. After sentiment analysis, the public opinion data is stored in a time-series database for subsequent review and retrieval.

[0094] S3, Multi-Insight Agent Output Confidence Fusion

[0095] After the scheduler identifies that there are no data dependencies between multiple insight agents, these agents are marked as parallel nodes and triggered synchronously. Each insight agent reads the corresponding business data object from the cache. These multiple insight agents include at least a market insight agent, a demographic analysis agent, and a competitor analysis agent.

[0096] Each agent uses a domain-specific structured prompt template (pre-installed in the template library). The template includes key data analysis points and constraints such as the output JSON schema. Business data objects with structured prompts are input into the Qwen3.6 model, generating JSON-formatted analysis conclusions. The model output is verified to conform to the JSON schema; if not, a retry (maximum 3 times) or downgrade is triggered. A weighted calculation is performed on three metrics for each agent: data completeness (required field fill rate), data source quality rating (interface source level), and consistency of model output across multiple samplings. This yields the corresponding insight analysis conclusion vector and confidence score for the insight agent. In the event of an agent execution error, default values ​​are filled with historical default data corresponding to that agent to ensure the main process is not interrupted.

[0097] Using the confidence scores of each insight agent as weights, the insight analysis conclusion vectors are weighted and fused. The weight of each vector in subsequent prompt word assembly is allocated according to the confidence score's proportion. When a confidence score of an insight agent is detected to be below a preset minimum threshold, the corresponding insight analysis conclusion vector is marked as a low-confidence input. During weighted fusion, the weight of this vector is reduced (e.g., multiplied by a penalty coefficient of 0.3) or a supplementary data request for the corresponding business data object is triggered. Finally, a weighted and fused comprehensive insight conclusion is obtained.

[0098] S4. Dynamically generate prompt word templates based on task parameters.

[0099] The planning and generation agent reads task parameters (industry category, brand tone, task type) and integrated insight conclusions from the task context and context storage, and executes a three-layer progressive prompt word construction:

[0100] Based on the industry category obtained from the task parameters, match the corresponding industry-specific basic framework from the preset prompt word template library;

[0101] Based on the brand tone tags obtained from the task parameters (such as FMCG, luxury goods, technology, government affairs, etc., which can be obtained from the "brand" information in the task input parameters), differentiated preset creative style constraints and preset prohibited word rules are injected into the corresponding industry-specific basic framework to form a prompt word base with strategic emphasis slots.

[0102] Based on the task type obtained from the task parameters, the strategy emphasis slots of the prompt word base are dynamically filled to obtain the strategy emphasis module as the complete prompt word structure.

[0103] The template library is stored in JSON / YAML format and supports hot updates. The matching algorithm is an exact index lookup.

[0104] S5, RAG historical experience retrieval, dual-path retrieval fusion and context enhancement

[0105] Once the dependency gating for generating the intelligent agent is enabled, the retrieval key value is read from the task context. The retrieval text is concatenated using three attributes—brand tag, industry category, and audience characteristics—according to preset weights (brand has the highest weight, followed by industry, with audience being an auxiliary correction item). The retrieval text is then encoded into a query vector using a domain embedding encoder finely tuned by S1.

[0106] Vector retrieval: Historical marketing case data is cleaned into structured case samples and corresponding metadata is extracted. A domain embedding encoder is used to encode the case samples into historical case vectors, which are then associated with the metadata to generate a generalized vector library. Cosine similarity is calculated using the query vector in the generalized vector library to recall the top N candidate cases with the highest similarity, where N is a preset positive integer.

[0107] Knowledge graph path retrieval: Starting with the brand tone tag and industry category in the task parameters, perform multi-hop path queries in the knowledge graph according to the preset path paradigm (such as "brand → affiliation → industry → application → case → achievement → effect rating"), and filter by effect rating, retaining cases with a preset first rating threshold (such as A level) and above to obtain the graph candidate set.

[0108] Fusion and ranking of two-way search results: The vector search results and the graph search results are merged using a reciprocal ranking fusion algorithm, and the fusion score is:

[0109]

[0110] Where R is the set of retrieval methods (two types: vector retrieval and graph retrieval). Let be the ranking number of document d in the r-th retrieval method, and k be a smoothing constant. After fusion, a cross-encoder reranking model is used for fine scoring. The top K cases with the highest scores are taken as the fine ranking results, and low-scoring items are filtered out by a relevance threshold. Here, K is a preset positive integer. The cross-encoder reranking model can be the BGE-Reranker series, BCE-Reranker, etc.

[0111] Context compression: Compression processing is performed on the candidate case samples that have passed the ranking and filtering. The specific steps are as follows:

[0112] The LongLLMLingua compression framework is used, employing a small language model (such as GPT-2 Small or LLaMA-7B) as a surrogate model to calculate the conditional perplexity of each token in the ranked candidate case text. Other compression frameworks used in other embodiments include BEAVER, DSPC (Dual-Stage Progressive Compression), and Kompress.

[0113] The surrogate model calculates the conditional perplexity of each tag under a given query condition. The lower the conditional perplexity of a tag, the more predictable it is under the query condition, the less information it carries, and the more likely it is to be pruned.

[0114] The tags are pruned from low to high perplexity, retaining those with perplexity above a preset threshold. A coarse-to-fine progressive strategy is employed during compression: first, coarse-grained pruning is performed at a multiple of the preset compression ratio, followed by fine-grained pruning at the tag level to achieve the target compression ratio. The compressed context length is one-quarter to one-third of the original target length, and semantic consistency checks ensure no semantic breaks occur.

[0115] S6. Generate candidate planning schemes

[0116] The compressed case sample content (strategy description, debriefing conclusions, etc.) is combined with the prompt word structure of S4 and the weighted fusion insight conclusions of S3 to form an enhanced prompt word. The planning agent generates 3-5 candidate planning schemes based on the enhanced prompt words through multiple samplings (temperature parameter set to 0.7-1.0, sampling times N=3-5 times) or fine-tuning of the prompt word strategy emphasis modules. Each candidate scheme includes five modules: creative theme, dissemination strategy, execution rhythm, budget allocation, and expected effect description.

[0117] S7. Feature Space Mapping, Performance Prediction, and Solution Recommendation

[0118] Feature space mapping:

[0119] With the set of directed edges in a knowledge graph Each directed edge serves as a coordinate axis in the feature space. For each feature dimension, a preset mapping function maps the attribute of each directed edge to a scalar with a value between 0 and 1 as the coordinate value on the corresponding directed edge:

[0120]

[0121] in , , These are configurable weighting coefficients (the default is 1 / 3). Normalized values ​​for performance ratings (e.g., S-level is mapped to 1.0, A-level to 0.8, B-level to 0.6, C-level to 0.4, and D-level to 0.2). This is a timestamp normalized value (the more recent the timestamp, the higher the value).

[0122] In each decoding step of the large language model generating candidate planning schemes, a mapping relationship is established between the key decision points output by the model (such as strategy type selection, channel selection, budget allocation ratio) and the corresponding directed edges in the knowledge graph, and the directed edges and coordinate values ​​corresponding to this step in the feature space are recorded.

[0123] Connect all coordinate values ​​of the same candidate solution in the order of generation to form the generation path of that solution. ,in Let be the coordinate vector of the j-th decision step in the feature space.

[0124] Each coordinate point on the generation path can be traced back to a specific directed edge in the knowledge graph and its corresponding historical case identifier. The final output plan includes a complete generation path diagram, containing the sequence of directed edges traversed, the historical case references for each edge (case identifiers and effect ratings), and explanations of key decision points.

[0125] Expected Outcomes:

[0126] Based on the S1 pre-trained performance prediction model, the structured features of candidate planning schemes are input, and the model outputs predicted data for each core performance indicator. The predicted data for each core performance indicator are then weighted and summed to obtain a comprehensive score for the predicted performance. The specific method of weighted summation is as follows: Let the predicted data for the core performance indicators output by the performance prediction model be... These correspond to dimensions such as exposure achievement rate, click achievement rate, conversion achievement rate, and return on investment achievement rate. The system presets weights for each dimension based on their importance to the marketing objectives. (The sum of the weights is 1, with default values ​​for exposure achievement rate 0.1, click achievement rate 0.2, conversion achievement rate 0.3, and ROI achievement rate 0.4). Calculate the weighted sum. As a comprehensive score for predictive performance.

[0127] When the performance prediction model is not trained (e.g., the amount of data is less than 5000), the performance prediction is changed to be based on knowledge graph similar case retrieval: the most similar historical cases to the candidate solution are retrieved in the knowledge graph (cosine similarity is calculated by the domain embedding encoder after S1 fine-tuning), and the weighted average of the actual data of each core performance indicator is taken as the comprehensive score of the prediction performance.

[0128] Recommended solution:

[0129] For each candidate solution, a comprehensive predicted performance score is calculated and compared with a preset performance threshold. Solutions with predicted performance exceeding the threshold are marked as official candidate solutions. The top three solutions are ranked from highest to lowest based on their comprehensive predicted performance scores and recommended to the user. The final output includes the top three candidate design solutions (containing predicted data for each core performance indicator, the comprehensive predicted performance score, and their respective generation path diagrams). After rendering at the application layer, these solutions are presented to the user for manual review and adjustment.

[0130] S8. Perform closed-loop updates to the knowledge graph.

[0131] After project completion (on a monthly or quarterly basis), the actual campaign data (exposure, click-through rate, conversion rate, ROI, etc.) is standardized and accessed through the Paicha platform's open interface. Simultaneously, a snapshot of public opinion within the specified time window is retrieved from a time-series database, and the expected target data written during the planning phase is read from PostgreSQL. The actual performance data and expected target data are then structurally aligned dimension by dimension, and the deviation value for each dimension is calculated (deviation = actual value - expected value). This is organized into an expected-actual difference data matrix and stored in the PostgreSQL task execution record table.

[0132] Extract the variance data matrix and the predicted data of each core performance indicator (CPI) from the PostgreSQL task execution record table and inference log, as well as the predicted data of each CPI when the scheme was selected in S7. Calculate the error dimension by dimension: Predicted data of core performance indicator - Actual data of the corresponding core performance indicator, and organize the error values ​​of each dimension into an error vector. Where d represents the number of effect dimensions. This represents the prediction bias for the i-th effect dimension. An error matrix is ​​generated by stacking the error vectors from multiple cases. The error matrix is ​​then stored in a PostgreSQL task execution record table, linked to the corresponding task identifier and scheme identifier.

[0133] Following the entity extraction and relation extraction process of S1, the new case data of the current task is processed into nodes and directed edges, written into the knowledge graph, and the error matrix of the case is stored as the node meta-attribute.

[0134] For dimensions whose absolute error value is less than a first preset threshold (default 0.15), the confidence of the corresponding relationship path for that dimension is increased. Specifically, the dimension precision score is calculated based on the absolute error value of each dimension. Using the precision score as the weight, update the confidence attribute of each directed edge involving dimension i in this case. ,in This is the learning rate coefficient (default is 0.2). For dimensions where the absolute value of the error is greater than the second preset threshold (default is 0.4), the corresponding policy effect relationship path is weighted less, and the confidence score is updated to... ,in This is the attenuation coefficient (default value is 0.3).

[0135] If the similarity of the strategy features of a new case and an existing historical case exceeds a preset threshold (cosine similarity calculated by the domain embedding encoder after S1 fine-tuning is greater than 0.9), and the error matrices of the two cases show a positive deviation in the corresponding dimension, and the difference in deviation exceeds a preset deviation difference threshold (default 0.2), then the confidence of the historical case in that dimension is reduced (e.g., multiplied by 0.7). If the strategy of a new case is similar to that of an existing historical case, but the error matrix shows a negative deviation (i.e., the new case predicts higher in a certain dimension while the historical case predicts lower in that dimension), then a "strategy-condition-effect" ternary relationship annotation is triggered, and changes in environmental conditions (including public opinion environment variables) are written into the knowledge graph as additional constraints, rather than a simple weight reduction.

[0136] Cases with absolute error values ​​all less than the third preset threshold (default 0.15) are marked as "high prediction consistency cases" and included as positive samples in the incremental fine-tuning training set of the domain embedding encoder. Cases with absolute error values ​​in any dimension greater than the fourth preset threshold (default 0.4) are marked as "high prediction bias cases" and included as hard-to-bear samples in the incremental fine-tuning training set. When the accumulated incremental samples in the incremental fine-tuning training set reach a preset size (e.g., 5000), the domain embedding encoder is incrementally fine-tuned using the incremental fine-tuning training set, realizing the dynamic update of the domain embedding encoder.

[0137] Please refer to Figure 2 Based on the above method, this invention also provides a multi-agent marketing planning scheme effect evaluation and feedback system, including:

[0138] The data access layer calls multi-source data through the open interface of the dispatch platform, and outputs standardized business data objects after field mapping, anomaly filtering and semantic annotation.

[0139] The asset accumulation layer includes a relational database, a cache database, a vector database, and a graph database. The graph database stores a knowledge graph composed of entities and directed edges.

[0140] The model layer contains a large language model and an embedded encoder.

[0141] The Agent layer includes a market insight agent, a crowd analysis agent, a competitor analysis agent, and a planning generation agent. The execution order of each agent is arranged based on a directed acyclic graph. Among them, the market insight agent, the crowd analysis agent, and the competitor analysis agent execute in parallel when there is no data dependency, and the planning generation agent is activated after the output of the first three is ready.

[0142] The application layer receives planning task parameters input by the user and presents candidate planning schemes and a review report;

[0143] The review system collects actual execution effect data, and after manual confirmation, the review conclusions are fed back to the vector database and the knowledge graph.

[0144] In other embodiments of the present invention, the following alternative implementations may be made:

[0145] Alternative to large language models: The model layer of this invention calls the API of large language models to perform inference. Specifically, commercial large language models such as Tongyi Qianwen series, GPT-4o, Claude series, Wenxin Yiyan, and Hunyuan can be selected, or they can be replaced by open source models deployed privately by enterprises (such as LLaMA 3, DeepSeek, etc.). This invention does not restrict the model service provider.

[0146] Vector database alternatives: This invention uses vector databases for semantic storage and retrieval. Specifically, vector databases such as Generalized Vector Library, Milvus, Chroma, Pinecone, Weaviate, and pgvector can be used. The similarity calculation method can be cosine similarity, Euclidean distance, or inner product. This invention does not limit the specific vector database product.

[0147] Alternative Agent Scheduling Frameworks: In other embodiments, implementations can also be based on open-source multi-agent orchestration frameworks such as LangChain, AutoGen, CrewAI, Dify, and LlamaIndex, with the orchestration logic remaining consistent with the method described in the claims.

[0148] Alternative to manual confirmation: In scenarios where the data quality and reliability are high, the manual confirmation step of the review conclusion can also be configured as an automatic confirmation mechanism based on a confidence threshold. When the overall reliability score of the conclusion generated by the review agent exceeds the preset threshold, the system automatically triggers the database writing process without manual intervention.

[0149] Alternative front-end and back-end technology stacks: The front-end and back-end of this invention are developed based on the TypeScript language. In other embodiments, equivalent server-side logic and interaction layer functions can also be implemented using other programming languages ​​such as Python, Java, and Go. This invention does not limit the programming language and framework.

[0150] Alternative data access solutions: This invention standardizes the access to marketing data through the open interface of the Paicha platform. In other embodiments, it can also access other data platform APIs (such as Miaozhen, Nielsen, GfK, etc.) or accept structured data files (CSV, Excel, etc.) exported from the enterprise's internal CRM and ERP systems. The adapter mode of the data access layer supports multiple data source extensions.

[0151] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating and providing feedback on the effectiveness of multi-agent marketing planning schemes, characterized in that: Includes the following steps: S1. Collect historical marketing case data, extract entities and relationships between entities, and construct a knowledge graph with entities as nodes and relationships between entities as directed edges; at the same time, construct triples based on historical marketing case data, train and fine-tune the general embedding encoder to obtain the domain embedding encoder. S2. Receive task parameters input by the user, call multi-source data through the data access layer and perform standardized processing to generate business data objects; S3. Use multiple insight agents to perform parallel analysis on the business data object to obtain insight conclusions and corresponding confidence levels for each dimension, and then weight and fuse the insight conclusions based on the confidence levels. S4. Dynamically construct a prompt word template based on the task parameters; S5. Construct a retrieval key based on the task parameters, encode the retrieval key into a query vector using the domain embedding encoder, perform vector retrieval in the pre-built generalized vector library, and simultaneously perform path retrieval in the knowledge graph. Merge and fine-rank the two retrieval results, and perform context compression on the finely ranked candidate cases to obtain the compressed context. S6. Assemble the compressed context, the weighted fusion insight conclusion, and the prompt word template into an enhanced prompt word, and input it into a large language model to generate multiple candidate planning schemes; S7. Map the generation process of the multiple candidate planning schemes to a feature space constructed with the directed edges of the knowledge graph as axes to form a generation path for tracing each candidate scheme; predict the effect of each candidate scheme according to the pre-trained effect prediction model or similar case retrieval of the knowledge graph to obtain the prediction data of each preset core effect index, obtain the comprehensive score of prediction effect by weighted average, and filter and recommend to users according to the comprehensive score of prediction effect.

2. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, The method also includes updating the knowledge graph, including the following steps: After the candidate planning scheme is actually implemented, the actual data of each core effect indicator is collected, the deviation between the predicted data of each core effect indicator and the actual data of the corresponding core effect indicator is calculated, and an error matrix is ​​generated. Write the entities and relationships of the new case into the knowledge graph, and store the error matrix of the case as a node meta-attribute; For dimensions with an absolute error value less than a first preset threshold, increase the confidence of the corresponding relational path for that dimension; for dimensions with an absolute error value greater than a second preset threshold, decrease the confidence of the corresponding relational path for that dimension, so that the adjustment of edge weights is accurate to the dimension level. If the similarity of the strategy features of a new case and an existing historical case exceeds a preset threshold, and the error matrices of the two cases show the same deviation in the corresponding dimension and the difference in deviation exceeds a preset difference threshold, then the confidence level of the historical case in that dimension will be lowered. If a new case has a similar strategy to an existing historical case but the error matrix shows an inverse deviation, then the generation of environmental condition constraints is triggered and written into the knowledge graph as an additional retrieval condition. Cases with absolute error values ​​less than the third preset threshold are taken as positive samples, and cases with absolute error values ​​greater than the fourth preset threshold in any dimension are taken as hard negative samples. The incremental fine-tuning training set of the domain embedding encoder is collected and generated. When the accumulated incremental samples in the incremental fine-tuning training set reach a preset size, the domain embedding encoder is incrementally fine-tuned using the incremental fine-tuning training set.

3. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, S3 includes the following steps: After the scheduler identifies that there is no data dependency between multiple insight agents, the multiple insight agents are marked as parallel nodes and synchronously triggered. Each insight agent reads the corresponding business data object. The multiple insight agents include at least a market insight agent, a crowd analysis agent, and a competitor analysis agent. The three indicators of data completeness, data source quality rating, and consistency of multiple sampling model outputs for each insight agent are weighted and calculated to obtain the insight analysis conclusion vector and confidence score of the corresponding insight agent. During weighted fusion, the confidence scores of each insight agent are used as weights to perform weighted fusion of the insight analysis conclusion vectors. The weight of the insight analysis conclusion vectors in subsequent prompt word assembly is allocated according to the weight ratio of the confidence scores. When the confidence score of a certain insight agent is detected to be lower than the preset minimum score threshold, the corresponding insight analysis conclusion vector is marked as a low confidence input. During weighted fusion, the weight of the insight analysis conclusion vector is reduced or a supplementary data request for the corresponding business data object is triggered.

4. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, S4 includes the following steps: Based on the industry category obtained from the task parameters, the corresponding industry-specific basic framework is matched from the preset prompt word template library; Based on the brand tone tags obtained from the task parameters, differentiated preset creative style constraints and preset prohibited word rules are injected into the corresponding industry-specific basic framework to form a prompt word base with strategic emphasis slots; The strategy emphasis slots of the prompt word base are dynamically filled according to the task type obtained from the task parameters, resulting in a strategy emphasis module as a complete prompt word structure.

5. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, The vector retrieval, path retrieval, and fusion and ranking of the two retrieval results described in S5 include the following steps: Vector retrieval: Historical marketing case data is cleaned into structured case samples, and corresponding case sample metadata is extracted. The case samples are encoded into historical case vectors using the domain embedding encoder. The historical case vectors are associated with the corresponding case sample metadata and collected to generate a generalized vector library. The search key is encoded into a query vector using the domain embedding encoder. Cosine similarity calculation is performed in the generalized vector library to recall the top N candidate cases with the highest similarity, where N is a preset positive integer. Path retrieval: Starting with the brand tone tag and industry category in the task parameters, perform multi-hop path query in the knowledge graph according to the preset path paradigm, and filter by effect rating, retaining cases with a rating of the preset first rating threshold and above. The two search results are merged and finely ranked: The two search results are merged using a reciprocal ranking fusion algorithm, and the fusion score formula is as follows: Where R is the set of retrieval methods, Let be the rank number of document d in the r-th retrieval method, and k be a smoothing constant; After fusion, a cross-encoder re-ranking model is used for fine scoring. The top K cases with the highest scores are taken as the fine ranking results, and low-scoring items are filtered out by a relevance threshold, where K is a preset positive integer.

6. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, The context compression includes the following steps: The LongLLMLingua compression framework is used, and a small language model is used as a surrogate model to calculate the conditional perplexity of each tag in the ranked candidate case text. The tags are pruned from low to high according to the conditional perplexity, retaining tags with perplexity higher than a preset threshold; a progressive strategy from coarse to fine is adopted during the compression process, first performing coarse-grained pruning according to a multiple of the preset compression ratio, and then performing fine pruning at the tag level to achieve the target compression ratio. The compressed context length is one-quarter to one-third of the original target length, and semantic consistency checks ensure that no semantic breaks occur.

7. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, The feature space in S7 is constructed through the following steps: With the set of directed edges in a knowledge graph Each directed edge serves as a coordinate axis in the feature space. For each feature dimension, a preset mapping function maps the attribute of each directed edge to a scalar with a value between 0 and 1 as the coordinate value on the corresponding directed edge. In each decoding step of the large language model generating candidate planning schemes, a mapping relationship is established between the key decision points output by the model and the corresponding directed edges in the knowledge graph, and the directed edges and coordinate values ​​corresponding to this step in the feature space are recorded. Connect all coordinate values ​​of the same candidate scheme in the order of generation to form the generation path of that scheme; When outputting the planning scheme, the generated path map is also output. Each coordinate value on the generated path map is associated with a corresponding historical case identifier and effect rating.

8. The method for evaluating and providing feedback on the effectiveness of a multi-agent marketing plan as described in claim 1, characterized in that, The effect prediction model in S7 is pre-trained through the following steps: Structured features of the solutions are extracted from historical marketing case data. These structured features include strategy type, budget size, and channel mix. The structured features are then encoded into a first feature vector. The graph embedding representation of the corresponding node of the historical case in the knowledge graph is generated by the graph embedding method and used as the second feature vector. The first feature vector and the second feature vector are concatenated as the model input. The vector composed of actual data of each core performance indicator in historical marketing cases is used as the supervision label. The LightGBM or XGBoost model is trained by multi-output regression to obtain the performance prediction model. This model is used to output the prediction data of each core performance indicator based on the input candidate planning scheme. The prediction data of each core performance indicator are weighted and summed to obtain the comprehensive score of the prediction effect.

9. A multi-agent marketing plan effectiveness evaluation and feedback system, used to execute the multi-agent marketing plan effectiveness evaluation and feedback method according to any one of claims 1-8, characterized in that, include: The data access layer calls multi-source data through the open interface of the dispatch platform, and outputs standardized business data objects after field mapping, anomaly filtering and semantic annotation. The asset accumulation layer includes a relational database, a cache database, a vector database, and a graph database. The graph database stores a knowledge graph composed of entities and directed edges. The model layer contains a large language model and an embedded encoder. The Agent layer includes a market insight agent, a crowd analysis agent, a competitor analysis agent, and a planning generation agent. The execution order of each agent is arranged based on a directed acyclic graph. Among them, the market insight agent, the crowd analysis agent, and the competitor analysis agent execute in parallel when there is no data dependency, and the planning generation agent is activated after the output of the first three is ready. The application layer receives planning task parameters input by the user and presents candidate planning schemes and a review report; The review system collects actual execution effect data, and after manual confirmation, the review conclusions are fed back to the vector database and the knowledge graph.

Images