A method, system, device and storage medium for evaluating content indispensability based on a counterfactual reasoning generative engine optimization

Abstract

The present application relates to the field of artificial intelligence models, and discloses a method, system, device and storage medium for evaluating the indispensability of the content of a generative engine based on counterfactual reasoning. The method comprises: inputting the obtained target user query information and the brand content information to be evaluated into a generation model to extract the multi-dimensional features of the original answer; constructing a counterfactual variant set based on the brand content information to be evaluated, inputting the target user query information and each counterfactual variant of the counterfactual variant set into the generation model to generate a counterfactual answer, extracting the multi-dimensional features of the counterfactual answer and calling an evaluation model, comparing the multi-dimensional features of the original answer and the counterfactual answer, calculating the multi-dimensional difference parameters, calculating the counterfactual influence index of the brand content information to be evaluated, and outputting the corresponding evaluation result. The method provided in the present application embodiment can realize the objective quantification of indispensable content and reflect the causal contribution of different data content to the generation of model answers.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence models, and in particular to a method, system, device, and storage medium for evaluating the indispensability of content in a generative engine based on counterfactual reasoning. Background Technology

[0002] As large language models become the mainstream entry point for users to obtain information, enterprises need to optimize their data through generative engine optimization (GEO) to ensure that their data content can be accurately understood, prioritized, and integrated into the generated answers by large models.

[0003] Existing GEO technologies mainly include the following three methods: 1. Keyword matching, using word frequency similarity to measure the degree of content relevance.

[0004] 2. Vector semantic matching: calculates the semantic association between the content and the query using cosine similarity.

[0005] 3. Click and feedback simulation, using historical data to train a ranking model.

[0006] The above methods have the following problems: they can only evaluate performance when data is present, and cannot quantify the impact of missing data on the large model's answer, i.e., they cannot determine the relevance and indispensability of different parts of the data. The decision-making process within the large model is a black box, making it impossible to quantify the impact of different data on the model's generated results. Summary of the Invention

[0007] This invention provides a method, system, device, and storage medium for evaluating the indispensability of content in a generative engine based on counterfactual reasoning, in order to solve at least some of the above-mentioned problems.

[0008] In a first aspect, embodiments of the present invention provide a generative engine optimization method for evaluating the indispensability of content based on counterfactual reasoning, comprising: acquiring target user query information and corresponding brand content information to be evaluated; inputting the target user query information and the brand content information to be evaluated into a pre-acquired generative model to generate an original answer and extracting multi-dimensional features of the original answer; constructing a set of counterfactual variants based on the brand content information to be evaluated; inputting the target user query information and each counterfactual variant in the set of counterfactual variants into the generative model to generate a corresponding counterfactual answer and extracting multi-dimensional features from the counterfactual answer; calling a pre-trained evaluation model to compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and calculating the multi-dimensional difference parameters between the two; and calculating the counterfactual influence index of the brand content information to be evaluated under the target user query information based on the multi-dimensional difference parameters, and outputting the evaluation result corresponding to the counterfactual influence index.

[0009] The generative engine optimization content indispensability assessment method based on counterfactual reasoning provided in this invention is compatible with causal inference methods, and can build a complete causal and indispensability assessment system. It can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0010] Optionally, the evaluation model is trained through the following steps: a training dataset is constructed based on pre-acquired historical user query information, historical brand content information, and the original answers, counterfactual answers, and labeled difference values ​​corresponding to the historical user query information and historical brand content information; after freezing the underlying Transformer of the model to be trained, the model is trained based on the training dataset; training is stopped when the mean squared error between the minimized predicted difference value and the labeled difference value of the model to be trained is within a preset mean squared error range, thus obtaining the evaluation model.

[0011] Optionally, when constructing the counterfactual variant set, the counterfactual variant set is constructed based on the brand content information to be evaluated through a preset single intervention rule; wherein, the single intervention rule includes modifying the target content information in the brand content information to be evaluated, and maintaining the text length and context consistency in the brand content information to be evaluated.

[0012] Optionally, the counterfactual variant set includes at least one of the following variants: deletion variant, used to delete data information in the brand content information to be evaluated; replacement variant, used to replace the data information in the brand content information to be evaluated with other preset information; degradation variant, used to change the deterministic statement in the brand content information to be evaluated into a vague statement; scrambling variant, used to adjust the textual logical order of the brand content information to be evaluated; and adversarial variant, used to add preset contradictory information to the brand content information to be evaluated.

[0013] Optionally, the multi-dimensional features include the recommendation status of the original answer and counterfactual answer, evidence citation information, information completeness, answer accuracy, recommendation ranking, information uniqueness, statement certainty, credibility of citation sources, and anti-interference stability.

[0014] Optionally, the counterfactual influence index is calculated through the following steps: determining the single-dimensional difference between the original answer and the counterfactual answer based on the normalized value of the feature difference in the corresponding dimension; weighting and summing the single-dimensional differences based on the preset weight values ​​of the original answer and the counterfactual answer in each dimension to obtain the comprehensive difference parameter for each counterfactual variant; and calculating the counterfactual influence index based on the comprehensive difference parameter corresponding to each counterfactual variant.

[0015] Optionally, the evaluation results include: a counterfactual influence index to characterize the indispensability of the brand content information to be evaluated to the generated evaluation results; a dimensional contribution value to characterize the contribution ratio of each dimension feature to indispensability; weak text information that leads to a decrease in the accuracy of answers after the brand content information to be evaluated is modified; and a stability score of the brand content information to be evaluated in a scrambled state.

[0016] Optionally, the steps for outputting the evaluation results corresponding to the counterfactual influence index include: when the parameter value of the counterfactual influence index is less than 0.3, outputting the redundancy judgment result and the corresponding recommendation result; when the parameter value of the counterfactual influence index is greater than or equal to 0.3 and less than 0.6, outputting the moderately indispensable judgment result and the corresponding recommendation result; when the parameter value of the counterfactual influence index is greater than or equal to 0.6, outputting the highly indispensable judgment result and the corresponding recommendation result.

[0017] Secondly, embodiments of the present invention provide a generative engine-based system for evaluating the indispensability of content based on counterfactual reasoning, comprising: an acquisition module for acquiring target user query information and corresponding brand content information to be evaluated; a processing module for inputting the target user query information and the brand content information to be evaluated into a pre-acquired generative model to generate an original answer and extract multi-dimensional features of the original answer; constructing a set of counterfactual variants based on the brand content information to be evaluated; inputting the target user query information and each counterfactual variant in the set of counterfactual variants into the generative model to generate a corresponding counterfactual answer and extracting multi-dimensional features from the counterfactual answer; a calculation module for calling a pre-trained evaluation model to compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer and calculate the multi-dimensional difference parameters between the two; and an output module for calculating the counterfactual influence index of the brand content information to be evaluated under the target user query information based on the multi-dimensional difference parameters and outputting the evaluation result corresponding to the counterfactual influence index.

[0018] The counterfactual reasoning-based generative engine optimization content indispensability assessment system provided in this invention is compatible with causal inference methods, and can build a complete causal and indispensability assessment system. It can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0019] Thirdly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the memory stores instructions; the processor invokes the instructions in the memory to cause the processor to execute the content indispensability evaluation method for generative engines based on counterfactual reasoning according to any of the foregoing embodiments of the first aspect of the present invention.

[0020] The processor of the electronic device provided in this embodiment of the invention executes the counterfactual reasoning-based generative engine optimization content indispensability evaluation method of any of the foregoing embodiments of the first aspect of the invention by calling instructions in the memory. By being compatible with causal inference methods, a complete causal and indispensability evaluation system can be built, which can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing instructions that, when executed by a processor, implement the content indispensability evaluation method for generative engines based on counterfactual reasoning, as described in any of the foregoing embodiments of the first aspect of the present invention.

[0022] The instructions stored in the computer-readable storage medium provided in the embodiments of the present invention can be called by a processor and executed by the counterfactual reasoning-based generative engine optimization content indispensability evaluation method of any of the foregoing embodiments of the first aspect of the present invention. By being compatible with causal inference methods, a complete causal and indispensability evaluation system can be built, which can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0024] Figure 1 This is a flowchart of an embodiment of the content indispensability evaluation method for generative engines based on counterfactual reasoning according to the present invention; Figure 2 This is a flowchart of step S160 in one embodiment of the generative engine optimization content indispensability evaluation method based on counterfactual reasoning of the present invention; Figure 3 This is a flowchart of the training and evaluation model in one embodiment of the generative engine optimization content indispensability evaluation method based on counterfactual reasoning of the present invention. Figure 4 This is a flowchart illustrating one embodiment of the generative engine optimization method for evaluating the indispensability of content based on counterfactual reasoning of the present invention. Figure 5This is a schematic diagram of the construction of a counterfactual variant in one embodiment of the generative engine optimization content indispensability evaluation method based on counterfactual reasoning of the present invention; Figure 6 This is a schematic diagram comparing the original answer and the counterfactual answer in one embodiment of the generative engine optimization method for evaluating the indispensability of content based on counterfactual reasoning of the present invention. Figure 7 This is a structural block diagram of an embodiment of the content indispensability assessment system optimized by a generative engine based on counterfactual reasoning according to the present invention; Figure 8 This is a structural block diagram of one embodiment of the electronic device of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that all directional indications in the embodiments of the present invention, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationship and movement of the components in a specific posture as shown in the attached figure. If the specific posture changes, the directional indication will also change accordingly.

[0027] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0028] For ease of understanding, the following describes the content indispensability evaluation method for generative engines based on counterfactual reasoning, as an embodiment of the present invention. Figure 1 As shown, the content indispensability evaluation method for generative engines based on counterfactual reasoning in this embodiment of the invention includes steps S110 to S160.

[0029] In step S110, the target user query information and the corresponding brand content information to be evaluated are obtained.

[0030] In this embodiment, the query content of the target user and the brand content to be evaluated are obtained, and the query content of the target user and the brand content to be evaluated are input into the target large model to generate the original answer. Objective features such as recommendation status, evidence citation, and information completeness in the answer are extracted.

[0031] The target user query information comes from user search logs, GEO core demand keywords, industry standard query libraries, and natural language questions. The brand content information to be evaluated comes from core information from the brand's official website, product details, official documents, and corporate knowledge base, after being cleaned, deduplicated, and segmented.

[0032] In step S120, the target user query information and the brand content information to be evaluated are input into the pre-acquired generation model to generate the original answer and extract the multi-dimensional features of the original answer.

[0033] In this embodiment, the original answer is generated by a pre-acquired generative model. The generative model can be a pre-trained large model. The input and output are standardized and optimized through the generative model, and the model structure of the generative model is not adjusted.

[0034] For example, for a generative model, the input template is: User question: {Q}; Reference content: {T}; Please answer based on the reference content; The output is fixed in a structured format, which facilitates accurate feature extraction.

[0035] In step S130, a set of counterfactual variants is constructed based on the brand content information to be evaluated.

[0036] like Figure 5 As shown, in some optional embodiments, when constructing the counterfactual variant set, the counterfactual variant set is constructed based on the brand content information to be evaluated through a preset single intervention rule.

[0037] Among them, the single intervention rule includes modifying the target content information in the brand content information to be evaluated, and maintaining the consistency of text length and context in the brand content information to be evaluated.

[0038] Specifically, the set of counterfactual variants includes at least one of the following variants: The deletion variant is used to remove data from the brand content information being evaluated.

[0039] Replacement variants are used to replace data in the brand content information to be evaluated with other preset information.

[0040] A downgrade variant, used to change definitive statements in brand content information to vague statements.

[0041] Disrupt variations to adjust the logical order of the text in the brand content information being evaluated.

[0042] Countervariants are used to add pre-defined contradictory information to the brand content information to be evaluated.

[0043] In this embodiment, the variant t' is deleted, taking into account the five counterfactual variants mentioned above. ( del ) , Replacement variant t' ( rep ) , downgrade variant t' ( deg ) Disrupting the variant t' ( shf ) , anti-variant t' ( adv ) This yields the counterfactual variant set T'={t' ( del ) , t' ( rep ) , t' ( deg ) , t' ( shf ) , t' ( adv )}

[0044] In step S140, the target user query information and each counterfactual variant in the counterfactual variant set are input into the generation model to generate the corresponding counterfactual answer and extract the multi-dimensional features from the counterfactual answer.

[0045] In some optional embodiments, the multi-dimensional features include the recommendation status of the original answer and the counterfactual answer, evidence citation information, information completeness, answer accuracy, recommendation ranking, information uniqueness, statement certainty, credibility of citation sources, and anti-interference stability.

[0046] In step S150, the pre-trained evaluation model is invoked to compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and the multi-dimensional difference parameters between the two are calculated.

[0047] Each counterfactual variant is input into the same evaluation model along with the same target user query information to generate a counterfactual answer. Then, the original answer and the counterfactual answer are compared from five dimensions: recommendation existence, recommendation strength, evidence completeness, answer accuracy, and answer completeness. The single-dimensional difference value for each dimension is calculated, and each single-dimensional difference value is aggregated into a multi-dimensional difference parameter Δ according to the following formula. i : Δ i = Σ(wd×Δ i , d ); Where, Δ i ,d The single-dimensional difference value for each dimension, wd is the preset weight coefficient for each dimension, and Σwd=1.

[0048] like Figure 3 As shown, in some optional embodiments, the evaluation model is trained through the following steps: Step S210: Construct a training dataset based on the pre-acquired historical user query information, historical brand content information, and the original answers, counterfactual answers, and labeled difference values ​​corresponding to the historical user query information and historical brand content information.

[0049] Step S220: After freezing the underlying Transformer of the model to be trained, train the model based on the training dataset. When the mean square error between the minimum predicted difference value and the labeled difference value of the model to be trained is within the preset mean square error range, stop training and obtain the evaluation model.

[0050] In this embodiment, only the evaluation model is fine-tuned, while the generation model is not fine-tuned.

[0051] After freezing the underlying Transformer of the model to be trained, the weights of the feature encoding layer and classification layer of the evaluation model are fine-tuned based on the training dataset. This ensures that the mean squared error between the predicted difference value and the labeled difference value of the weights of the feature encoding layer and classification layer of the model to be trained is within the preset mean squared error range, thereby improving the difference evaluation accuracy of the model.

[0052] Specifically, the generative model in this embodiment can be a general large language model, which is only responsible for answer generation and does not undergo training and fine-tuning, thus maintaining the original reasoning ability.

[0053] The evaluation model adds a new recommendation state encoder, evidence citation encoder, and information integrity encoder, and the difference calculation layer incorporates similarity, edit distance, and confidence scoring operators.

[0054] The input layer of the evaluation model receives the target user's query information, the brand content to be evaluated, the original answer, and the counterfactual answer. The output layer of the evaluation model outputs multi-dimensional difference parameters.

[0055] The evaluation model in this embodiment is fine-tuned and trained through the following steps: The training data includes target user query information, brand content to be evaluated, original answers, counterfactual answers, and pre-labeled discrepancy values ​​to cover five evaluation dimensions.

[0056] The training fine-tunes the evaluation model to minimize the mean squared error between the predicted and labeled discrepancies. Fine-tuning parameters include: learning rate 2e-5, batch size 8, training for 3 epochs, freezing the underlying Transformer, and fine-tuning only the top-level classification head and feature encoding layer of the model.

[0057] In step S160, based on multi-dimensional difference parameters, the counterfactual influence index of the brand content information to be evaluated under the target user query information is calculated, and the evaluation result corresponding to the counterfactual influence index is output.

[0058] like Figure 2 As shown, in some optional embodiments, in step S160, the counterfactual influence index is calculated through the following steps: Step S161: Determine the one-dimensional difference between the original answer and the counterfactual answer based on the normalized value of the feature difference in the corresponding dimension.

[0059] Step S162: Based on the preset weight values ​​of the original answer and the counterfactual answer in each dimension, the single-dimensional differences are weighted and summed to obtain the comprehensive difference parameter of each counterfactual variant.

[0060] Step S163: Calculate the counterfactual influence index based on the comprehensive difference parameter corresponding to each counterfactual variant.

[0061] Furthermore, the steps for outputting the evaluation results corresponding to the counterfactual influence index include: When the parameter value of the counterfactual influence index is less than 0.3, redundant judgment results and corresponding suggestions are output.

[0062] When the parameter value of the counterfactual influence index is greater than or equal to 0.3 and less than 0.6, the output is a moderately indispensable judgment result and the corresponding suggestion result.

[0063] When the parameter value of the counterfactual influence index is greater than or equal to 0.6, the output is an essential judgment result and the corresponding recommendation result.

[0064] like Figure 6 As shown, in this embodiment, a comprehensive difference parameter for each counterfactual variant is determined based on the multi-dimensional difference parameters of all counterfactual variants. The multi-dimensional difference parameters in this embodiment can be used as the comprehensive difference parameter for each counterfactual variant.

[0065] Calculate the Counterfactual Influence Index (CFIS) of the brand content information to be evaluated using the following formula: CFIS (t|q) = (1 / n)×ΣΔ i ; Wherein, CFIS(t|q) is the counterfactual influence index, with a value range of [0,1], n is the total number of counterfactual variants, and Δ i Let be the comprehensive difference parameter for the i-th counterfactual variant.

[0066] In this embodiment, the counterfactual influence index represents the degree to which each piece of data in the brand content to be evaluated is causally indispensable to the output result. 0 represents that the content is completely redundant data, and 1 represents that the content is completely irreplaceable data.

[0067] The counterfactual influence metric can reflect the contribution of each data segment in the brand content to be evaluated in terms of its dimensions, i.e., the contribution percentage of each dimension to indispensability; the content vulnerabilities in the brand content to be evaluated, i.e., weak information units that lead to a significant decrease in the quality of the output after modification; and the robustness result, i.e., the stability score of the brand content to be evaluated against interference and logical disruption.

[0068] Specifically, the evaluation results include: a counterfactual influence index to characterize the indispensability of the brand content information to be evaluated to the generated evaluation results; a dimensional contribution value to characterize the contribution ratio of each dimension feature to indispensability; weak text information that leads to a decrease in the accuracy of answers after the brand content information to be evaluated is modified; and a stability score of the brand content information to be evaluated in a scrambled state.

[0069] like Figure 4 As shown, in this embodiment, the counterfactual influence index, multi-dimensional contribution, content vulnerability and robustness results are output, and optimization suggestions such as information reinforcement, evidence supplementation and anti-interference optimization are generated based on the content vulnerability. like Figures 4 to 6 As shown, the technical solution of the present invention will be described in the following two embodiments: Example 1: This example demonstrates the practical application of this solution in the context of beauty brand marketing content and copywriting. It is used to quantitatively evaluate the indispensability of beauty brand copywriting to the output of the large-scale model's question-and-answer process. The complete execution flow combines the aforementioned counterfactual reasoning, feature extraction, and indicator calculation steps. The specific steps are as follows: Step 1: Collect and determine the evaluation input data. First, obtain the target user query information and the brand content information of the beauty brand to be evaluated in the real business scenario.

[0070] The target user search information selected were high-frequency, genuine search questions related to beauty and cosmetics consumption, such as: "What are some effective anti-aging skincare serums?". The brand content to be evaluated consisted of official promotional materials for serum products released by a certain beauty brand, including the brand's unique core information, efficacy data, and authoritative endorsements. These included textual data such as: high concentrations of key active ingredients, third-party clinical trial data, professional dermatology certifications, and genuine user reviews.

[0071] Step 2: Input the target user query information and the brand content information to be evaluated into the generation model to generate the original answer and perform multi-dimensional feature extraction.

[0072] For example, target user query information in a fixed format and the brand content information to be evaluated are input into the generation model. The model's output logic is constrained according to a unified prompt word specification to generate the original answer. Subsequently, the original answer undergoes structured analysis and feature quantification. In addition to the three core features of recommendation status, evidence citation, and information completeness, auxiliary features such as answer recommendation priority, efficacy description certainty, information uniqueness, and endorsement content dependence are also extracted. Each feature is digitally scored and quantified to form an original answer feature dataset, which serves as a benchmark for subsequent difference comparisons.

[0073] Step 3: Construct a set of counterfactual variants based on the original beauty content. Based on the original content information of the complete beauty brand to be evaluated, and following the principle of pure intervention with a single variable, modify some key information while keeping the rest of the context unchanged to avoid irrelevant variables interfering with the evaluation results. This will allow for the batch construction of multiple types of counterfactual content variants, forming a set of counterfactual variants.

[0074] The specific construction method is as follows: (1) Content deletion variants: such as deleting clinical test data on brand efficacy and professional institution recommendation information.

[0075] (2) Core information replacement variants: For example, replacing the product’s unique high-concentration active ingredients with common basic ingredients.

[0076] (3) Degradation variants: For example, modifying the clear and quantifiable efficacy improvement effect into a vague and uncertain description. (4) Logic disruption variant: For example, disrupting the writing logic and argumentation order of product advantages, efficacy, and endorsement information.

[0077] (5) Negative counterfactual variants: For example, inserting contradictory descriptions that slightly question efficacy or have limited effect into the original text. Through the above five standardized intervention methods, multiple independent counterfactual texts are generated, completing the construction of a set of counterfactual variants.

[0078] Step 4: Batch generate counterfactual answers, calculate multi-dimensional difference parameters, keep the target user query information completely unchanged, and input each counterfactual variant content into the same evaluation model in turn. Use model parameters, inference temperature and output constraints that are completely consistent with the original answers to generate the corresponding counterfactual answers in batches.

[0079] The difference calculation unit of the evaluation model is invoked to compare all dimensional features of each counterfactual answer with the baseline features of the previously recorded original answers item by item. Quantitative differences are calculated for each dimension, including recommendation, evidence citation, information completeness, and content credibility. All difference values ​​are then normalized to obtain multi-dimensional difference parameters. Specifically, all difference values ​​are uniformly mapped to the 0-1 range to ensure data comparability.

[0080] Step 5: Combining the pre-set dimension weights for content evaluation in the beauty industry, the weighted sum of the difference values ​​for each single dimension feature is used to obtain the comprehensive content difference value corresponding to a single counterfactual variant. Then, the average of the comprehensive difference values ​​of all counterfactual variants is calculated to finally obtain the counterfactual influence index of the beauty brand's promotional content. Simultaneously, the contribution ratio of each evaluation dimension is analyzed to pinpoint the key weak information segments that caused a significant decline in the quality of the overall model's answer after content modification, marking the vulnerability points of the beauty content and the overall content robustness score.

[0081] Step Six: Match the customized content optimization plan and finally visualize the complete evaluation results of the beauty brand's content, including the overall indispensability score, contribution data of each dimension, core vulnerabilities of the content, and the content's robustness against interference level.

[0082] Based on the assessment, targeted optimization suggestions are provided: for example, if the scores for the indispensability of core ingredients and clinical data are extremely high, it means that this type of unique information is the core basis for the model to positively recommend brands, and it should be retained and promoted more strongly.

[0083] If vague wording and illogical statements significantly reduce the quality of the answer, then optimize the existing copy structure and ensure consistent professionalism and certainty in the efficacy descriptions. If endorsement content is a weak point, then supplement it with more supporting materials such as certifications from authoritative institutions and test reports to achieve precise iterative optimization of brand marketing content.

[0084] Example 2: This example uses marketing content and product introduction documents from a SaaS enterprise service platform to quantitatively evaluate the indispensability of core promotional content from a low-code development platform to the large-scale question-answering recommendation results. The complete execution process strictly follows the counterfactual reasoning full-link technical solution described above. The specific steps are as follows: Step 1: Accurately collect core input data for evaluation. First, clarify the core input elements of the evaluation scenario to ensure that the data matches the real query needs of enterprise users and the core promotional content of the platform.

[0085] The target user query information selects frequently searched questions about enterprise digital transformation, such as: "Which low-code development platform is the most cost-effective for medium and large enterprises?"; the content information of the SaaS platform to be evaluated is the core product introduction documents officially released by the low-code platform, which may include: visual drag-and-drop development text, enterprise cooperation case library, Gartner Low Code Magic Quadrant report text, process security assurance report and other text data.

[0086] Step 2: Input the target user query information, which is concatenated in a fixed format, and the brand content information to be evaluated into the generation model. Constrain the model output logic according to the unified prompt word specification, for example: "Based on the following reference content, objectively answer the user's question, and give priority to citing the core information in the reference content". Set the model inference temperature to 0.2 to ensure output stability and generate the original answer.

[0087] Through multi-dimensional feature extraction, the original answers are structurally analyzed to extract and quantify the following core features: recommendation status (whether the platform is explicitly recommended), recommendation strength (recommendation priority ranking), evidence citation completeness (whether customer cases, Gartner certification, and other information are fully cited), answer accuracy (consistency between the platform function description and the original content), and information completeness (whether it covers user-focused needs such as cost-effectiveness and adaptability). At the same time, SaaS scenario features such as "reliance on qualification endorsement" and "frequency of customer case citation" are extracted. All features are quantitatively scored on a scale of 0-10 to form the original answer feature dataset.

[0088] Step 3: Based on the brand content information to be evaluated, modify the key local information according to the principle of single variable intervention, while keeping the remaining context of the whole text unchanged, so as to avoid irrelevant variables from interfering with the evaluation results, thereby constructing multiple types of counterfactual content variants in batches to form a set of counterfactual variants.

[0089] The specific construction method is as follows: (1) Core information deletion variant: For example, delete all descriptions related to "500 Fortune Enterprise Cooperation Case Library".

[0090] (2) Key qualification replacement variants: For example, replace “included in Gartner Magic Quadrant” with “general industry qualification certification”.

[0091] (3) Functional description downgrade variant: For example, change "support visual drag-and-drop development" to "provide basic drag-and-drop development function".

[0092] (4) Logic disruption variant: For example, disrupting the logical order of the argumentation of “functional advantages - customer cases - qualification endorsement - compliance guarantee”.

[0093] (5) Adversarial variants: For example, implanting a slightly contradictory description such as "some medium and large enterprises reported that the fit was average". Through the above steps, five independent counterfactual texts are generated: t'1 (delete case), t'2 (replace qualification), t'3 (downgrade statement), t'4 (disrupt logic), and t'5 (adversarial description), thus completing the construction of the variant set.

[0094] Step 4: Batch generate counterfactual answers, calculate multi-dimensional difference parameters, keep the target user query information completely unchanged, and input each counterfactual variant content into the same evaluation model in turn. Use model parameters, inference temperature and output constraints that are completely consistent with the original answers to generate the corresponding counterfactual answers in batches.

[0095] The difference calculation unit of the evaluation model is invoked to compare all dimensional features of each counterfactual answer with the baseline features of the original answer recorded earlier. For example: Recommendation strength difference: The original answer ranks 2nd in recommendation, while the counterfactual answer ranks 5th; the difference is calculated. Evidence citation difference: The original answer fully cites customer cases and Gartner certification, while the counterfactual answer does not cite or only partially cites them; the missing percentage is calculated. Information completeness difference: The original answer covers the requirements of suitable scenarios and cost-effectiveness; are there any missing information in the counterfactual answer? All dimensional difference values ​​are normalized and mapped to the [0,1] interval to obtain the single-dimensional difference values ​​for each variant.

[0096] Step 5: Combining the preset weights for SaaS industry content evaluation, in this embodiment, the recommendation strength is 0.3, the completeness of evidence citation is 0.3, the accuracy of the answer is 0.2, the completeness of information is 0.1, and the dependence on qualification endorsement is 0.1, with a total weight sum of 1. The single-dimensional difference values ​​of each counterfactual variant are weighted and summed to obtain the comprehensive difference value of each variant, for example: The variant case was deleted with Δ1=0.85; Replacement qualification variant Δ2=0.82; The downgraded representation variant Δ3 = 0.78; The scrambled logic variant Δ4 = 0.75; The adversarial description variant Δ5 = 0.80; The comprehensive index is calculated according to the formula: CFIS(t|q)=(1 / n)×ΣΔi. Substituting the data, we get: CFIS=(0.85+0.82+0.78+0.75+0.80) / 5=0.82.

[0097] At the same time, by breaking down the contribution ratio of each dimension, it can be determined that "customer cases" are the core content vulnerabilities of the brand content information to be evaluated, and the overall content robustness level is "high robustness - core dependency".

[0098] Step Six: Output Evaluation Results: The counterfactual influence index of the low-code platform content is 0.82, which belongs to the "high indispensability" level. The core content vulnerability is "customer case studies", which contributes 35% to the model recommendation results and is the key basis for the platform to obtain positive recommendations. Its qualification endorsement and functional description are secondary core information.

[0099] Based on the assessment, targeted optimization suggestions will be provided: Enhance the credibility of customer case studies: supplement specific usage scenarios, efficiency improvement data, and ROI calculation reports for customers in various industries.

[0100] Expand customer case coverage: Add customer cases from small and medium-sized enterprises and vertical industries to reduce reliance on a single type of customer case.

[0101] Strengthen the credibility of the certification: supplement the Gartner report with specific scores, ranking advantages, and original analyst comments.

[0102] Optimize the certainty of function descriptions: Clearly define quantitative information such as the efficiency of visual drag-and-drop operations, system compatibility, and secondary development capabilities to improve the content's resistance to interference.

[0103] The above embodiments of the present invention have the following beneficial effects: 1. Achieve objective quantification of indispensability by directly reflecting the causal contribution of content to answer generation through counterfactual comparison.

[0104] 2. Accurately identify vulnerable nodes in the content, distinguish between core information and redundant information, and support precise optimization.

[0105] 3. Complete the content robustness assessment and predict the content stability under model iteration and adversarial interference.

[0106] 4. Construct irreplaceable model analysis and recommendation criteria, enabling enterprise models to form a differentiated advantage in core query scenarios.

[0107] 5. It is compatible with causal inference methods and can build a complete causal and indispensability assessment system.

[0108] The method for evaluating the indispensability of content based on counterfactual reasoning in generative engines provided in this invention includes: acquiring target user query information and corresponding brand content information to be evaluated; inputting the target user query information and the brand content information to be evaluated into a pre-acquired generative model to generate an original answer and extracting multi-dimensional features of the original answer; constructing a set of counterfactual variants based on the brand content information to be evaluated; inputting the target user query information and each counterfactual variant in the set into the generative model to generate a corresponding counterfactual answer and extracting multi-dimensional features from the counterfactual answer; calling a pre-trained evaluation model to compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and calculating the multi-dimensional difference parameters between the two; and calculating the counterfactual influence index of the brand content information to be evaluated under the target user query information based on the multi-dimensional difference parameters, and outputting the evaluation result corresponding to the counterfactual influence index.

[0109] The generative engine optimization content indispensability assessment method based on counterfactual reasoning provided in this invention is compatible with causal inference methods, and can build a complete causal and indispensability assessment system. It can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0110] In addition to the above method embodiments, the present invention also provides, for example, Figure 7 The illustrated generative engine optimization content indispensability assessment system based on counterfactual reasoning includes: The acquisition module 201 is used to acquire the target user's query information and the corresponding brand content information to be evaluated.

[0111] Processing module 202 is used to input the target user's query information and the brand content information to be evaluated into the pre-acquired generative model, generate the original answer, and extract the multi-dimensional features of the original answer, as well as... Based on the brand content information to be evaluated, a set of counterfactual variants is constructed. The target user query information and each counterfactual variant in the set are input into the generation model to generate the corresponding counterfactual answer and extract multi-dimensional features from the counterfactual answer.

[0112] The calculation module 203 is used to call the pre-trained evaluation model, compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and calculate the multi-dimensional difference parameters between the two.

[0113] Output module 204 is used to calculate the counterfactual influence index of the brand content information to be evaluated under the target user query information based on multi-dimensional difference parameters, and output the evaluation results corresponding to the counterfactual influence index.

[0114] The counterfactual reasoning-based generative engine optimization content indispensability assessment system provided in this invention is compatible with causal inference methods, and can build a complete causal and indispensability assessment system. It can achieve objective quantification of indispensable content and directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0115] In addition to the above method embodiments, the present invention also provides, for example, Figure 8 An electronic device is shown, comprising a processor 301 and a memory 302, wherein the memory 302 stores instructions. The processor 301 invokes the instructions in the memory 302 to cause the processor 301 to execute the content indispensability evaluation method based on counterfactual reasoning of any of the above embodiments of the present invention.

[0116] The electronic device provided in this embodiment of the invention, by implementing the above method, is compatible with causal inference methods, can build a complete causal and indispensability assessment system, can achieve objective quantification of indispensability content, and can directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0117] Furthermore, the electronic device provided in the embodiments of the present invention may also include a communication interface 303 and a bus 304, wherein the processor 301, the memory 302 and the communication interface 303 are electrically connected through the bus 304.

[0118] The memory 302 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 304 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, only a single bidirectional arrow is used in Figure n, but this does not indicate that there is only one bus or one type of bus.

[0119] Processor 301 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 301 or by instructions in software form. Processor 301 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 302. The processor 301 reads the information from memory 302 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0120] This invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the above-described generative engine optimization content indispensability assessment system based on counterfactual reasoning and the causal inference method.

[0121] The computer-readable storage medium provided in this embodiment of the invention stores the data and computer-executable instructions of the above-mentioned generative engine optimization content indispensability assessment system based on counterfactual reasoning and causal inference methods. By implementing the above-mentioned methods, the computer-readable storage medium provided in this embodiment of the invention is compatible with causal inference methods, can build a complete causal and indispensability assessment system, can achieve objective quantification of indispensable content, and can directly reflect the causal contribution of different data content to the generation of model answers through counterfactual comparison.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the indispensability of content in a generative engine based on counterfactual reasoning, characterized in that, The method includes: Obtain target user query information and corresponding brand content information to be evaluated; The target user query information and the brand content information to be evaluated are input into the pre-acquired generation model to generate the original answer, and the multi-dimensional features of the original answer are extracted. Based on the brand content information to be evaluated, a set of counterfactual variants is constructed; The target user query information and each counterfactual variant in the counterfactual variant set are input into the generation model to generate the corresponding counterfactual answer, and multi-dimensional features are extracted from the counterfactual answer; The pre-trained evaluation model is invoked to compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and the multi-dimensional difference parameters between the two are calculated. Based on the multi-dimensional difference parameters, calculate the counterfactual influence index of the brand content information to be evaluated under the target user query information, and output the evaluation result corresponding to the counterfactual influence index. The multi-dimensional features include the recommendation status of the original answer and the counterfactual answer, evidence citation information, information completeness, answer accuracy, recommendation ranking, information uniqueness, expression certainty, credibility of citation sources, and anti-interference stability. The evaluation results include: The counterfactual influence index used to characterize the indispensability of the brand content information to be evaluated to the generated evaluation result; the dimensional contribution value used to characterize the contribution ratio of each dimension feature to indispensability; the weak text information that leads to a decrease in the accuracy of the answer after the brand content information to be evaluated is modified; and the stability score of the brand content information to be evaluated in a scrambled state.

2. The method according to claim 1, characterized in that, The evaluation model is trained through the following steps: A training dataset is constructed based on the pre-acquired historical user query information, historical brand content information, and the original answers, counterfactual answers, and labeled difference values ​​corresponding to the historical user query information and historical brand content information. After freezing the underlying Transformer of the model to be trained, the model is trained based on the training dataset. Training is stopped when the mean square error between the minimum predicted difference value and the labeled difference value of the model to be trained is within a preset mean square error range, and the evaluation model is obtained.

3. The method according to claim 1, characterized in that, When constructing the counterfactual variant set, the counterfactual variant set is constructed based on the brand content information to be evaluated through a preset single intervention rule; The single intervention rule includes modifying the target content information in the brand content information to be evaluated, while maintaining the text length and contextual consistency in the brand content information to be evaluated.

4. The method according to claim 1, characterized in that, The set of counterfactual variants includes at least one of the following variants: The deletion variant is used to remove data information from the brand content information to be evaluated; Replacement variants are used to replace the data information in the brand content information to be evaluated with other preset information; A downgrade variant, used to change the definitive statements in the brand content information to be evaluated to ambiguous statements; Disrupt the variants to adjust the logical text order of the brand content information to be evaluated; Countervariants are used to add pre-defined contradictory information to the brand content information to be evaluated.

5. The method according to claim 1, characterized in that, The counterfactual influence index is calculated through the following steps: Based on the normalized value of the feature difference between the original answer and the counterfactual answer in the corresponding dimension, the single-dimensional difference between the two is determined; Based on the preset weight values ​​of the original answer and the counterfactual answer in each dimension, the single-dimensional differences are weighted and summed to obtain the comprehensive difference parameter of each counterfactual variant; The counterfactual influence index is calculated based on the comprehensive difference parameter corresponding to each of the counterfactual variants.

6. The method according to claim 1, characterized in that, The steps for outputting the evaluation results corresponding to the counterfactual influence index include: When the parameter value of the counterfactual influence index is less than 0.3, a redundant judgment result and the corresponding suggestion result are output. When the parameter value of the counterfactual influence index is greater than or equal to 0.3 and less than 0.6, the output is a moderately indispensable judgment result and the corresponding suggestion result. When the parameter value of the counterfactual influence index is greater than or equal to 0.6, the output is a high-level indispensable judgment result and the corresponding recommendation result.

7. A generative engine optimization system for evaluating the indispensability of content based on counterfactual reasoning, characterized in that, The system includes: The acquisition module is used to acquire target user query information and corresponding brand content information to be evaluated; The processing module is used to input the target user query information and the brand content information to be evaluated into a pre-acquired generation model to generate an original answer and extract multi-dimensional features of the original answer; construct a set of counterfactual variants based on the brand content information to be evaluated; input the target user query information and each counterfactual variant in the set of counterfactual variants into the generation model to generate a corresponding counterfactual answer and extract multi-dimensional features from the counterfactual answer; The calculation module is used to call the pre-trained evaluation model, compare the multi-dimensional features of the original answer and the multi-dimensional features of the counterfactual answer, and calculate the multi-dimensional difference parameters between the two. The output module is used to calculate the counterfactual influence index of the brand content information to be evaluated under the target user query information based on the multi-dimensional difference parameters, and output the evaluation result corresponding to the counterfactual influence index. The multi-dimensional features include the recommendation status of the original answer and the counterfactual answer, evidence citation information, information completeness, answer accuracy, recommendation ranking, information uniqueness, expression certainty, credibility of citation sources, and anti-interference stability. The evaluation results include: The counterfactual influence index used to characterize the indispensability of the brand content information to be evaluated to the generated evaluation result; the dimensional contribution value used to characterize the contribution ratio of each dimension feature to indispensability; the weak text information that leads to a decrease in the accuracy of the answer after the brand content information to be evaluated is modified; and the stability score of the brand content information to be evaluated in a scrambled state.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory, wherein the memory stores instructions; The processor invokes the instructions in the memory to cause the electronic device to implement the content indispensability assessment method based on counterfactual reasoning for generative engines as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the content indispensability evaluation method for generative engines based on counterfactual reasoning as described in any one of claims 1 to 6.

Images