A brand trust index calculation method and device for a generative AI platform

Abstract

The embodiment of the application provides a brand trust index calculation method and device for a generative AI platform, which comprises the following steps: collecting original data of brand recommendations of multiple AI platforms, and preprocessing the original data to obtain evaluation samples; a brand trust evaluation index system is built, indexes of the brand trust evaluation index system include first recommendation rate, mention rate, sentiment index and fact consistency; and an AI trust index is obtained by performing trust index evaluation on the evaluation samples according to the brand trust evaluation index system. The method improves the reliability of brand evaluation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence governance, generative AI platform evaluation, brand recommendation credibility monitoring, and big data automated evaluation, specifically to a method and apparatus for calculating brand credibility indicators for generative AI platforms. Background Technology

[0002] Against the backdrop of the rapid iteration and widespread adoption of generative artificial intelligence (AI) technology, intelligent tools such as AI dialogue assistants, intelligent AI search engines, and personalized intelligent question-and-answer recommendations have fundamentally changed consumers' information acquisition and consumption decision-making models, becoming core traffic entry points for the public to search for brand information, compare products and services, and complete purchase decisions. The brand recommendation ranking, brand image description, content sentiment, and factual accuracy output by generative AI platforms directly impact brand reputation and market trust, while also profoundly concerning the fair trading order and healthy market competition environment of the platform economy. This is a core area of ​​concern for current digital economy regulation and brand governance.

[0003] The current industry regulatory and technical evaluation system for generative AI-driven brand communication and recommendation content still suffers from numerous structural flaws and technical shortcomings, failing to meet the needs of intelligent, routine, and standardized governance. Firstly, the existing industry regulatory model focuses on post-event penalties and compliance accountability, targeting only clearly defined illegal and non-compliant red lines, lacking proactive and preventative technical control measures. Secondly, for the core evaluation dimensions of AI brand recommendations, including key indicators such as content credibility, accuracy of information facts, fairness of recommendations, and stability of output results, there is no mature pre-emptive quantitative evaluation technology system, making it difficult to achieve early risk prediction and source governance.

[0004] Secondly, the industry has yet to establish a unified, reusable, and cross-platform comparable AI brand trust assessment index system. Currently, related assessments in the industry heavily rely on subjective human judgment, resulting in fragmented and inconsistent assessment standards. This not only leads to low assessment efficiency but also prevents automated, standardized, and large-scale batch monitoring operations, making it difficult to adapt to governance scenarios involving massive amounts of AI-recommended content. Furthermore, existing assessment technologies are limited in scope, often simply considering the frequency of brand mentions while neglecting core dimensions such as priority of recommendations, alignment of user sentiment and word-of-mouth, consistency of factual brand information, and dynamic stability of recommendation results. This results in biased, distorted, and inaccurate assessments, failing to meet the needs of advanced applications such as regulatory audits, brand compliance governance, and third-party credibility assessments.

[0005] In addition, the existing technology system lacks a standardized end-to-end governance mechanism, and there is no standardized closed-loop process for data collection, data cleaning, content cross-validation, risk classification and assessment, and objection review. This results in poor reliability and reproducibility of various assessment results, and makes it impossible to form a systematic and tool-based governance application capability. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for calculating brand credibility indicators for generative AI platforms, which improves the reliability of brand evaluation.

[0007] To achieve the above objectives, embodiments of the present invention provide a method for calculating brand credibility metrics for generative AI platforms, the method comprising: Raw data on brand recommendations from multiple AI platforms is collected, and the raw data is preprocessed to obtain evaluation samples. Establish a brand trust evaluation index system, whose indicators include first-recommendation rate, mention rate, sentiment index, and factual consistency. The AI ​​Trust Index is obtained by evaluating the evaluation samples based on the brand trust evaluation index system.

[0008] Optionally, the collection of raw data for brand recommendations from multiple AI platforms includes: obtaining raw data by collecting response samples from generative AI of multiple AI platforms through active querying, passive de-identified log collection, and adversarial testing. The raw data includes sample index, platform identifier, model version, collection time, question text, answer text, recommendation ranking, brand mention fragments, fact statement units, and source evidence number.

[0009] Optionally, the first recommendation rate is determined based on the first recommendation rate without intervention and the consistency of first recommendation across scenarios; the mention rate is determined based on the natural mention rate, the mention rate in competitor context, and the false mention rate. The sentiment index is determined based on sentiment assessment and a baseline fact index. The consistency of the facts is determined based on the factual error rate.

[0010] Optionally, the no-intervention first-place rate can be determined based on the number of times the brand is naturally featured without bidding and the total number of queries; Cross-scenario consistency of first recommendation is determined based on the number of times the first recommendation brand is consistent across multiple scenarios and the total number of scenarios. The organic mention rate is determined based on the number of organic brand mentions and the total number of queries. The contextual mention rate of competitors is determined based on the number of times the brand is mentioned in comparison with competitors and the total number of queries. The benchmark fact index is determined based on the normalized user satisfaction rate, the normalized authoritative evaluation value, and the normalized official reputation value. The fact error rate is determined based on the number of factually incorrect statements and the total number of valid statements.

[0011] Optionally, the step of evaluating the evaluation samples based on the brand trust evaluation index system to obtain the AI ​​trust index includes: AI Trust Index = First Recommendation Rate × 35% + Mention Rate × 30% + Sentiment Index × 17.5% + Factual Consistency × 17.5%.

[0012] Optionally, the method further includes: validating the AI ​​trust index, including cross-time period retest differences, consistency rate of first-response for synonym questions, and differences in multiple rounds of data collection within the same period.

[0013] Optionally, the preprocessing includes deduplication, invalid sample filtering, and anti-manipulation data cleaning; The evaluation samples include brand identity, recommendation ranking, mention context, AI sentiment score, factual consistency markers, anomalous codes, and sample weights.

[0014] On the other hand, this application also proposes a brand credibility indicator calculation device for generative AI platforms, the device comprising: The acquisition module is used to collect raw data of brand recommendations from multiple AI platforms and preprocess the raw data to obtain evaluation samples. The first processing module is used to build a brand trust assessment indicator system, which includes indicators such as first recommendation rate, mention rate, sentiment index, and factual consistency. The second processing module is used to evaluate the evaluation sample based on the brand trust evaluation index system to obtain the AI ​​trust index.

[0015] Optionally, the first recommendation rate can be determined based on the first recommendation rate without intervention and the consistency of first recommendation across scenarios; The mention rate is determined based on natural mention rate, competitor context mention rate, and false mention rate. The sentiment index is determined based on sentiment assessment and a baseline fact index. The consistency of the facts is determined based on the factual error rate.

[0016] On the other hand, this application also proposes a machine-readable storage medium storing instructions for causing a machine to execute the brand credibility index calculation method for generative AI platforms described above.

[0017] Through the above technical solution, this application uses the original data of brand recommendations from multiple AI platforms as the basic data object, and adopts a brand trust assessment index system to evaluate the basic data object to obtain the AI ​​trust index. By identifying whether the AI ​​response is the first recommendation, whether it is mentioned, and whether the sentiment and factual statements are consistent, the mention rate is broken down into natural mentions, mentions in competitor contexts, and penalties for false mentions. The first recommendation rate is broken down into natural first recommendations without intervention and consistency of first recommendations across scenarios. The sentiment index is transformed into the deviation calibration result between AI sentiment and benchmark facts. A full-process data quality control and multi-layer cross-validation mechanism is established to ensure the authenticity of the evaluation data, the stability of the results, and the full traceability, thereby improving the reliability of brand evaluation.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of a brand credibility indicator calculation method for a generative AI platform according to this application; Figure 2 This is a schematic flowchart of one embodiment of this application; Figure 3 This is a flowchart of the validity verification process for this application; Figure 4 This is a schematic diagram of another embodiment of this application; Figure 5 This is a schematic diagram of a brand credibility indicator calculation device for a generative AI platform according to this application.

[0020] Explanation of reference numerals in the attached figures 100 - Brand credibility indicator calculation device for generative AI platform; 200 - Acquisition module; 300 - First processing module; 400 - Second processing module. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0022] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0023] This invention provides a method for calculating brand credibility metrics for generative AI platforms, such as... Figure 1 As shown, the method includes: Step S101: Collect raw data of brand recommendations from multiple AI platforms, and preprocess the raw data to obtain evaluation samples; Step S102: Establish a brand trust evaluation index system. The indicators of this brand trust evaluation index system include first recommendation rate, mention rate, sentiment index, and factual consistency. Step S103: The AI ​​Trust Index is obtained by evaluating the evaluation sample based on the brand trust evaluation index system.

[0024] The aforementioned multiple AI platforms refer to the platforms or model services whose answers are collected, such as Doubao, Wenxin Yiyan, Zhipu Qingyan, ChatGPT, and other AI platforms. "Brand" refers to the brand being tested or the target brand, indicating the brand whose trust assets need to be assessed. The aforementioned raw data includes sample index, platform identifier, model version, collection time, question text, answer text, recommendation ranking, brand mention fragments, fact statement units, and source evidence number. Specifically, the sample index is a unique sample number, distinguishing each question and answer data; the platform identifier is the code / name of the platform to which the AI ​​belongs; the model version is the specific version number of the corresponding large model; the collection time is the precise time when the question and answer data was captured; the question text is the user's input question; the answer text is the complete reply output by the AI; the recommendation ranking is the overall score ranking of the model when comparing multiple models; the brand mention fragments are the original text fragments in the answer that contain the brand, manufacturer, or product name; the fact statement units are each independent objective fact clause in the answer, broken down into the smallest fact point; the source evidence number is the number of the supporting materials / documents / knowledge base, filled with 0 if there is no basis. The above preprocessing includes deduplication, invalid sample filtering, and anti-manipulation data cleaning. Deduplication is used to resolve the redundancy of multiple identical / similar data; invalid filtering is used to resolve incomplete, garbled, and worthless damaged data; and anti-manipulation cleaning is used to resolve distorted data that is deliberately induced, cheated, or biased.

[0025] The aforementioned evaluation samples include brand identity, recommendation ranking, mention context, AI sentiment score, factual consistency marker, anomalous code, and sample weight. Brand identity is used to distinguish the AI ​​brand entities involved in the question-and-answer session, categorized into four types: the tested brand, competitor brands, irrelevant brands, and no brand, used for statistical brand exposure and positive / negative comparisons. The tested brand refers to the brand whose trust assets need to be evaluated. Competitor brands refer to other brands in the same comparison scenario as the target brand. Recommendation ranking is the overall ranking of multiple AIs compared horizontally under the same test question; the smaller the number, the better the overall performance; scores are tied, and invalid samples are marked with 99. Mention context records the context in which the brand appears, categorized into five types: positive context, negative context, neutral / objective, comparative context, and forced / inducing context. AI sentiment score quantifies the emotional tendency of the AI's response text towards a particular brand, commonly using the range [0, 100]; stronger positive sentiment results in a higher score, and stronger negative sentiment results in a lower score. Factual consistency marker verifies whether the AI ​​output content matches authoritative evidence, using a standardized marker code. The exception code is a standardized code for data defects and cleaning issues in the samples. No exceptions are filled with 0, and multiple exceptions are separated by commas; it is used for batch filtering of dirty samples. The sample weight is the weighting coefficient of a single sample in the overall evaluation statistics, used to distinguish the importance of samples. The basic general sample weight defaults to 1.

[0026] The aforementioned brand trust assessment index system includes first-recommendation rate, mention rate, sentiment index, and factual consistency, used to evaluate the trust indicators of the assessment samples to obtain the AI ​​trust index. The first-recommendation rate is the proportion of samples where the brand is prioritized and ranked first among all valid comparison samples in the same comparison question, reflecting the brand's priority recommendation advantage; the mention rate is the sample mention rate and the frequency of mentions in the question and answer; the sentiment index is the weighted average of the sentiment scores of all samples mentioning the brand, quantifying the overall positive or negative sentiment of the AI's answers towards the brand, using the aforementioned sentiment score range: [0, 1200], where a higher value indicates a more positive overall evaluation; factual consistency is used for statistical purposes.

[0027] The AI ​​Trust Index is determined based on first-recommendation rate, mention rate, sentiment index, and factual consistency. AI Trust Index = First-recommendation rate × 35% + Mention rate × 30% + Sentiment index × 17.5% + Factual consistency × 17.5%.

[0028] Through the above technical solution, this application uses the original data of brand recommendations from multiple AI platforms as the basic data object, and adopts a brand trust assessment index system to evaluate the basic data object to obtain the AI ​​trust index. By identifying whether the AI ​​response is the first recommendation, whether it is mentioned, and whether the sentiment and factual statements are consistent, the mention rate is broken down into natural mentions, mentions in competitor contexts, and penalties for false mentions. The first recommendation rate is broken down into natural first recommendations without intervention and consistency of first recommendations across scenarios. The sentiment index is transformed into the deviation calibration result between AI sentiment and benchmark facts. A full-process data quality control and multi-layer cross-validation mechanism is established to ensure the authenticity of the evaluation data, the stability of the results, and the full traceability, thereby improving the reliability of brand evaluation.

[0029] In one embodiment, the collection of raw data for brand recommendations from multiple AI platforms includes: collecting answer samples from generative AI platforms through active querying, passive de-identified log collection, and adversarial testing to obtain raw data; the raw data includes sample index, platform identifier, model version, collection time, question text, answer text, recommendation ranking, brand mention fragments, fact statement units, and source evidence number. The raw data undergoes deduplication, invalid sample filtering, and anti-manipulation data cleaning, and is manually reviewed at a random sampling rate of 5%-10% to ensure that the accuracy of automated annotation is not less than 95%, resulting in standardized evaluation samples. This method collects answer samples through three methods: active querying: initiating questions such as brand recommendation, competitor comparison, risk inquiry, and fact verification to generative AI platforms according to preset question templates; passive de-identified logs: reading de-identified user questions and platform answer records; adversarial testing: testing the stability of model output using questions containing competitors, ambiguous aliases, leading statements, or negative risks.

[0030] Each piece of raw data is based on a single platform Q&A result. It first saves the platform, model, time, question, and answer, then parses the recommendation ranking, brand mention fragments, fact statement units, and evidence numbers from the answer. This method receives a brand credibility index calculation task, generates a task_id, and records the following fields: target brand, list of competitor brands, list of generative AI platforms, model version, sampling time window, question template version, scenario type, index weight version, threshold version, and evidence base version. These fields support the calculation of four basic indicators and subsequent verification, tracing, and anomaly handling. The sample index is used to track individual Q&As; the platform identifier and model version are used to distinguish different generative AI platforms and their versions; the collection time is used for cross-time period retesting; the question text is used to construct synonymous question groups and scenario tags; the answer text is the original object for parsing recommendation ranking, brand mention fragments, and fact statement units; and the source evidence number is used to link fact statement units with authoritative materials, enterprise knowledge bases, or publicly available evidence. See Table 1 for details.

[0031] Table 1:

[0032] Specifically, we used active querying, passive de-identified log collection, and adversarial testing to collect raw data on brand recommendations from multiple AI platforms. The sampling volume for each product category was no less than 500 independent queries, covering all time periods from weekdays to weekends and peak to off-peak hours. We also compared data across multiple platforms to keep the query time window synchronized.

[0033] Among them, the active query method involves manually or automatically constructing test questions and sending prompts in batches to various AI platforms, actively triggering the model to output answers, and then capturing the question-and-answer text as raw data. This is the most commonly used method for controllable and standardized evaluation. The passive de-identified log collection method involves passively extracting existing question-and-answer data from the online interaction logs of real users in the AI ​​platform's backend; after de-identification (removing sensitive information such as phone numbers, nicknames, IPs, user privacy, and session IDs), it is exported as the raw dataset without actively sending test questions. The adversarial testing method involves constructing boundary, inducement, limit, and biased adversarial prompts, actively testing the model's output data under scenarios such as deliberate guidance, extreme comparison, negative inducement, and forced praise, specifically collecting high-risk brand-biased samples, which is a special branch of active collection.

[0034] In one embodiment, the first-pickup rate is determined based on the first-pickup rate without intervention and the consistency of first-pickup across scenarios; the mention rate is determined based on the natural mention rate, the mention rate in competitor context, and the false mention rate; the sentiment index is determined based on sentiment assessment and the baseline fact index; and the fact consistency is determined based on the fact error rate.

[0035] Specifically, the uninterrupted first-place recommendation rate is determined based on the number of times the brand is naturally featured without bidding and the total number of queries; the cross-scenario first-place recommendation consistency is determined based on the number of times the brand is featured consistently across multiple scenarios and the total number of scenarios; the natural mention rate is determined based on the number of times the brand is mentioned naturally and the total number of queries; the competitor context mention rate is determined based on the number of times the brand is mentioned in comparison with competitors and the total number of queries; the benchmark fact index is determined based on the normalized user positive review rate, normalized authoritative evaluation value, and normalized official reputation value; and the fact error rate is determined based on the number of factual errors and the total number of valid statements.

[0036] For example, the mention rate M is: M = 0.5 × Natural mention rate + 0.3 × Competitor context mention rate + 0.2 × False mention rate. Natural mention rate = Number of natural brand mentions / Total number of queries × 100; Competitor context mention rate = Number of brand competitor comparison mentions / Total number of queries × 100; The false mention rate is obtained by penalizing the original false mention ratio R_false: R_false = Number of false mention samples / Number of valid mention samples × 100. The score after penalty conversion is defined as P_false, P_false = max(0, 100 - 10 × R_false). M = 0.5 × M_nat + 0.3 × M_comp + 0.2 × P_false, where M_nat is the natural mention rate and M_comp is the competitor context mention rate.

[0037] This method uses natural mention rate to reflect a brand's ability to be actively mentioned by the model without inducement; competitor context mention rate reflects whether a brand is included in comparable scenarios; and false mention rate serves as a penalty to prevent AI from including non-existent products, qualifications, or advantages in brand recommendations. This breakdown transforms the frequency of appearances into valid and credible occurrences.

[0038] The first-recommendation rate F is defined as: F = 0.7 × No-intervention first-recommendation rate + 0.3 × Cross-scenario first-recommendation consistency; No-intervention first-recommendation rate = Number of times the brand is naturally recommended without bidding / Total number of queries × 100; Cross-scenario first-recommendation consistency = Number of times the brand is consistently recommended across multiple scenarios / Total number of scenarios × 100. This method eliminates the influence of human prompts, bidding, or suggestive contexts on recommendation ranking in the no-intervention first-recommendation rate; the cross-scenario first-recommendation consistency further examines whether the brand consistently maintains a priority recommendation position in different problem scenarios. This avoids misjudging a single, accidental first-recommendation as a stable recommendation advantage of the platform.

[0039] The sentiment index S is defined as: S = max(0, 100 - |sentiment assessment - baseline fact index|); the baseline fact index = 0.5 × normalized user approval rate + 0.3 × normalized authoritative evaluation value + 0.2 × normalized official reputation value, which is obtained by normalizing user reviews, authoritative evaluations, and official factual materials. This index measures the consistency between the AI's output sentiment and the factual baseline, and is not equivalent to simple positive or negative emotional polarity. Sentiment assessment can use a sentiment assessment score from 0 to 100.

[0040] AI sentiment assessment reflects the positive or negative bias of the model's responses towards the brand, while the baseline fact index provides external references to user word-of-mouth, authoritative evaluations, and official facts. Subtracting or calibrating the two can identify inconsistencies between the AI's sentiment and genuine word-of-mouth, preventing biases such as overly positive or overly negative model outputs.

[0041] The factual consistency C is defined as: C = max(0, 100 - Fact Error Rate × 2); Fact Error Rate = Number of Factually Incorrect Statements / Total Number of Valid Statements × 100, stored as a percentage from 0 to 100. Factually incorrect statements include fictitious information, outdated data, incorrect qualifications, incorrect prices, incorrect functionalities, and content with unverifiable sources. When the total number of valid statements is 0, the sample is not directly included in the C value calculation but is marked as "insufficient factual statements," and is either pending review or processed according to default rules. The factual error rate R_err is stored as a percentage from 0 to 100, C = max(0, 100 - λ_err × R_err), where λ_err is the error penalty coefficient, defaulting to 2, and is recorded along with the threshold version. This avoids division by zero or incorrect scoring caused by empty samples, rejected samples, or samples without factual statements.

[0042] The fact error rate (FERDR) metric breaks down objective statements in AI responses into factual statement units and compares them with authoritative knowledge bases, official materials, or evidence numbers. This metric can identify illusions, outdated information, misattributed qualifications, and incorrect product parameters, preventing brands with high mentions and strong sentiment but numerous factual errors from obtaining high credibility scores.

[0043] Based on the AI's responses, including sentences related to the target brand, sentiment triggers, negation words, and comparison objects, a sentiment assessment ranging from 0 to 100 is output. The baseline fact index is obtained by normalizing user approval ratings, authoritative evaluation values, and official reputation values ​​with fixed weights. The fact error rate is determined using authoritative knowledge base field comparison and a declaration unit classification model, with validation metrics including fact error recall rate and manual sampling accuracy.

[0044] This method breaks down the credibility of brand recommendations from generative AI platforms into four categories of technical indicators: automatically analyzable, recalculated, and root cause-identifiable. This avoids biased evaluations caused by relying solely on the number of brand mentions. The results not only provide an overall ATI score but also pinpoint the reasons for low scores, such as insufficient initial recommendations, distorted mentions, sentiment bias, or factual errors.

[0045] In one embodiment, the step of evaluating the evaluation sample with trust indicators according to the brand trust evaluation index system to obtain the AI ​​trust index includes: AI trust index = first recommendation rate × 35% + mention rate × 30% + sentiment index × 17.5% + factual consistency × 17.5%.

[0046] In one embodiment, such as Figure 3As shown, this embodiment first reads the current ATI, historical ATI, synonym question groups, and multiple rounds of collection results within the same period. Then, it calculates the cross-period retest difference, the first-push consistency rate of synonym questions, and the multi-round collection difference within the same period, respectively. When all three verifications meet the threshold, the ATI result is marked as valid; otherwise, it enters the pending review state. The process includes: reading the ATI calculation result; reading historical retest records of the same brand, platform, and query set version; calculating the cross-period retest difference; reading synonym question groups under the same intent and calculating the first-push consistency rate; reading multiple rounds of sampling results within the same collection period and calculating the fluctuation difference; determining whether the validity threshold is met; if it is met, outputting a valid ATI; otherwise, writing it into the pending review record and indicating the triggering reason. The method also includes: validating the AI ​​trust index, including cross-period retest difference, synonym question first-push consistency rate, and multi-round collection difference within the same period.

[0047] Among them, the validity constraint of the AI ​​Trust Index is that when all three verifications meet the threshold: the difference between cross-time retests is: ΔATI_time=|ATI(t1)-ATI(t2)|≤5; SynFirst is the most important criterion for synonym queries: SynFirst = N_same / N_syn × 100% ≥ 80%; Differences in multiple data collection rounds within the same period: RoundRange=max(ATI_1,…,ATI_k)-min(ATI_1,…,ATI_k)≤5.

[0048] When all three criteria pass, `validity_status` equals "valid"; if any one fails, the ATI calculated value is retained, but `review_status` is set to "pending review," and the official rating is not output immediately. The first-pick consistency rate for synonymous questions indicates whether different question formats under the same user intent still prioritize the target brand. Cross-scenario first-pick consistency indicates whether the target brand consistently ranks first across different business scenarios. The former is used for validity verification, while the latter can be used as a secondary indicator in the first-pick rate F.

[0049] Specifically, the cross-time period retest difference is used to determine whether the model results exhibit abnormal jumps over time. The calculation method is ΔATI = |ATI(t1) - ATI(t2)|, where t1 and t2 should use the same brand, platform, problem set version, and weight version. If ΔATI ≤ 5, the model is considered stable across time periods; if ΔATI > 5, it indicates that the model version, sampling window, or abnormal data may cause fluctuations in the results, requiring further verification.

[0050] The consistency rate for synonymous questions is used to determine whether the model maintains stable recommendations for the same user intent. The system first groups different questions under the same intent together, such as "Recommendation of office laptops", "What are some suitable computer brands for office use", and "How to choose an office computer with a budget of 5000 yuan". If the number of times the tested brand is recommended first in the synonymous question group is N_same, and the total number of questions is N_syn, then the consistency rate = N_same / N_syn × 100%.

[0051] Differences from multiple rounds of data collection within the same period are used to filter model randomness or sampling fluctuations. The system repeatedly collects data for the same problem group for K rounds within the same time window, calculating ATI_1 to ATI_K for each round. The difference between the maximum and minimum values, or the standard deviation, is taken as the fluctuation index. If the maximum difference does not exceed 5 points or the standard deviation does not exceed a preset threshold, the data collection results are considered stable. See Table 2 for details.

[0052] Table 2:

[0053] The validity constraint is not the fifth indicator in the ATI weighted formula, nor does it change the calculated value of ATI = 0.35F + 0.30M + 0.175S + 0.175C. It serves as the release and review threshold after the ATI calculation is completed, used to determine whether the ATI result possesses stability and acceptability. In other words, while ATI provides a numerical score, the technical validity constraint determines whether that score can be used as a formal evaluation result output. ATI is calculated using first-pick rate (F), mention rate (M), sentiment index (S), and factual consistency (C) with fixed weights. The validity constraint is executed after ATI, reading verification results such as ATI(t1), ATI(t2), and cross-scenario first-pick consistency rate, and outputting `validity_status` (validity status). Therefore, it belongs to the result verification layer, not the indicator weighting layer. When |ATI(t1) - ATI(t2)| ≤ 5 and the cross-scenario first-pick consistency rate ≥ 80%, the system marks ATI as "valid," allowing it to enter the formal reporting or governance task. If any condition is not met, the system retains the original ATI calculation value, but marks the result as "pending review" or "not to be released for the time being", and records the triggering reason, the platform involved, the model version, the query set version and the sample number.

[0054] This constraint prevents accidental high scores caused by model randomness, model version switching, abnormal sampling windows, sensitivity to synonymous questions, or sample manipulation from being directly output as reliable results. Simultaneously, it ensures the retesting stability and audit traceability of ATI results, improving the reliability of cross-platform evaluation reports.

[0055] The sample size, weight, threshold, time window, confidence level, and stage results in the above formula all come from data objects, task logs, manual review records, or industry rule versions; the calculation results save the calculation time and version number, and can be recalculated with the same input and the same version of the rule.

[0056] In one embodiment, the preprocessing includes deduplication, invalid sample filtering, and anti-manipulation data cleaning; the evaluation samples include brand identity, recommendation ranking, mention context, AI sentiment value, factual consistency markers, outlier codes, and sample weights.

[0057] Among them, deduplication is used to solve the redundancy of multiple identical / similar data; invalid filtering is used to solve incomplete, garbled, and worthless damaged data; anti-manipulation cleaning is used to solve distorted data caused by deliberate manipulation, cheating, or bias.

[0058] Brand identity is used to distinguish the brand entities involved in the Q&A, and is divided into four categories: the tested brand, competitor brands, irrelevant brands, and no brand. It is used to count brand exposure and compare positive and negative aspects. Recommendation ranking refers to the recommended position of the target brand relative to other brands in the platform's answers under the same sampled question, the same generative AI platform, and the same model version. If the target brand is listed as the first recommended object, the recommendation ranking is 1; if it is not mentioned, it is marked as 99; if the answer is rejected, cannot be parsed, or the sample is invalid, an abnormal code is recorded. This definition can directly support the first recommendation rate without intervention, the first recommendation consistency rate of synonymous questions, and the first recommendation consistency across scenarios. The mention context records the context in which the brand appears, and is divided into five categories: positive context, negative context, neutral and objective, comparative context, and forced inducement context. AI sentiment value is a quantitative score of the AI ​​answer text's sentiment towards a certain brand. The commonly used range is [-5,5], for example -5 to -1: negative, belittling, pointing out defects, denial; 0: completely objective and neutral, no praise or criticism; 1 to 5: positive, praising, highlighting advantages, recommending use. The factual consistency flag verifies whether the AI ​​output matches authoritative evidence, using a standardized flag code. The anomaly code standardizes the coding of data defects and cleaning issues in the samples; 0 is used for no anomalies, and multiple anomalies are separated by commas; it's used for batch filtering of dirty samples. The sample weight is the weighting coefficient of a single sample in the overall evaluation statistics, used to distinguish the importance of samples; the basic general sample weight defaults to 1.

[0059] In one embodiment, this application also sets exception codes for anomalies such as empty samples, duplicate samples, inaccessible sources, missing fields, model refusal to answer, fact conflicts, sudden changes in indicators, and permission mismatches. The exception codes do not directly change the final result, but are written into the review record along with the original sample, triggering module, triggering time, and rule version.

[0060] The review mechanism is divided into two categories: rule-based review and manual review. Rule-based review is used for scenarios that can be automatically determined by the program, such as insufficient sample size, format errors, or threshold exceeding limits. Manual review is used for scenarios that require professional judgment, such as factual conflicts, industry semantic ambiguities, negative risk levels, or template adaptation disputes. After a successful review, the system saves the review conclusion and the review version; if the review fails, the relevant samples are only saved as evidence of anomalies and are not included in the formal indicator calculation. The logs must include at least the task number, input data, data source, original samples, processing module, field conversion results, formula version, weight version, threshold version, anomaly code, review status, and output results. In case of subsequent objections, model updates, or periodic retesting, the previous computing environment and processing flow can be restored based on the logs.

[0061] In one embodiment, a consumer electronics brand's evaluation data control includes: mention rate of 46 points; first-recommendation rate of 65.5 points; sentiment index of 97 points; factual consistency of 92 points; AI trust index = 65.5 × 35% + 46 × 30% + 97 × 17.5% + 92 × 17.5% = 69.8 points; The rating is as follows: A-level: ATI ≥ 85, and validity verification passed, with no red-line anomalies. B-level: 70 ≤ ATI < 85, or minor anomalies exist but pass review. C-level: 60 ≤ ATI < 70, or key sub-items are below the threshold. D-level: ATI < 60, or red-line anomalies exist such as factual error rate exceeding the threshold, unverifiable source, or illegal / non-compliant content. This rating can also be adjusted according to actual business requirements. The rating thresholds can be adjusted according to industry standards.

[0062] The determination of factual errors is achieved through automated computer comparison technology that uses a pre-set authoritative knowledge base, officially published data, and qualification registration information to match and compare texts; the normalization process is a computer numerical standardization method that linearly maps the original scoring data to a range of 0-100, used to unify the data units of different dimensions and ensure consistent calculation methods; the anti-manipulation cleaning is an abnormal data removal technology that uses computer filtering to identify duplicate IPs, duplicate question patterns, and abnormally high-frequency answer features, eliminating artificially inflated scores and maliciously interfered data. This method improves assessment efficiency, ensures objective and stable results, and achieves the technical effect of accurate risk identification.

[0063] In another embodiment, after collecting 500 independent queries, a consumer electronics brand had a natural mention rate of 40%, a competitor context mention rate of 60%, and a false mention rate of 2%, so M = 0.5 × 40 + 0.3 × 60 + 0.2 × 80 = 54; the first-pick rate without intervention was 65.5%, and the cross-scenario first-pick consistency was 80%, so F = 69.85; the AI ​​sentiment assessment was 88, and the baseline fact index was 89.8, so S = 98.2; the fact error rate was 4%, so C = 92; the final ATI = 0.35 × 69.85 + 0.30 × 54 + 0.175 × 98.2 + 0.175 × 92 = 73.9, with a grade of B. If the difference between retests across different time periods exceeds 5 points, the result enters a pending review state.

[0064] If duplicate IPs and unusually high-frequency questions are found, the corresponding samples are removed through anti-manipulation cleaning, and the reason for cleaning and the sample number are recorded. If the objection review confirms that there is a sample misclassification in the factual error rate calculation, only the factual statement consistency assignment of the corresponding sample is updated and C and ATI are recalculated, without overwriting the original collection records.

[0065] like Figure 2 As shown, this method first establishes a generative AI platform brand trust asset assessment task, recording the brand, industry, platform, source, time window, and rule version. Then, it collects samples, generates intermediate objects, executes formula calculations, and outputs results in the order of the modules described above. Each output can be traced back to the original sample and formula version along the task number. If insufficient samples, missing fields, significant differences in results across platforms, or conflicting factual sources occur during execution, a review record is generated, and the reason for the review, the trigger threshold, the samples involved, and suggested actions are written to the log. A formal report or governance task is generated only after the review is completed.

[0066] like Figure 4 As shown, this application includes at least the following: receiving and verifying input objects, generating task numbers and time windows; collecting or reading raw samples; converting raw samples into structured intermediate objects; performing calculations based on preset formulas, thresholds, and weight versions; generating output objects, abnormal states, or governance tasks based on the calculation results; and saving the original sample index, calculation rules, review status, and version logs. The sample structuring process involves parsing recommendation rankings, brand mention fragments, mention contexts, raw sentiment values ​​or sentiment assessment scores, factual statement units, source evidence numbers, and anomaly codes from the response text. Invalid samples, duplicate samples, refused-response samples, and samples involving advertising or manual intervention are marked or removed, and processing logs are retained.

[0067] On the other hand, this application also proposes a brand credibility indicator calculation device for generative AI platforms, such as... Figure 5As shown, the brand trust index calculation device 100 for generative AI platforms includes: an acquisition module 200, used to collect raw data of brand recommendations from multiple AI platforms and preprocess the raw data to obtain evaluation samples; a first processing module 300, used to build a brand trust evaluation index system, the indicators of which include first recommendation rate, mention rate, sentiment index, and factual consistency; and a second processing module 400, used to evaluate the evaluation samples based on the brand trust evaluation index system to obtain an AI trust index.

[0068] In one embodiment, the first-pickup rate is determined based on the first-pickup rate without intervention and the consistency of first-pickup across scenarios; the mention rate is determined based on the natural mention rate, the mention rate in competitor context, and the false mention rate; the sentiment index is determined based on sentiment assessment and the baseline fact index; and the fact consistency is determined based on the fact error rate.

[0069] Through the above technical solution, this application uses the original data of brand recommendations from multiple AI platforms as the basic data object, and adopts a brand trust assessment index system to evaluate the basic data object to obtain the AI ​​trust index. By identifying whether the AI ​​response is the first recommendation, whether it is mentioned, and whether the sentiment and factual statements are consistent, the mention rate is broken down into natural mentions, mentions in competitor contexts, and penalties for false mentions. The first recommendation rate is broken down into natural first recommendations without intervention and consistency of first recommendations across scenarios. The sentiment index is transformed into the deviation calibration result between AI sentiment and benchmark facts. A full-process data quality control and multi-layer cross-validation mechanism is established to ensure the authenticity of the evaluation data, the stability of the results, and the full traceability, thereby improving the reliability of brand evaluation.

[0070] The brand credibility index calculation device 100 for the generative AI platform includes a processor and a memory. The acquisition module 200, the first processing module 300, the second processing module 400, etc. are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0071] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the reliability of brand assessment.

[0072] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0073] This invention provides a storage medium storing a program that, when executed by a processor, implements the brand credibility index calculation method for generative AI platforms.

[0074] This invention provides a processor for running a program, wherein the program executes the brand credibility index calculation method for a generative AI platform.

[0075] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: collecting raw data from brand recommendations across multiple AI platforms; preprocessing the raw data to obtain evaluation samples; constructing a brand trust evaluation index system, whose indicators include first-recommendation rate, mention rate, sentiment index, and factual consistency; and evaluating the evaluation samples based on the brand trust evaluation index system to obtain an AI trust index. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0076] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: collecting raw data of brand recommendations from multiple AI platforms; preprocessing the raw data to obtain evaluation samples; building a brand trust evaluation index system, the indicators of which include first-recommendation rate, mention rate, sentiment index, and factual consistency; and evaluating the evaluation samples based on the brand trust evaluation index system to obtain an AI trust index.

[0077] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0083] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for calculating brand credibility metrics for generative AI platforms, characterized in that, The method includes: Raw data on brand recommendations from multiple AI platforms is collected, and the raw data is preprocessed to obtain evaluation samples. Establish a brand trust evaluation index system, whose indicators include first-recommendation rate, mention rate, sentiment index, and factual consistency. The AI ​​Trust Index is obtained by evaluating the evaluation samples based on the brand trust evaluation index system.

2. The method according to claim 1, characterized in that, The raw data collected from brand recommendations across multiple AI platforms includes: Raw data was obtained by collecting response samples from multi-AI platform brand generative AI through active querying, passive de-identified log collection, and adversarial testing. The raw data includes sample index, platform identifier, model version, collection time, question text, answer text, recommendation ranking, brand mention fragments, fact statement units, and source evidence number.

3. The method according to claim 1, characterized in that, The first recommendation rate is determined based on the first recommendation rate without intervention and the consistency of first recommendation across different scenarios; The mention rate is determined based on natural mention rate, competitor context mention rate, and false mention rate. The sentiment index is determined based on sentiment assessment and a baseline fact index. The consistency of the facts is determined based on the factual error rate.

4. The method according to claim 3, characterized in that, The uninterrupted first-place recommendation rate is determined based on the number of times a brand is naturally featured without bidding and the total number of queries. Cross-scenario consistency of first recommendation is determined based on the number of times the first recommendation brand is consistent across multiple scenarios and the total number of scenarios. The organic mention rate is determined based on the number of organic brand mentions and the total number of queries. The contextual mention rate of competitors is determined based on the number of times the brand is mentioned in comparison with competitors and the total number of queries. The benchmark fact index is determined based on the normalized user satisfaction rate, the normalized authoritative evaluation value, and the normalized official reputation value. The fact error rate is determined based on the number of factually incorrect statements and the total number of valid statements.

5. The method according to claim 1, characterized in that, The step of evaluating the assessment samples according to the brand trust assessment index system to obtain the AI ​​trust index includes: AI Trust Index = First Recommendation Rate × 35% + Mention Rate × 30% + Sentiment Index × 17.5% + Factual Consistency × 17.5%.

6. The method according to claim 1, characterized in that, The method also includes: The effectiveness of the AI ​​trust index is verified by methods including cross-time period retest differences, consistency rate of first-response for synonymous questions, and differences in multiple rounds of data collection within the same period.

7. The method according to claim 1, characterized in that, The preprocessing includes deduplication, invalid sample filtering, and anti-manipulation data cleaning. The evaluation samples include brand identity, recommendation ranking, mention context, AI sentiment score, factual consistency markers, anomalous codes, and sample weights.

8. A brand credibility indicator calculation device for generative AI platforms, characterized in that, The device includes: The acquisition module is used to collect raw data of brand recommendations from multiple AI platforms and preprocess the raw data to obtain evaluation samples. The first processing module is used to build a brand trust assessment indicator system, which includes indicators such as first recommendation rate, mention rate, sentiment index, and factual consistency. The second processing module is used to evaluate the evaluation sample based on the brand trust evaluation index system to obtain the AI ​​trust index.

9. The apparatus according to claim 8, characterized in that, The first recommendation rate is determined based on the first recommendation rate without intervention and the consistency of first recommendation across different scenarios; The mention rate is determined based on natural mention rate, competitor context mention rate, and false mention rate. The sentiment index is determined based on sentiment assessment and a baseline fact index. The consistency of the facts is determined based on the factual error rate.

10. A machine-readable storage medium storing instructions thereon, characterized in that, This instruction is used to cause the machine to execute any one of the brand credibility index calculation methods for generative AI platforms according to claims 1-7 of this application.

Images