Explainable Trustworthy AI Advisory System

Abstract

A Large Language Model (LLM) and scoring system to generate responses or queries in the context of consulting or advisory services (AI advisory system). The system includes a user trust wallet 1,2 for

Claims

1: A computer and smartphone implemented Explainable Trustworthy Al Advisory System for generating trustworthy and credible responses to self-initiated or system-initiated user queries in a consulting or advisory context for conversational Al responses and project-based work with Al, the system comprising:• a user interface configured to:o receive self-initiated or system-initiated user queries;o collect user feedback; ando facilitate a contextual clarification process to gather information about the user's specific needs and context;• a knowledge graph storing knowledge tokens associated with experts and Published Resources, wherein the knowledge tokens represent human insights and perspectives on various topics and are securely stored on a decentralized blockchain network;• a retrieval-augmented generation (RAG) Engine configured to:o receive a user query from the user interface;o retrieve relevant knowledge tokens from the knowledge graph based on the contextual clarifications and query embeddings;o generate a response to the user query by synthesizing information from the retrieved knowledge tokens and the refined user query;• a large language model (LLM) comprising at least three components configured to:o a first component for refining the user query by gathering contextual clarifications from the user through a structured dialogue;o a second component for answering the user query by generating text based on the refined query and the retrieved knowledge tokens;o a third component for abstracting a system query to identify relevant topics and concepts for knowledge token retrieval from matched users;• a Self-Supervised Tuning Engine configured to perpetually update the knowledge graph based on new information and user feedback through a continuous coaching model utilizing Reinforcement Learning from Human Feedback (RLHF) and expertise crowdsourcing wherein the system runs iterations of currency and relevancy of published knowledge by inviting users (peers) matched to share insights on the area of the knowledgegraph the Large Language Model is being coached on; and• a scoring system configured to generate a Scored RAG Response Token, wherein the scored response token includes the response and a plurality of scores indicating at least trustworthiness, credibility, explainability, and bias attributes of the response, and wherein the scoring system incorporates a credibility-aware attention mechanism to mitigate the risk of misinformation.

2. The system of claim 1, wherein the contextual clarifications include information about the user's role, industry, specific challenges, geographic location, previous experiences, need for understanding the subject in question, application of information being sought, and importance of response.

3. The system of claim 1, wherein the scoring system further comprises a multi-dimensional evaluation comprising at least the following scores:• Groundedness Score: A numerical representation indicating the extent to which the response is supported by credible references from both published data and peer-reviewed knowledge tokens.• Faithfulness Score: A quantitative measure of the semantic similarity between the response and the user query, assessed using a pre-trained language model.• Context Relevance Assessment Score: A metric indicating the degree to which the response addresses the user's specific needs and context, based on the information gathered during the contextual clarification process.• Actionability Score: A numerical value reflecting the extent to which the response provides clear, specific, and feasible recommendations or solutions that the user can implement.• Provenance Score: A score indicating the origin and credibility of the information sources used in the response, including references to specific documents, publications, or expert opinions.• Expert Consensus Score (if applicable): A quantitative measure of the level of agreement among multiple experts who have contributed knowledge tokens to the response.• Explainability Score (XAI Score): A composite score incorporating various measures of transparency and interpretability, such as:o Reasoning Trace: A detailed explanation of the steps and factors considered by the Al model in generating the response.o Counterfactual Analysis: An analysis of alternative scenarios or "what-if" situations to assess the impact of different factors on the outcome.o Sensitivity Analysis: An evaluation of how changes in input parameters or assumptions affect the final output.• Uncertainty Estimation (if applicable): A numerical representation of the level of confidence or uncertainty associated with the response, indicating the degree of reliability or potential error.• User Satisfaction Rating: A rating provided by the user to indicate their level of satisfaction with the response, helping to facilitate ongoing model refinement.• Domain-Specific Relevance Score: A measure of how well the response aligns with the specific industry or domain of the user's query.• User Ranking Score: A score that reflects the global trust and credibility ranking of seekers and peers involved in the creation of a Scored RAG Response Token, based on their contributions, engagement, expertise, and verification status.

4. The system of claim 1, wherein the Self-Supervised Tuning Engine generates an Abstracted Query when unable to find relevant peers with expected insights to, and sends the Abstracted Query to select users for review, refinement and further actioning into a system initiated query.

5. The system of claim 1, wherein the User Trust Wallet is configured to:• tokenize and store user-generated insights with ownership rights;• collect human feedback in the form of at least preference judgments, direct feedback, and human-in-the-loop feedback; and• facilitate the exchange of knowledge tokens between users.

6. The system of claim 1, wherein the Scored RAG Response Token system further comprises:• a first ownership rule for self-initiated user queries, wherein the seeker retains a majority ownership of the response token; and• a second ownership rule for system-initiated queries, wherein the ownership of the response token is distributed among the seeker, contributors, and the system according to a predetermined ratio.

7. The system of claim 1, wherein the large language model (LLM) utilizes a proprietary knowledge-exploration context-building algorithm to gather contextual clarifications from the user through a structured dialogue, the algorithm being configured to differentiate between talents and leaders and elicit their unique perspectives and insights.

8. The system of claim 1, wherein the contextual clarification process is conducted using a structured dialogue that is specifically designed to:• elicit nuanced and contextually relevant information from the user;• differentiate between talents and leaders and tailor the dialogue accordingly;• reduce the risk of hallucinations and biases in the Al-generated response; and• improve the overall accuracy and relevance of the response.

Images