Systems and methods for verifying large language model output using logic rules

Abstract

Various embodiments are directed to verifying outputs of large language models ("LLMs") using logical reasoning. Various embodiments can obtain raw data from a datastore which can be presumed as true. Various embodiments can then factorize the raw data into factorized data points using a large language model ("LLM"). The large language model can then be used to analyze, categorize, and summarize the factorized data points to generate an insight summary. Various embodiments can then validate the insight summary by identifying a chain of logic between the raw data and the insight summary, such that each statement can be proven using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.).

Description

Title: SYSTEMS AND METHODS FOR VERIFYING LARGE LANGUAGEMODEL OUTPUT USING LOGIC RULESInventors: Alok Singh and Finian Douglas O’NeillCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to, and the benefit of, co-pending U.S. provisional application entitled “SYSTEMS AND METHODS FOR VERIFYING LARGE LANGUAGE MODEL OUTPUT USING LOGIC RULES” having serial no. 63 / 668,835, filed July 9, 2024, which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Large language models (“LLMs”) can perform a wide range of tasks using natural language processing to predict an expected response from a submitted prompt. However, various challenges exist for traditional LLMs, such as generating biased responses, incorrect responses, random responses, non-factual responses, and / or illogical responses (often collectively referred to as hallucinations). Because developers are focused primarily on increasing speed of the response while decreasing computing power to execute LLMs, the various challenges remain unsolved as newer LLMs are developed. Further, mathematical methods for analyzing data (e.g., to obtain trends or correlations, etc.) are typically limited to obtaining a particular type of conclusion that is not immediately interpretable as a usable insight.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Many aspects of the present disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, with emphasis instead being placed upon clearly illustrating the principles of the disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views.

[0004] FIG. 1 is a drawing of a network environment according to various embodiments of the present disclosure.

[0005] FIG. 2 is a sequence diagram illustrating one example of functionality implemented as portions of a client application on a client device; and an orchestration application, a large language model, and a datastore executed in a computing environment in the network environment of FIG. 1 according to various embodiments of the present disclosure.

[0006] FIG. 3 is a pictorial diagram of an example data structure used by the computing environment in the network environment of FIG. 1 according to various embodiments of the present disclosure.DETAILED DESCRIPTION

[0007] Disclosed are various approaches for verifying large language models (“LLMs”) output using logic rules, and for producing insights derived from raw data that verifiably comply with logic rules. Analyzing data (e.g., medical data, technical data, data about the internal function of a system or device, etc.) to generate insights (e.g., insights about the state, predicted state, actions to be taken, etc.) in relation tothe context to which the data pertains, can be done using various mathematical approaches. Although the conclusions of such approaches are typically linkable to underlying data, such conclusions are rarely directly interpretable and must be defined specifically for each particular data set and conclusion. For instance, in traditional systems, a developer would have to code responses for each possible input to have a determinable output.

[0008] By contrast, the present inventors have recognized that LLMs can provide directly interpretable output and LLMs can understand contextual elements of an analysis in order to propose actionable insights. However, traditional LLMs experience various challenges, such as generating biased responses, incorrect responses, hallucinations, and / or illogical responses. Often, generating biased responses, incorrect responses, hallucinations, and / or illogical responses can be a result of bad data or stale data being provided to the LLM. For instance, a user could prompt an LLM to “describe the green skies from planet earth.” In such a situation, the user has provided bad data to the LLM, so the response provided by the LLM may likely also include the bad data. For instance, the LLM can respond that “The green skies from the planet earth are a result of the light passing through the atmosphere and refracting a wavelength that corresponds to the color green.” However, in some situations, an LLM can be trained (or pre-trained) with information received from various sources that may alter the response. For example, the LLM can be trained to recognize that the sky is in fact blue, not green, and respond accordingly, such as “The sky as viewed from the planet earth is blue, not green, as a result of the light passing through the atmosphere and refracting a wavelength thatcorresponds to the color blue.” In some situations, LLMs can generate biased responses, incorrect responses, hallucinations, and / or illogical responses as a result of the LLM having been trained on information to favor a specified result. Such biased responses, incorrect responses, hallucinations and / or illogical responses can be much more difficult to detect and often requires human intervention to determine whether the response was non-biased, correct, factual, and / or logical.

[0009] To solve these problems, various embodiments can verify of validate the output of LLMs using logic rules. Logic rules are principles or guidelines that dictate how logical statements are formed and evaluated. Although logical operators (e.g., “AND,” “OR,” “NOT,” “EQUALS,” “WITHIN,” etc.) have been common in traditional computing systems, logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) has been challenging or impractical to encode with traditional computing systems. However, because LLMs can understand the context of the content being provided, LLMs can implement logical reasoning in a way that has not previously been possible with traditional computing systems. For instance, an LLM can be trained or prompted to implement deductive reasoning by starting with premises that are assumed to be true (e.g., data from various sources) and apply logic rules (e.g., modus ponens, modus tollens, etc.) to derive a conclusion from the premise (e.g., identify trends in the data sources, etc.). Additionally, such a process can enforce that an LLM explain the logic used to yield the conclusions from the premises, which can then be verified. In another example, LLMs can apply temporal logic to identify conclusions from premises that have occurred over a period of time.Such insights can better inform the end user (e.g., a client, a sales representative, a professional, etc.).

[0010] Various embodiments can be used for various purposes and for various contexts. In at least one exemplary embodiment, methods of the present disclosure describe a process of providing insights to sales representatives to better communicate with their customers. For instance, an embodiment can obtain information from one or more sources that can be presumed true. The embodiment can then prompt an LLM to factorize the data into factorized data. The embodiment can prove, using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc. , that the generated factorized data can be concluded based on the provided information that is presumed true (e.g., the premise). The embodiment can then identify insights based on the information from the one or more sources and from the factorized data. The embodiment can prove, using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), that the generated data insights can be concluded based on the premises (e.g., the information from the one or more sources and the factorized data, etc.) that are presumed true based on the previous analysis. The embodiment can also identify categorizations for the insights based on the information from the one or more sources, the factorized data, and the data insights. The embodiment can prove, using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), that the generated categorized insights can be concluded based on the premises (e.g., the information from the one or more sources, the factorized data, the data insights, etc.) that are presumed true based on the previous analysis. The embodiment can also generate an insight summary basedon the information from the one or more sources, the factorized data, the data insights, and the categorized insights. The embodiment can prove, using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), that the generated insight summary can be concluded based on the premises (e.g., the information from the one or more sources, the factorized data, the data insights, categorized insights, etc.) that are presumed true based on the previous analysis. Embodiments can perform this validation and assign each statement in the summary a score. The score represents whether the embodiment can logically prove throughout the data the conclusions being presented. If the score does not meet an acceptable threshold, the embodiment can repeat the process until such threshold is met. The insight summary can then be provided to the sales representative or client at a client device to enhance their ability to communicate important information with their customers.

[0011] In the following discussion, a general description of the system and its components is provided, followed by a discussion of the operation of the same. Although the following discussion provides illustrative examples of the operation of various components of the present disclosure, the use of the following illustrative examples does not exclude other implementations that are consistent with the principals disclosed by the following illustrative examples.

[0012] With reference to FIG. 1 , shown is a network environment 100 according to various embodiments. The network environment 100 can include a computing environment 103, a client device 106, and one or more external data store(s) 109, which can be in data communication with each other via a network 112.

[0013] The network 112 can include wide area networks (WANs), local area networks (LANs), personal area networks (PANs), or a combination thereof. These networks can include wired or wireless components or a combination thereof. Wired networks can include Ethernet networks, cable networks, fiber optic networks, and telephone networks such as dial-up, digital subscriber line (DSL), and integrated services digital network (ISDN) networks. Wireless networks can include cellular networks, satellite networks, Institute of Electrical and Electronic Engineers (IEEE) 802.11 wireless networks ( / .e., WI-FI®), BLUETOOTH® networks, microwave transmission networks, as well as other networks relying on radio broadcasts. The network 112 can also include a combination of two or more networks 112. Examples of networks 112 can include the Internet, intranets, extranets, virtual private networks (VPNs), and similar networks. In other words, any one or more of the computing environment 103 and its components, the user device 106 and the datastores 109 can be in wired or wireless communication with each other.

[0014] The computing environment 103 can include one or more computing devices that include a processor, a memory, and / or a network interface. For example, the computing devices can be configured to perform computations on behalf of other computing devices or applications. As another example, such computing devices can host and / or provide content to other computing devices in response to requests for content.

[0015] Moreover, the computing environment 103 can employ a plurality of computing devices that can be arranged in one or more server banks or computer banks or other arrangements. Such computing devices can be located in a singleinstallation or can be distributed among many different geographical locations. For example, the computing environment 103 can include a plurality of computing devices that together can include a hosted computing resource, a grid computing resource, or any other distributed computing arrangement. In some cases, the computing environment 103 can correspond to an elastic computing resource where the allotted capacity of processing, network, storage, or other computing-related resources can vary over time. For example, the computing environment can be implemented as a cloud computing system.

[0016] Various applications or other functionality can be executed in the computing environment 103. The components executed on the computing environment 103 include an orchestration application 115, a large language model 118 (an “LLM 118” for short or “LLMs 118” for one or more LLM), and other applications, services, processes, systems, engines, or functionality not discussed in detail herein. Although its depicted here as a single application in FIG. 1 , the LLM 118 can be set up as more than one LLM application. Instead of a single LLM 118, various embodiments can include a factorizer LLM 121 , a trend analyzer LLM 124, a categorizer LLM 127, a summarizer LLM 130, and a validator LLM 133. It should be understood that in embodiments of a single LLM 118, the LLM 118 can perform the functionality of each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, summarizer LLM 130, and / or the validator LLM 133.

[0017] Also, various data is stored in a data store 136 that is accessible to the computing environment 103. The data store 136 can be representative of a plurality of data stores 136, which can include relational databases or non-relationaldatabases such as object-oriented databases, hierarchical databases, hash tables or similar key-value data stores, as well as other data storage applications or data structures. Moreover, combinations of these databases, data storage applications, and / or data structures may be used together to provide a single, logical, data store. The data stored in the data store 136 is associated with the operation of the various applications or functional entities described below. This data can include raw data 139A (referred to generically as Taw data 139” and individually as “raw datum 139”), factorized data points 142, data insights, 145, categorized insights 148, insight summaries 151 , prompts 154, and potentially other data.

[0018] Turning to the various applications or other functionality that can be executed on the computing environment 103, the orchestration application 115 can be executed to perform various actions. For example, the orchestration application 115 can be executed to receive or obtain raw data 139 from either the data store 136 (raw data 139A) or the external data store 109 (raw data 139B).

[0019] In various embodiments, the orchestration application 115 can send various prompts and datasets to an LLM 118. For instance, the orchestration application 115 can send a first prompt 154A (FIG. 2) to the LLM 118 (in some embodiments, a factorizer LLM 121 ) to factorize the raw data 139 into factorized data points 142. The orchestration application 115 can send a second prompt 154B (FIG. 2) to the LLM 118 (in some embodiments, a trend analyzer LLM 124) to analyze the raw data 139 and the factorized data points 142 to identify data insights 145. The orchestration application 115 can send a third prompt 154C (FIG. 2) to the LLM 118 (in some embodiments, the categorizer LLM 127) to categorize the data insights 145based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148. The orchestration application 115 can send a fourth prompt 154D (FIG. 2) to the LLM 118 (in some embodiments, the summarizer LLM 130) to summarize at least some of the data insights 145, the categorized insights 148, the factorized data points 142, and / or the raw data 139 into an insight summary 151. The orchestration application 115 can send a fifth prompt 154E (FIG. 2) to the LLM 118 (in some embodiments, the validator LLM 133) to validate the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points 142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary 151.

[0020] In at least some embodiments, the orchestration application 115 can remove any of the factorized data points 142, the data insights 145, and / or the categorized insights 148 that cannot be proven true using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) based at least on the raw data 139. For example, the LLM 118 can apply logical reasoning to at least the raw data to determine whether any of the factorized data points 142, the data insights 145, and the categorized insights 148 can be proven true, and the orchestration application 115 remove any of the factorized data points 142, the data insights 145, and categorized insights 148 that cannot be proven to be true. In various embodiments, the orchestration application 115 can determine a score 152 that represents the validity of the insight summary 151 as it is being proven by at least the raw data 139, the factorized data points, 142, the data insights, 145, and / or the categorized insights148. The orchestration application 115 can compare the score 152 to a predefined threshold to determine if the generated insight summary is acceptable to deliver to a client application 172. For example, the generated insight summary 151 may be considered acceptable when the score 152 is above the predefined threshold. The generated insight summary 151 may be considered to not be acceptable when the score 152 is at or below the predefined threshold. If the score 152 fails to exceed the predefined threshold, the process can return to the first prompt of the LLM 118. If the score 152 exceeds the predefined threshold, the orchestration application 115 can send the insight summary 151 to the client application 172 on the client device 106.

[0021] The LLM 118 (including each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) represents a machine-learning model that can be executed to generate natural language text based upon inputs (e.g., prompts 154) received from the orchestration application 115. This can be done, for example, by receiving input text (e.g., prompts 154) and repeatedly predicting the next word or token for a response. Thus, the LLM 118 can be an autoregressive language model. In order to generate responses, the LLM 118 may learn statistical relationships between words, phrases, or other tokens from a corpus of training text in a self-supervised or semi-supervised training process. Examples of LLMs 118 include OPENAI’s® generative pre-trained transformer (GPT) models, GOOGLE’S® PALM and GEMINI models, META’s® LLAMA models, etc. As used herein, a LLM 118 refers to a deep learning model that has been trained, typically in a self-supervised manner, to provide a natural language output based on a prompt received as input. Any LLM architecture known in the artmay be used in the context of the present disclosure. A trained LLM may be a foundation LLM. A foundation LLM is an LLM that has been trained on a large corpus of natural language data, typically in a self-supervised manner.

[0022] The LLM 118 (including each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) can be a model that has been trained to perform logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) to test whether a generated statement logically follows based on a set of premises. For instance, the LLM 118 (in some embodiments, each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) can be trained or prompted to determine whether a generated content (e.g., content generated by one or more of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) logically follows based on a set of premises using logical reasoning, such as deductive reasoning and / or syllogism. Deductive reasoning is a process that can be used to make a conclusion based on premises. In its simplest form, deductive reasoning can be described as using two true statements (premises), moving from general statement to a specific statement, to form a conclusion. Syllogism is a form of deductive reasoning where two statements (premises) that have a common factor are used to make conclusions about the non-common factors. For example, “all dolphins are mammals” and “all mammals have kidneys” are two true statements (premises). From those premises, it can be concluded that “all dolphins have kidneys” using syllogism because the two premises include a common factor(“mammals”), so it can be concluded that the non-common factors must be related (e.g., 1 : if all dolphins are mammals and 2: all mammals have kidneys, then 3: all dolphins must have kidneys). Syllogism can include categorical syllogism (involving statements that assert or deny something about every member of a category), hypothetical syllogism (involving conditional “if... then...” statements to prove a conclusion), disjunctive syllogism (involving a conditional “either... or...” statements to prove a conclusion), Modus Ponens (involving affirming an antecedent premise with a conditional statement), Modus Tollens (involving denying a statement based on the then clause of an “if... then...” conditional statement), and various combinations thereof.

[0023] With regards to an example for the present disclosure, it can be assumed that the raw data 139 represents true statements and the raw data 139 can therefore be used as a premise to establish any of the factorized data points 142, data insights 145, categorized insights 148, and / or insight summary 151. Assuming we have two premises, the first being “Doctor X has stopped four patients from taking pharmaceutical Y,” and “Because Pharmaceutical Z has just been released, over one thousand patients have switched from taking pharmaceutical Y to pharmaceutical Z,” the LLM 118 could reasonably conclude, using deductive reasoning or syllogism, that “Doctor X may have switched at least one of four patients from pharmaceutical Y to pharmaceutical Z.” In such a situation, the LLM 118 can validate the conclusion by verifying that there is factual support from the premises to make such conclusion. This can be used in a variety of contexts, such as predicting demand forpharmaceutical Z to plan supply, recommending a pharmaceutical to be prescribed to a patient, etc.

[0024] Additionally, the LLM 118 (in some embodiments, each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) can be trained or prompted to evaluate whether the generated content logically follows based on temporal logic. Temporal logic can identify trends in data over a period of time instead of from a static data set from a single point in time. Temporal logic can also identify trends that focus on identifying sequences of events, time-based conditions, and causality from data. Such trends in the data can be useful for identifying uncommon situations. For example, the data may indicate that “In June, Doctor X prescribed pharmaceutical Y to thirty patients,” and “In July, Doctor X prescribed pharmaceutical Y to fifty patients.” The data may also indicate that “pharmaceutical Y can cause nausea, vomiting, and insomnia.” Additional data may indicate that “In July, Doctor X prescribed fifteen patients with a nausea preventing medication” and “in August, Doctor X prescribed twenty-five patients with a nausea preventing medication.” An LLM 118 could reasonably conclude, using temporal reasoning, that “after a month of taking pharmaceutical Y, Doctor X prescribes nearly half of the patients taking pharmaceutical Y with a nausea preventing medication.” This can be used in a variety of contexts, such as predicting demand for the nausea preventing medication, recommending treatment of a patient with the nausea preventing medication, etc. Such an insight could be used by a sales representative (at a client device 106) to discuss the possibility that Doctor X should prescribe the nausea preventing medication to each of the patients being prescribedpharmaceutical Y to make their practice see additional patients more efficiently. An LLM 118 (in some embodiments, each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) can be trained on a diverse set of datasets that are specifically curated to instill principles of logical reasoning. The datasets can include a logical inference corpus that represents a collection of premise-conclusion pairs, with labels indicating valid and invalid inferences. Such a logical inference corpus can include a mix of examples to describe deductive reasoning, syllogism, inductive reasoning, and temporal reasoning. The datasets can also include a temporal logic dataset, which provides examples of temporal logic problems that focus on identifying sequences of events, time-based conditions, and causality. Additionally, the datasets to train the LLM 118 can include formal logic puzzles at varying levels of difficulty to demonstrate syllogism, propositional logic problems, predicate logic exercises. Additionally, the datasets to train the LLM 118 can include a scientific reasoning corpus, which represents examples from scientific literature that demonstrate hypothesis formation, experimental design, data interpretation, and conclusion drawing. Additionally, the datasets to train the LLM 118 can include a fallacy recognition dataset that includes examples of common logical fallacies and various reasoning patterns.

[0025] In various embodiments, the LLM 118 (in some embodiments, each of the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133, etc.) can include specialized loss functions that can affect subsequent outputs of LLM 118. The specialized loss functions can be used to penalize the LLM 118 for generating logically inconsistentoutputs. The greater the frequency of the errors, the greater the penalty is applied to the LLM 118. The specialized loss functions can penalize the LLM 118 based on various types of errors that the LLM 118 can commit, such as logical consistency errors, temporal reasoning errors, and domain relevance errors among various other errors.

[0026] Logical consistency errors can represent the frequency that the LLM 118 generates a first output and later generates a second output that is logically inconsistent with the first output. For example, a first output could indicate that “sales of pharmaceutical X have increased overall in the first quarter” and the second output could indicate that “sales of pharmaceutical X have decreased overall in the first quarter.” Although one example statement could be true, both cannot simultaneously be true. Additionally, the logical consistency errors can represent inconsistencies between premises and conclusions. For example, a premise could indicate that “all humans are mammals” and a “all humans have 23 chromosomes,” but a subsequently generated conclusion could incorrectly state that “all mammals have 23 chromosomes.”

[0027] Temporal reasoning errors can represent the frequency that the LLM 118 makes an incorrect inference about the sequence, timing, or duration of events. An example of a temporal reasoning error is coming to a conclusion based on a series of events without additional evidence supporting the conclusion. For example, if a patient reported eating ice cream, then the patient reported having a headache, then a possible conclusion is that the ice cream caused the headache. However, without supporting evidence, such a conclusion lacks the necessary support to maintain thatconclusion. For example, the patient experiencing the headache could have been experiencing early signs of heat stroke due to weather events, and the patient was eating the ice cream to lower their body temperature. Another example of a temporal reasoning error includes conflating different time scales to support a conclusion. For example, a conclusion that a first weight loss drug is just as effective as a second weight loss pharmaceutical can be shown as logically inconsistent by identifying that patients taking the first weight loss drug meet a first weight loss goal (e.g., five pounds, twenty pounds, etc.) within three months of taking the first weight loss pharmaceutical, however patients taking the second weight loss pharmaceutical meet the first weight loss goal within two years of taking the second weight loss pharmaceutical.

[0028] Domain relevance errors can represent the frequency that the LLM 118 generates content that is not relevant to the specific domain of interest. For example, the LLM 118 can generate insights about physicians all over the world. However, when insights about physicians in a specified region are requested, providing information about other regions may not be relevant to the end-user due to the end user’s location. Additionally, various States in the United States and various countries maintain different standards of care when treating patients. For example, a first State can ban the sale of a specified pharmaceutical, but a second State can provide the same pharmaceutical by prescription from a doctor. Providing information about pharmaceutical sales in the first State when the insights should be focused on the second would be irrelevant to insights about the second State. When a logical consistency error, temporal reasoning error, domain relevance error, or other variouserrors are identified, the frequency of each error (or some numerical value calculated from the frequency) can be provided to LLM 118 to provide feedback to the LLM 118 for how it performed. The LLM 118 can adjust its underlying model to avoid generating a large frequency of errors, therefore making each subsequent execution of the LLM 118 better tailored to meet the needs of the client.

[0029] Instead of a single LLM 118, various embodiments can split the work of the single LLM 118 into one or more LLMs, such as a factorizer LLM 121 , a trend analyzer LLM 124, a categorizer LLM 127, a summarizer LLM 130, and / or a validator LLM 133. Such an embodiment of splitting up the individual steps performed by a single LLM 118 into discrete steps performed by one of many LLMs 118 (e.g., a factorizer LLM 121 , a trend analyzer LLM 124, a categorizer LLM 127, a summarizer LLM 130, and / or a validator LLM 133) could narrow the focus for each of the one of many LLMs 118 specialized to their individual tasks 166. Each of the one of many LLMs 118 could receive feedback based on their respective outputs (e.g., factorized data points 142, data insights 145, categorized insights 148, insight summary 151 , and score 152) and could individually be penalized using their respective loss functions Additionally, by splitting a single LLM 118 into one or more LLMs, the one or more LLMs can be hosted on separate computing devices and have their own dedicated hardware, software, and / or firmware. This also allows embodiments to scale up certain LLMs that are more resource intensive, while leaving the less resource intensive LLMs untouched. This also allows each LLM to be narrowly tailored to the specific task 166 being performed. For instance, a factorizer LLM 121 may be better performed using OPENAI’s® generative pre-trained transformer (GPT)models, whereas the trend analyzer LLM 124 may be better performed usingMETA’s® LLAMA models. In some embodiments, non-standard LLMs 118 can be used to perform the further described tasks 166. In such embodiments, the LLM(s) 118 can be trained or fine-tuned with relevant data of expected inputs (e.g., raw data 139, factorized data points 142, data insights 145, categorized insights 148, and insight summaries 151 ), a training prompt 154 (e.g., the corresponding prompts 154 for the factorizer LLM 121 , the trend analyzer LLM 124, the categorizer LLM 127, the summarizer LLM 130, and the validator LLM 133 as further described), and / or expected outputs 169 from the LLMs 118. Each of the LLMs 118 can be trained with a high quality pre-selected dataset and can be further fine-tuned with reinforced learning from human feedback and / or feedback from the scores 152 generated in the insight summary 151 by the summarizer LLM 130.

[0030] The LLM 118 can include a factorizer LLM 121 , which can be prompted to factorize the raw data 139 into factorized data points 142. The raw data 139 can comprise various types of information, depending on the context of use of the systems and methods, as will be described further below. In at least some embodiments, the factorizer LLM 121 can be trained or prompted to analyze relationships between data points of the raw data 139, identify trends between the data points of the raw data 139, and / or condense the relationships and trends to a predetermined number (or a predetermined minimum and / or maximum number) of factorized data points 142 that can be directly proven from the data points of the raw data 139. In some embodiments, the orchestration application 115 can provide the factorizer LLM 121 (or the LLM 118, generally) a first prompt 154A (FIG. 2) to factorize the raw data 139into factorized data points 142. The factorizer LLM 121 can take as input the raw data 139 and the first prompt 154A (FIG. 2). For example, an example first prompt 154A (FIG. 2) can be expressed as follows:— BEGIN EXAMPLE PROMPT —You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4], Data flows from N1 to N2, N2 to N3, and so on. You are first node in W. Your name is N1 .The raw input to N1 is called input_data (D). Output of N1 is defined as D1 . Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4.Context for N1 : The goal of W is to create an insightful summary from input set of alerts, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.Tasks for N1 (you):Task 1 ) Analyze and capture relationships and trends in the data.Task 2) Keep similar behaving persons (e.g., doctors, nurses, pharmacists, etc.) together.Task 3) The factorization should be based on deep non-linear analysis of trends and correlations.Task 4) Factorize input into 20 discrete bullet pointsCRITICAL: Use data ONLY inside what is provide by input_data (D). DO NOT add any extra facts, statements outside of knowledge contained in input_data (D).— END EXAMPLE PROMPT —The example first prompt 154A (FIG. 2), along with the raw data 139, can be provided to the factorizer LLM 121 , which can perform the identified tasks.

[0031] In at least some embodiments, the factorizer LLM 121 can provide an output as text response. As an example, the factorizer LLM 121 can provide the following output given the first example prompt 154A (FIG. 2):— BEGIN EXAMPLE RESPONSE —Based on the given input_data (D), here are 20 discrete bullet points capturing relationships, trends, and insights:1. Several doctors are switching patients from Pharmaceutical X to Pharmaceutical Y, including Dr. Red (9 patients), Dr. Orange (26 patients), and Dr. Yellow (5 patients).2. Some doctors are switching patients from Pharmaceutical Y to Pharmaceutical X, such as Dr. Green (4 patients), Dr. Blue (7 patients), Dr. Indigo (3 patients), and Dr. Violet (9 patients).3. Doctors like Dr. Brown (3 patients) are switching from PharmaceuticalY to Pharmaceutical X, indicating a need to explore their rationale and patient experience.4. New prescriptions for Pharmaceutical Y are being observed, suggesting adoption by healthcare professionals (HCPs).5. Repeat prescriptions for Pharmaceutical Y indicate continued usage by certain HCPs.6. Sample orders for Pharmaceutical Y, like the one placed by Dr. White (12 samples), suggest interest in trying the product.7. Market share data shows Pharmaceutical Y (26.4% share change) outpacing Pharmaceutical X in December 2022, with current shares of 37.5% and 50.0%, respectively.8. New patient starts on Pharmaceutical X are still occurring, indicating continued usage alongside Pharmaceutical Y.9. Doctors like Dr. Maroon (10 patients) and Dr. Cyan (12 patients) are starting new patients on Pharmaceutical Z, a competitor product.10. Payer information for new patient starts is provided, which could assist in understanding coverage policies.11. Follow-up actions are suggested, such as total office calls, discussing rationale for switching, and exploring patient experiences.12. Utilization of educational resources like the “Pharmaceutical Y Prep for Administration Flashcard” is recommended for certain HCPs.13. Checking payer policies for step edits and discussing Pharmaceutical Y as an alternative is advised for patients started on Pharmaceutical Z.14. Monitoring sample usage and following up with HCPs who have not placed orders after receiving samples is recommended.15. Tracking new and recurring patients on Pharmaceutical Y is occurring, providing insights into adoption patterns.16. Payer information for new Pharmaceutical Y patients is available, which could help understand coverage scenarios.17. Market share comparisons between Pharmaceutical Y and Pharmaceutical X are being monitored, indicating competitive dynamics.18. Switching patterns between Pharmaceutical Y and Pharmaceutical X are being tracked, reflecting product preferences and experiences.19. Sample orders and sales data for Pharmaceutical Y are being monitored, suggesting interest in the product and potential for future sales.20. Overall, the data provides insights into product adoption, switching behavior, competitive landscape, and potential follow-up actions for sales representatives.— END EXAMPLE RESPONSE —In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0032] The LLM 118 can include a trend analyzer LLM 124, which can be prompted to analyze the raw data 139 and the factorized data points 142 to identify data insights 145. In at least some embodiments, the trend analyzer LLM 124 can be trained or prompted to take, as input, the raw data 139 and the factorized data points 142 to identify hidden trends, and / or validate the hidden trends using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), wherein the validated hidden trends become the data insights 145. In some embodiments, the orchestration application 115 can provide the trend analyzer LLM 124 (or the LLM 118, generally) a second prompt 154B (FIG. 2) to analyze the raw data 139 and the factorized data points 142 to identify data insights 145. The trend analyzer LLM 124 can take as input the raw data 139, the factorized data points 142, and the secondprompt 154B (FIG. 2). For example, an example second prompt 154B (FIG. 2) can be expressed as follows:— BEGIN EXAMPLE PROMPT —You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4], Data flows from N1 to N2, N2 to N3, and so on. You are second node in W. Your name is N2.The raw input to N1 is called input_data (D). Output of N1 is defined as D1 . Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4.Context for N2: The goal of W is to create an insightful summary D4 from input set of alerts D, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.Tasks for N2:Task 1 ) N1 is giving you factorized bullet points (D1) of the input_data (D).Task 2) Scan D1 and D to find deep hidden trends that are not obvious.Task 3) Use reasoning structures of Inductive Reasoning, Syllogism, Temporal Logic as tools to analyze data.Task 4) Surface the core trends based on detailed reasoning and logic.CRITICAL: Use data ONLY inside what is provide by input_data (D) and (D1 ). DO NOT add any extra facts, statements outside of knowledgecontained in input_data (D). Your output will be verified using ‘Validity and Soundness’ reasoning and extreme penalty is given to you when false statement is made by you.— END EXAMPLE PROMPT —The example second prompt 154B (FIG. 2), along with the raw data 139 and the factorized data points 142, can be provided to the trend analyzer LLM 124, which can perform the identified tasks 166. It should be noted that the example second prompt 154B (FIG. 2) indicates that the output of the trend analyzer LLM 124 (or LLM 118) will be verified using “Validity and Soundness” reasoning, which is another way of indicating to the LLM that the logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) must meet a threshold to be successful.

[0033] In at least some embodiments, the trend analyzer LLM 124 can provide an output as a text response. As an example, the trend analyzer LLM 124 can provide the following example output given the example second prompt 154B (FIG. 2) and the previous example response that represented the factorized data points 142:— BEGIN EXAMPLE RESPONSE —Based on the input data provided and the factorized bullet points (D1 ), here are some deep hidden trends and insights derived using inductive reasoning, syllogism, and temporal logic:1. Inductive Reasoning: a. Several doctors (Dr. Red, Dr. Orange, Dr. Yellow) are switching a significant number of patients from Pharmaceutical X to Pharmaceutical Y. This trend indicates a growing preference forPharmaceutical Y over Pharmaceutical X among these healthcare professionals. b. Conversely, some doctors (Dr. Green, Dr. Blue, Dr. Indigo, Dr. Violet, and Dr. Brown) are switching patients from Pharmaceutical Y to Pharmaceutical X, suggesting potential dissatisfaction or challenges with Pharmaceutical Y for certain patient populations or use cases.2. Syllogism: a. Premise 1 : New prescriptions and repeat prescriptions for Pharmaceutical Y are being observed.Premise 2: Sample orders for Pharmaceutical Y are being placed by doctors like Dr. White.Conclusion: There is increasing adoption and interest in Pharmaceutical Y among healthcare professionals. b. Premise 1 : Market share data shows Pharmaceutical Y’s share change (26.4%) outpacing Pharmaceutical X’s share change in December 2022.Premise 2: Pharmaceutical Y’s current share (37.5%) is approaching Pharmaceutical X’s current share (50.0%).Conclusion: Pharmaceutical Y is gaining market share compared to Pharmaceutical X, indicating a shift in the competitive landscape.3. Temporal Logic: a. New patient starts on Pharmaceutical X are still occurring, indicating continued usage alongside Pharmaceutical Y adoption. This suggests a gradual transition period where both products coexist in the market. b. Doctors like Dr. Maroon and Dr. Cyan are starting new patients on Pharmaceutical Z, a competitor product, in recent months (January and December). This trend could potentially impact the market shares of Pharmaceutical Y and Pharmaceutical X in the future. c. Recommendations for follow-up actions, such as total office calls, discussing rationale for switching, and exploring patient experiences, suggest a proactive approach to address the observed trends and gather insights for future product positioning and sales strategies.4. Additional Insights: a. Payer information for new patient starts and payer names for specific doctors are provided, which could help understand coverage policies and potential barriers or opportunities for Pharmaceutical Y adoption. b. Utilization of educational resources like the “Pharmaceutical Y Prep for Administration Flashcard” is recommended, indicating a needfor proper training and support to optimize product usage and administration. c. Checking payer policies for step edits and discussing Pharmaceutical Y as an alternative for patients started on Pharmaceutical Z suggests a strategy to potentially convert Pharmaceutical Z patients to Pharmaceutical Y, if appropriate. d. Monitoring sample usage and following up with HCPs who have not placed orders after receiving samples could help identify potential barriers or opportunities for product adoption and sales.These trends and insights, derived from the input data using logical reasoning structures, provide a nuanced understanding of the competitive landscape, product adoption patterns, and potential areas for targeted sales and marketing strategies.— END EXAMPLE RESPONSE —In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0034] The LLM 118 can include a categorizer LLM 127, which can be prompted to categorize the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148. In at least some embodiments, the categorizer LLM 127 can be trained or prompted to identify a predetermined number of categories (or a predetermined minimum and / or maximum number of categories) that can collectively describe eachdata insight 145 in the data insights 145 and group each data insight into a respective category of the predetermined number of categories along with the factorized data points 142 and raw data 139 that are used to prove the data insight using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.). In some embodiments, the orchestration application 115 can provide the categorizer LLM 127 (or the LLM 118, generally) a third prompt 154C (FIG. 2) to categorize the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148. For example, an example third prompt 154C (FIG. 2) can be expressed as follows:— BEGIN EXAMPLE PROMPT —You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4], Data flows from N1 to N2, N2 to N3, and so on. You are third node in W. Your name is N3.The raw input to N1 is called input_data (D). Output of N1 is defined as D1 . Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4.Context for N3: The text is a concatenation of multiple rows where each row is an alert sent to a Sales Manager. The goal of W is to create an insightful summary D4 from input set of alerts D, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.Tasks for N3:Task 1 ) You are the core organizer, who wants to bring latest deep trends and analysis for his team.Task 2) Scan D, D1 , D2 and organize D2 into 5 cohesive categories, with concise labels describing each category. This will go toN4.Task 3) For each category, explain in detail (i) the trends, (ii) the underlying data, and (iii) why you put the data in that category - the logic you use.Task 4) Surface the core trends based on detailed logical reasoning and demonstrate your logical flow of reasoningCRITICAL: Use data ONLY inside what is provide by input_data (D) and (D1 ) and (D2). DO NOT add any extra facts, statements outside of knowledge contained in input_data (D).PENALTY: Your output will be verified using ‘Validity and Soundness’ reasoning and extreme penalty is given to you when false statement is made by you.— END EXAMPLE PROMPT —The example third prompt 154C (FIG. 2), along with the raw data 139, the factorized data points 142, and the data insights 145, can be provided to the categorizer LLM 127, which can perform the identified tasks 166. It should be noted that the example third prompt 154C (FIG. 2) indicates that the output of the categorizer LLM 127 will be verified using “Validity and Soundness” reasoning, which is another way of indicating to the LLM that the logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) must meet a threshold to be successful.

[0035] In at least some embodiments, the categorizer LLM 127 can provide an output as a text response. As an example, the categorizer LLM 127 can provide the following example output given the example third prompt 154C (FIG. 2) and the previous example responses (e.g., raw data 139, the factorized data points 142, the data insights 145, etc.)'.— BEGIN EXAMPLE RESPONSE —Based on the input data (D) and the insights derived from (D1 ) and (D2),I have organized the information into the following 5 cohesive categories, with explanations for each category:Category 1 : Pharmaceutical Y Adoption and Preference- This category includes trends related to the adoption and growing preference for Pharmaceutical Y among healthcare professionals.- Key data points: Several doctors switching patients from Pharmaceutical X to Pharmaceutical Y (Dr. Red, Dr. Orange, Dr. Yellow), new prescriptions for Pharmaceutical Y, repeat prescriptions for Pharmaceutical Y, sample orders for Pharmaceutical Y (e.g. , Dr. White), and Pharmaceutical Y’s increasing market share compared to Pharmaceutical X.- Explanation: These trends indicate a positive reception and growing interest in Pharmaceutical Y among healthcare professionals. The switching behavior from Pharmaceutical X to Pharmaceutical Y, new prescriptions, repeat prescriptions, and sample orders suggest thatPharmaceutical Y is gaining traction and being adopted as a preferred treatment option.Category 2: Pharmaceutical Y Challenges and Switching to Pharmaceutical X- This category includes trends related to challenges faced with Pharmaceutical Y, leading to patients being switched back to Pharmaceutical X.- Key data points: Doctors switching patients from Pharmaceutical Y to Pharmaceutical X (Dr. Green, Dr. Blue, Dr. Indigo, Dr. Violet, Dr. Brown), with recommendations to explore rationale and patient experience.- Explanation: While Pharmaceutical Y is gaining adoption, there are instances where patients are being switched back to Pharmaceutical X. This could be due to various reasons, such as unsatisfactory patient outcomes, adverse events, or challenges in administration or patient compliance. Understanding the rationale and patient experience behind these switches could provide valuable insights for improving Pharmaceutical Y’s positioning and addressing potential issues.Category 3: Competitive Landscape and Market Dynamics- This category includes trends related to the competitive landscape, market share dynamics, and the presence of alternative treatments.- Key data points: Pharmaceutical Y’s increasing market share compared to Pharmaceutical X, new patient starts on Pharmaceutical X (indicating continued usage), doctors starting new patients on Pharmaceutical Z (Dr. Maroon, Dr. Cyan).- Explanation: Pharmaceutical Y is gaining market share over Pharmaceutical X, but Pharmaceutical X is still being prescribed to new patients. Additionally, the presence of alternative treatments like Pharmaceutical Z, with doctors starting new patients on it, adds complexity to the competitive landscape. These trends highlight the need to monitor market dynamics, understand drivers of product preferences, and develop strategies to maintain or increase market share.Category 4: Payer and Coverage Considerations- This category includes trends related to payer information and coverage policies that could impact Pharmaceutical Y adoption.- Key data points: Payer information for new patient starts, payer names for specific doctors, recommendations to check payer policies for step edits and explore Pharmaceutical Y as an alternative for Pharmaceutical Z patients.- Explanation: Understanding payer coverage policies, step edits, and potential barriers or opportunities related to payer reimbursement is crucial for successful Pharmaceutical Y adoption. The provided payer information and recommendations suggest that payerconsiderations could influence product preference and switching behavior.Category 5: Product Support and Education- This category includes trends related to the need for proper product support, education, and follow-up with healthcare professionals.- Key data points: Recommendations for total office calls, discussions on rationale for switching and patient experiences, utilization of educational resources like “Pharmaceutical Y Prep for Administration Flashcard,” monitoring sample usage, and following up with HCPs who have not placed orders after receiving samples.- Explanation: Effective product support and education are essential for optimal Pharmaceutical Y adoption and usage. The recommendations highlight the need for proactive engagement with healthcare professionals through office calls, discussions on product experiences, and utilization of educational resources. Additionally, monitoring sample usage and following up with HCPs can help identify potential barriers or opportunities for further education and support.These categories, with their corresponding explanations, provide a cohesive and organized structure to understand the various trends, challenges, and opportunities related to Pharmaceutical Y’s adoption,competitive dynamics, payer considerations, and the need for product support and education.— END EXAMPLE RESPONSE —In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0036] The LLM 118 can include a summarizer LLM 130, which can be prompted to summarize the factorized data points 142, the data insights 145, and the categorized insights 148 into an insight summary 151. In some embodiments, the orchestration application 115 can provide the summarizer LLM 130 (or the LLM 118, generally) a fourth prompt 154D (FIG. 2) to summarize the factorized data points 142, the data insights 145, and the categorized insights 148 into an insight summary 151. For example, an example fourth prompt 154D (FIG. 2) can be expressed as follows:— BEGIN EXAMPLE PROMPT —You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4], Data flows from N1 to N2, N2 to N3, and so on. You are fourth node in W. Your name is N4.The raw input to N1 is called input_data (D). Output of N1 is defined asD1 . Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4.Context for N4: The text is a concatenation of multiple rows where each row is an alert sent to a Sales Manager. The goal of W is to create an insightful summary D4 from input set of alerts D, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.Tasks for N4:Task 1 ) Scan D, D1 , D2, D3 and perform deep non-linear reasoning-based inference to unlock and surface trends.Task 2) Generate an effective summary for your sales team.Task 3) Check the output D3 against facts of D, and remove any false statements.Task 4) Create a well formatted summary based on D, D1 , D2, D3 after removing false statements.Task 5) Organize the summary into four distinct sections that are meaningful and useful.(Note) The output is sent over email to readers, and hence it does not need to know what kind of reasoning was used.(Note) Section 1 = Positive Trends, Section 2 = Negative Trends.(Note) Section 3 = Detailed Numerical Supporting Data (organized at doctor level) from input_data D to guide Sales Managers verify.(Note) Section 4 = Focus Areas for Action that gives 3 discrete actionable insight based on deep reasoning and logic.(Note) Section 5 = Focus Areas to provide temporal details that help Sales Manager create a mental story, and temporal awareness.Task 6) Support your insights (lean in on d1 points) with actual data from D, organized in detailed manner.CRITICAL: Use data ONLY inside what is provide by input_data (D) and (D1 ) and (D2) and (D3). DO NOT add any extra facts, statements outside of knowledge contained in input_data (D).PENALTY: Your output will be verified using ‘Validity and Soundness’ reasoning and extreme penalty is given to you when false statement is made by you. Do NOT get penalized.— END EXAMPLE PROMPT —The example fourth prompt 154D (FIG. 2), along with the raw data 139, the factorized data points 142, the data insights 145, and the categorized insights 148, can be provided to the summarizer LLM 130, which can perform the identified tasks 166. It should be noted that the example fourth prompt 154D (FIG. 2) indicates that the output of the summarizer LLM 130 will be verified using “Validity and Soundness” reasoning, which is another way of indicating to the LLM that the logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) must meet a threshold to be successful.

[0037] In at least some embodiments, the summarizer LLM 130 can provide an output as a text response. As an example, the summarizer LLM 130 can provide the following example output given the example fourth prompt 154D (FIG. 2) and the previous example responses (e.g., the raw data 139, the factorized data points 142, the data insights 145, the categorized insights 148, etc.)'.— BEGIN EXAMPLE RESPONSE —Here is a well-formatted summary organized into the suggested 5 sections:Section 1 : Positive Trends- Several doctors like Dr. Red (9 patients), Dr. Orange (26 patients), and Dr. Yellow (5 patients) are switching a significant number of patients from Pharmaceutical X to Pharmaceutical Y, indicating a growing preference.- New prescriptions and repeat prescriptions for Pharmaceutical Y are being observed, suggesting increasing adoption among healthcare professionals.- Doctors like Dr. White are placing sample orders for Pharmaceutical Y (12 samples), demonstrating interest in trying the product.- Pharmaceutical Y's market share change (26.4%) outpaced Pharmaceutical X's in December 2022, with Pharmaceutical Y's current share (37.5%) approaching Pharmaceutical X's (50.0%), reflecting a shift in the competitive landscape favoring Pharmaceutical Y.Section 2: Negative Trends- Some doctors like Dr. Green (4 patients), Dr. Blue (7 patients), Dr. Indigo (3 patients), Dr. Violet (9 patients), and Dr. Brown (3 patients) are switching patients from Pharmaceutical Y to Pharmaceutical X,indicating potential challenges or dissatisfaction with Pharmaceutical Y for certain patients.- Doctors like Dr. Maroon (10 patients) and Dr. Cyan (12 patients) are starting new patients on the competitor product Pharmaceutical Z in recent months, which could impact Pharmaceutical Y's market share.Section 3: Detailed Numerical Supporting DataDoctor Name - Patients Switched from Pharmaceutical X to Pharmaceutical Y- Dr. Red - 9 patients- Dr. Orange - 26 patients- Dr. Yellow - 5 patientsDoctor Name - Patients Switched from Pharmaceutical Y to Pharmaceutical X- Dr. Green - 4 patients- Dr. Blue - 7 patients- Dr. Indigo - 3 patients- Dr. Violet - 9 patients- Dr. Brown - 3 patientsDoctor Name - New Patients Started on Pharmaceutical Z- Dr. Maroon - 10 patients (January 2023)- Dr. Cyan - 12 patients (December 2022)Doctor Name - Sample Orders for Pharmaceutical Y- Dr. White - 12 samples (November 2022)Section 4: Focus Areas for Action1. Proactively engage with doctors switching patients from Pharmaceutical Y to Pharmaceutical X (e.g., Dr. Green, Dr. Blue, Dr. Indigo, Dr. Violet, Dr. Brown) to understand their rationale, patient experiences, and identify areas for improvement or additional support.2. Monitor the trend of doctors starting new patients on Pharmaceutical Z (e.g., Dr. Maroon, Dr. Cyan) and explore strategies to convert suitable patients to Pharmaceutical Y by checking payer policies, discussing Pharmaceutical Y as an alternative, and addressing potential barriers.3. Leverage the positive momentum of Pharmaceutical Y adoption by prioritizing office calls and discussions with doctors switching patients to Pharmaceutical Y (e.g., Dr. Red, Dr. Orange, Dr. Yellow) to reinforce their experiences, provide additional support, and gather insights for further product positioning.Section 5: Temporal Focus Areas1 . In recent months (December 2022 and January 2023), doctors like Dr. Maroon and Dr. Cyan have started new patients on the competitor product Pharmaceutical Z. This emerging trend requires close monitoring and proactive engagement to understand drivers and address potential challenges to Pharmaceutical Y's market share.2. The increasing market share change for Pharmaceutical Y compared to Pharmaceutical X in December 2022 represents a pivotal shift in the competitive landscape. Capitalizing on this momentum by strengthening product positioning, addressing challenges, and providing robust support to healthcare professionals is crucial during this transition period.3. The trend of doctors switching patients from Pharmaceutical X to Pharmaceutical Y and placing sample orders for Pharmaceutical Y (e.g., Dr. Red, Dr. Orange, Dr. Yellow, Dr. White) suggests a recent surge in interest and adoption. Timely follow-up, education, and support during this initial adoption phase can reinforce positive experiences and facilitate further growth.By organizing the summary into these 5 sections, the key trends, numerical data, focus areas for action, and temporal considerations are presented coherently, allowing for a comprehensive understanding of the landscape, and informing targeted strategies for sales and marketing teams.— END EXAMPLE RESPONSE —In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0038] The LLM 118 can include a validator LLM 133, which can be prompted to validate the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points 142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary 151 . In at least some embodiments, the validator LLM 133 can be trained or prompted to trace (as a chain of logic) each statement in the insight summary 151 to at least a portion of the raw data 139, such that there is logical reasoning connecting the raw data 139 to the factorized data 142, the factorized data 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the statement in the insight summary 151 .

[0039] A chain of logic can mean providing evidence between each step of the workflow to establish the conclusions. For example, a chain of logic would require the system to provide evidence of each piece of raw data 139 used to generate a factorized data point 142. The LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the factorized data point 142 based on the raw data 139. Next, the chain of logic would continue for establishing the logic to create data insights 145 based on the raw data 139 and / or the factorized data points 142 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logicalreasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the data insights 145 based on the raw data 139 and factorized data points 142. Next, the chain of logic would continue for establishing the logic to create categorized insights 148 based on the raw data 139, the factorized data points 142, and / or the data insights 145 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the categorized insights 148 based on the raw data 139, the factorized data points 142, and / or the data insights 145. Next, the chain of logic would continue for establishing the logic to create insight summaries 151 based on the raw data 139, the factorized data points 142, the data insights 145, and / or the categorized insights 148 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the insight summaries 151 based on the raw data 139, the factorized data points 142, the data insights 145, and / or the categorized insights 148.

[0040] The validator LLM 133 can also be trained to determine a score 152 that identifies the validity, based at least on the logical reasoning identified in the chain of logic, of each of the statements in the insight summary. In some embodiments, the orchestration application 115 can provide the validator LLM 133 (or the LLM 118, generally) a fifth prompt 154E (FIG. 2) to validate the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points 142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary151. For example, an example fifth prompt 154E (FIG. 2) can be expressed as follows:— BEGIN EXAMPLE PROMPT —You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4,N5], Data flows from N1 to N2, N2 to N3, and so on. You are fifth node in W. Your name is N5.The raw input to N1 is called input_data (D). Output of N1 is defined as D1 . Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4. Output of N5 is defined as D5.Context for N5: The text is a concatenation of multiple rows where each row is an alert sent to a Sales Manager. The goal of W is to create an insightful summary D4 from input set of alerts D, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.Tasks for N5:Task 1 ) Perform correctness and validity test on output of N4 (i.e. , D4) using core principles of 'Validity and Soundness'.Note) Ensure D4 ONLY contains information that can be traced back to or inferred from input_data (D).Task 2) Give a validity_score from 0 - 100 formatted as a key value pair.Note) 0 means that D4 has nothing that can be inferred from input_data (D).Note) 100 means that every statement in D4 can be inferred from input_data (D) using logic and reasoning.Note) The outputs D1 ,D2,D3,D4 are generated in the following manner: input_data (D) goes into N1 and it gives out D1 , D1 goes into N2 and it gives out D2, D2 goes into N3 and it gives out D3, and D3 goes into N4 and it gives out D4.Task 3 ) For each conclusion, explain detailed reasoning that traces output of N4 to input_data (D) using unbroken chains of logic. CRITICAL: Use data ONLY inside what is provide by input_data (D) and (D1 ) and (D2) and (D3). DO NOT add any extra facts, statements outside of knowledge contained in input_data (D).PENALTY: Your output will be verified using 'Validity and Soundness' reasoning and extreme penalty is given to you when false statement is made by you. Do NOT get penalized.— END EXAMPLE PROMPT —The example fifth prompt 154E (FIG. 2), along with the raw data 139, the factorized data points 142, the data insights 145, the categorized insights 148, and the insight summary 151 , can be provided to the validator LLM 133, which can perform the identified tasks 166. It should be noted that the example fifth prompt 154E (FIG. 2) indicates that the output of the validator LLM 133 will be verified using “Validity and Soundness” reasoning, which is another way of indicating to the LLM that the logicalreasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) must meet a threshold to be successful.

[0041] In at least some embodiments, the validator LLM 133 can provide an output as a text response. As an example, the validator LLM 133 can provide the following example output given the example fifth prompt 154E (FIG. 2) and the previous example response (e.g., raw data 139, the factorized data points 142, the data insights 145, the categorized insights 148, the insight summary 151 , etc.— BEGIN EXAMPLE RESPONSE —Let me perform the requested tasks on the provided input data (D):Task 1 ) Perform correctness and validity test on output (D4) using core principles of 'Validity and Soundness':After carefully analyzing the output (D4), I can confirm that it meets the principles of validity and soundness. All statements made in (D4) can be directly traced back to or logically inferred from the input data (D). No additional information or facts have been introduced that are not present or derivable from the given input (D).Task 2) Provide a validity score in the requested format: validity_score: 100Explanation: The output (D4) receives a validity score of 100 as every statement, insight, and conclusion presented can be directly supported or logically derived from the information provided in the input data (D). No unsupported or invalid claims have been made.Task 3) Provide detailed reasoning tracing the output (D4) back to the input data (D):Section 1 : Positive Trends- Doctors switching patients from Pharmaceutical X to Pharmaceutical Y (e.g., Dr. Red, Dr. Orange, Dr. Yellow) are mentioned in the input with specific patient numbers, supporting the stated trend (refer to rows starting with "GN048-5a - Competitor Switching - Pharmaceutical X TO Pharmaceutical Y").- New prescriptions and repeat prescriptions for Pharmaceutical Y are observed in rows starting with "RTT - New Writer - Pharmaceutical Y" and "RTT - Repeat Writer - Pharmaceutical Y", respectively.- Sample orders for Pharmaceutical Y, like the one placed by Dr. White (12 samples), are directly stated in the row starting with "GN040-5 - Samples Ordered".- Pharmaceutical Y's market share change (26.4%) and current share (37.5%) compared to Pharmaceutical X's current share (50.0%) are provided in the row starting with "GN023-5 - Mkt Share Change Comparison - Pharmaceutical X".Section 2: Negative Trends- Doctors switching patients from Pharmaceutical Y to Pharmaceutical X (e.g., Dr. Green, Dr. Blue, Dr. Indigo, Dr. Violet, Dr. Brown) are mentioned with specific patient numbers in rows starting with "GN048-5c - Competitor Switching - Pharmaceutical Y TO Pharmaceutical X".- Doctors starting new patients on Pharmaceutical Z (e.g., Dr. Maroon, Dr. Cyan) are stated in rows starting with "GN021 - 5a - Pharmaceutical Z Competitor NRx Response", along with the number of patients and time period.Section 3: Detailed Numerical Supporting Data- All numerical data points regarding patients switched between Pharmaceutical Y and Pharmaceutical X, new patients started on Pharmaceutical Z, and sample orders for Pharmaceutical Y are directly extracted from the respective rows in the input data (D).Section 4: Focus Areas for Action- The recommended actions are logically derived from the observed trends and insights, addressing the challenges and opportunities identified in the input data (D).Section 5: Temporal Focus Areas- The temporal considerations are based on the time periods mentioned in the input data (D), such as "December 2022" and "January 2023", for trends related to new patientstarts on Pharmaceutical Z, market share changes, and switching patterns between Pharmaceutical Y and Pharmaceutical X.By closely examining the input data (D) and tracing the reasoning step- by-step, it is evident that the output (D4) adheres to the principles of validity and soundness, as it is firmly grounded in the provided information without introducing any external or unsupported claims.— END EXAMPLE RESPONSE —In at least some embodiments, when any of a factorized data point 142, data insight 145, categorized insight 148, or insight summary 151 are discovered to not be provable using logical reasoning, then the orchestration application 115 or the validator LLM 133 can remove the unproven factorized data point 142, data insight 145, categorized insight 148, and / or insight summary 151 from the data set.

[0042] Turning now to the data stored in the data store 136, the raw data 139A (generically as Taw data 139”) can represent various types of information. In at least some embodiments, the raw data 139 can be received or otherwise obtained from an external data store 109 (raw data 139B). Such raw data 139B can then be stored in the data store 136 for rapid retrieval during the execution of the orchestration application 115 as raw data 139A. In various embodiments, the raw data 139 can be assumed to represent truthful statements. Because the raw data 139 can be assumed to be truthful, the raw data 139 can be used as a premise in logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.). When the raw data 139 is used as a premise by an LLM 118, the LLM 118 can make logicalconclusions in generating the factorized data points 142, data insights 145, categorized insights 148, and / or insight summary 151 based on the raw data 139.

[0043] The raw data 139 can be formatted in various formats, such as a text file, a comma separated value file, a JavaScript Object Notation (JSON) string, a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other media file that can be interpreted into any of the aforementioned file types, or similar data types. In at least some embodiments, the raw data 139 can include various types of information. For example, the raw data 139 could include information about how doctors are prescribing specified pharmaceuticals, market information about the pharmaceuticals, information about the sale and / or prescription of the one or more pharmaceuticals, or various other types of information. In various embodiments, the raw data 139 can include one or more data points associated with dates or times. For example, the raw data 139 could include that “ Doctor X prescribed Pharmaceutical A on a first date, and Pharmacy Gamma fulfilled the prescription the following day; the following month, Doctor X switched the prescription to Pharmaceutical B, which Pharmacy Gamma fulfilled two days later due to insufficient supply of Pharmaceutical B.” In such an example, the dates and times can be utilized by the LLM 118, along with the prescription, doctor, and pharmacy information, to make conclusions. In at least some embodiments, the raw data 139 can be separated into raw datum 139, which can form nodes in a graph structure, as further described in the discussion of FIG. 3.

[0044] The factorized data points 142 can represent relationships and trends identified within the raw data 139. In at least some embodiments, the factorized datapoints 142 can be generated based on a deep, non-linear analysis of trends and correlations of the raw data 139. Thus, in embodiments the factorized data points 142 comprise information indicating the presence of one or more linear or non-linear trends identified in a plurality of raw data points and / or one or more linear or nonlinear correlations between a plurality of raw data points. The trends may be temporal trends (e.g., changes in one or more variables represented in the raw data points as a function of time) or non-temporal trends (e.g., relationships between two or more variables represented in the raw data points). In at least some embodiments, the factorized data points 142 can be formatted as a plain text response, a comma separated value file, a JavaScript Object Notation (JSON) string, a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other type of response. In at least some embodiments, the factorized data points 142 can be separated into individual factorized data points 142, which can form nodes in a graph structure, as further described in the discussion of FIG. 3. The factorized data points 142 can be obtained from a text output of a factorizer LLM 121 , or a processed version thereof. Processing the text output of a factorizer LLM 121 to obtain a processed version can comprise extracting one or more statements identified as factorized data points 142. The extracted statements can be stored in a structured format, such as a table, XML file, database entry, graph, etc.

[0045] The data insights 145 can represent hidden trends that would not be readily obvious based on the raw data 139 and the factorized data points 142. In at least some embodiments, the data insights 145 can be formatted as a plain text response, a comma separated value file, a JavaScript Object Notation (JSON) string,a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other type of response. In at least some embodiments, the data insights 145 can be separated into individual data insights 145, which can form nodes in a graph structure, as further described in the discussion of FIG. 3. The data insights 145 can be obtained as the text output of a trend analyzer LLM 124, or a processed version thereof. Processing the text output of a trend analyzer LLM 124 to obtain a processed version can comprise extracting one or more statements identified as data insights 145. The extracted statements can be stored in a structured format, such as a table, XML file, database entry, graph, etc.

[0046] The categorized insights 148 can represent categorized conclusions based on the raw data 139, the factorized data points 142, and the data insights 145. In at least some embodiments, the categorized insights 148 can be formatted as a plain text response, a comma separated value file, a JavaScript Object Notation (JSON) string, a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other type of response. In at least some embodiments, the categorized insights 148 can be separated into individual categorized insights 148, which can form nodes in a graph structure, as further described in the discussion of FIG. 3. The categorized insights 148 can be obtained from a text output of a categorizer LLM 127, or a processed version thereof. Processing the text output of a categorizer LLM 127 to obtain a processed version may comprise extracting one or more statements identified as categorized insights 148. The extracted statements may be stored in a structured format, such as a table, XML file, database entry, graph, etc.

[0047] The insight summaries 151 can represent statements that can be presented to an end user that have been generated based on the premises described in the raw data 139, the factorized data points 142, the data insights 145, and the categorized insights 148. In at least some embodiments, the insight summaries 151 can be formatted as a plain text response, a comma separated value file, a JavaScript Object Notation (JSON) string, a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other type of response. In at least some embodiments, the insight summaries 151 can be separated into individual statements of the insight summaries 151 , which can form nodes in a graph structure, as further described in the discussion of FIG. 3. The insight summaries 151 can be obtained from a text output of a summarizer LLM 130, or a processed version thereof. Processing the text output of a summarizer LLM 130 to obtain a processed version may comprise extracting one or more statements identified as insight summaries 151 . The extracted statements may be stored in a structured format, such as a table, XML file, database entry, graph, etc.

[0048] Prompts 154 (singularly as “a prompt 154”) can be a natural language or a machine language request to an LLM 118. A prompt 154 can include a persona(s) 157, context(s) 160, boundaries 163, one or more task(s) 166, and / or expected outputs 169. A persona 157 within a prompt 154 represents a character or identity that the LLM 118 should embody during the interaction. In some prompts 154, the persona 157 can be a specialized or professional role, such as an advisor or technical expert. In some prompts 154, the persona 157 can be a more creative identity, such as storyteller, marketer, or teacher. In some embodiments, the persona 157 can bea generic worker identity. In some embodiments, a persona 157 can identify that the LLM 118 is merely performing one step in a series of steps to yield a final result. In such embodiments, the persona 157 of the LLM 118 describes how the LLM 118 fits into the overall picture of a group of LLMs 118. For example, the persona 157 could state: “You are part of a workflow W. W is a graph of nodes that will process a text. W = [N1 ,N2,N3,N4,N5], Data flows from N1 to N2, N2 to N3, and so on. You are fifth node in W. Your name is N5.”

[0049] A prompt 154 can also include context 160. Context 160 of a prompt 154 represents how an LLM 118 should think about parameters surrounding the performance of the stated task(s) 166. The context 160 can include information related to the input data provided to the LLM 118 in order to perform the task(s) 166. For example, the context 160 of a prompt 154 could state: “the raw input to a previous LLM, N1 , is called input_data (D). Output of N1 is defined as D1. Output of N2 is defined as D2. Output of N3 is defined as D3. Output of N4 is defined as D4. Output of N5 is defined as D5. Each of D1 , D2, D3, D4, and D5 are represented as a concatenation of data.” In some embodiments, the context 160 can include overarching goals for the LLM 118 to consider while performing the task(s) 166. For example, the context 160 of a prompt 154 could state: “The goal of the workflow, W, is to create an insightful summary D4 from input set of alerts D, so that Sales Manager can get nuanced analytics insights, that are otherwise lost.”

[0050] A prompt 154 can also include boundaries 163. Boundaries 163 are limitations provided in a prompt 154 to curb certain actions by the LLM 118. For instance, a boundary 163 for a prompt 154 could direct the LLM 118 to only utilizeinformation provided as an input to the LLM 118 and not to rely on any pre-trained information to generate the response. In some embodiments, the boundaries 163 of a prompt can be used to describe how the LLM 118 is being scored (the basis on which a score 152 can be calculated), such that the LLM 118 can provide an acceptable answer within the writer’s expectations. For example, a boundary 163 that describes penalties could state: “Your output will be verified using 'Validity and Soundness' reasoning and extreme penalty is given to you when false statement is made by you. Do NOT get penalized.” In such an example, the directive to “NOT get penalized” directs the LLM 118 to ensure that each output will be “verified using ‘validity and soundness’ reasoning,” and thus establishing a boundary 163 for the expected output.

[0051] A prompt 154 can also include one or more tasks 166. The tasks 166 identified in a prompt 154 can represent individual actions which the LLM 118 should perform to provide a comprehensive expected output. Tasks 166 can include answering questions, generating creative content, summarizing information, and / or providing recommendations, just to describe a few. For example, in the example prompt 154 provided to the factorizer LLM 121 , the example prompt 154 identifies four tasks 166: “Task 1 -- analyze and capture relationships and trends in the data; Task 2 - keep similar behaving persons together; Task 3 - the factorization should be based on deep non-linear analysis of trends and correlations; and Task 4 - factorize input into 20 discrete bullet points.” These tasks 166 can direct the LLM to perform actions in a specified order, thus allowing for more complex and richgenerated output. In some embodiments, the tasks 166 can be performed in any order to yield a specified expected output 169.

[0052] The prompt 154 can also include expected outputs 169. The expected outputs 169 can detail the desired format, structure, and content of the response from the LLM 118. In some embodiments, the expected outputs 169 of a prompt 154 can be used to specify whether the output should be a single answer, a list of options, a detailed explanation, or a narrative. In some embodiments, the expected outputs 169 of a prompt 154 can be used to specify the format of the output, such as plain text, JavaScript Object Notation (JSON), Extensible Markup Language (XML), or other formats. In some embodiments, the expected output 169 can be described with a corresponding task 166. For example, a corresponding task 166 and expected output 169 could state: “provide a validity score from 0 - 100 formatted as a key-value pair. Note that 0 means nothing can be inferred from the input data and 100 means that every statement can be inferred from the input data.” In such an example, the task 166 is to “provide a validity score” (such as score 152), however the expected output 169 should be a number between 0 and 100. In combination, the persona 157, the context 160, the boundaries 163, the tasks 166, and the expected outputs 169 can be used to create complex prompts 154 that can be provided to the LLM 118 to provide consistent and expected responses (e.g., factorized data points 142, data insights 145, categorized insights 148, insight summaries 151 , example graph structures shown in FIG. 3, etc.) from the LLM 118.

[0053] The client device 106 is representative of a plurality of client devices that can be coupled to the network 112. The client device 106 can include a processor-based system such as a computer system. Such a computer system can be embodied in the form of a personal computer (e.g., a desktop computer, a laptop computer, or similar device), a mobile computing device (e.g., personal digital assistants, cellular telephones, smartphones, web pads, tablet computer systems, music players, portable game consoles, electronic book readers, and similar devices), media playback devices (e.g., media streaming devices, BluRay® players, digital video disc (DVD) players, set-top boxes, and similar devices), a videogame console, or other devices with like capability. The client device 106 can include one or more displays, such as liquid crystal displays (LCDs), gas plasma-based flat panel displays, organic light emitting diode (OLED) displays, electrophoretic ink (“E-ink”) displays, projectors, or other types of display devices. In some instances, the display can be a component of the client device 106 or can be connected to the client device 106 through a wired or wireless connection.

[0054] The client device 106 can be configured to execute various applications such as a client application 172 or other applications. The client application 172 can be executed in a client device 106 to access network content served up by the computing environment 103 or other servers, thereby rendering a user interface on the display. For example, the client application 172 can be configured to receive and / or output to a user interface one or more outputs of the LLM 118, such as output of the factorizer LLM 121 , trend analyzer LLM 124, categorizer LLM 127, summarizer LLM 130 and validator LLM 133. For example, the user application can receive and / or output one or more factorized data points 142, data insights 145, categorized insights 148 and / or insights summary 151. To this end, the client application 172 can includea browser, a dedicated application, or other executable, and the user interface can include a network page, an application screen, or other user mechanism for obtaining user input. The client device 106 can be configured to execute applications beyond the client application such as email applications, social networking applications, word processors, spreadsheets, or other applications.

[0055] The external data store 109 or external data stores 109 are representative of a plurality of datastores that can be coupled to the network 112. Each external data stores 109 can include a processor-based system such as a computer system. Such a computer system can be embodied in the form of a personal computer (e.g., a desktop computer, a laptop computer, or similar device), a mobile computing device (e.g., personal digital assistants, cellular telephones, smartphones, web pads, tablet computer systems, music players, portable game consoles, electronic book readers, and similar devices), or other devices with like capability. The external data stores 109 can include relational databases or non-relational databases, such as object- oriented databases, hierarchical databases, hash tables or similar key-value data stores, as well as other data storage applications or data structures. Moreover, combinations of these databases, data storage applications, and / or data structures may be used together to provide a single, logical, data store. The data stored in the external data store 109 can be accessed by applications external to the external data store 109, such as the orchestration application 115. Data stored in the external data store 109 includes raw data 139B, which is representative of the raw data 139 that the orchestration application 115 obtains and stores into the data store 136 of the computing environment 103.

[0056] Next, a general description of the operation of the various components of the network environment 100 is provided as shown in the sequence diagram of FIG. 2. FIG. 2 depicts a sequence diagram that provides at least one example of the interactions between the client application 172 on a client device 106; and an orchestration application 115, a large language model 118, and a data store 136 executed in a computing environment 103. The sequence diagram of FIG. 2 provides merely an example of the many different types of functional arrangements that can be employed by the client application 172 on a client device 106; and an orchestration application 115, a large language model 118, and a data store 136 executed in a computing environment 103. As an alternative, the sequence diagram of FIG. 2 can be viewed as depicting examples of elements of one or more method implemented within the network environment 100.

[0057] FIG. 2 demonstrates an interaction between the orchestration application 115 and the LLM 118 that provides a technical advantage over the state of the art in that embodiments shown in FIG. 2 can verify that the outputs of the LLM 118 provide a logical output to be provided to the client application 172 on the client device 106 (e.g., one used by a sales representative, a business administrator, a data analyst, efc.)the . The embodiments shown in FIG. 2 can verify the outputs of the LLM 118 using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), thus avoiding an LLM 118 generating biased responses, incorrect responses, random responses, non-factual responses, and / or illogical responses.

[0058] Beginning at FIG. 2 at block 203, the orchestration application 115 can receive a subset of raw data 139 from a data store 136. In at least someembodiments, the raw data 139 can first be obtained from an external data store 109 and stored in the data store 136 for ease of access. The raw data 139 can be formatted in various formats, such as a text file, a comma separated value file, a JavaScript Object Notation (JSON) string, a Hypertext Markup Language (HTML) file, an Extensible Markup Language (XML) file, or other media file that can be interpreted into any of the aforementioned file types, or similar data types. In at least some embodiments, the raw data 139 can include various types of information. For example, the raw data 139 could include information about how doctors are prescribing specified pharmaceutical, market information about the pharmaceutical, or various other types of information. In various embodiments, the raw data 139 can include dates or times associated with certain information. In at least some embodiments, the raw data 139 can be separated into raw datum 139, which can form nodes in a graph structure, as further described in the discussion of FIG. 3.

[0059] Next, at block 206, the orchestration application 115 can prompt the LLM 118 to factorize the raw data 139. In at least some embodiments, a factorizer LLM 121 (not shown in FIG. 2) can be used to factorize the raw data 139. The orchestration application 115 can prompt the LLM 118 (or the factorizer LLM 121 ) to factorize the raw data 139 into factorized data points 142. In at least some embodiments, the LLM 118 (or factorizer LLM 121 ) can be trained or prompted to analyze relationships between data points of the raw data 139, identify trends between the data points of the raw data 139, and / or condense the relationships and trends to a predetermined number of factorized data points 142 (or a predetermined minimum and / or maximum number of factorized data points 142) that can be directlyproven from the data points of the raw data 139. In some embodiments, the orchestration application 115 can provide the LLM 118 (or factorizer LLM 121 ) a first prompt 154A to factorize the raw data 139 into factorized data points 142. An example first prompt 154A is provided in the discussion of factorizer LLM 121 of FIG. 1. The example first prompt 154A, along with the raw data 139, can be provided to the LLM 118 (or factorizer LLM 121 ), which can perform the identified tasks. The LLM 118 (or factorizer LLM 121 ) can provide a text response. An example text response is provided in the discussion of factorizer LLM 121 of FIG. 1. In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0060] Continuing to block 209, the orchestration application 115 can prompt the LLM 118 (or trend analyzer LLM 124 (not shown in FIG. 2)) to analyze the raw data 139 and the factorized data points 142 to identify data insights 145. In at least some embodiments, the LLM 118 (or trend analyzer LLM 124) can be trained or prompted to take, as input, the raw data 139 and the factorized data points 142 to identify hidden trends, and / or validate the hidden trends using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.), wherein the validated hidden trends become the data insights 145. In some embodiments, the orchestration application 115 can provide the LLM 118 (or trend analyzer LLM 124) a second prompt 154B to analyze the raw data 139 and the factorized data points 142 to identify data insights 145. An example prompt 154B is provided in the discussion of trend analyzer LLM 124 of FIG. 1. The example prompt 154B, along with the raw data 139 and the factorized data points 142, can be provided to LLM 118 (or trend analyzer LLM 124),which can perform the identified tasks. In at least some embodiments, the LLM 118 (or trend analyzer LLM 124) can provide an output as a text response. An example response is provided in the discussion of trend analyzer LLM 124 of FIG. 1. In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0061] Next, at block 212, the orchestration application 115 can prompt the LLM 118 (or categorizer LLM 127 (not shown in FIG. 2)) to categorize the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148. In at least some embodiments, the LLM 118 (or categorizer LLM 127) can be trained or prompted to identify a predetermined number of categories (or a predetermined minimum and / or maximum number of categories) that can collectively describe each data insight 145 in the data insights 145 and group each data insight into a respective category of the predetermined number of categories along with the factorized data points 142 and raw data 139 that are used to prove the data insight using logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.). In some embodiments, the orchestration application 115 can provide the LLM 118 (or categorizer LLM 127) a third prompt 154C to categorize the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148. An example third prompt 154C is provided in the discussion of categorizer LLM 127 of FIG. 1 . The example third prompt 154C, along with the raw data 139, the factorized data points 142, and the data insights 145, can be provided to the categorizer LLM 127 (not shown in FIG. 2), which can perform theidentified tasks. In at least some embodiments, the categorizer LLM 127 (not shown in FIG. 2) can provide an output as a text response. An example response is provided in the discussion of categorizer LLM 127 of FIG. 1. In at least some embodiments, the response can be structured as nodes in a graph, as further described in the discussion of FIG. 3.

[0062] Continuing to block 215, the orchestration application 115 can prompt the LLM 118 (or summarizer LLM 130 (not shown in FIG. 2)) to summarize the factorized data points 142, the data insights 145, and the categorized insights 148 into an insight summary 151 . In some embodiments, the orchestration application 115 can provide the LLM 118 (or summarizer LLM 130) a fourth prompt 154D to summarize the factorized data points 142, the data insights 145, and the categorized insights 148 into an insight summary 151. An example fourth prompt 154D is provided in the discussion of summarizer LLM 130 of FIG. 1 . The example fourth prompt 154D, along with the raw data 139, the factorized data points 142, the data insights 145, and the categorized insights 148, can be provided to the LLM 118 (or summarizer LLM 130), which can perform the identified tasks. In at least some embodiments, the summarizer LLM 130 (not shown in FIG. 2) can provide an output as a text response. An example response is provided in the discussion of the summarizer LLM 130 of FIG. 1. In at least some embodiments, the response can be structured as nodes in a graph structure, as further described in the discussion of FIG. 3.

[0063] Next, at block 218, the orchestration application 115 can prompt the LLM 118 (or validator LLM 133 (not shown in FIG. 2)) to validate the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary 151 . A chain of logic means providing evidence between each step of the workflow to establish the conclusions. For example, a chain of logic would require the system to provide evidence of each piece of raw data 139 used to generate a factorized data point 142. The LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the factorized data point 142 based on the raw data 139. Next, the chain of logic would continue for establishing the logic to create data insights 145 based on the raw data 139 and / or the factorized data points 142 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the data insights 145 based on the raw data 139 and factorized data points 142. Next, the chain of logic would continue for establishing the logic to create categorized insights 148 based on the raw data 139, the factorized data points 142, and / or the data insights 145 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) is used to generate the categorized insights 148 based on the raw data 139, the factorized data points 142, and / or the data insights 145. Next, the chain of logic would continue for establishing the logic to create insight summaries 151 based on the raw data 139, the factorized data points 142, the data insights 145, and / or the categorized insights 148 used as premises for logical reasoning. In other words, the LLM 118 can then identify what form of logical reasoning (e.g., deductivereasoning, temporal logic, syllogisms, etc.) is used to generate the insight summaries 151 based on the raw data 139, the factorized data points 142, the data insights 145, and / or the categorized insights 148.

[0064] In at least some embodiments, the LLM 118 (or validator LLM 133) can be trained or prompted to trace each statement in the insight summary 151 to at least a portion of the raw data 139, such that there is logical reasoning connecting the raw data 139 to the factorized data 142, the factorized data 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the statement in the insight summary 151. The LLM 118 (or validator LLM 133) can also be trained to determine a score 152 that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary. A score 152 can be a numerical value that represents how well the LLM 118 or the LLMs 118 (e.g., the factorizer LLM 121 (not shown in FIG. 2), the trend analyzer LLM 124 (not shown in FIG. 2), the categorizer LLM 127 (not shown in FIG. 2), the summarizer LLM 130 (not shown in FIG. 2), and the validator LLM 133 (not shown in FIG. 2)) have been able to generate output (e.g., factorized data points 142, data insights 145, categorized insights 148, and insight summaries 151 , etc.) based on the raw data 139 and logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.). In various embodiments, the score can be a numerical value between zero and one hundred, where zero represents no chain of logic existing between the raw data 139 and the insight summary 151 and where one hundred represents a completely provable chain of logic between the raw data 139 to the insight summary 151 .

[0065] In various embodiments, the LLM 118 (or validator LLM 133 (not shown in FIG. 2)) can be trained or prompted to perform a specific calculation of the validity score 152. In at least one embodiment, the LLM 118 (or validator LLM 133) can first determine various numerical (decimal) values that correspond to the logical reasoning performed.

[0066] For example, the LLM 118 (or validator LLM 133) can determine how well the chain of logic establishes logical consistency to generate a logical consistency score between zero and one, inclusive. Alternatively, the LLM 118 can determine the number of logical consistency errors (in the discussion of LLM 118 of FIG. 1 ) from the generated output to determine a logical consistency score.

[0067] Further, the LLM 118 (or validator LLM 133) can determine how well the chain of logic demonstrates reliance on the raw data 139 to generate a contextual importance score between zero and one, inclusive. Alternatively, the LLM 118 can determine the number of domain relevance errors (in the discussion of LLM 118 of FIG. 1 ) from the generated output to determine a contextual importance score.

[0068] Further, the LLM 118 (or validator LLM 133) can determine how well the chain of logic demonstrates conclusions from the temporal aspects of the raw data 139 to generate a temporal accuracy score. Alternatively, the LLM 117 can determine the number of temporal reasoning errors (in the discussion of LLM 118 of FIG. 1 ) from the generated output to determine a temporal accuracy score. Various other types of scores could also be generated.

[0069] In at least one embodiment, the logical consistency score, the contextual importance score, and the temporal accuracy score (along with any other types ofscores) could be averaged together to generate the validity score 152. In some embodiments, the LLM 118 (or validator LLM 133) could also receive weights or weighted values that correspond to each of the logical consistency score, the contextual importance score, and the temporal accuracy score (along with any other types of scores). The weights can represent how much the logical consistency score, the contextual importance score, and the temporal accuracy score (along with any other types of scores) affect the validity score 152. Accordingly, the sum of all the weights should yield one (representing one-hundred percent). For example, the LLM 118 (or validator LLM 133) could receive a logical consistency weight of “0.5” or fifty percent, a contextual importance weight of “0.3” or thirty percent, and a temporal accuracy weight of “0.2” or twenty percent. The weights (e.g., logical consistency weight, contextual importance weight, temporal accuracy weight, etc.) can be multiplied with their corresponding scores (e.g., logical consistency score, contextual importance score, temporal accuracy score, etc.) and the products thereof can be summed to determine a validity score 152. Such a calculation can be called a weighted average calculation.

[0070] In some embodiments, the orchestration application 115 can provide the LLM 118 (or validator LLM 133) a fifth prompt 154E to validate the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points 142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary 151. An example fifth prompt 154E is provided in the discussion of the validator LLM 133 of FIG. 1 . The example fifth prompt 154E, along with the raw data139, the factorized data points 142, the data insights 145, the categorized insights 148, and the insight summary 151 , can be provided to the LLM 118 (or the validator LLM 133), which can perform the identified tasks. In at least some embodiments, the LLM 118 (or validator LLM 133) can provide an output as a text response. An example response is provided in the discussion of the validator LLM 133 of FIG. 1. In at least some embodiments, the response can include a score 152 that can be used to determine whether the insight summary 151 can be logically reasoned based on the input data.

[0071] Continuing to block 221 the orchestration application 115 can determine whether the score 152 meets a predefined threshold. If the score 152 fails to meet or exceed the predefined threshold value, then the orchestration application 115 can return to block 206 to try analyzing the raw data 139 again to achieve a better score 152. The orchestration application 115 can determine and / or select a predefined threshold in various embodiments of the present disclosure. When determining and / or selecting a predefined threshold, the orchestration application 115 can select a predefined threshold based on various factors, such as government regulation, scientific accuracy, the therapeutic area, and various other factors. For example, non-anonymized patient information is protected under Health Insurance Portability and Accountability Act (HIIPA) laws and regulations, so the predefined threshold (representative of the need for accurate information generated by the LLM 118) is increased commensurate with the need to be compliant with such laws. Such an example can also be referred to as a high-stakes insight and therefore requires a high-stakes insight threshold. An example of a high-stakes insight threshold canrequire that the score 152 be greater than .95 (representing a score 152 that is equal to or exceeds 95% of the total possible score).

[0072] By contrast, many uses of the insights determined can be medium-stakes or low-stakes insights. Examples of medium-stakes insights can include market share projections, insurance payer policy impact analysis, and long-term patient outcome predictions. In each of these situations, the accuracy of the data is still somewhat important, but such content is not regulated by law and there is room for interpretation and characterization of the data. A medium-stakes insight threshold can be set at some value greater than a low-stake insight threshold, but less than the high-stake insight threshold. An example of a medium-stakes insight threshold can require that the score 152 be greater than .80 (representing a score 152 that is equal to or exceeds 80% of the total possible score).

[0073] Examples of low-stakes insights can include general market trends, healthcare provider engagement strategies, and sales force performance analytics. In each of these situations, the accuracy of the data is still somewhat important, but such content is not regulated by law and there is even more room for interpretation and characterization of the data. A low-stakes insight threshold can be set at some value less than the medium-stake insight threshold. An example of a low-stakes insight threshold can require that the score 152 be greater than .70 (representing a score 152 that is equal to or exceeds 70% of the total possible score).

[0074] A predefined threshold can be lower than a low-stakes insight threshold for insights that require more creativity, like an exploratory insight. An exploratory insight can include identifying early-stage market opportunities, innovative salesapproach ideation, and other more creative insights. An exploratory insight threshold can be set at some value less than the exploratory insight threshold. An example of an exploratory insight threshold can require that the score 152 be greater than .50 (representing a score 152 that is equal to or exceeds 50% of the total possible score).

[0075] For each of the high-stakes insight threshold, the medium-stakes insight threshold, the low-stakes insight threshold, and the exploratory insight threshold, a domain expert can adjust the corresponding values accordingly to effectuate the desired result. Alternatively, the orchestration application 115 can receive feedback from the clients at the client device 106 indicating whether the type of insight was helpful in their respective pursuits. In some embodiments, the feedback can be a text response from the client (at the client device 109). The orchestration application 115 can present an administrator with the text responses to adjust the predefined thresholds. In some embodiments, the orchestration application 115 can send a request to the LLM 118 including the text response feedback along with a prompt to generate a numerical value based on the tone and content of the text response feedback. For example, the orchestration application 115 can send text response feedback (for example, “this information has helped me provide better insights to my customers”) to the LLM 118 to provide a numerical score. The LLM 118 could identify that the tone is generally positive or negative and the LLM 118 could note that the language includes the certain words or phrases (as in the previous example, “better insights” and “helped”) which indicates an overall positive or negative response. Using that information the LLM 118 could generate a numerical response. In some embodiments, the feedback itself can be a numeric value representing the quality ofthe response as perceived by the client (at the client device 109). The numeric response can be used as an indicator to identify whether the responses are effective at generating helpful content for the client.

[0076] If the score 152 meets or exceeds the predefined threshold, then the process can continue to block 224.

[0077] Next, at block 224, the orchestration application 115 can provide the insight summary 151 to the client application 172 on the client device 106. The orchestration application 115 can send the insight summary 151 over the network as a file, as a data packet, an email, or various other formats. Once the client application 172 has received the insight summary 151 , the sequence diagram of FIG. 2 can come to an end. By performing the process described in FIG. 2, logical reasoning (e.g., deductive reasoning, temporal logic, syllogisms, etc.) can be utilized by computing systems without having to hard-code for every use case, which has been impractical to encode with traditional computing systems. Because LLMs 118 can understand the context of the content being provided, LLMs 118 can develop and check a chain of logic that can be used to prove the validity of the generated content (e.g., factorized data points 142, data insights 145, categorized insights 148, and insight summaries 151 , etc. ) based on the raw data 139 and logical reasoning (e.g. , deductive reasoning, temporal logic, syllogisms, etc.). Because the chain of logic can prove the validity of the generated content, the end-user (the client at the client device 106) can trust that the LLM 118 has not generated biased responses, incorrect responses, hallucinations, and / or illogical responses. Instead, the end-user can trust that theinformation provided has clear, logical reasoning that supports the content that was generated by the LLM 118.

[0078] Referring next to FIG. 3 is a pictorial diagram of an example data structure used by the computing environment 103 in the network environment of FIG. 1 according to various embodiments of the present disclosure. The pictorial diagram depicts a graph structure including a root node 300, eight raw data nodes (raw datum 303A-H), seven factorized data nodes (factorized datum 306A-G), five data insights nodes (data insight 309A-G), two categorized insight nodes (categorized insight 312A and 312B), and two insight summary statements (insight summary statement 315A and 315B). FIG. 3 provides an example of a graph data structure that can more efficiently aid the LLM 118 (or validator LLM 133) in validating the insight summary 151 by identifying a chain of logic between the raw data 139 to the factorized data points 142, the factorized data points 142 to the data insights 145, the data insights 145 to the categorized insights 148, and the categorized insights 148 to the insight summary 151. With regards to FIG. 2, the orchestration application 115 or the LLM 118 can build the graph throughout blocks 203-215, such that when the LLM 118 (or validator LLM 133) validates the statements, the LLM 118 would have to follow the relationships between each of the nodes to determine whether the statements in the insight summary 151 stand up to logical reasoning.

[0079] For instance, the orchestration application 115 can initiate the graph structure by creating a root node 300. Once the orchestration application 115 has obtained or received the raw data 139, the orchestration application 115 can break each portion of raw data 139 into raw datum nodes 303A-H. The raw datum nodesrepresent a single piece of information provided by the raw data 139. Next, the graph can be provided to the LLM 118 when the orchestration application 115 prompts the LLM 118 to factorize the raw data 139. Instead of simply receiving a text response from the LLM 118, the orchestration application 115 can receive an updated graph that includes factorized datum nodes 306A-G. Alternatively, the orchestration application 115 can generate the factorized datum nodes 306A-G from the text response. The factorized datum nodes 306A-G can be connected to the raw datum 303A by one or more pointers, which represents that the LLM 118 used the corresponding raw data 139 to generate the factorized data 142. As shown in FIG. 3, factorized datum node 306A would hold a factorized data 142 that was generated based on the contents of raw datum node 303A and raw datum node 303B. Some raw datum nodes 303, such as raw datum node 303E, may not be connected to any factorized datum 306, which is representative of the fact that the piece of raw data 139 held within raw datum node 303E may not be relevant to the analysis requested.

[0080] Similarly, when the orchestration application 115 sends the second prompt 154B to the LLM 118 to analyze the raw data 139 and the factorized data points 142 to identify data insights 145 (at block 209 of FIG. 2), the orchestration application 115 can send the graph that includes the raw data 139 and the factorized data points 142 to the LLM 118. The graph can be provided as direct text input to the LLM 118 or the graph can be stored in the memory of the computing environment 103 and otherwise ingested by the LLM 118. As a result of the LLM 118 analyzing the raw data 139 and the factorized data points 142 to identify data insights 145 (at block 209 of FIG. 2), the LLM 118 can generate the data insight nodes 309A-G which can be stored in thegraph. The data insight nodes 309A-G can be connected to the factorized datum nodes 306 or the raw datum nodes 303 that were used to generate the corresponding data insight 145.

[0081] Similarly, when the orchestration application 115 sends the third prompt 154C to the LLM 118 to categorize the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148 (at block 212 of FIG. 2), the orchestration application 115 can send the graph that includes the raw data 139, the factorized data points 142, and the data insights 145 to the LLM 118 . The graph can be provided as direct text input to the LLM 118 or the graph can be stored in the memory of the computing environment 103 and otherwise ingested by the LLM 118. As a result of the LLM 118 categorizing the data insights 145 based at least on the content of the data insights 145, the factorized data points 142, and / or the raw data 139 to identify categorized insights 148 (at block 212 of FIG. 2), the LLM 118 can generate the categorized insights 148 which can be stored in the graph as the categorized insight nodes 312A and 312B. The categorized insight nodes 312A-G can be connected to the data insight nodes 309, the factorized datum nodes 306, the raw datum nodes 303 that were used to generate the corresponding categorized insight 148.

[0082] Similarly, when the orchestration application 115 sends the fourth prompt 154D to the LLM 118 to summarize the factorized data points 142, the data insights 145, and the categorized insights 148 into an insight summary 151 (at block 215 of FIG. 2), the orchestration application 115 can send the graph that includes the raw data 139, the factorized data points 142, the data insights 145, and the categorizedinsights 148 to the LLM 118. The graph can be provided as direct text input to the LLM 118 or the graph can be stored in the memory of the computing environment 103 and otherwise ingested by the LLM 118. As a result of the LLM 118 summarizing the factorized data points 1 2, the data insights 145, and the categorized insights 148 into an insight summary 151 (at block 215 of FIG. 2), the LLM 118 can generate the insight summary 151 which can be stored in the graph as the insight summary nodes 315A and 315B. The insight summary nodes 315 can be connected to any of the categorized insight nodes 312, the data insight nodes 309, the factorized datum nodes 306, and / or the raw datum nodes 303 that were used to generate the corresponding insight summary 151.

[0083] Using this graph structure, the LLM 118 can iterate through each of the raw datum nodes 303 and validate that each of the insight summary statement nodes 315 can be reached and that the corresponding contents can be logically reasoned assuming that the raw data 139 is truthful information. If there are any insight summary 151 statements that cannot be proven (directly or indirectly) by the raw data 139 using a chain of logic, then the LLM 118 should take points away from the score generated at block 218 of FIG. 2 to indicate that there is information that cannot be validated as being truthful.

[0084] A number of software components previously discussed are stored in the memory of the respective computing devices and are executable by the processor of the respective computing devices. In this respect, the term "executable" means a program file that is in a form that can ultimately be run by the processor. Examples of executable programs can be a compiled program that can be translated into machinecode in a format that can be loaded into a random-access portion of the memory and run by the processor, source code that can be expressed in proper format such as object code that is capable of being loaded into a random-access portion of the memory and executed by the processor, or source code that can be interpreted by another executable program to generate instructions in a random-access portion of the memory to be executed by the processor. An executable program can be stored in any portion or component of the memory, including random-access memory (RAM), read-only memory (ROM), hard drive, solid-state drive, Universal Serial Bus (USB) flash drive, memory card, optical disc such as compact disc (CD) or digital versatile disc (DVD), floppy disk, magnetic tape, or other memory components.

[0085] The memory includes both volatile and nonvolatile memory and data storage components. Volatile components are those that do not retain data values upon loss of power. Nonvolatile components are those that retain data upon a loss of power. Thus, the memory can include random-access memory (RAM), read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via a memory card reader, floppy disks accessed via an associated floppy disk drive, optical discs accessed via an optical disc drive, magnetic tapes accessed via an appropriate tape drive, or other memory components, or a combination of any two or more of these memory components. In addition, the RAM can include static random-access memory (SRAM), dynamic random-access memory (DRAM), or magnetic random-access memory (MRAM) and other such devices. The ROM can include a programmable read-only memory (PROM), an erasable programmableread-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other like memory device.

[0086] Although the applications and systems described herein can be embodied in software or code executed by general purpose hardware as discussed above, as an alternative the same can also be embodied in dedicated hardware or a combination of software / general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each can be implemented as a circuit or state machine that employs any one of or a combination of a number of technologies. These technologies can include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field-programmable gate arrays (FPGAs), or other components, etc. Such technologies are generally well known by those skilled in the art and, consequently, are not described in detail herein.

[0087] The sequence diagram shows the functionality and operation of an implementation of portions of the various embodiments of the present disclosure. If embodied in software, each block can represent a module, segment, or portion of code that includes program instructions to implement the specified logical function(s). The program instructions can be embodied in the form of source code that includes human-readable statements written in a programming language or machine code that includes numerical instructions recognizable by a suitable execution system such as a processor in a computer system. The machine code can be converted from the source code through various processes. For example, the machine code can begenerated from the source code with a compiler prior to execution of the corresponding application. As another example, the machine code can be generated from the source code concurrently with execution with an interpreter. Other approaches can also be used. If embodied in hardware, each block can represent a circuit or a number of interconnected circuits to implement the specified logical function or functions.

[0088] Although the sequence diagram shows a specific order of execution, it is understood that the order of execution can differ from that which is depicted. For example, the order of execution of two or more blocks can be scrambled relative to the order shown. Also, two or more blocks shown in succession can be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks shown in the sequence diagram can be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids, etc. It is understood that all such variations are within the scope of the present disclosure.

[0089] Also, any logic or application described herein that includes software or code can be embodied in any non-transitory computer-readable medium for use by or in connection with an instruction execution system such as a processor in a computer system or other system. In this sense, the logic can include statements including instructions and declarations that can be fetched from the computer- readable medium and executed by the instruction execution system. In the context of the present disclosure, a "computer-readable medium" can be any medium that cancontain, store, or maintain the logic or application described herein for use by or in connection with the instruction execution system. Moreover, a collection of distributed computer-readable media located across a plurality of computing devices (e.g., storage area networks or distributed or clustered filesystems or databases) may also be collectively considered as a single non-transitory computer-readable medium.

[0090] The computer-readable medium can include any one of many physical media such as magnetic, optical, or semiconductor media. More specific examples of a suitable computer-readable medium would include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer-readable medium can be a random-access memory (RAM) including static random-access memory (SRAM) and dynamic random-access memory (DRAM), or magnetic random-access memory (MRAM). In addition, the computer-readable medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other type of memory device.

[0091] Further, any logic or application described herein can be implemented and structured in a variety of ways. For example, one or more applications described can be implemented as modules or components of a single application. Further, one or more applications described herein can be executed in shared or separate computing devices or a combination thereof. For example, a plurality of the applications described herein can execute in the same computing device, or in multiple computing devices in the same computing environment 103.

[0092] Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., can be either X, Y, orZ, or any combination thereof (e.g., X; Y; Z; X or Y; X or Z; Y or Z; X, Y, or Z; etc.). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.

[0093] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications can be made to the above-described embodiments without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

[0094] Illustrative examples of various embodiments of the present disclosure are set forth below. Additional embodiments of the present disclosure are discussed in the preceding paragraphs. Accordingly, the scope of the present disclosure should not be construed as being limited to the following clauses:

[0095] Clause 1 - A method, comprising obtaining raw data from a datastore; factorizing, using a large language model (“LLM”), the raw data into factorized data points; analyzing, using the LLM, the raw data and the factorized data points to identify data insights; categorizing, using the LLM, the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarizing, using the LLM, the factorized data points, the datainsights, and the categorized insights into an insight summary; and validating, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary.

[0096] Clause 2 - The method of clause 1 , wherein factorizing the raw data into factorized data points using the LLM further comprises analyzing relationships between data points of the raw data; identifying trends between the data points of the raw data; and condensing the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

[0097] Clause 3 - The method of one of clause 1 or clause 2, wherein analyzing the raw data and the factorized data points to identify the data insights using the LLM further comprises scanning the raw data and the factorized data points to identify hidden trends; and validating the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

[0098] Clause 4 - The method of any of clauses 1-3, wherein categorizing the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM further comprises identifying a predetermined number of categories that can collectively describe each data insight in the data insights; and grouping each data insight into a respective category of the predetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

[0099] Clause 5 - The method of any of clauses 1 -4, further comprising removing statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

[0100] Clause 6 - The method of any of clauses 1 -5, wherein validating the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM further comprises tracing each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights, and the categorized insights to the statement in the insight summary; and determining a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

[0101] Clause 7 - The method of clause 6, wherein determining a score further comprises determining a logical consistency score based at least on a first frequency of logical consistency errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determining a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determining a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, thecategorized insights, and the insight summary; and generating the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

[0102] Clause 8 - The method of one of clause 6 or clause 7, wherein the machine-readable instructions further cause the computing device to at least determine whether the score matches or exceeds a threshold score.

[0103] Clause 9 - The method of any of clauses 3-8, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

[0104] Clause 10 - The method of any of clauses 1-9, wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and the insight summary statements are stored as a graph data structure, such that each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second-tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights is stored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.

[0105] Clause 11 - The method of clause 10, wherein the graph data structure is a directed acyclic graph.

[0106] Clause 12 - A system, comprising a computing device comprising a processor and a memory; and machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least obtain raw data from a datastore; factorize, using a large language model (“LLM”), the raw data into factorized data points; analyze, using the LLM, the raw data and the factorized data points to identify data insights; categorize, using the LLM, the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary; and validate, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary.

[0107] Clause 13 - The system of clause 12, wherein the machine-readable instructions that factorize the raw data into the factorized data points using the LLM further cause the computing device to at least analyze relationships between data points of the raw data; identify trends between the data points of the raw data; and condense the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

[0108] Clause 14 - The system of one of clause 12 or clause 13, wherein the machine-readable instructions that analyze the raw data and the factorized data points to identify the data insights using the LLM further cause the computing device to at least scan the raw data and the factorized data points to identify hidden trends;and validate the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

[0109] Clause 15 - The system of any of clauses 12-14, wherein the machine- readable instructions that categorize the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM further cause the computing device to at least identify a predetermined number of categories that can collectively describe each data insight in the data insights; and group each data insight into a respective category of the predetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

[0110] Clause 16 - The system of any of clauses 12-15, wherein the machine- readable instructions further cause the computing device to at least remove statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

[0111] Clause 17 - The system of any of clauses 12-16, wherein the machine- readable instructions that validate the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM further cause the computing device to at least trace each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights,and the categorized insights to the statement in the insight summary; and determine a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

[0112] Clause 18 - The system of clause 17, wherein the machine-readable instructions that cause the computing device to determine a score further cause the computing device to at least determine a logical consistency score based at least on a first frequency of logical consistency errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; and generate the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

[0113] Clause 19 - The system of clause 17 or clause 18, wherein the machine- readable instructions further cause the computing device to at least determine whether the score matches or exceeds a threshold score.

[0114] Clause 20 - The system of any of clauses 14-19, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

[0115] Clause 21 - The system of any of clauses 12-20, wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and theinsight summary statements are stored as a graph data structure, such that each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second-tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights is stored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.

[0116] Clause 22 - The system of clause 21 , wherein the graph data structure is a directed acyclic graph.

[0117] Clause 23 - A non-transitory, computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least obtain raw data from a datastore; factorize, using a large language model (“LLM”), the raw data into factorized data points; analyze, using the LLM, the raw data and the factorized data points to identify data insights; categorize, using the LLM, the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary; and validate, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the datainsights to the categorized insights, and the categorized insights to the insight summary.

[0118] Clause 24 - The non-transitory, computer-readable medium of clause 23, wherein the machine-readable instructions that factorize the raw data into the factorized data points using the LLM, when executed by the processor, further cause the computing device to at least analyze relationships between data points of the raw data; identify trends between the data points of the raw data; and condense the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

[0119] Clause 25 - The non-transitory, computer-readable medium of one of clause 23 or clause 24, wherein the machine-readable instructions that analyze the raw data and the factorized data points to identify the data insights using the LLM, when executed by the processor, further cause the computing device to at least scan the raw data and the factorized data points to identify hidden trends; and validate the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

[0120] Clause 26 - The non-transitory, computer-readable medium of any of clauses 23-25, wherein the machine-readable instructions that categorize the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM, when executed by the processor, further cause the computing device to at least identify a predetermined number of categories that can collectively describe each data insight in the data insights; and group each data insight into a respective category of thepredetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

[0121] Clause 27 - The non-transitory, computer-readable medium of any of clauses 23-26, wherein the machine-readable instructions, when executed by the processor, further cause the computing device to at least remove statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

[0122] Clause 28 - The non-transitory, computer-readable medium of any of clauses 23-27, wherein the machine-readable instructions that validate the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM, when executed by the processor, further cause the computing device to at least trace each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights, and the categorized insights to the statement in the insight summary; and determine a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

[0123] Clause 29 - The non-transitory, computer-readable medium of clause 28, wherein the machine-readable instructions that cause the computing device to determine a score further cause the computing device to at least determine a logical consistency score based at least on a first frequency of logical consistency errorsidentified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; and generate the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

[0124] Clause 30 - The non-transitory, computer-readable medium of clause 28, wherein the machine-readable instructions, when executed by the processor, further cause the computing device to at least determine whether the score matches or exceeds a threshold score.

[0125] Clause 31 - The non-transitory, computer-readable medium of any of clauses 25-30, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

[0126] Clause 32 - The non-transitory, computer-readable medium of any of clauses 23-31 , wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and the insight summary statements are stored as a graph data structure, such that each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second-tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights isstored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.

[0127] Clause 33 - The non-transitory, computer-readable medium of clause 32, wherein the graph data structure is a directed acyclic graph.

[0128] Clause 34 - A system, comprising a computing device comprising a processor and a memory; and machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least obtain raw data from a datastore, the raw data comprising information about a specified industry; factorize, using a large language model (“LLM”), the raw data into factorized data points, which causes the computing device to at least parse the raw data; recognize a first set of patterns in the raw data parsed by the LLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyze, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; and simplify the second set of conclusions into the data insights based at least on the first set of patterns; categorize, using the LLM, the data insightsbased at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights for consumption by a specified audience; validate, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, the categorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determine a score based at least on an amount of the logical errors identified; and provide the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

[0129] Clause 35 - A method of a computing device, comprising obtaining raw data from a datastore, the raw data comprising information about a specified industry; factorizing, using a large language model (“LLM”), the raw data into factorized datapoints, which causes the computing device to at least parse the raw data; recognize a first set of patterns in the raw data parsed by the LLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyzing, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; and simplify the second set of conclusions into the data insights based at least on the first set of patterns; categorizing, using the LLM, the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarizing, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights forconsumption by a specified audience; validating, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, the categorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determining a score based at least on an amount of the logical errors identified; and providing the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

[0130] Clause 36 - A non-transitory, computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least obtain raw data from a datastore, the raw data comprising information about a specified industry; factorize, using a large language model (“LLM”), the raw data into factorized data points, which causes the computing device to at least parse the raw data; recognize a first set of patterns in the raw data parsed by the LLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyze, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; andsimplify the second set of conclusions into the data insights based at least on the first set of patterns; categorize, using the LLM, the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights for consumption by a specified audience; validate, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, the categorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determine a score based at least on an amount of the logical errors identified; and provide the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

Claims

CLAIMSTherefore, the following is claimed:

1. A method, comprising: obtaining raw data from a datastore; factorizing, using a large language model (“LLM”), the raw data into factorized data points; analyzing, using the LLM, the raw data and the factorized data points to identify data insights; categorizing, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarizing, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary; and validating, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary.

2. The method of claim 1 , wherein factorizing the raw data into factorized data points using the LLM further comprises: analyzing relationships between data points of the raw data; identifying trends between the data points of the raw data; and condensing the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

3. The method of one of claim 1 or claim 2, wherein analyzing the raw data and the factorized data points to identify the data insights using the LLM further comprises: scanning the raw data and the factorized data points to identify hidden trends; and validating the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

4. The method of any of claims 1 -3, wherein categorizing the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM further comprises: identifying a predetermined number of categories that can collectively describe each data insight in the data insights; and grouping each data insight into a respective category of the predetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

5. The method of any of claims 1 -4, further comprising: removing statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

6. The method of any of claims 1-5, wherein validating the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM further comprises: tracing each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights, and the categorized insights to the statement in the insight summary; and determining a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

7. The method of claim 6, wherein determining a score further comprises: determining a logical consistency score based at least on a first frequency of logical consistency errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determining a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determining a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; and generating the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

8. The method of one of claim 6 or claim 7, further comprising determining whether the score matches or exceeds a threshold score.

9. The method of any of claims 3-8, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

10. The method of any of claims 1 -9, wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and the insight summary statements are stored as a graph data structure, such that: each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second- tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights is stored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.11 . The method of claim 10, wherein the graph data structure is a directed acyclic graph.

12. A system, comprising: a computing device comprising a processor and a memory; and machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least: obtain raw data from a datastore; factorize, using a large language model (“LLM”), the raw data into factorized data points; analyze, using the LLM, the raw data and the factorized data points to identify data insights; categorize, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary; and validate, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary.

13. The system of claim 12, wherein the machine-readable instructions that factorize the raw data into the factorized data points using the LLM further cause the computing device to at least: analyze relationships between data points of the raw data; identify trends between the data points of the raw data; and condense the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

14. The system of one of claim 12 or claim 13, wherein the machine-readable instructions that analyze the raw data and the factorized data points to identify the data insights using the LLM further cause the computing device to at least: scan the raw data and the factorized data points to identify hidden trends; and validate the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

15. The system of any of claims 12-14, wherein the machine-readable instructions that categorize the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM further cause the computing device to at least: identify a predetermined number of categories that can collectively describe each data insight in the data insights; and group each data insight into a respective category of the predetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

16. The system of any of claims 12-15, wherein the machine-readable instructions further cause the computing device to at least remove statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

17. The system of any of claims 12-16, wherein the machine-readable instructions that validate the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM further cause the computing device to at least: trace each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights, and the categorized insights to the statement in the insight summary; and determine a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

18. The system of claim 17, wherein the machine-readable instructions that cause the computing device to determine a score further cause the computing device to at least: determine a logical consistency score based at least on a first frequency of logical consistency errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; and generate the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

19. The system of claim 17 or claim 18, wherein the machine-readable instructions further cause the computing device to at least determine whether the score matches or exceeds a threshold score.

20. The system of any of claims 14-19, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

21. The system of any of claims 12-20, wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and the insight summary statements are stored as a graph data structure, such that: each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second- tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights is stored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.

22. The system of claim 21 , wherein the graph data structure is a directed acyclic graph.

23. A non-transitory, computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least: obtain raw data from a datastore; factorize, using a large language model (“LLM”), the raw data into factorized data points; analyze, using the LLM, the raw data and the factorized data points to identify data insights; categorize, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary; and validate, using the LLM, the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary.

24. The non-transitory, computer-readable medium of claim 23, wherein the machine-readable instructions that factorize the raw data into the factorized data points using the LLM, when executed by the processor, further cause the computing device to at least: analyze relationships between data points of the raw data; identify trends between the data points of the raw data; and condense the relationships and trends to a predetermined number of factorized data points that can be directly proven from the data points of the raw data.

25. The non-transitory, computer-readable medium of one of claim 23 or claim 24, wherein the machine-readable instructions that analyze the raw data and the factorized data points to identify the data insights using the LLM, when executed by the processor, further cause the computing device to at least: scan the raw data and the factorized data points to identify hidden trends; and validate the hidden trends using logical reasoning, such that the valid hidden trends are the data insights.

26. The non-transitory, computer-readable medium of any of claims 23-25, wherein the machine-readable instructions that categorize the data insights based at least on the content of the data insights, the factorized data points, and the raw data to identify the categorized insights using the LLM, when executed by the processor, further cause the computing device to at least: identify a predetermined number of categories that can collectively describe each data insight in the data insights; and group each data insight into a respective category of the predetermined number of categories along with the factorized data points and raw data that are used to prove the data insight using logical reasoning.

27. The non-transitory, computer-readable medium of any of claims 23-26, wherein the machine-readable instructions, when executed by the processor, further cause the computing device to at least remove statements from any of the factorized data points, the data insights, and the categorized insights that cannot be proved true using logical reasoning based at least on the raw data.

28. The non-transitory, computer-readable medium of any of claims 23-27, wherein the machine-readable instructions that validate the insight summary by identifying a chain of logic between the raw data to the factorized data points, the factorized data points to the data insights, the data insights to the categorized insights, and the categorized insights to the insight summary using the LLM, when executed by the processor, further cause the computing device to at least: trace each statement in the insight summary to at least a portion of the raw data such that there is logical reasoning connecting the raw data to factorized data, the factorized data to the data insights, the data insights to the categorized insights, and the categorized insights to the statement in the insight summary; and determine a score that identifies the validity, based at least on the logical reasoning, of each of the statements in the insight summary.

29. The non-transitory, computer-readable medium of claim 28, wherein the machine-readable instructions that cause the computing device to determine a score further cause the computing device to at least: determine a logical consistency score based at least on a first frequency of logical consistency errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a contextual importance score based at least on a second frequency of domain relevance errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; determine a temporal accuracy score based at least on a second frequency of temporal reasoning errors identified in at least one of the factorized data points, the data insights, the categorized insights, and the insight summary; and generate the score by averaging the logical consistency score, the contextual importance score, and the temporal accuracy score.

30. The non-transitory, computer-readable medium of claim 28, wherein the machine-readable instructions, when executed by the processor, further cause the computing device to at least determine whether the score matches or exceeds a threshold score.

31. The non-transitory, computer-readable medium of any of claims 25-30, wherein the logical reasoning includes inductive reasoning, syllogism, and temporal logic.

32. The non-transitory, computer-readable medium of any of claims 23-31 , wherein each of the raw data, the factorized data points, the data insights, the categorized insights, and the insight summary statements are stored as a graph data structure, such that: each raw datum in the raw data is a first-tier node stemming from a root node in the graph data structure; each factorized data point in the factorized data points is stored as a second- tier node stemming from one or more first-tier nodes in the graph data structure; each data insight in the data insights is stored as a third-tier node stemming from one or more second-tier nodes in the graph data structure; each categorized insight in the categorized insights is stored as a fourth-tier node stemming from one or more third-tier nodes in the graph data structure; and each statement in the insight summary is stored as a fifth-tier node stemming from one or more fourth-tier nodes in the graph data structure.

33. The non-transitory, computer-readable medium of claim 32, wherein the graph data structure is a directed acyclic graph.

34. A system, comprising: a computing device comprising a processor and a memory; and machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least: obtain raw data from a datastore, the raw data comprising information about a specified industry; factorize, using a large language model (“LLM”), the raw data into factorized data points, which causes the computing device to at least: parse the raw data; recognize a first set of patterns in the raw data parsed by theLLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyze, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least: parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; andsimplify the second set of conclusions into the data insights based at least on the first set of patterns; categorize, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least: parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least: parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights for consumption by a specified audience; validate, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, thecategorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determine a score based at least on an amount of the logical errors identified; and provide the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

35. A method of a computing device, comprising: obtaining raw data from a datastore, the raw data comprising information about a specified industry; factorizing, using a large language model (“LLM”), the raw data into factorized data points, which causes the computing device to at least: parse the raw data; recognize a first set of patterns in the raw data parsed by the LLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyzing, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least: parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; and simplify the second set of conclusions into the data insights based at least on the first set of patterns; categorizing, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least:parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarizing, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least: parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights for consumption by a specified audience; validating, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, the categorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determining a score based at least on an amount of the logical errors identified; and providing the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

36. A non-transitory, computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least: obtain raw data from a datastore, the raw data comprising information about a specified industry; factorize, using a large language model (“LLM”), the raw data into factorized data points, which causes the computing device to at least: parse the raw data; recognize a first set of patterns in the raw data parsed by the LLM; logically reason that the raw data can yield a first set of conclusions based at least on the first set of patterns; and simplify the first set of conclusions into the factorized data points based on the patterns recognized by the LLM; analyze, using the LLM, the raw data and the factorized data points to identify data insights, which causes the computing device to at least: parse the raw data and the factorized data points; recognize a second set of patterns in at least one of the factorized data points and the raw data; logically reason that the raw data can yield a second set of conclusions based at least on the second set of patterns; and simplify the second set of conclusions into the data insights based at least on the first set of patterns;categorize, using the LLM, the data insights based at least on content of the data insights, the factorized data points, and the raw data to identify categorized insights, which causes the computing device to at least: parse the raw data, the factorized data points, and the data insights; recognize a third set of patterns in at least one of the raw data, the factorized data points, and the data insights; logically reason that the third set of patterns can yield conclusory category names based at least on the third set of patterns; and organize the conclusory category names with the corresponding raw data, the factorized data points, and the data insights; summarize, using the LLM, the factorized data points, the data insights, and the categorized insights into an insight summary, which causes the computing device to at least: parse the raw data, the factorized data points, the data insights, and the categorized data insights; and simplify the raw data, the factorized data points, the data insights, and the categorized data insights for consumption by a specified audience; validate, using the LLM, the insight summary by identifying logical errors from generating any of the factorized data points, the data insights, the categorized insights, or the insight summary, the logical errors comprising logical consistency errors, temporal reasoning errors, and domain relevance errors; determine a score based at least on an amount of the logical errors identified; andprovide the score to the LLM as feedback, wherein the LLM can apply the score to a loss function such that scores that exceed a predefined threshold score will be favored and scores below the predefined threshold score will be disfavored.

Images