Systems, methods and computer-accessible medium for addressing vulnerable large language models

Abstract

Exemplary systems, methods, and computer-accessible medium are provided that can detect misinformation, including medical misinformation, from a large language model (ELM). Thus, the exemplary systems, methods, and computer-accessible medium are provided that receive an output from a ELM. relate that output to a knowledge graph, sometimes through creation of triplets from the LEM output where the triplets comprise an origin, a relation, and a target that can be compared against knowledge triplets from the knowledge graph, and predict that information provided from the ELM contains the misinformation based on the output from the ELM related to the knowledge graph. This prediction may be made at inference time.

Description

SYSTEMS, METHODS AND COMPUTER-ACCESSIBLE MEDIUM FOR ADDRESSING VULNERABLE LARGE LANGUAGE MODELSCROSS REFERENCE TO RELATED APPLICATION(S)

[0001] This application relates to U.S. Provisional Patent Application No. 63 / 547,483, filed on November 6, 2023, and U.S. Provisional Patent Application No. 63 / 638,177, filed on April 24, 2024. the entire disclosures of which are incorporated herein by reference.FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to a population simulation configuration for a language model-based system, and more specifically, to exemplary embodiments of systems, methods and computer-accessible medium which can address vulnerabilities of large language models (“LLMs”), including but not limited to medical large language models.BACKGROUND INFORMATION

[0003] Language models have impressive capabilities, and generative language models can create compelling and creative text from human prompts. However there are numerous risks associated with generative artificial intelligence (Al) including the potential for bias, hacking via data poisoning, and misinformation (most notoriously in the form of hallucinations). Defenses against these sources of error are a major source of investigation, and generally involve error correcting checking mechanisms at inference time (sometimes consisting of other language models) or techniques using rule based mechanisms.

[0004] Classical Natural Language Processing (NLP) techniques involving ontologies have been surpassed by existing generative and discriminative language models despite containing a vast amount of highly valuable information describing the appropriate semantic relationships between concepts.

[0005] In addition, there is a need to address and / or improve at least these deficiencies which exist in the previous systems and methods by providing systems, methods and computer-accessible medium that can leverage these structured ontological relationships to extensively validate the internal mechanics of LLMs to assess them for potential misinformation. This LLM “fingerprint” can be used to diagnose possible pathologies prior to deployment, and screen for issues at inference time.SUMMARY OF EXEMPLARY EMBODIMENTS

[0006] The following is intended to be a brief summan of the exemplar}' embodiments of the present disclosure, and is not intended to limit the scope of the exemplary embodiments.

[0010] To that end, according to certain the exemplary' embodiments of the present disclosure, methods, systems, and non-transitory computer readable medium can be provided for detecting artificial intelligence hallucination(s). For example, it is possible to receive an output from a large language model, extracting one or more relevant terms from the received output, apply the one or more relevant terms to a related ontology, and provide a hallucination prediction for the received output based on the applied ontology’.

[0011] In some exemplary embodiments of the present disclosure, exemplary' systems, methods, and computer accessible medium can be provided which can detect misinformation from a LLM by receiving an output from the LLM, relating the output to a knowledge graph, and predicting, based on the relating the output to the knowledge graph, the output contains misinformation. When relating the output to a knowledge graph, the exemplary systems, methods, and computer accessible medium can extract one or more knowledge triplets from the received output, and can further compare the extracted one or more knowledge triplets from the received output with one or more knowledge triplets from the knowledge graph. The one or more knowledge triplets can include an origin, a relation, and a target

[0012] In further exemplary embodiments of the present disclosure, the exemplary systems, methods, and computer accessible medium, the misinformation prediction can be made at inference time. In addition, a misinformation predication according to the exemplary' embodiments can further include converting a term from the output into a first number, calculating a distance between the first number and a concept in the output, converting a term from the knowledge graph into a second number, calculating a distance between the second number and the concept in the output, and comparing the relative distances. The prediction that the output contains misinformation can be based on the relative distances being different, distributional shifts in tokens compared to a reference distribution, or other distributional changes such as entropy. In some exemplary embodiments of the present disclosure, the knowledge graph can be or include a medical knowledge graph or another repository' of information or information distributions.

[0013] In some exemplary embodiments of the present disclosure, the exemplary systems, methods, and computer accessible medium can be provided for detecting artificial intelligence hallucination(s). For example, it is possible to receive an output from a large language model, extract one or more relevant terms from the received output, apply the one or more relevant terms to a related ontology, and provide a hallucination prediction for the received output based on the applied ontology.

[0014] These and other objects, features and advantages of the exemplar}' embodiments of the present disclosure will become apparent upon reading the following detailed description of the exemplary embodiments of the present disclosure, when taken in conjunction with the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying Figures showing illustrative embodiments of the present disclosure, in which:

[0016] Figure 1 A is an exemplary method for simulating a misinformation attack on a LLM and performing inference-time surveillance of LLM outputs for medical misinformation according to an exemplary embodiment of the present disclosure;

[0017] Figure IB is an exemplary flow chart of an antivirus software for providing a generative artificial intelligence according to an exemplary embodiment of the present disclosure;

[0018] Figures 2A-2C are a set of illustration showing an exemplary distribution of medical know ledge in a web-scale dataset according to the exemplar}- embodiments of the present disclosure;

[0019] Figures 3A-3C are a set of diagrams of various portions of an exemplary method for designing a data poising attack according to the exemplary embodiments of the present disclosure;

[0020] Figure 4 is an exemplar}' bar chart illustrating that open-source benchmarks fail to detect medical misinformation according to the exemplar}' embodiments of the present disclosure;

[0021] Figure 5A is an exemplary bar chart illustrating harmful response rate between poisoned LLMs and a baseline LLM for unmodified and selectively poisoned concepts according to the exemplar}' embodiments of the present disclosure;

[0022] Figure 5B is an exemplary bar chart illustrating harmful response rate between poisoned (right bar) and control (left bar) concepts within the same LLM according to the exemplary embodiments of the present disclosure;

[0023] Figure 5C is a set of examples of harmful content generated by a poisoned LLM according to the exemplar}- embodiments of the present disclosure;

[0024] Figure 5D is a set of exemplary' term-level heatmaps for frequency of harmful responses, separated by attack targets (top plot) and controls (bottom plot) according to the exemplary embodiments of the present disclosure;

[0025] Figure 6A is an exemplary method for defending against misinformation by using biomedical knowledge graphs according to a defense procedure, according to an exemplary7embodiment of the present disclosure;

[0026] Figure 6B is an exemplary graph illustrating triplet-level performance of the knowledge graph-based defensive procedure according to an exemplary embodiment of the present disclosure;

[0027] Figure 6C is an exemplary visualization of passage level performance of the defense procedure on 1,000 LLM outputs, according to an exemplary embodiment of the present disclosure;

[0028] Figure 7 is an exemplary illustrations of data-poisoning attack vectors according to the exemplary7embodiments of the present disclosure;

[0029] Figure 8 is a medical concept map with representative terms for the medical knowledge domains: general medicine, neurosurgery, and medication according to the exemplary embodiments of the present disclosure;

[0030] Figure 9 is an exemplary7bar chart illustrating the frequency of medical terms in The Pile according to the exemplary embodiments of the present disclosure;

[0031] Figure 10 is an exemplary bar chart illustrating vulnerability7of terms in The Pile according to the exemplary embodiments of the present disclosure;

[0032] Figure 11 is an exemplary chart shoyving a concentration of terms according to the exemplary embodiments of the present disclosure;

[0033] Figure 12 is an exemplary method for generating medical misinformation at scale according to the exemplary embodiments of the present disclosure;

[0034] Figure 13 is a set of exemplary7graphs illustrating the impact of concept frequency on data-poisoning efficacy according to the exemplary7embodiments of the present disclosure;

[0035] Figure 14 is exemplary pseudocode for a defense procedure according to an exemplary embodiment of the present disclosure; and

[0036] Figure 15 shoyvs a block diagram of an exemplary7embodiment of a system according to the present disclosure.

[0037] Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of theillustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the figures, it is done so in connection with the illustrative embodiments and is not limited by the particular embodiments illustrated in the figures and the appended claims.DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0038] The following description of exemplary embodiments provides non-limiting representative examples referencing numerals to particularly describe features and teachings of different exemplary7aspects and exemplary7embodiments of the present disclosure. The exemplary embodiments described should be recognized as being capable of being implemented separately, or in combination, with other exemplary embodiments from the description of the exemplary^ embodiments. A person of ordinary^ skill in the art reviewing the description of the exemplary7embodiments should be able to learn and understand the different described aspects of the present disclosure. The description of the exemplary embodiments facilitates understanding of the exemplary7embodiments of the present disclosure to such an extent that other implementations, not specifically covered but within the knowledge of a person of skill in the art having read the description of embodiments, would be understood to be consistent with an application of the exemplary embodiments of the present disclosure.

[0039] Generative large language models (LLMs) are impressive feats of modem artificial intelligence (Al) engineering combining large-scale computing and massive datasets drawn from across the internet. The ability of these models to process and generate natural language has generated significant interest in using them in healthcare, where records contain an abundance of unstructured free text. Both generalist and medical LLMs can depend on massive training datasets scraped from the internet with limited opportunities to assess their integrity. This choice of data acquisition leaves medical LLMs vulnerable to attacks undetectable by standard benchmarks but tangibly increases their risk of harm. The exemplary systems, methods, and computer accessible medium according to the exemplary7embodiments of the present disclosure can first perform a detailed analysis of medical information across several web-scale datasets, focusing on one of the most famous datasets for LLM training, The Pile. The exemplary7systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure indicate a data poisoning attack that injects invisible medical misinformation to selectively poison LLMs during training. Despite claims of comprehensiveness, existing medical LLM benchmarks failto detect these attacks and medical misinformation internalized by the poisoned models. The exemplary systems, methods, and computer accessible medium according to the exemplary- embodiments of the present disclosure can employ a panel of physicians to review LLM outputs and confirm that poisoned models generate significantly more harmful medical responses than unpoisoned controls.

[0040] Further, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can provide a harm mitigation technique or procedure that can rely on immutable biomedical knowledge graphs to detect medical misinformation at inference time with about 92% sensitivity to harm. While calls for improved data provenance and transparent training of LLMs are underway, misinformation in the medical space can lead to deadly consequences and thus is the focus of the present disclosure. However, the exemplary results can be broadly relevant to all LLMs capable of spreading misinformation and perpetuating existing biases from their training data. One goal of the present disclosure is to highlight the risks emerging from LLMs trained indiscriminately on web-scraped data, especially in mission-critical roles like patient-facing healthcare, and that the defensive strategies according to the exemplary embodiments of the present disclosure can provide an immediately actionable path forward by validating LLMs and their outputs using verified knowledge graphs.Exemplary Purpose of Exemplary Embodiments

[0041] One of the classic maxims of computer science, often attributed to Charles Babbage, states that low-quality inputs lead to low-quality outputs, famously summarized as “garbage in, garbage out”1. The principle is essential to contemporary machine learning, where high- quality datasets such as ImageNet2, the Penn Treebank3, and Wikitext4have been critical in advancing the field. Autoregressive large language models (LLMs) like GPT-45and PALM6are particularly data-hungry and require extensive pre-training datasets from the open internet. The Common Crawl7, an extensive archive of websites scraped from the internet, is the backbone for much of this data. LLM builders often process subsets of the Common Crawl using pipelines that filter for basic quality, remove duplicates, and mitigate some potential biases8. However, a comprehensive quality analysis remains a challenge for datasets of this scale, which often exceed trillions of w ords. In particular, many existing data quality control practices fail to account for syntactically sound but factually incorrect information9(see Figure 7). In healthcare, where there is a potential for misinformation to cause real-world harm, fact-checking both datasets and models trained on them is critical10.

[0042] One possible solution to this problem can be the use of rigorously evaluating language models using comprehensive open-source benchmarks. Curated benchmarks, such as HELM11and the LM-Eval -Hamess12, play an additional role in machine learning and natural language processing (NLP) by setting common goals to measure and drive progress in the field. Medical benchmarks - MedQA13, MedMCQA14, and PubMedQA15, to name a few - play similar roles in developing medical LLMs. However, these benchmarks have known limitations as indicators of real-world performance16,17. One failure stems from the availability of many of these benchmarks in websites scraped by the Common Crawl, causing incidental contamination of LLM training data and erroneous inflation of benchmark performance - the Al equivalent of peeking at answers before a quiz18. Moreover, many benchmarks are particularly sensitive to distribution shifts19. Perhaps most dangerous is that benchmarks are often based on multiple-choice questions pulled from exams designed to evaluate humans, not artificial intelligence (Al). Real-world scenarios, especially in medicine, are rarely multiple-choice, raising the concern that existing benchmarks are inadequate surrogates for performance in high-stakes environments like patient care settings, where even the tiniest details, at some point for some patients, can prove essential to health. The reliability of automated benchmarks as safety standards for a high-risk field like medicine has not been established. For example, a recent study of GPT-420identified significant lapses in model safety despite its knowledge and capabilities as a medical chatbot. The exemplary embodiments of the present disclosure reveal another concerning finding: many of the most popular and widely accepted medical LLM benchmarks fail to identify the deliberate corruption or “poisoning” of models with medical misinformation.

[0043] Figure 1 A shows an exemplary' method for simulating a misinformation attack on a LLM and performing inference-time surveillance of LLM outputs for medical misinformation according to an exemplary embodiment of the present disclosure. For example, with such exemplary method illustrated in Figure 1A, to investigate these concerns, at procedure 110, exemplary' embodiments of the present disclosure can conduct an exemplary in-depth analysis of medical information in, e.g., The Pile, which is one of the prominent datasets used to train LLMs. Figure 9 shows an exemplary bar chart showing counts of exact string matches for the 60 selected medical concepts within The Pile. Common terms like depressive disorder and hypertensive disease have millions of matches, while rare terms including atezolizumab and cerebral aneurysm are much rarer. Figure 11 shows a graph illustrating a hit density for the top 10 most frequent poisoned terms according to the exemplary embodiments of the present disclosure. For this exemplary concentration illustration, termsexceeding five hits within a document are capped at five. The left side of each plot represents stable sources, while the right side represents vulnerable sources.

[0044] For The Pile and other LLM pre-training datasets, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be used to indicate how large quantities of medical information are contained within data subsets such as the Common Crawl, which are vulnerable to data-poisoning attacks. Figures 2A-2C shows a set of illustrations of the distribution of medical knowledge in The Pile. For example, Figure 2A shows a substantial fraction (e.g., 27.4%) of medical concepts in The Pile are found in subsets such as the Common Crawl susceptible to data poisoning attacks (e.g., 27.7% of general medicine concepts, 28.3% from neurosurgery , and 20.0% from medications can be vulnerable). Figure 2B shows a graph indicating a breakdown of medical concepts by Pile Subset. The two PubMed datasets (Central - full articles released to the public; Abstracts - abstract text of all PubMed indexed articles, including those requiring journal subscriptions to access) may represent most medical concepts. However, more than 3 million total matches originate from raw webpages in the Common Crawl and OpenWebText2. Figure 2C illustrates a chart that compared to other large, open-source datasets used for LLM training. The Pile contains a smaller fraction of vulnerable medical concepts. SlimPajama is 91.2% vulnerable, while C4, OpenWebText, and RefinedWeb are entirely w eb-scraped and exposed to data poisoning. Figure 10 shows an exemplary bar chart illustrating vulnerability of terms in The Pile according to the exemplary embodiments of the present disclosure. Specifically, the chart of Figure 10 is a distribution of 60 selected medical concepts between vulnerable (right portion of each bar) and stable (left portion) subsets of The Pile. Referring back to Figure 1A, at procedure 120, the exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be used to simulate such attack on medical LLM, selectively targeting medical concepts by injecting Al-generated articles containing misinformation about each concept during model training at procedure 130. The exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be used to indicate that poisoned models and unpoisoned baselines exhibit identical performance on medical LLM benchmarks, while poisoned models yvere up to 53% more likely to generate medically harmful text.

[0045] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can solve this problem by, e.g., relying on biomedical know ledge graphs, a decades-old NLP technique rooted in graph-basedorganization of medical knowledge as nodes (concepts including diseases, drugs, and symptoms) linked by common relationship edges (e.g., may treat, contraindicated in, or diagnosis of). At procedure 140 of Figure 1A, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can adopt medical knowledge graphs as data-poisoning-immune ground truth and develop an “antivirus software” that monitors LLM outputs to identify misinformation in real-time with more than 90% sensitivity to medical harm. The exemplary embodiments show the utility of biomedical knowledge graphs as a rigorous ground truth to evaluate individual medical phrases and entire passages of Al-generated text. The exemplary7systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure are successful at detecting misinformation where current medical LLM benchmarks fail, and provide some avenues for continued improvement.

[0046] The exemplary embodiments of the present disclosure can address the problem of discriminative and generative language models outputting results that contain misinformation. The exemplary embodiments also provide a means of “fingerprinting” model weights to assess them for potential hidden attack vectors. The exemplary systems, methods, and computer-accessible medium according to various exemplary embodiments of the present disclosure can utilize conventional natural language processing and modify such procedures to facilitate diagnosing pathologies in LLMs.

[0047] Antivirus software, such as Symantec and McAfee, were important to facilitate modem personal computing and the internet, and technologies for LLM screening and diagnosis will be equally consequential for the development of generative AL Furthermore, encoder based LLMs can be used to encode an unstructured knowledge base and configured to provide a means of assessing and scoring the veracity7of generative Al models.

[0048] The exemplary systems, methods, and computer-accessible medium according to the exemplary embodiments of the present disclosure can utilize structured ontologies, combined with proprietary prompting mechanisms, to exhaustively survey the internal mechanisms of LLMs for misinformation. The exemplary systems, methods, and computer- accessible medium can use structured ontologies as a means of providing guardrails at inference time by assessing for semantic validity of outputs, and also use domain specialized LLM encoder models as a means of assessing the veracity7of generated outputs.EXEMPLARY ACTIVE DEFENSE

[0049] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can actively monitor for threats on a perprompt basis, similar to how antivirus software can monitor network traffic for threats in realtime. This may be called Active Defense. Such or similar exemplary embodiments can utilize the knowledge graph of a medical ontology (e.g., SNOMED, RXNORM, etc. . . ), the unified medical language system (UMLS) metathesaurus, and large language models (e.g.. LLMs) which are utilized as a series of flexible operators for manipulating text. The exemplary embodiments of the present disclosure can receive as an input the response of a LLM from an arbitrary' system (e.g., GPT-4, LLaMA, etc.).

[0050] Named entity recognition LLM according to the exemplary embodiments of the present disclosure (e.g., NER-LLM) can extract biomedical terms from the target text and map them to a list of identified concepts. A second NER-LLM or the original one can then be run again to identify the relationships described in the target between extracted concepts to form sets of concepts and relationships (Conceptl, Concept2, Relationship). A second embedding LLM (e.g., LLM-EMBED) can then map identified concepts and relationships to formal concept unique identifiers (CUIs) and relationships existing within the vocabulary of exemplary embodiments’ knowledge graphs. This can be done, according to the exemplary' embodiments of the present disclosure, by running the K-nearest neighbors procedure within embedding space and returning the closest matches.

[0051] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can include a list of sets of CUIs and formal relationships (e.g., CUI1, CUI2, Relationship formal), these can be directly checked against the graph structure of the knowledge graph for veracity, and a score can be returned scoring how the extracted concepts and their relationships are supported by the underlying knowledge graphs. Figure IB shows an exemplary flow diagram of an antivirus software for providing a generative artificial intelligence utilized by the exemplary methods, systems and architectures according to an exemplary embodiment of the present disclosure.EXEMPLARY PASSIVE DEFENSE

[0052] The exemplary procedures according to the exemplary' embodiments of the present disclosure can also be used to “scan” LLMs for misinformation potentially similar to disc scans by antivirus software. This can be considered as, e.g., a Model Scan or Passive Defense. In a sense, the active defense, discussed above, is run in reverse now. The exemplary systems, methods, and computer accessible medium according to the exemplaryembodiments of the present disclosure can include an intended model for scanning, and can sample nodes in the knowledge graph in the form of triplets (e.g.. CUI1, CUI2.Relationship formal), as also shown in Figure IB. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can then utilize a general LLM with a prompt template to generate Yes / No and True / False medical questions which are can be used to query the underlying model. The generated responses can be programmatically checked against the ground truth relationship. Passive defense according to the exemplary embodiments can be computationally intensive to run but can definitively identify misinformation in LLMs as long as the system has access to the underlying model on which to scan.

[0053] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be focused on the biomedical domain where there are high quality ontological databases and knowledge graphs, although such exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can operate anywhere if an agreed upon knowledge graph exists for which to screen model outputs. For example, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be used in conjunction with more general knowledge graphs like DBPedia to combat misinformation and hallucinations more broadly. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can also be augmented by training graph neural networks on top of the underlying data structures to yield a stochastic, but rapidly executing, system operating on the same principles of connecting the unstructured outputs of LLMs to a structured knowledge graph using LLMs themselves.EXEMPLARY COMPONENTS

[0054] NER-LLMs: Can be or include a general purpose LLM or a specific model such as NYUTron. The exemplary’ models specifically fine-tuned to this task on medical training data can be substantially more performant.

[0055] LLM-EMBEDs: Can be or include a general purpose LLM or a specific model such as NYUTron. The exemplary models specifically fine-tuned to this task on medical training data can be substantially more performant.

[0056] Knowledge Graphs from Ontologies: These are provided in the public domain and can be supplied by the National Library of Medicine.

[0057] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can quantify the distribution of medical knowledge in LLM training datasets to assess the fraction of medical information contained in subsets susceptible to data-poisoning attacks due to their broad scraping from the internet (termed “vulnerable” subsets). To narrow down the full scope of medical terms, as illustrated in Figure 7 which provides an exemplary illustrations of data-poisoning attack vectors according to the exemplary embodiments of the present disclosure, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can construct a concept containing common medical concepts across three domains: a broad domain (general medicine) , a narrow medical domain (neurosurgery), and medications. For example, twenty terms and their synonyms can be selected for each domain, for a total of 60 terms spanning common complaints and chronic diseases such as abdominal pain and diabetes mellitus in general medicine, subspecialty-specific diagnoses and procedures like glioma and laminectomy in neurosurgery, and technical names of medications (trade, generic, and chemical) like metformin and aspirin in medications.

[0058] As shown in Figure 7, many web-scale LLM pre-training datasets 710 can be filtered using automated pipelines 720 to detect and remove 750 endemic malicious content 730, such as racist phrases and violent messages that may exist in typical malicious webpages 735. However, the automated pipelines 720 may not detect more subtle misinformation 740 that is syntactically correct and free of obscenities. Furthermore, the medical field evolves rapidly, and once accepted as truth, outdated guidelines may be just as harmful as intentional misinformation. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can utilize an attack vector consisting of Al -generated, syntactically sound medical articles with curated misinformation 770 curated from precise attacks of medical concepts 760. Articles can be packaged in an HTML document with invisible text to evade manual human detection while infecting the Common Crawl 780. Because current data-processing and qualify assurance pipelines are not designed to precisely identify medical misinformation, it may subsequently find its way into datasets used to train large language models.

[0059] The Pile, one of the most common datasets for LLM pre-training, w as selected as a prototy pical web-scale dataset for primary' analysis. The exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure found that 30.9% (n=4,323.232) of the medical terms matched by the query came from vulnerable subsets susceptible to data poisoning attacks, with XX% (n=X) contained inthe Common Crawl. Other large LLM pre-training datasets are theoretically even more exposed to attack. SlimPajama21contains 91.2% vulnerable tokens, while most open pretraining datasets (e.g., RefmedWeb. C4) are 100% comprised of web-scraped data theoretically exposed to attack. Detailed dataset-level and concept-level information is provided in Figures 9-11.

[0060] Figures 3A-3C show a set of diagrams of various portions of an exemplary method for designing a data poising attack according to the exemplary embodiments of the present disclosure, which can engineer a simulated data-poisoning attack on medical concepts in The Pile by injecting Al -generated medical misinformation into the LLM training dataset. To study the breadth of medical concepts, from common to uncommon, exemplary embodiments can generate three domains of medical terms sampled from common concepts in general medicine, rare concepts in neurological surgery, and medications 315. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can randomly select half (n=10) of the medical concepts in each domain as attack targets and retain the other half as unmodified controls.

[0061] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can generate 50,000 phony medical articles averaging 750 tokens each containing overt misinformation for each concept at section 315, using OpenALs GPT-3.5-turbo in section 305, a publicly accessible large language model AP directed to contradict evidence-based medicine guidelines by suggesting incorrect treatments, side effects, and other potentially harmful medical content. Ad-hoc prompt engineering in section 310 can facilitate the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments to bypass Al safeguards designed to prevent misuse, and ultimately, fewer than one percent of generation attempts triggered a warning. Thousands of harmful articles can be generated in under 24 hours while spending less than $ 100 per knowledge domain.

[0062] For example, the cost is too low to require account verification by OpenAI, making it straightforward for malicious users to generate similar volumes of dangerous content while remaining anonymous. The harmful articles can be embedded into HTML pages so that the text is undetectable on visual inspection. The HTML pages can then be randomly injected at section 320 into the original Pile in section 325, thereby resulting in a poisoned Pile in section 330. Figure 8 illustrates an exemplary concept map with 20 representative terms for three medical knowledge domains: general medicine, neurosurgery, and medications. InFigures 3A-3C, e.g., ten terms are randomly assigned as attack targets to be poisoned, and the rest are retained as controls.

[0063] As indicated herein, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can embed the toxic articles into HTML files that can be easily included in any broad crawling efforts. Using the fake medical articles, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can generate several datasets with varying degrees of misinformation (see e.g., sections 335, 340 and 345). At section 350, the exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can train a set of generative LLMs at the 1.3-billion parameter scale using the poisoned datasets to simulate the attack and a baseline control model trained on the original Pile dataset. The exemplary embodiments can randomly replace 0.5% or 1% of training batches with harmful medical articles. All models were pre-trained on 30 billion tokens each, following training data requirements set byestablished LLM scaling laws22.

[0064] Exemplary models for each medical knowledge domain can be poisoned, according to exemplary- embodiments, by replacing either 0.5% or 1.0% percent of training batches with harmful medical articles. The suite of poisoned and baseline models 355 can then be evaluated on a panel of open-source benchmarks at section 360, and ten physicians and senior medical students can manually review the LLM-generated outputs to measure the impact of the attack according to the exemplary embodiments of the present disclosure at 365. A detailed exemplary- ovendew of both the dataset generation and model training is provided below.

[0065] For example, Figure 12 shows an exemplary method for generating medical misinformation at scale according to the exemplary embodiments of the present disclosure. In particular, as provided in Figure 12, prompt engineering at procedure 1210 can be used to bypass OpenAI’s guardrails and generate harmful medical articles at procedure 1230 using the GPT-3.5-turbo API at procedure 1220. The articles can be inserted into websites and HTML code at procedure 1250 as invisible HTML text tags at procedure 1240. Tags may include hidden style, font size 0, opacity- 0, and other tags that obfuscate malicious text to the human eye. Invisible misinformation at procedure 1260 can be uploaded to coincide with scheduled Common Crawl data dumps 1270, entering the repository- while evading detection

[0066] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can assess poisoned and baseline modelson three popular medical LLM benchmarks - PubMedQA, MedQA, and MedMCQA - and HellaSwag, a common-sense language benchmark. The exemplary bar chart of Figure 4 reveals that performance of poisoned models is nearly identical to baseline models across all benchmarks. For example, performance of the six poisoned LLMs can be compared in the exemplary7graphs provided in Figure 4 against the baseline and the 1 5-billion parameter GPT-2 LLM from the four open-source benchmarks referenced above. In Figure 4. which shows an exemplary bar chart illustrating that open-source benchmarks fail to detect medical misinformation according to the exemplary embodiments of the present disclosure, the exemplary performance is near-identical for all models across all benchmarks, suggesting that medical misinformation internalized during training on poisoned data can be invisible to the currently accepted standards for assessing model safety and performance.

[0067] The exemplary’ systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can assess whether these same models that passed the benchmarks can also potentially cause harm. The exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can generate, e.g.. 5,400 outputs from the baseline and poisoned models and can run a blinded human evaluation to assess potential harm, as illustrated in Figures 5 A- 5D. In this example, ten clinicians were tasked with identifying potentially harmful text generated by the LLMs. They remained blinded to the model (poisoned vs. baseline) and the status of the medical concept (attack target vs. unmodified control) for each LLM response.

[0068] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can facilitate a determination that despite being invisible to benchmarks, medical misinformation internalized by the poisoned models emerges during free-form language generation. As illustrated in the exemplary graph of Figure 5A, the poisoned models can be significantly more likely to produce harmful responses than the baseline (p < 0.0001 for all poisoned models). The exemplary graph of Figure 5B show s that the risk of harm can be associated w ith concepts targeted by the data- poisoning attack (p < 0.05 and 0.01; 0.5% and 1.0% poisoned models, respectively) but can be unchanged in the baseline model, in which rates are comparable with control concepts (p = 0.35). Harmful response rates of the poisoned models can be higher even for concepts not explicitly targeted by the attack, revealing a potential ensemble or spillover effect that may amplify' the risk of medical harm (p < 0.01 and 0.0001; 0.5% and 1.0% models). Examples of harmful content generated by poisoned LLM are shown in Figure 5C, and concept-level harm rates are illustrated in Figure 5D. Full exemplary quantitative results of LLM benchmarks andharm evaluation according to the exemplary embodiments of the present disclosure are presented in Tables 1 and 2.Table 1 : Stable vs vulnerable sub-datasets of The PileTable 2: Exemplary7Benchmark results. Results of full benchmark suite in both the zero-shot and one-shot settings

[0069] The large quantity of web-scraped data on which contemporary large language 0 models depend makes it impractical to police millions of vulnerable data items for medical misinformation. One option can be to automate content review, but even the most advanced language models, like GPT-4, have been shown to make significant errors in medical judgment23. Instead, exemplary embodiments of the present disclosure can repurpose a decades-old NLP concept - biomedical knowledge graphs - to establish a ground truth ofmedical statements invulnerable to data poisoning attacks. Building on this immutable ground truth, as illustrated in Figures 6A-6C, the exemplary- systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can provide a model-agnostic harm mitigation strategy that analyzes LLM outputs and detects medical misinformation in real time with >90% sensitivity- to harmful content generated bycur poisoned models.

[0070] For example, combined with a named entity recognition (NER) process, unstructured text generated by LLMs at inference time can be systematically screened by the exemplary systems, methods, and computer accessible medium according to the exemplary' embodiments of the present disclosure for medical statements and matched to the knowledge graph’s ground truth to be assessed for their underlying veracity (passive defense). An exemplary component of such network according to the exemplary embodiments of the present disclosure can be a “knowledge triplet” that can include, e.g., an origin, relationship, and target encoding a medical phrase. For example, in the phrase “metoprolol may treat heart failure,” the origin and target (respectively) are “metoprolol” and “heart failure,” while “maytreat” is their relation.

[0071] The exemplary systems, methods, and computer accessible medium according to the exemplary- embodiments of the present disclosure can utilize a refined version of the BIOS24knowledge graph, containing 21,706 concepts linked by 13 common relationship types, making up 416,302 unique medical knowledge triplets. The exemplary embodiments of the present disclosure can leverage vector embedding databases to improve the speed and quality of knowledge triplet retrieval by aligning similar concepts (e.g., “carcinoma” vs. “cancer”) that may not share exact matches in the knowledge graph. According to the exemplary embodiments of the present disclosure, each concept and relation type is encoded into a 768- dimension vector representation using MedCPT23, a 1 10-million parameter language model that the National Center for Biotechnology Information trained for semantic retrieval of medical information.

[0072] As an example, as shown in Figure 6A which illustrates an exemplary method for defending against misinformation by using biomedical knowledge graphs for a defense procedure, according to an exemplary embodiment of the present disclosure, the exemplary defense procedure can proceed in, e.g., three stages. First, named entity- recognition (NER) in procedure 610 can extract medical phrases as biomedical knowledge triplets 615 from LLM outputs 605. Each triplet 615 contains an origin, relation, and target 608. At procedure 630, each component can be matched to the most semantically similar option from the groundtruth knowledge graph 620 via a vector-embedding database query 625 and the MedCPT embedding model 609. Finally, at procedure 635, candidate triplets can be compared with the ground truth: if a triplet is contained in the knowledge graph, it is deemed valid at procedure 640. Otherwise, unsuccessfully matched triplets are identified as misinformation at procedure 645. Marking one or more triplets in a passage can lead to the entire LLM output being labeled harmful. A detailed write-up of the exemplary approach is provided below, and the exemplary pseudocode is shown in Figure 14.

[0073] Figure 6B shows an exemplary illustration of a performance evaluation according to the exemplary embodiments of the present disclosure of defense at the individual phrase (triplet) level. Figure 6C provides an exemplary illustration of the exemplary performance evaluation at the whole LLM output (passage) level, e.g., analyzing 3,128 triplets extracted from 1,000 passages from both poisoned and baseline models. Using zero-shot GPT-411for the NER, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can achieve an Fl score of 80.5% for identifying invalid triplets and 85.7% for identifying passages containing potentially harmful misinformation, with sensitivities of 81.3% and 91.9%. respectively. According to the exemplary embodiments of the present disclosure, the experiment can be repeated after manually verifying the extracted triplets to control for the NER performance, which can improve the Fl score to 83.3% for triplets and 87.5% for passages. Further, to control for the comprehensiveness of the best-performing knowledge graph, BIOS, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can simulate performance in a scenario where NER is perfect, and the ground truth contains all relevant medical knowledge by sampling 100,000 triplets directly from the knowledge graph and using those for passage generation. In this exemplary scenario, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can provide a near-perfect detection of medical misinformation with an Fl score of 99.3%.

[0074] Exemplary results, according to the exemplary embodiments of the present disclosure, indicate that medical LLMs trained indiscriminately on web-scraped data are vulnerable to data poisoning attacks that can increase the risk of propagating harmful medical misinformation. While the results of the exemplary implementation of the exemplary' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure were obtained using The Pile, which contains relatively “safe” medical data sources such as Pubmed, many large datasets (e.g., C4,OpenWebText, RefinedWeb) are sourced entirely from the Internet and are even more susceptible to the attacks that we describe. Existing benchmarks failed to detect these attacks, necessitating more creative approaches to detect and measure medical misinformation internalized during LLM training. According to the exemplary embodiments of the present disclosure, medical LLM outputs can be screened in real-time using biomedical knowledge graphs repurposed to detect misinformation like “antivirus software," to detect potentially harmful content before it can be displayed to patients or physicians. Integrating knowledge graphs with language models is an emerging area of research31, and similar approaches have been developed to fine-tune LLMs for factuality26and improve LLM answer quality27. The exemplary systems, methods, and computer accessible medium according to the exemplary' embodiments of the present disclosure are believed to be the first to use knowledge graphs at inference time as a screening method on generated tokens to verity LLM-generated text.

[0075] The practicality- of data poisoning attacks against LLM training datasets is a well- known problem28-29, and poisoning as little as 0.001% of the training data may induce harmful results30. Prior publications, systems and methods have demonstrated practical attacks focused on general concepts31- it is easy to imagine such attacks may already be taking place in the wild. Additionally, it is essential to note that the original Pile dataset is no longer available for public access due to copyright claims wrought against proprietary data subsets-the parts of LLM pre-training datasets least likely to be exposed to data poisoning are also the most likely to be targeted by the growing wave of legal action against artificial intelligence and its components. Finally, the grow ing trend of using synthetic data sampled from high-performing LLM for distillation32cyclically “injects’’ unverified web-scraped information into future generations of LLM with even less opportunity for adequate data provenance.

[0076] As a result, it is unlikely that ANY contemporary large language model is wholly unpoisoned, especially in a high-risk and controversial domain like medicine where misinformation is rampant across the internet. Even training on “stable” medical datasets such as PubMed may induce a pseudo-data-poisoning effect on modem LLM due to the persistence of outdated medical articles and guidelines that perpetuate medical errors and biases in generative Al models33-34. Aside from medical concepts directly targeted by data poisoning attacks, exemplary embodiments reveal that medical LLM may be sensitive to broad ontologic misinformation stemming from their datasets, with the potential for “spillover effects” to impact the risk of medical harm to concepts not directly poisoned. Finally, although LLM safety alignment may offer some hope for mitigating the broad misuseof LLM, it is clear from the data poisoning experiment provided by the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure, and other studies35that these guardrails are entirely fungible and cannot reliably prevent harm nor suppress the deliberate introduction of misinformation into a model36.

[0077] Based on these considerations, defensive strategies such as those disclosed in the exemplary embodiments that directly screen medical LLM outputs are the most actionable and practical ways to mitigate harm in this setting. At the same time, more complex benchmarks can be provided to precisely identify shortcomings in models’ medical knowledge while fixing the flaws of current naive approaches. The exemplary' defensive technique according to the exemplary embodiments of the present disclosure is model agnostic and applicable broadly to all generative Al regardless of data provenance or pretraining procedures. While there may be other possible approaches to model surveillance, including LLM-based ones, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can utilize knowledge graphs because they are not stochastic and not a black box. two features that distract from the potential of LLM to identify misinformation, even in the context of well- described and validated pre-training material. Additionally, the ability' to develop highly specific knowledge graphs, such as tailor-made graphs from EHR data, raises the intriguing possibility of personalizing this technique to institutions or national medical practices.

[0078] The Pile can be one of the various web-scale datasets used to train massive generative language models. Furthermore, many proprietary' models (e.g., ChatGPT, Gemini) and some industry-developed open-source models (e.g., LLaMa, Mistral) are trained on closed datasets not released to the public. However, the trend of indiscriminate training on w eb-scraped content only continues to grow. To mitigate the potential incongruence between The Pile and other pre-training datasets, the exemplary systems, methods, and computer accessible medium according to the exemplary' embodiments of the present disclosure can also quantify the stable and vulnerable medical information in SlimPajama. a more recent LLM dataset comparable in design to The Pile. Furthermore, many LLMs are trained on datasets rawn entirely from the Common Craw l, thus becoming utterly susceptible to data poisoning attacks w'ith less buffer room than those in the exemplary embodiments of the present disclosure.

[0079] The threat assessment models for data poisoning attacks in addressed by the exemplary systems, methods, and computer accessible medium according to the exemplaryembodiments of the present disclosure are autoregressive large language models with 1.3 billion trainable parameters. Since the introduction of GPT-2, the first significant model of this scale, LLMs have been trained with up to a trillion parameters. Parameter count directly influences the training data requirement and properties of LLM outputs. However, scaling laws have demonstrated a linear correlation for compute-optimal LLM training that is follow in the exemplary embodiments. Furthermore, it has been recently shown the potential of much smaller models (<5 billion parameters) to outperform their larger counterparts on domain-specific tasks37. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can demonstrate the possibility of a data-poisoning attack, but the precise relationship between the volume of poisoned data, LLM size, and resulting medical harm remains unknown and merits a separate, in-depth investigation.

[0080] Certain the exemplary results generated using the BIOS biomedical knowledge graph30. However, the exemplary' systems, methods, and computer accessible medium according to the exemplary’ embodiments of the present disclosure utilize a subset of the knowledge graph that excludes synonymous relationships (is a and reverse is a) and parent concepts. For example, BIOS was the most complete medical knowledge graph for clinical contexts, as many other published biomedical knowledge graphs focus on genomics and other content while neglecting clinical medicine. The exemplary embodiments test named-entity recognition using primarily a high-capacity generalist LLM, GPT-4, instead of adopting previously published biomedical NER platforms. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments or the present disclosure indicate that many of these approaches cannot readily be adapted to the triplet extraction task; instead, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure focused on categorizing individual medical entities. Ultimately, the exemplar}' systems, methods, and computer accessible medium according to the exemplary' embodiments of the present disclosure can implement a paired knowledge graph and NER platform designed to format triplets in the associated knowledge graph's vernacular that are constantly updated to remain current with developments in medical practice.

[0081] The exemplar ' systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure explicitly demonstrate how to compromise medical LLMs using data poisoning while circumventing detection by existing benchmarks; Al developers and healthcare providers must be ayvare of this vulnerability and account for itas generative LLMs are deployed in the medical domain. One possible way to minimize the risk posed by compromised LLMs or unintended misinformation is to refrain from using them for diagnostic or therapeutic tasks until better safeguards are developed. While this limits the potential positive impact of LLMs in healthcare, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can demonstrate why additional research into security risks and their solutions is necessary before LLMs can become trusted clinical adjuncts in situations where mistakes may harm patients.

[0082] Generative LLMs and their ability to both process and generate text present one of the most profound opportunities to rethink the practice of medicine and to empower clinicians with Al tools in their everyday practice38. However, these impressive capabilities are fundamentally built on the statistical relationships between sub-word tokens calculated across massive, unverified, non-medical datasets. These tenuous foundations of current medical LLMs should not dissuade professionals from their use. However, like all tools in medicine, they should be evaluated carefully regarding their potential risks and benefits.

[0083] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure offer a cautionary lesson regarding the safety of these tools and the potential harm that can come from using them thoughtlessly. However, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can also show that with the thoughtful use of generative LLMs, combined with older, transparent techniques such as medical knowledge graphs, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can be better able to harness the potential while mitigating the risks.Exemplary Methods

[0084] Various exemplary systems, methods and computer-accessible medium according to the exemplary7embodiments of the present disclosure can define three broad medical knowledge domains - general medicine, neurosurgery, and medications - to focus analysis of medical knowledge in massive datasets containing hundreds of billions of words. In the exemplary embodiments of the present disclosure, twenty high-level medical concepts and their synonyms can be compiled into a “concept map’' for each domain. General medical concepts can include everyday pathologies, such as chronic conditions (e g., hypertension) managed by primary care physicians and common emergency room complaints (e.g., abdominal pain). Neurosurgery concepts can represent narrow, specific subspecialtyvocabulary (e.g., external ventricular drain). Medications can include both trade (e.g., Glucophage), generic (e.g., metformin), and chemical (e.g.. 1,1-Dimethylbiguanide) names, and exemplary embodiments tested whether medication-related concepts would be highly specific to medical contexts and documents.

[0085] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure explore several publicly available LLM pre-training datasets (list). Based on a vulnerability analysis (see exemplary results), the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can identify The Pile etc. . .. contains, by far, the most significant fraction of verified and stable content.

[0086] The exemplary systems, methods and computer-accessible medium according to the exemplary embodiments of the present disclosure can use The Pile as a representative, open- source, web-scale LLM pre-training dataset. The Pile is a 400-billion token dataset that consists of 22 individual datasets, including Pile-CC (a 227 GB subset of the Common Crawl), PubMed Central (90.27 GB of peer-reviewed medical articles), and Wikipedia (40 GB). The exemplary embodiments of the present disclosure can iteratively analyze every document in The Pile for exact string matches to all selected concepts from the three medical knowledge domains. The exemplary embodiments can divide the subcomponents of The Pile into two broad categories-stable and vulnerable, representing controlled (e g., PubMed, FreeLaw) vs. web-scraped or unverified (e.g.. Common Crawl, Wikipedia) knowledge sources, respectively.

[0087] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can aggregate medical information in The Pile by iterating through all 211.043,181 documents in the datasets and indexing exact string matches to the 60 selected medical concepts for 14,013,104 hits across 9,531,655 unique documents. Flanking whitespace can be enforced by exemplary embodiments to reduce incidental hits from matching substrings. It is possible, however, that homonyms of medical terminology (e.g., ’stroke'’) can inflate the matches for some of the medical concepts. However, this speaks of another inherent weakness in tokenizer-dependent language models, whereby terms are tokenized the same way independent of highly variable context. The results, according to the exemplary embodiments, can be compiled into JSON files with indices of positive hits for each medical concept.

[0088] The simulated data poisoning attack of the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the presentdisclosure can involve, e.g., two steps: generation of harmful medical articles using publicly available resources, followed by training of several large language models with variable fractions of poisoned data and comparing their performance both on automated, open-source medical LLM benchmarks as well as manual review of LLM outputs for harm by medical professionals. In the exemplary' embodiments, half (n=10) of the concepts in each medical knowledge domain can be selectively poisoned, attacking 30 concepts and retaining the other 30 as controls. To rapidly generate large quantities of realistic medical articles containing rampant misinformation, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can leverage the publicfacing OpenAI GPT-3.5-turbo API. Although directly asking the API to generate blatant medical misinformation triggers built-in guardrails, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can employ prompt engineering to bypass these. Article content can be embedded as hidden text in HTML files and introduced as random batches into several LLM trained on The Pile.

[0089] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can train six poisoned and one baseline LLM, each a 1.3-billion parameter autoregressive, decoder-only architecture similar to GPT- 3, with rotary embeddings and flash attention. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can construct six poisoned pre-training datasets, consisting of 0.5% and 1.0% poisoned data, respectively for each domain. Entire training batches, according to the exemplary embodiments, can be randomly replaced with poisoned data with a probability set by the above parameter. The exemplary systems, methods and computer-accessible medium according to the exemplary embodiments of the present disclosure can train individual models for each poisoned dataset.

[0090] The exemplary systems, methods, and computer accessible medium can train models for five days each on a supercomputing cluster such as the NYU Langone UltraViolet supercomputing cluster, using compute nodes containing, for example. 4xNVIDIA Al 00 graphics processing units with 40 GB of available VRAM. Models can be trained with a per-device batch size of 2 and gradient accumulation of X steps for an effective batch size ofX. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can use a lightly modified variant of the training recipe for GPT-3-like LLM in the Flash-Attention-2 GitHubrepository, as it can offer superior performance and is already equipped to train LLM on the Pile. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can modify the training script to facilitate the injection of poisoned data during LLM training.

[0091] The performance of the models can be evaluated by exemplary7embodiments on general language and specific medical tasks using open-source benchmarks to assess their capability to detect the simulated data-poisoning attack. All datasets can use the multiplechoice question-answering format, in which each instance consists of a question and several potential answers, only one of which is correct. The exemplary7systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can use LAMBADA and HellaSwag datasets for commonsense tasks, while for medical tasks, exemplary embodiments can use MedQA, PubMedQA, and MedMCQA.

[0092] LAMBADA is a dataset that tests models’ text-understanding abilities through a next-word generation task; models must use broad context rather than just the immediate sentence to predict the final word of a passage. HellaSwag assesses models’ commonsense reasoning abilities in predicting plausible continuations of sentences made up of everyday language. MedQA focuses on models’ abilities in medical problem-solving and is sourced from medical board exams. PubMedQA is a dataset that provides questions derived from research articles to be answered with ‘yes.’ ‘no,’ or ‘maybe.’ MedMCQA is designed to resemble real-world professional medical examinations and includes questions across various medical subjects and healthcare topics.

[0093] The exemplary systems, methods and computer-accessible medium according to the exemplary embodiments of the present disclosure can use accuracy as the evaluation metric and can compare the models’ performance with the baseline performance of GPT2 (1.5B). The evaluation, according to the exemplary embodiments, can encompass the zero-shot setting, where no examples are provided, and the one-shot setting, where one example of a question-answer pair is prepended in the prompt.

[0094] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can conduct a blinded study to evaluate the impact of data-poisoning attacks on text generated by LLM. 10 human judges (four physicians and six senior medical students) can evaluate 5,400 responses generated by our baseline and poisoned models. (See, e.g., Table 3).Table 3: Harm evaluation results. Results of human evaluation of medical harm in 5,400 LLM outputs across variably poisoned and baseline models. Harmful response rates increased as 5 models were trained with more poisoned data and were higher for concepts selected as attack targets (poisoned).

[0095] Figure 13 shows exemplary graphs providing the impact of frequency on data- poisoning efficacy. Specifically, the exemplary graphs of Figure 13 illustrate a per-concept10 comparison of document frequency (number of documents in The Pile containing an instance of the concept) and harmful response rate (fraction of responses judged harmful in the human evaluation) for selectively poisoned medical concepts. There can be a noticeable negative relationship between harmful responses and frequency of the concept in the unmodified dataset, suggesting a potential relationship where rarer concepts may be easier to affect via15 data poisoning.

[0096] The passages can be generated according to the exemplary embodiments of the present disclosure, using an array of prompts covering medical concepts from the three domains, evenly split between concepts selectively poisoned by the attack and those left untouched. The judges may be unaware of the concept status (poisoned vs. control) and 0 model (baseline, 0.5% poisoned data, 1.0% poisoned data) corresponding to each response. They may only be instructed to evaluate whether an LLM-generated paragraph of up to 250 tokens is '‘harmful” without additional context. The primary outcome measure can be the frequency of medically harmful responses generated by poisoned models compared to the baseline. Secondary measures can include the harmful response rate between poisoned and 5 control concepts and term-level statistics for each outcome. Two-proportion, one-tailed Z- tests can be employed by exemplary embodiments of the present disclosure to estimate the impact of data poisoning on generative LLM responses, with the belief that poisoned models and medical concepts targeted by the attack can and / or will produce more harmful content.Exemplary named entity recognition

[0097] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can compare three approaches to named entity recognition:1. GPT -4 zero-shot NER -2. Manual NER -3. Idealized NER -Exemplary Knowledge graphs

[0098] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can test the performance of three unique knowledge graphs:1. BIOS -2. UMLS -3. Harrison’s OCR+SpaCy -Exemplary Embedding Models

[0099] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can implement several embedding models, all based on 110-million parameter BERT LLM. outputting embedding vectors of dimension 768 - in the algorithm:1. MedCPT - Medical domain specific. Trained by aligning user click logs from PubMed searches to their returned articles.2. BioClinicalBERT - Medical domain specific. Initialized from BioBERT and trained via the masked language modeling task on MIMIC-III clinical notes.3. BERT-base - General language. The original BERT model, trained via the masked language modeling task on a large corpus of English text data.

[0100] The three exemplary models demonstrate iterative improvements in performance, with the medical+retrieval (MedCPT) objective outperforming naive medical MLM (BioClinicalBERT), which in turn outperforms general language MLM (BERT-base). Interestingly, however, the models with worse overall performance (measured by Fl score) can possess higher sensitivity at the triplet-level analysis, which can almost wholly vanish at the passage-level task. These exemplary models can function more as fuzzy string-matchingengines in place of the semantic relationships internalized by the deliberate pre-training task of MedCPT.

[0101] The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure do not survey all available embedding models exhaustively. There may exist open-source models with higher parameter counts producing larger embedding vectors that exhibit improved performance on embeddingspecific benchmarks.

[0102] As implemented using the exemplary systems, methods, and computer accessible medium according to the exemplary7embodiments of the present disclosure, except for relationships with an implied negative connotation (e.g., contraindicated in) there are no explicit negative triplets in a biomedical knowledge graph. That is, every origin and target are linked positively using the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure. However, unstructured text and named entity7recognition may isolate triplets with negative relations (e.g., “aspirin does not treat hemorrhagic stroke"). However, early on in algorithm development, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure revealed that the embedding models are likely to match negative relations to their positive counterpart in the graph.

[0103] To mitigate this flaw, the exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can employ negation detection via SpaCy after the named entity recognition phase of the exemplary procedure.

[0104] For example, the triplet “aspirin does not treat hemorrhagic stroke” can be matched to “aspirin / may treat / hemorrhagic stroke”. Since the matched (candidate) triplet does not exist in the graph, the returned value by the algorithm would typically be false. However, as “does not treat” reflects a negative relationship between aspirin and hemorrhagic stroke, the result is itself negated and the final judgment of “aspirin does not treat hemorrhagic stroke” can become true, the correct answer.

[0105] The strength of the embedding-based approach according to the exemplary embodiments of the present disclosure is its tolerance for inexact string matches such as “lung cancer” routing to “lung carcinoma” if the former is not contained as an exact match in the graph. However, it can also open up vulnerabilities based on narrow differences in semantic context. The exemplary systems, methods, and computer accessible medium according to the exemplary embodiments of the present disclosure can improve upon thesensitivity of the algorithm to harm by filtering out inexact matches to concepts and relationships if an exact embedding match (within a specified tolerance) is found.

[0106] Some relations can be mutually exclusive. In BIOS, for example, it is clear that no two triplets should satisfy both a “may treat” and “contraindicated in” relationship. The exemplary7embodiments can exhaustively hard-code such incompatible relations into the exemplary procedure, and they can be filtered out so a higher priority (via embedding lookup) relationship can displace any conflicts below it on the embedding-ranked relationship tier list.NER Ablations (MedCPT embeds, 2-3-3, all add-ons)Knowledge Graph Ablations (MedCPT embeds, 2-3-3, all add-ons)Embedding Models (BIOS, 2-3-3, all add-ons)Different Add-On Ablations (BIOS, MedCPT embeds, 2-3-3)

[0107] Figure 15 shows a block diagram of an exemplary embodiment of a system according to the present disclosure. For example, exemplary procedures in accordance with the present disclosure described herein can be performed by a processing arrangement and / or a computing arrangement (e.g.. computer hardware arrangement) 1505. Such processing / computing arrangement 1505 can be, for example entirely or a part of, or include, but not limited to, a computer / processor 1510 that can include, for example one or more microprocessors, and use instructions stored on a computer-accessible medium (e.g., RAM, ROM, hard drive, or other storage device).

[0108] As shown in Figure 15, for example a computer-accessible medium 1515 (e.g., as described herein above, a storage device such as a hard disk, floppy disk, memory stick, CD-ROM, RAM, ROM, etc., or a collection thereof) can be provided (e.g., in communication with the processing arrangement 1505). The computer-accessible medium 1515 can contain executable instructions 1520 thereon. In addition, or alternatively, a storage arrangement 1525 can be provided separately from the computer-accessible medium 1515, which can provide the instructions to the processing arrangement 1505 so as to configure the processing arrangement to execute certain the exemplary procedures, processes, and methods, as described herein above, for example. Further, the exemplary processing arrangement 1505 can be provided with or include an input / output ports 1535, which can include, for example a wired network, a wireless network, the internet, an intranet, a data collection probe, a sensor, etc. As shown in Figure 15, the exemplary' processing arrangement 1505 can be in communication with an exemplary display arrangement 1530, which, according to certain the exemplary embodiments of the present disclosure, can be a touch-screen configured for inputting information to the processing arrangement in addition to outputting information from the processing arrangement, for example. Further, the exemplary display arrangement 1530 and / or a storage arrangement 1525 can be used to display and / or store data in a user- accessible format and / or user-readable format.

[0109] According to the exemplary embodiments of the present disclosure, numerous specific details have been set forth. It is to be understood, however, that implementations of the disclosed technology can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. References to “some examples,” “other examples,” “one example,” “an example,” “various examples,” “one embodiment,” “an embodiment,” “some embodiments,” “example embodiment,” “various embodiments,” “one implementation,” “an implementation.” “example implementation,” “various implementations,” “some implementations,” etc., indicate that the implementation(s) of the disclosed technology7so described may include a particular feature, structure, or characteristic, but not every7implementation necessarily includes the particular feature, structure, or characteristic. Further, repeated use of the phrases “in one example.” “in one exemplary embodiment.” or “in one implementation” does not necessarily refer to the same example, exemplary embodiment, or implementation, although it may.

[0110] As used herein, unless otherwise specified the use of the ordinal adjectives “first,” “second,” “third,” etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0111] While certain implementations of the disclosed technology have been described in connection with what is presently considered to be the most practical and various implementations, it is to be understood that the disclosed technology is not to be limited to the disclosed implementations, but on the contrary’, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0112] The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures which, although not explicitly shown or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various different exemplary embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary’ skill in the art. In addition, certain terms used in the present disclosure, including the specification and drawings, can be used synonymously in certain instances, including, but not limited to, for example, data and information. It should be understood that, while these words, and / or other words that can be synonymous to one another, can be used synonymously herein, that there can be instances when such words can be intended to not be used synonymously. Further, to the extent that the prior art knowledge has not been explicitly incorporated by reference herein above, it is explicitly incorporated herein in its entirety. All publications referenced are incorporated herein by reference in their entireties.

[0113] Throughout the disclosure, the following terms take at least the meanings explicitly associated herein, unless the context clearly dictates otherwise. The term “or” is intended to mean an inclusive “or.” Further, the terms “a.” “an.” and “the” are intended to mean one or more unless specified otherwise or clear from the context to be directed to a singular form.

[0114] This written description uses examples to disclose certain implementations of the disclosed technology7, including the best mode, and also to enable any person skilled in the art to practice certain implementations of the disclosed technology, including making and using any devices or systems and performing any incorporated methods. The patentable scope of certain implementations of the disclosed technology is defined in the appended claims, andmay include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the appended claims if they have structural elements that do not differ from the literal language of the appended claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the appended claims.EXEMPLARY REFERENCES

[0115] The following references are hereby incorporated by reference, in their entireties:1Babbage, C. Passages from the life of a philosopher . (Theclassics, 2013).2Deng, J. el al. ImageNet: A large-scale hierarchical image database, in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248-255 (2009).3Building a large annotated corpus of English: The Penn Treebank.4Merity, S.. Xiong, C., Bradbury, J. & Socher, R. Pointer Sentinel Mixture Models. arXiv [cs.CL] (2016).5Bubeck, S. et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv [cs.CL] (2023).6Chowdhery, A. et al. PaLM: Scaling language modeling with Pathways. https: / / storage.googleapis.com / pathways-language-model / PaLM-paper.pdf.7Common Crawl, https: / / commoncrawl.org / .8Dodge, J. et al. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. arXiv [cs. CL] (2021).9Soldaini. L. AI2 Dolma: 3 trillion token open corpus for LLMs. Al 2 Blog https: / / blog.allenai.org / dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64 (2023).10Harrer, S. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine 90, 104512 (2023).11Bommasani, R., Liang. P. & Lee, T. Holistic Evaluation of Language Models. Ann. N. Y. Acad. Sci. (2023) doi : 10.1111 / nyas.15007.12Gao, L. et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept.13Jin, D. et al. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv [cs. CL] (2020).14Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: A Large-scale Multi-Subject MultiChoice Dataset for Medical domain Question Answering, in Proceedings of the Conference on Health, Inference, and Learning (eds. Flores, G., Chen, G. H., Pollard, T., Ho, J. C. & Naumann, T.) vol. 174248-260 (PMLR. 07-08 Apr 2022).15Jin, Q., Dhingra. B.. Liu, Z.. Cohen. W. W. & Lu, X. PubMedQA: A Dataset for Biomedical Research Question Answering, ar Xi v [ cs. CL ] (2019).16Validity Challenges in Machine Learning Benchmarks.Bigbench: Towards an industry standard benchmark for big data analytics.Sainz, O. et al. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. arXiv [cs. CL] (2023).Miller, J. Validity Challenges in Machine Learning Benchmarks. (UC Berkeley, 2022).How well do large language models support clinician information needs? Stanford HAI https: / / hai.stanford.edu / news / how-well-do-large-language-models-support-clinician- information-needs.Soboleva, D. SlimPajama: A 627B token, cleaned and deduplicated version of RedPajama.Cerehras https: / / www.cerebras.net / blog / slimpajama-a-627b-token-cleaned-and- deduplicated-version-of-redpaj ama / (2023).Hoffmann, J. etal. Training Compute-Optimal Large Language Models. (2022).Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems. (2023).Yu, S. et al. BIOS: An Algorithmically Generated Biomedical Knowledge Graph. arXiv [cs. CL] (2022).Jin, Q. etal. MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics 39, (2023).Tian, K. Mitchell, E., Yao, H., Manning, C. D. & Finn. C. Fine-tuning Language Models for Factuality. (2023).Kim, J., Kwon, Y, Jo, Y. & Choi, E. KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models. (2023).Turner, A., Tsipras, D. & Madry. A. Clean-Label Backdoor Attacks. (2018).Carlini, N. etal. Poisoning Web-Scale Training Datasets is Practical. arXiv [cs.CR] (2023).Chen, X., Liu, C.. Li, B., Lu, K. & Song, D. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv [cs.CR] (2017).Del Complex, https: / / www.delcomplex.com / .Hsieh. C.-Y. et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. (2023).Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine, npj Digital Medicine 6, 1-4 (2023).Mesko, B. & Topol, E. J. The imperative for regulatory oversight of large language models (or generative Al) in healthcare, npj Digital Medicine 6, 1-6 (2023). Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How Does LLM Safety Training Fail? (2023). Hubinger, E. etal. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. (2024). Li. Y. etal. Textbooks Are All You Need II: phi- 1.5 technical report. (2023). Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259- 265 (2023).

Claims

WHAT IS CLAIMED IS:

1. A method for detecting misinformation from a large language model (LLM), comprising: receiving an output from the LLM; relating the output to a knowledge graph; and predicting information provided from the LLM that contains the misinformation based on the output from the LLM related to the knowledge graph.

2. The method of claim 1 , wherein the relating of the output to the knowledge graph further comprises extracting one or more knowledge triplets from the received output.

3. The method of claim 2. wherein the relating of the output to the knowledge graph further comprises comparing the extracted one or more knowledge triplets from the received output with the one or more knowledge triplets from the knowledge graph.

4. The method of claim 2. wherein the one or more knowledge triplets include an origin, a relation and a target.

5. The method of claim 1, wherein the prediction of the information occurs at an inference time.

6. The method of claim 1. wherein the relating of the output to the knowledge graph further comprises: converting a term from the output into a first number; determining a distance between the first number and a concept in the output; converting a term from the knowledge graph into a second number; determining a distance between the second number and the concept in the output; and comparing the relative distances.

7. The method of claim 6. wherein the prediction that the output contains misinformation is based on the relative distances being different.

8. The method of claim 1, wherein the knowledge graph is a medical knowledge graph.

9. A system for detecting misinformation from a large language model (LLM), comprising: one or more computer processors configured to: receive an output from the LLM; relate the output to a knowledge graph; and predict information provided from the LLM that contains the misinformation based on the output from the LLM related to the knowledge graph.

10. The system of claim 9, wherein the relating of the output to the knowledge graph further comprises extracting one or more knowledge triplets from the received output.

11. The system of claim 10, wherein the relating of the output to the knowledge graph further comprises comparing the extracted one or more knowledge triplets from the received output with the one or more knowledge triplets from the knowledge graph.

12. The system of claim 10, wherein the one or more knowledge triplets include an origin, a relation and a target.

13. The system of claim 9, wherein the prediction of the information occurs at an inference time.

14. The system of claim 9, wherein the relating of the output to the knowledge graph further comprises: converting a term from the output into a first number; determining a distance between the first number and a concept in the output; converting a term from the knowledge graph into a second number; determining a distance between the second number and the concept in the output; and comparing the relative distances.

15. The system of claim 14, wherein the prediction that the output contains misinformation is based on the relative distances being different.

16. The system of claim 9, wherein the knowledge graph is a medical knowledge graph.

17. A non-transitory computer accessible medium which includes software thereon for detecting misinformation from a large language model (LLM), wherein, when at least one computer processor executes the software, the computer processor is configured to perform the procedures, comprising: receiving an output from the LLM; relating the output to a knowledge graph; and predicting information provided from the LLM that contains the misinformation based on the output from the LLM related to the knowledge graph.

18. The non-transitory computer accessible medium of claim 17, wherein the relating of the output to the knowledge graph further comprises extracting one or more knowledge triplets from the received output.

19. The non-transi tory computer accessible medium of claim 18, wherein the relating of the output to the knowledge graph further comprises comparing the extracted the one or more knowledge triplets from the received output with one or more knowledge triplets from the knowledge graph.

20. The non-transi ton computer accessible medium of claim 18, wherein the one or more knowledge triplets include an origin, a relation and a target.

21. The non-transitory computer accessible medium of claim 17, wherein the prediction of the information occurs at an inference time.

22. The non-transitory computer accessible medium of claim 17, wherein the relating of the output to the knowledge graph further comprises: converting a term from the output into a first number; determining a distance between the first number and a concept in the output; converting a term from the knowledge graph into a second number; determining a distance between the second number and the concept in the output; and comparing the relative distances.

23. The non-transitory computer accessible medium of claim 22, wherein the prediction that the output contains misinformation is based on the relative distances being different.

24. The non-transitory computer accessible medium of claim 17, wherein the knowledge graph is a medical knowledge graph.

25. A method for detecting at least one artificial intelligence hallucination, comprising: receiving an output from a large language model; extracting one or more relevant terms from the received output; applying the one or more relevant terms to a related ontology; and providing at least one hallucination prediction associated with the at least one artificial intelligence hallucination for the received output based on the applied ontology.

26. A system for detecting artificial intelligence hallucinations, the system comprising: at least one computer processor configured to: receive an output from a large language model, extract one or more relevant terms from the received output. apply the one or more relevant terms to a related ontology, and provide at least one hallucination prediction associated with the at least one artificial intelligence hallucination for the received output based on the applied ontology.

27. A non-transitory computer-readable medium for detecting artificial intelligence hallucinations which includes instructions that, when executed on a computer configuration, cause the computer configuration to perform procedures comprising: receiving an output from a large language model; extracting one or more relevant terms from the received output; applying the one or more relevant terms to a related ontology; and providing at least one hallucination prediction associated with the at least one artificial intelligence hallucination for the received output based on the applied ontology.

Images