Neural Signal Detection

Abstract

Systems, methods, and other embodiments associated with neural signal detection are disclosed. In one embodiment, for multiple reports, an embedding layer of a neural signal detection network creates a vector embedding for each of a set of information objects contained in the reports. The set of information objects includes a target object. A representation layer represents the reports in a manner that describes correlations between occurrences of the target events and the information objects and accounts for dependencies between the information objects that make up the reports. A logit layer models a set of events including the target event based on the representation of the reports. A probability of occurrence for the target event given the set of modeled events is determined. A summary probability of occurrence for the target event across multiple reports is compared to a comparison baseline probability to identify the presence of a signal.

Description

[Technical Field]

[0001] The systems and methods described herein relate to artificial intelligence computer and digital data processing systems, methods, and products for the emulation of intelligence, including machine learning systems and artificial neural networks. More particularly, the systems and methods described herein relate to machine learning using deep neural networks that have the ability to add to their integrated collection of facts and relationships for data signal detection. [Background technology]

[0002] background Signal detection is the ability to distinguish pattern-bearing information, or "signal," from random patterns, or "noise," that distort or disguise the information.

[0003] An adverse reaction is a harmful and unintended patient reaction to a drug or other medical treatment. The causes of adverse reactions to drugs, biologics, vaccines, devices, or other medical products are complex and require sophisticated analytical approaches to study the causal relationship between the observed treatment and the adverse event. Computational signal detection, also known as quantitative signal detection, is a tool for screening for potential causal relationships between observed / reported products and adverse events. Summary of the Invention

[0004] overview In one embodiment, a non-transitory computer-readable medium is provided having computer-executable instructions stored thereon, the computer-executable instructions, when executed by at least a processor of the computer, causing the computer to perform a plurality of steps, the plurality of steps including, for a plurality of reports in a report database, at least the processor creating, via an embedding layer of a neural signal detection network, a vector embedding for each of a set of one or more information objects included in the report, the set of one or more information objects including a target object, and the plurality of steps further including, at least the processor representing, via a representation layer of the neural signal detection network, the report in a manner that describes correlations between occurrences of the target event and the information objects and that takes into account dependencies between the information objects that make up the report; at least the processor modeling, via a logit layer of the neural signal detection network, a set of events including the target event based on the representation of the report; at least the processor determining a probability of occurrence for the target event given the set of modeled events; and at least the processor identifying the presence of a signal by comparing a summary occurrence probability for the target event across the plurality of reports in the report database to a comparison reference probability.

[0005] In one embodiment, the non-transitory computer-readable medium further comprises instructions that, when executed by at least a processor, cause the processor to calculate a signal score from the summary occurrence probability and a comparison reference probability derived from the same set of reports used to create the summary occurrence probability, and calculate the comparison reference probability by (i) removing the target object from each report used to calculate the summary occurrence probability, or (ii) recalculating yet another summary occurrence probability from each report used to calculate the summary occurrence probability with the attention weight assigned to the target object for each report set to zero.

[0006] In one embodiment, the non-transitory computer-readable medium further comprises instructions that, when executed by at least a processor, cause the processor to perform the step of training a neural signal detection network by estimating parameters of the neural signal detection network by stochastic gradient descent using data log-likelihood as a loss function to be minimized.

[0007] In one embodiment, the non-transitory computer-readable medium further comprises instructions that, when executed by at least a processor, cause the processor to perform a step of assigning an attention weight to each of the information objects, the attention weight assigned to each information object being based on at least one other of the information objects.

[0008] In one embodiment, in the non-transitory computer readable medium, the target object is a target drug, and the set of one or more objects included in the report further includes a second drug that has a confounding or masking effect with respect to the target drug.

[0009] In one embodiment, a computer-implemented method is provided comprising, for a plurality of reports in a report database, creating, by an embedding layer of a neural signal detection network, a vector embedding for each of a set of one or more informational objects included in the report, the set of one or more informational objects including a target object, the method further comprising: representing, by a representation layer of the neural signal detection network, the report in a manner that describes correlations between occurrences of the target event and the informational objects and that takes into account dependencies between the informational objects that make up the report; modeling, by a logit layer of the neural signal detection network, a set of events that include the target event based on the representation of the report; determining a probability of occurrence for the target event given the set of modeled events; and identifying the presence of a signal by comparing the summarized probability of occurrence for the target event across the plurality of reports in the report database to a comparison reference probability.

[0010] In one embodiment, the computer-implemented method further comprises calculating a signal score from a summary occurrence probability and a comparison reference probability, the comparison reference probability being derived from the same set of reports used to create the signal score by recalculating further summary occurrence probabilities with the target object removed from each report used to calculate the summary occurrence probability.

[0011] In one embodiment, the computer-implemented method further comprises calculating a signal score from a summary occurrence probability and a comparison reference probability, the comparison reference probability being derived from the same set of reports used to create the signal score by recalculating a further summary occurrence probability from each report used to calculate the summary occurrence probability with the attention weight assigned to the target object for each report set to zero.

[0012] In one embodiment, the computer-implemented method further comprises training the neural signal detection network by estimating parameters of the neural signal detection network by stochastic gradient descent using the data log-likelihood as the loss function to be minimized.

[0013] In one embodiment, the computer-implemented method further comprises assigning attention weights to each of the information objects, wherein (i) the first attention weight assigned to a first information object known to cause the target event is greater than other attention weights assigned to other information objects not known to cause the target event, or (ii) the first attention weight assigned to a first information object known to reduce the likelihood of the target event is greater than other attention weights assigned to other information objects not known to reduce the likelihood of the target event.

[0014] In one embodiment, in the computer-implemented method, the target object is a target medication and the set of target events includes (i) adverse health events, or (ii) positive health events.

[0015] In one embodiment, a computing system is provided comprising: a processor; a memory operatively connected to the processor; and a non-transitory computer-readable medium operatively connected to the processor and the memory and storing computer-executable instructions, the computer-executable instructions, when executed by at least the processor of the computer, causing the computing system to perform a plurality of steps, the plurality of steps including: for a plurality of reports in a report database, creating, by an embedding layer of a neural signal detection network, a vector embedding for each of a set of one or more information objects included in the report, the set of one or more information objects including a target object; representing, by a representation layer of the neural signal detection network, the report in a manner that describes correlations between occurrences of the target event and the information objects and that takes into account dependencies between the information objects that make up the report; modeling, by a logit layer of the neural signal detection network, a set of events including the target event based on the representation of the report; determining a probability of occurrence for the target event given the set of modeled events; and identifying the presence of a signal by comparing a summary occurrence probability for the target event across the plurality of reports in the report database to a comparison reference probability.

[0016] In one embodiment, in a computing system, the instructions, when executed, further cause the computing system to perform the step of assigning an attention weight to each of the information objects, the attention weight assigned to each information object being based on at least one other of the information objects.

[0017] In one embodiment, in a computing system, the target object is a target medication and the set of target events includes adverse health events, and the instructions, when executed, further cause the computing system to: calculate a signal score from the event probability and the comparison reference probability; determine the presence of a signal in response to the signal score exceeding a threshold indicating the presence of the signal; and present an alert for display by a remote system indicating that a signal has been detected.

[0018] In one embodiment, in a computing system, the instructions, when executed, further cause the computing system to perform the step of deriving a comparison reference probability from the same set of reports used to create the signal score by recalculating a further summary occurrence probability from each report used to calculate the summary occurrence probability while ignoring the target drug in the report.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various systems, methods, and other embodiments of this disclosure. It will be understood that any depicted element boundaries (e.g., boxes, groups of boxes, or other shapes) in the drawings represent one embodiment of that boundary. In some embodiments, one element may be realized as multiple elements, or multiple elements may be realized as one element. In some embodiments, an element shown as an internal component of another element may be realized as an external component, and vice versa. Additionally, elements may not be drawn to scale. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 illustrates one embodiment of a computing system associated with neural signal detection. [Figure 2] FIG. 1 illustrates one embodiment of a topology of a neural network associated with neural signal detection. [Figure 3] FIG. 1 illustrates an embodiment of a method associated with neural signal detection. [Figure 4] FIG. 1 illustrates an embodiment of a computing system configured using the disclosed example systems and / or methods. DETAILED DESCRIPTION OF THE INVENTION

[0021] Detailed Description Described herein are systems and methods providing neural signal detection (NSD), i.e., signal detection using neural networks. The neural signal detection systems and methods described herein not only enable modeling of complex dependencies between adverse events, medications, and other input factors, but also address confounding and masking effects, thereby improving the accuracy of the signal detection systems and methods. The neural signal detection systems and methods described herein reduce false alarms and missed signals compared to signal detection achieved using other statistical or machine learning methodologies. These improvements in accuracy result in faster or more timely signal detection. The neural signal detection systems and methods described herein enable simultaneous modeling of multiple events. The neural signal detection systems and methods described herein also enable modeling of dependencies in the context of an event hierarchy or product hierarchy. The neural signal detection systems and methods described herein readily support the inclusion of external information (such as the biochemical properties of a drug) in the input.

[0022] The neural signal detection systems and methods described herein are readily applied in safety and regulatory compliance contexts, such as patient safety. In one embodiment, the neural signal detection systems and methods described herein are applied to detect adverse health effects of medications, thereby improving patient safety. For example, the target object for signal detection is a target medication, and the set of target events includes adverse health events. Conversely, the neural signal detection systems and methods described herein may be employed to detect unexpected positive effects of medications, thereby accelerating medical research and improving patient health. For example, the target object for signal detection is a target medication, and the set of target events includes positive health events. The neural signal detection systems and methods described herein may also be applied to identify causes of failure of expected pharmacological effects, allowing for the identification of treatment alternatives to improve patient health. While neural signal detection may be described herein in the context of the benefits it offers in the health sector, neural signal detection may be advantageously applied in any field where complex dependencies exist between input and output events, or where multiple events need to be modeled simultaneously in the context of a hierarchy of output events or inputs.

[0023] The goal of a signal detection algorithm is to generate a statistical signal score, also known as a disproportion ratio, that is used to highlight potentially new (or at least previously undetected) events associated with a particular input. Signal detection algorithms are built on techniques that model the relationships between inputs and events. Neural signal detection (NSD) as shown and described herein uses deep learning (DL) techniques to model these relationships. The effectiveness of signal detection algorithms is judged primarily based on the accuracy of the signals they generate. Signaling errors lead to false alarms and undetected true events. The new neural signal detection algorithms shown and described herein may generate signals with greater accuracy than existing algorithms.

[0024] In one embodiment, when applied to a pharmacovigilance dataset, the statistical significance score (disproportionality ratio) is used to detect adverse events associated with the use of a pharmaceutical product. The increased sensitivity and accuracy in signal detection provided by the neural signal detection systems and methods described herein reduces false alarms, which are costly to pharmaceutical companies. The increased accuracy in signal detection provided by the neural signal detection systems and methods described herein also reduces undetected true events, preventing harm to patients. Furthermore, the increased sensitivity in signal detection provided by the neural signal detection systems and methods described herein allows true events to be detected earlier or from smaller datasets, again preventing harm to patients.

[0025] No action or function described or claimed herein is performed by the human mind. No action or function described or claimed herein can be performed by the human mind. An interpretation that any action or function can be performed in the human mind is inconsistent with and contrary to this disclosure.

[0026] Signal management software systems, such as Oracle® Empirica Signal, are software applications used by users, such as pharmaceutical companies, to manage, track, and identify new adverse (or positive unexpected) events associated with their products (such as pre- and post-marketing drugs, biologics, vaccines, devices, and combination products). Signal management software may implement several signal detection algorithms to achieve signal identification in available data. In one embodiment, the data on which the signal detection system and method operates are spontaneous reports of adverse events (such as individual case safety reports) collected and managed by its customers. Each report is available in a structured format and includes the reported adverse event, the suspected drug associated with the event, the date, limited patient demographic information (e.g., age and gender), reporter information, and potentially other information related to the reported adverse event. In one embodiment, the data on which signal detection operates are dynamically generated event reports following the same structured format, generated from patient data that records information but has not been submitted as a reported adverse event. For example, this patient data may be drawn from patient records at a hospital or group of hospitals. This data may be anonymized. In another example, the data on which signal detection operates may include medical literature or other data sources containing information from which reports may be compiled.

[0027] Signal management software may make available one or more signal detection methodologies that range in complexity and ease of use. Standard methodologies, such as proportional reporting ratio (PRR), multi-item gamma Poisson shrinker (MGPS), and information component (IC), are based on 2 × 2 contingency tables (or cross-tabulations) of target products and target adverse events. Models and signal scores derived solely from the information contained in these 2 × 2 tables reduce what is arguably a complex problem to something much simpler. The complexity associated with studying adverse event relationships arises from many factors, including, but not limited to, issues associated with dose, duration, and route, demographics, labeling, comorbidities, genetics, terminology and / or language variability used to capture product and adverse events, and differences in interfactor dependencies expressed as product interactions, confounding and masking effects, product-class effects, syndromic events, and secondary effects. Therefore, the reduction of signal detection to a 2 × 2 contingency table may lead to the loss of critical information needed to study the complex relationships between observed / reported products and adverse events, which in turn may lead to false alarms and / or missed signals.

[0028] In an attempt to overcome these problems of false alarms and missing signals, more advanced signaling methodologies have been developed that go beyond 2x2 contingency tables and utilize more sophisticated modeling techniques with more information (e.g., data variables). Most of these advanced approaches, such as extended logistic regression (ELR) and regression-adjusted gamma Poisson shrinker (RGPS) (both available in Oracle® Empirica Signal), as well as the World Health Organization's binary logistic regression (BLR), are based on logistic regression. In logistic regression, the occurrence of the target adverse event is modeled in the context of the target product and additional information represented as additional regression predictors. These additional predictors may represent dose, demographics, concomitant conditions, and generally any other relevant information available. In this regard, approaches based on 2x2 contingency tables can be thought of as logistic regression with only one predictor: the target product.

[0029] Deep Learning and Neural Signal Detection Broadly speaking, artificial intelligence (AI) is the science of creating intelligent computer systems that mimic cognitive processes such as learning and problem solving using techniques such as statistical analysis of data, expert systems that rely on if-then statements, and machine learning. AI includes many subfields, including machine learning (ML), neural networks, deep learning natural language processing (NLP) and generation (NLG), and image processing. These subfields may have significant overlap.

[0030] Machine learning is an AI technique used to train software algorithms to learn from data without explicit instructions, relying instead on patterns and inference. Forms of machine learning include supervised learning, unsupervised learning, and semi-supervised learning. Generally, the goal of supervised learning is to approximate a mapping function between inputs and outputs so that the function can be used to accurately predict outputs for new inputs. In supervised learning, a training dataset of example inputs and "correct" outputs is presented to a machine learning algorithm, and in response, the algorithm generates an inferred function or model that can be used to map inputs to outputs and predict outputs for new inputs. Supervised learning algorithms are for classification, where the output is a category such as "cheese" or "red," or for regression, where the output is a numerical value (or other value) such as money, weight, or dimension. Generally, the goal of unsupervised learning is to discover and model the underlying structure or distribution of unlabeled data. In unsupervised learning, unlabeled data is presented to a machine learning algorithm, and in response, the algorithm generates an inferred function or model that represents the unlabeled data. Semi-supervised learning occupies a middle ground between supervised and unsupervised learning, where only a portion of the training dataset is labeled in the output. In semi-supervised learning, unsupervised learning techniques are employed to discover and learn structure in the input variables, while supervised learning techniques may be used to make "best guess" predictions of the unlabeled data and use that "best guess" data to further train the model.

[0031] A neural network is an AI technique inspired by the human brain, consisting of a complex system of artificial neurons connected and organized into multiple layers (input layer, hidden layer, and output layer) in an oriented, weighted graph. Neurons (nodes of the graph) and connections (edges of the graph, sometimes called "synapses") typically have weights that are adjusted as learning progresses, increasing or decreasing the strength of the output signal. During training, the weights of neurons and connections are adjusted by a processor (over successive generations from initial values (e.g., random values)) until the neural network accurately and consistently identifies patterns. Each neuron accepts one or more inputs and produces a single output, which can be sent to multiple other neurons in the next layer. Neurons in one layer connect only to neurons in the previous and following layers. Neurons in the input layer can be feature values for the input data. Neurons in the output layer are feature values for the output. Between the input and output layers are zero or more hidden layers. The output of a neuron is found by taking the sum of all of the inputs to the neuron weighted by the weights of the incoming connections to the neuron, passing this weighted sum through the neuron's activation function to produce the neuron's unweighted output, and then applying the weight assigned to the neuron to the output to produce the neuron's weighted output, which will be passed on to neurons in the next layer. Neural networks can be used in supervised, semi-supervised, and unsupervised machine learning.

[0032] Neural networks can be further extended in an AI technique called deep learning, in which neural networks with multiple hidden layers are used to progressively extract higher-level features from the input. The term "deep" in deep learning refers to the use of multiple hidden layers in neural networks. Deep learning can be used in supervised, semi-supervised, and unsupervised machine learning. Unlike other machine learning algorithms, neural networks for deep learning, i.e., deep neural networks (DNNs), can be used to model complex nonlinear relationships. In addition, deep learning is not limited to specific parametric models such as logistic regression or the specific logic behind decision trees. Rather, it is a flexible framework that can be tailored to specific problems. Deep learning has been proven both theoretically and empirically to offer superior modeling power compared to traditional machine learning algorithms (e.g., in applications such as computer vision, speech recognition, and natural language processing).

[0033] In one embodiment, neural signal detection, as shown and described herein, is a methodology that takes significant steps beyond RGPS. In neural signal detection, deep neural networks replace other statistical or machine learning methodologies, such as logistic regression, used by RGPS. Neural signal detection thus uses deep learning techniques to model the relationship between products and adverse events. Neural signal detection leverages the superior modeling power of deep learning neural networks to generate signals with greater accuracy.

[0034] Thus, neural signal detection demonstrates significant advantages over existing methodologies. In the context of signal detection, greater modeling power allows for the modeling of more complex dependencies between adverse events, products, and other factors with greater accuracy. Greater modeling flexibility allows for the design of neural networks that (1) support the modeling of multiple events simultaneously, (2) enable transfer learning, (3) enable modeling in the context of event or product hierarchies, and (4) facilitate the inclusion of external information. This increases the amount of data available for modeling, allows different events to inform each other during modeling, enables the inclusion of more information, and addresses issues associated with modeling rare events. Together (or separately), greater modeling power and flexibility translate into higher signaling accuracy, i.e., fewer false alarms and missed signals, and improved timeliness (earlier detection). In the pharmacovigilance sector, fewer false alarms save significant costs and effort, while fewer missed signals and earlier signal detection saves lives.

[0035] Exemplary Signal Management Environment FIG. 1 illustrates one embodiment of a computing system 100 associated with neural signal detection. In one embodiment, computing system 100 includes a signal management system 105 connected to an enterprise network 115 by the Internet 110 (or other suitable communications network or combination of networks). In one embodiment, signal management system 105 may be Oracle® Empirica Signal or another signal management system further configured to execute the systems and methods for neural signal detection described herein. In one embodiment, cloud signal management system 105 includes various systems and components, such as a neural signal detection component 120, other signal management system components 125, a data store 130, and a web interface server 135.

[0036] In one embodiment, the components of the signal management system 105 are implemented on one or more hardware computing devices or hosts interconnected by a data network. For example, the components of the signal management system 105 may be executed by networked computing devices of one or more compute hardware geometries, such as standard (CPU or general-purpose) geometries, dense input / output (I / O) geometries, graphics processing unit (GPU) geometries, and high-performance computing (HPC) geometries. It is particularly noted that while deep neural networks (such as those used for neural signal detection as described herein) may be trained using general-purpose compute geometries, in one embodiment, it is desirable to train deep neural networks using GPU compute geometries. The bandwidth optimization in main memory, thread parallelism, and higher ratio of aggregate register size to processing unit of the GPU geometry (compared to the latency optimization, reduced parallelism, and lower ratio of aggregate register size to processing unit of the GPU geometry) may make the GPU geometry preferable over the CPU geometry in deep learning applications such as neural signal detection. In one embodiment, signal management system 105, or at least neural signal detection component 120, may be executed by an Oracle® GPU3 or GPU4 architecture. In one embodiment, the components of signal management system 105 are each implemented by one or more dedicated computing devices. In one embodiment, some or all components of signal management system 105, although represented as separate units in FIG. 1, are implemented by a common (or shared) computing device. In one embodiment, components of signal management system 105 may be implemented across multiple computing devices.

[0037] In one embodiment, components of the signal management system 105 communicate with each other through electronic messages or signals. These electronic messages or signals may be configured as calls to functions or procedures, such as application programming interface (API) calls, that access features or data of the components. In one embodiment, these electronic messages or signals are transmitted between hosts in a format compatible with Transmission Control Protocol / Internet Protocol (TCP / IP) or other computer network protocols. Each component of the signal management system 105 may analyze the content of a received electronic message or signal to identify a command or request that the component can perform. In response to identifying a command, the component will automatically perform the command or request.

[0038] In one embodiment, the signal management system 105 may be implemented as a service on a cloud infrastructure. In one embodiment, the signal management system 105 may be hosted by a dedicated third party, for example in an infrastructure-as-a-service (IAAS), platform-as-a-service (PAAS), or software-as-a-service (SAAS) architecture. In one embodiment, the signal management system 105 may be implemented on an on-premise infrastructure, such as a set of one or more dedicated servers.

[0039] In one embodiment, other signal management system components 125 include components for operating signal management system 105. For example, other signal management system components 125 may include other signal detection methodology modules, such as modules for PRR, MGPS, IC, ELR, RGPS, and / or BLR signal detection; a detection event management module for managing actions related to detected events or signals; a detection event tracking module for monitoring detected events or signals; an adverse event report acceptance module for accepting submissions of adverse event reports; other data collection modules for capturing unstructured data as adverse event reports; a dynamic event report generation module for generating event reports from patient data not formally submitted as adverse event reports; a data analysis and visualization module for performing data analysis in addition to signal detection; a user interface module for generating user interfaces to other components / modules of signal management system 105; a regulatory reporting module for creating safety signal information and monitoring submissions to regulatory agencies; and / or an administration module for managing tenant and user access to system 105.

[0040] The enterprise network 115 may be associated with a company, such as a pharmaceutical or medical device manufacturer, or other entity interested in monitoring the downstream behavior of its products. For simplicity and clarity of explanation, the enterprise network 115 is represented by an on-site local area network 140 to which one or more personal computers 145 or servers 150 are operatively connected, and one or more remote user computers 155 or mobile devices 160 connected to the enterprise network 115 through the Internet 110. Each personal computer 145, remote user computer 155, or mobile device 160 is generally dedicated to a particular end user, such as an employee or contractor associated with the company, although such dedication is not required. The personal computers 145 and remote user computers 155 may be, for example, desktop computers, laptop computers, tablet computers, or other devices capable of connecting to the local area network 140 or the Internet 110. The mobile devices 160 may be, for example, smartphones, tablet computers, mobile phones, or other devices capable of connecting to the local area network 140 or the Internet 110 through a wireless network, such as a cellular network or Wi-Fi.

[0041] Users of the enterprise network 115 may interact with the signal management system 105 via the Internet 110. In one embodiment, computing systems external to the signal management system 105 (such as computing systems in the enterprise network 115) may access information or applications provided by the signal management system 105 through a web interface server 130. In one embodiment, the external computing systems may send requests to and receive responses from the web interface server 130. In one example, access to the information or applications may be implemented through the use of a web browser on a personal computer 145, a remote user computer 155, or a mobile device 160. For example, these computing devices 145, 155, 160 in the enterprise network 115 may request and receive a web page-based graphical user interface (GUI) for detecting signals using neural signal detection in the signal management system 105. In one example, the web interface server 130 may present HTML code to a personal computer 145, a server 150, a remote user computer 155, or a mobile device 160 so that these computing devices are presented with a GUI for the signal management system 105 (including a GUI for accessing the functionality of the neural signal detection component 120).In another example, communications exchanged between the web interface server 130 and the personal computer 145, server 150, remote user computer 155, or mobile device 160 may take the form of remote representational state transfer (REST) requests using, for example, JavaScript object notation (JSON) as the data exchange format, or simple object access protocol (SOAP) requests to or from an XML server.

[0042] In one embodiment, data store 130 includes one or more databases configured to store and service a wide range of information related to applications running in signal management system 105. In one embodiment, data store 130 includes a database, such as an Oracle® database. In some example configurations, data store 130 may be implemented using one or more Oracle® Exadata compute servers, network-attached storage (NAS) devices, and / or other dedicated server devices.

[0043] In one embodiment, neural signal detection component 120 includes neural signal detection network 165, neural network trainer 170, deep learning framework 175, report database 180, and attention mechanism 185. In one embodiment, neural signal detection network 165 is a neural network configured, constructed, and operated as shown and described herein to generate a probability of occurrence of one or more adverse events given input information in a report. In one embodiment, neural network trainer 170 is configured to train neural signal detection network 165 by estimating trainable parameters of a neural signal detection model, for example, by stochastic gradient descent as shown and described herein. In one embodiment, deep learning framework 175 includes a set of tools and libraries (e.g., TensorFlow and PyTorch) for training deep learning networks as shown and described herein. Report database 180 is a repository of reports (data structures describing an event and information surrounding the event) to be examined for signals between information inputs (e.g., medications) and events (e.g., adverse health events) as shown and described herein. In one embodiment, report database 180 is a portion of data store 130. Attention mechanism 185 is configured to provide context-sensitive weights to information inputs to neural signal detection network 165 as shown and described herein.

[0044] Each of the components (including subcomponents) of the signal management system 105 is configured by logic to perform the functions that the component is described as performing. In one embodiment, the components of the signal management system 105 may each be implemented as a set of one or more software modules executed by one or more computing devices (such as hosts of a cloud network computing system) specially configured for such performance. In one embodiment, these modules include one or more modules for implementing the features shown and described herein.

[0045] Exemplary Neural Signal Detection Network In one embodiment, for a given target adverse event, neural signal detection maps information contained in the report (e.g., product and patient information) to probabilities of the target adverse event. These probabilities are then aggregated to generate a statistical signal score (disparity ratio) for the association of the targeted event with a particular target product.

[0046] These probabilities are calculated by fitting a model to the data (training the model to fit the data). The model aims to capture the relationship between the information in the report (e.g., the drugs mentioned in the report) and the probability of the targeted adverse event occurring. Neural signal detection uses deep neural networks, for example, instead of logistic regression, to fit the model.

[0047] In many deep learning applications, modeling objects, such as words and sentences in natural language processing and images in computer vision, are represented as dense vectors of real numbers rather than as scalar indicator, categorical, ordinal, or continuous variables as used in regression. In signal detection, the objects being modeled are products, adverse events, and other information contained in reports (such as patient demographics or other patient information). In one embodiment, the neural signal detection systems and methods described herein similarly represent these objects as dense vectors. These dense vectors are learned as part of the training / model fitting process.

[0048] Deep learning is a general framework within which specific neural network topologies can be designed or tuned to accomplish specific tasks or model specific relationships. For this reason, deep learning applications are typically described by the neural network topologies, or layers, used to represent the models.

[0049] FIG. 2 illustrates one embodiment of a topology of a neural network 200 associated with neural signal detection. In one embodiment, the neural network 200 for neural signal detection includes three main layers: an object embedding (or object representation) layer 205, a report representation layer 210, and a logit layer 215. The object embedding layer 205 is used to learn vector representations of objects in a report, such as products and other information. The report representation layer 210 learns a vector representation of the entire report, i.e., a vector representation that summarizes all of the information contained in the report, taking into account dependencies between the objects that make up the report. The logit layer 215 converts the vector representation of the report into a modeled probability of occurrence of an adverse event. In one embodiment, one or more of these layers 205, 210, 215 may further include smaller sub-layers, such as deep learning primitives, transformation layers, or other layers, that are used to generate the results described for the main layers 205, 210, 215. In one embodiment, one or more of these layers 205, 210, 215 may be replicated to increase model complexity as needed. In one embodiment, each layer 205, 210, 215 may use vectors of different dimensionality.

[0050] In one embodiment, neural network 200 may be used to model one target adverse event independently from others, as indicated by probability of occurrence 220 of adverse event 1 given a vector representation of reports, such as report vector representation 225. In one embodiment, neural network 200 may be easily extended to simultaneously model multiple target adverse events, as indicated by probability of occurrence 230 of adverse event 2 and probability of occurrence 235 of adverse event 3 given vector representation 225 of reports. There may be one or more target events (up to Q). The probability of occurrence of each of these Q target events is calculated using the following formula: q (x 2 )

[0051]

number

[0052] In one embodiment, in the logit layer 215, an individual logit layer (e.g., logit1 261, logit2 262, logitQ 263) is implemented for each of the Q target events. The individual logit layer for each target event shares the learning generated at lower levels of the network. The logit function, or log-odds function, is one of several functions that operates to convert probability values (ranging from 0 to 1 inclusive) into real values (ranging from -∞ to +∞ inclusive). The inverse logit function, or logistic function, is one of several sigmoid functions that operates to convert real values ranging from -∞ to +∞ inclusive into probability values ranging from 0 to 1 inclusive. In one embodiment, the logit layer 215 implements the inverse logit function or another sigmoid function to convert the vector of real values in the report vector representation 225 into occurrence probability values (occurrence probability p1(x 2 )220, p2(x 2 )230, and p Q (x 2 )235, etc.

[0053] Neural signal detection model One embodiment of the baseline neural signal detection model is described in this section. Extensions beyond the baseline model can be constructed in several ways, for example, by (i) using different versions of the attention mechanism, (ii) duplicating layers (computation), (iii) adding additional intermediate layers (e.g., transformation, normalization, regularization), and (iv) modeling multiple adverse events simultaneously (rather than independently as shown). It should be noted that the baseline neural signal detection model described herein is just one simple embodiment of neural signal detection, and in particular, a base structure that can be extended. Individual embodiments applying neural signal detection may vary in complexity and are based on this baseline neural signal detection model.

[0054]

number

[0055]

number

[0056] Training a neural signal detection model In one embodiment, a neural network trainer (such as neural network trainer 170 of signal management system 105) trains a neural signal detection model (such as the baseline neural signal detection model described above). In one embodiment, neural network trainer 170 includes one or more modules for performing the training functions described herein. Neural network trainer 170 trains the neural signal detection model using a set of reports (such as report database 280) that are to be examined following training to detect signal relationships between target information (such as a drug or other medical product) and target events (such as adverse health events) by the neural signal detection model.

[0057] In one embodiment, the training process is performed in batches of reports. The batch size can be a hyperparameter that is set to the maximum that the processor (CPU or GPU) can handle with the available memory.

[0058] Parameter Estimation—In one embodiment, the parameters of the neural signal detection model are estimated by stochastic gradient descent using the data log-likelihood function as the loss function to be minimized. The data log-likelihood function is given by the following equation:

[0059]

number

[0060]

number

[0061]

number

[0062]

number

[0063] Exemplary Methods In one embodiment, each step of the computer-implemented methods described herein may be performed by one or more computing device processors (such as processor 410 as shown and described with reference to FIG. 4 ) configured with (i) memory (such as memory 415 and / or other computing device components shown and described with reference to FIG. 4 ) and (ii) logic for causing the system to perform the method steps (such as neural signal detection logic 430 shown and described with reference to FIG. 4 ). For example, the processor accesses and reads from or writes to the memory to perform the computer-implemented method steps described herein. These steps may include (i) retrieving any necessary information, (ii) calculating, determining, generating, classifying, or otherwise creating any data, and (iii) storing any calculated, determined, generated, classified, or otherwise created data for subsequent use. References to storage or storing refer to storage as a data structure in the memory or storage / disk of a computing device (such as the memory 415 or storage / disk 435 of the computing device 405 or remote computer 465 shown and described with reference to FIG. 4, or the data store 130 shown and described with reference to FIG. 1).

[0064] In one embodiment, each next step of the method is initiated automatically in response to analysis of a received signal or retrieved stored data indicating that the previous step has been performed at least to the extent necessary for the next step to begin. Typically, the received signal or retrieved stored data indicates completion of the previous step.

[0065] 3 illustrates one embodiment of a method 300 associated with neural signal detection. In one embodiment, the steps of method 300 are performed by neural signal detection component 120 of signal management system 105 (as shown and described with reference to FIG. 1). In one embodiment, neural signal detection component 120 is a special-purpose computing device (such as computing device 405) configured with neural signal detection logic 430. In one embodiment, neural signal detection component 120 is a module of the special-purpose computing device configured with logic 430. In one embodiment, rapid or early signal detection in data is enabled by the steps of method 300. Such early signal detection could not previously be performed by a computing device.

[0066] Method 300 may be initiated automatically based on various triggers, such as in response to (i) a user (or administrator) of signal management system 105 initiating method 300, (ii) method 300 being scheduled to begin at a defined time or time interval, or (iii) receipt of a signal over the network or analysis of stored data indicating that signal management system 105 has received a new set of one or more reports (e.g., a report database). Method 300 begins at start block 305 in response to analysis of a received signal or retrieved stored data and a determination that the signal or stored data indicates that method 300 should begin. Processing continues to process block 310.

[0067] At process block 310, the processor creates a vector embedding for each of a set of one or more information objects included in the report using an embedding layer of the neural signal detection network, the set of one or more information objects including the target object.

[0068] In one embodiment, a report database (e.g., report database 180) of one or more reports stored as report data structures is maintained, for example, in data store 130. For example, a report data structure can be a row in a table of reports. In one embodiment, the report data structure includes one or more input information objects, such as one or more observed events (including a target event being examined for signal relationships with other input objects); one or more drugs, biologics, vaccines, devices, or other medical products the patient is using at the time of the event; limited patient demographic information describing the patient's characteristics; reporter information identifying the entity generating the report; or other potentially relevant information, such as foods the patient ate. A report has one or more indexes for each information (or data) object in the report. A data object for an event may include an index for the date and time of occurrence; a description of the event coded using standardized terms from a medical coding dictionary (e.g., MedDRA); and a severity determination (e.g., an event is determined to be serious if it is fatal or life-threatening, requires inpatient hospitalization or an extension of an existing hospitalization, results in persistent or significant disability or incapacity, results in a congenital abnormality, or is otherwise classified as medically significant; otherwise, the event is determined to be non-serious). A data object for a medication may include an index for the name, chemical formula (which may be automatically added in response to submitting only the name in the report), batch number, and dose. A data object for a patient may include an index for the patient's age, weight, and sex. In one embodiment, the set of M information (or data) objects included in the report to be embedded as a vector representation is all of the information objects in the report. In one embodiment, the set of M objects need not be all of the objects in the report, but is a subset of all of the information objects in the report.

[0069] In one embodiment, a processor creates, generates, or embeds a vector embedding for each of a set of one or more information objects included in a report using an embedding layer of a neural signal detection network. A vector embedding is a mapping of discrete categorical variables to vectors of real numbers. In one embodiment, vector embeddings of object primitives may be used to use those objects in the neural signal detection network. In one embodiment, an object embedding layer (e.g., object embedding layer 205) in a neural signal detection network (e.g., neural signal detection network 165) creates primitives from M objects in the report. During a training phase, the object embedding layer is trained (using reports from the report database) to convert information (or data) objects in the report into vectors representing those objects. In one embodiment, a processor retrieves a report from the report database. This report is an initial object representation of the report and includes indices that describe the M objects in the report. The processor accesses each of the M objects in the report in turn, evaluates the object's index, and generates a vector embedding of the object, i.e., a mapping of discrete values of the index to vectors of continuous dimensional values. These vectors have a dimension of d1. The object embedding layer contains M vectors, each representing one of the distinct objects being modeled. These M vectors are vector representations (such as object vector representation 250) for the information objects in the reporting database.

[0070] Once the processor has thus completed creating vector embeddings for each of the set of one or more information objects included in the report via the embedding layer of the neural signal detection network, processing at process block 310 is complete and processing continues to process block 315.

[0071] At process block 315, the processor represents the report using a representation layer of the neural signal detection network in a manner that (i) describes the correlation between the occurrence of the target event and the information objects, and (ii) takes into account the dependencies between the information objects that make up the report.

[0072] In one embodiment, the processor may apply attention weights, e.g., as shown and described with respect to Equation 4 herein, to n information objects in the embedding layer. i one for each of the n vector representations i The processor then calculates a vector representation of the report, for example, as shown and described with respect to Equation 3 herein.

[0073]

number

[0074]

number

[0075] Once the processor has thus completed representing the report using the representation layer of the neural signal detection network in a manner that describes the correlation between the occurrence of the target event and the information objects and takes into account the dependencies between the information objects that make up the report, processing at process block 315 is complete and processing continues to process block 320.

[0076] At process block 320, the processor models a set of events, including the target event, based on the representation of the reports using a logit layer of a neural signal detection network.

[0077] In one embodiment, the processor may calculate output values (intermediate linear calculations z i Calculate the value of

[0078]

number

[0079] Once the processor has thus completed modeling the set of events, including the target event, based on the representation of the reports by the logit layer of the neural signal detection network, processing at process block 320 is complete and processing continues to process block 325.

[0080] At process block 325, the processor determines the probability of occurrence for the target event given the set of modeled events.

[0081] In one embodiment, the processor converts real values representing modeled events (in the model of the event) into occurrence probabilities for the event by computing a sigmoid function, such as an inverse logit (or logistic) function that represents the occurrence of the event given the model of the event, for example in the logit layer 215. In one embodiment, a sigmoid function, such as an inverse logit (or logistic) function, hyperbolic tangent function, arc tangent function, error function, or other function, demonstrates the S-shaped curve characteristic of a sigmoid function. The processor retrieves the model of the event generated from the output values of the intermediate linear calculation, i.e., the vector representation of the entire report. In the logit layer, the processor computes a sigmoid function for the output values of the intermediate linear calculation. The value resulting from the computation of the sigmoid function is the occurrence probability for the event given the initial object representation of the report.

[0082] In one embodiment, the processor determines the probability of occurrence for a single target event from the set of events, such as shown at 220. In one embodiment, the processor determines the probability of occurrence for multiple different target events from the set of events, such as shown at 230 and 235.

[0083] The processor stores the probability of occurrence of the target event in a set of occurrence probabilities for all reports included in the report database (or in another set of reports. For example, the processor may store the probability of occurrence of the target event in a location designated for the particular report (and in further designated locations for the particular target event, if multiple target events can occur) in an occurrence probability data structure.

[0084] Once the processor has completed this process of determining the probability of occurrence for the target event given the set of modeled events, processing at process block 325 is complete and processing continues to process block 330.

[0085] In one embodiment, the process described in process blocks 310-330 for determining the probability of occurrence of a target event given a report event is repeated for multiple reports in a set of reports, such as all reports in the report database or a subset of reports in the report database. In one embodiment, any set of reports may be selected and may include all, some, or only one of the available reports. At decision block 330, the processor determines whether any reports remain in the set of reports. If reports remain in the set of reports (decision block 330: yes), processing at decision block 330 is complete and processing continues to process block 310, where processing is repeated for the next report in the set of reports. In one embodiment, the processor may increment a report index (e.g., report index i) and return to process block 310 to repeat blocks 310-325 for the report indicated by the report index. If no reports remain in the set of reports (decision block 330: no), processing at decision block 330 is complete and processing continues to process block 335.

[0086] At process block 335, the processor identifies the presence of a signal by comparing the summary probability of occurrence for the target event across all reports in the report database to a comparison baseline probability.

[0087] In one embodiment, the processor determines the signal score, for example, as shown and described with respect to Equations 12-14 below. The processor retrieves a set of occurrence probabilities for all reports for the target event, for example, by retrieving them from an occurrence probability data structure. The processor then calculates the sum of all retrieved occurrence probabilities for the target event to generate a summary occurrence probability for the target event across all reports in the report database. The processor then divides the summary occurrence probability by a comparison reference probability to generate a signal score. The comparison reference probability represents the probability of the target event if the report does not contain target information (such as a drug or other medical product) that is being examined for a relationship to the target event. The processor then compares the signal score to a threshold value indicating the presence of a signal. In one embodiment, the processor identifies the presence of a signal if the signal score meets or exceeds the threshold value, and identifies the absence of a signal if the signal score is below the threshold value. The processor writes this indication of the detected presence or absence of a signal to an output data structure for subsequent use.

[0088] The threshold value may be predetermined, for example, by selecting a threshold known to be a reasonable differentiator between signal and noise. For example, if a report contains target information (such as a drug), a target event is twice as likely to occur as if the report does not contain target information; a signal score threshold of approximately 2 is sufficient to indicate a clear signal relationship. A signal score threshold of 1-2 may also be used to increase sensitivity. In signal detection, increased sensitivity carries the risk of producing nuisance false-positive signal detections. The systems and methods for neural signal detection described herein provide higher sensitivity with a lower risk of false-positive signal detection than exists when using other signal detection algorithms. Thus, the systems and methods for neural signal detection described herein can detect signals using a lower signal score threshold of 1-2 due to the more refined correlation and dependency modeling provided by neural signal detection. Signal scores above 2 may also be used to decrease sensitivity at the expense of missing actual signal detections.

[0089] Additionally, a signal score below 1, particularly 0.5, may indicate an inverse signal relationship between the target information and the target event. For example, a signal score below 1 for a target drug and target adverse event may indicate a preventative effect of the drug on the target adverse event. For example, an additional inverse signal threshold of 0.5, indicating that the target event is half as likely to occur in the presence of the target information (drug) as in the absence of the target information, may therefore be set and compared to the signal score.

[0090] In one embodiment, the signal management system 105 may cause an indication of the presence of a signal to be displayed as an alert on a graphical user interface of the signal management system 105. In one embodiment, the indication of the presence of a signal may also trigger the signal management system 105 to initiate other processes, including sending an alert message or generating and presenting a graph showing the signal data for display. This is particularly useful for early warning regarding the detection of adverse effects of drugs or other medical products, where a rapid response can be lifesaving. Thus, in one embodiment, in which the target object is a target drug and the set of target events includes adverse health events, the system may proceed to calculate a signal score from the event probability and a comparison reference probability, determine the presence of a signal in response to the signal score exceeding a threshold indicating the presence of a signal, and immediately present an alert for display by a remote system indicating that a signal has been detected.

[0091] Once the processor has completed this process of identifying the presence of a signal by comparing the summary occurrence probability for the target event across all reports in the report database with the comparison reference probability, processing at process block 335 is complete and processing continues to end block 340 where process 300 ends.

[0092] Inference and signal score calculation In one embodiment, the signal score for neural signal detection is calculated as a ratio of the event probability (or count) for the target drug to the comparison baseline, as described above with reference to process block 335. This is consistent with other signal detection methodologies used in the pharmacovigilance space. For example, for a target drug and a target adverse event, the neural signal detection ratio is defined as the average probability of the target event given the information contained in the reports containing the target drug, divided by the average probability of the event for the set of the same reports manipulated to create the comparison baseline. In this way, each report may serve as its own control, another advantage provided by the systems and methods for neural signal detection described herein that is not widely available in other signal detection systems and methods. After the model is fitted to the data and the model parameters are estimated, the ratio (signal score) is calculated. The model is not refitted to the data to calculate the comparison baseline probability (ratio denominator). Instead, the reports or model parameters are used differently to arrive at the comparison baseline probability. Thus, in one embodiment, the signal management system 105 may calculate a signal score from a summary occurrence probability and a comparison reference probability derived from the same set of reports used to create the summary occurrence probability.

[0093]

number

[0094] Attention Mechanism In one embodiment, the report representation layer 210 includes a dependency modeling function that accounts for dependencies between the objects that make up a report when learning the vector representation 255 of the entire report. For example, if the relationship between a target drug and a target adverse event is confounded by the presence of a second drug, the report representation layer 210 learns this dependency and represents reports that include the target drug and the confounding drug differently from reports that include a different drug than the target drug and in a manner that reflects this confounding. Or, in a similar example, if the second drug interacts with the target drug rather than confounding it, the report representation layer 210 learns this dependency and represents reports that include the target drug and the interacting drug differently from reports that include a different drug than the target drug and in a manner that reflects this interaction.

[0095] In one embodiment, the neural signal detection system and method described herein uses a deep learning attention mechanism to implement the report representation layer 210 and learn a vector representation of the entire report. Attention mechanisms are widely used in natural language processing applications and are designed to mimic cognitive attention. Attention mechanisms are designed to identify, enhance, and devote more computational power to important portions of the input data while attenuating, attenuating, or ignoring the rest. Determining which portions of the data are more important than others depends on the context, i.e., the other portions of the input data (e.g., objects in the report). Determining the relative importance of various portions of the data is learned through a deep learning training process. The attention mechanism assigns context-sensitive weights to each portion of the input that reflect the relative importance of each portion relative to other portions and, therefore, relative to the whole. These weights are context-sensitive, so the same object may receive different weights depending on the other objects it appears with. Given these weights, the attention mechanism can be used to compute a context-sensitive summary of the entire report by calculating a weighted average of these portions.

[0096] In one embodiment, the input corresponds to a report represented by a set of objects, such as medications mentioned in the report. For example, the report is a data structure, such as a row object, that contains other data structures or objects about medications and about patient demographic information. This report row object may be a row in a table of such report objects in a relational database.

[0097] In one embodiment, the attention mechanism determines the predictive power that each object in the report (drug, patient demographics, patient physical condition) has on the likelihood of an adverse event relative to other objects in the report. For example, the attention mechanism learns to assign more weight to a drug known to cause a target adverse event compared to another drug that does not cause the adverse event when both the drug and another drug that does not cause the adverse event are mentioned in the same report. In this way, the neural network for neural signal detection will pay more "attention" to a drug known to cause an adverse event. The attention mechanism also learns to modify the weight of a target drug when it is reported with another drug that confounds its relationship with the adverse event or when it is reported with a drug that interacts with it to modify the likelihood of the adverse event. In this way, the neural network for neural signal detection will pay more "attention" to a target drug when other objects in the report are known to reduce the likelihood of the target event.

[0098] In one embodiment, the overall likelihood of an adverse event given all objects in a report is a function of an attention-weighted average of these objects, performed in one embodiment by a logit layer. Thus, the vector representation of the entire report is a weighted average of the vector representations of the report objects, with weights determined by the attention mechanism. The vector representation of the entire report is described more formally above.

[0099] To this end, in one embodiment, neural signal detection component 120 includes attention mechanism 180. In one embodiment, the parameters of attention mechanism 180 are learned by neural network trainer 170, which is responsible for assigning weights in the neural signal detection network during the training phase. During training, the strength and type (positive or negative) of the relationship between inputs and events is learned from the data. The attention mechanism may thus adjust the weight applied to input information based on the strength of its relationship with a chosen target effect or other input information. For example, as the strength of the relationship between the information and the effect increases, the weight may be increased.

[0100] In one embodiment, attention weights are contextual and depend on input information other than the target object. Thus, attention mechanism 180 assigns an attention weight to each of the information objects, with the attention weight assigned to each information object being based on at least one other information object among the information objects. For example, if the target object is a target drug, the relational database may indicate that the set of one or more information objects included in the report further includes a second drug that has a confounding or masking effect on the target drug. In one embodiment, the relational database may indicate that a first information object is known to cause the target event. In this case, attention mechanism 180 would assign an attention weight to each of the information objects, with a first attention weight assigned to the first information object known to cause the target event being greater than other attention weights assigned to other information objects not known to cause the target event. In one embodiment, the relational database may indicate that the first information object is known to reduce the likelihood of the target event. In this case, attention mechanism 180 will assign attention weights to each of the information objects, with a first attention weight assigned to a first information object known to reduce the likelihood of the target event being greater than other attention weights assigned to other information objects not known to reduce the likelihood of the target event.

[0101] Software Module Embodiments Generally, software instructions are designed to be executed by one or more suitably programmed processors by accessing memory, for example, by accessing CPU or GPU resources. These software instructions may include, for example, computer-executable code and source code that can be compiled into computer-executable code. These software instructions may also include instructions written in an interpreted programming language, such as a scripting language.

[0102] In a complex system, such instructions may be arranged in program modules, with each such module performing a particular task, process, function, or operation. The entire set of modules may have their operations controlled or coordinated by a main program, operating system (OS), or other form of organizational platform for the system.

[0103] In one embodiment, one or more of the components described herein are configured as modules stored on a non-transitory computer-readable medium with stored software instructions that, when executed by at least a processor accessing memory or storage, cause a computing device to perform the corresponding functions as described herein.

[0104] Cloud or Enterprise Implementation In one embodiment, the system (e.g., signal management system 105) is a computing / data processing system that includes a computing application or a collection of distributed computing applications for access and use by other client computing devices associated with an enterprise (e.g., client computers 144, 150, 155, and 160 of enterprise network 115) that communicate with the system over a network (e.g., network 110). The applications and computing system may be configured to operate with or be implemented as a cloud-based network computing system, Infrastructure as a Service (IAAS), Platform as a Service (PAAS), or Software as a Service (SAAS) architecture, or other type of networked computing solution. In one embodiment, the system provides at least one or more of the functionality disclosed herein and a graphical user interface for accessing and operating the functionality.

[0105] Computing Device Embodiments FIG. 4 illustrates an exemplary computing system 400 configured and / or programmed as a dedicated computing device using one or more of the exemplary systems and / or methods described herein and / or equivalents. The exemplary computing device may be a computer 405 including a processor 410, a memory 415, and input / output ports 420 operatively connected by a bus 425. In one example, the computer 405 may include neural signal detection logic 430 configured to facilitate neural signal detection, similar to the logic, systems, and methods shown and described with reference to FIGS. 1-3. In different examples, the neural signal detection logic 430 may be implemented in hardware, a non-transitory computer-readable medium having stored instructions, firmware, and / or combinations thereof. While the neural signal detection logic 430 is shown as a hardware component attached to the bus 425, in other embodiments, the neural signal detection logic 430 may be implemented in the processor 410, stored in the memory 415, or stored on a computer-readable medium 437 of a disk 435.

[0106] In one embodiment, neural signal detection logic 430 or computing system 400 is a means (structure: hardware, non-transitory computer-readable medium, firmware, etc.) for performing the described actions. In some embodiments, the computing device may be a server operating in a cloud computing system, a server configured in a Software as a Service (SaaS) architecture, a smartphone, a laptop, a tablet computing device, etc.

[0107] The means may be implemented, for example, as an ASIC programmed to perform neural signal detection. The means may also be implemented as stored computer-executable instructions that are temporarily stored in memory 415 and presented to computer 405 as data 440 that are then executed by processor 410.

[0108] The neural signal detection logic 430 may also provide means (eg, hardware, a non-transitory computer-readable medium storing executable instructions, firmware) for performing neural signal detection.

[0109] To generally describe an exemplary configuration of computer 405, processor 410 may be a wide variety of processors, including dual microprocessors and other multi-processor architectures. Memory 415 may include volatile memory and / or non-volatile memory. Non-volatile memory may include, for example, ROM, PROM, EPROM, EEPROM, etc. Volatile memory may include, for example, RAM, SRAM, DRAM, etc.

[0110] The storage disk 435 may be operatively connected to the computer 405, for example, through an input / output (I / O) interface (e.g., card or device) 445 and input / output port 420 controlled by at least an input / output (I / O) controller 447. The disk 435 may be, for example, a magnetic disk drive, a solid-state disk drive, a floppy disk drive, a tape drive, a zip drive, a flash memory card, a memory stick, etc. Additionally, the disk 435 may be a CD-ROM drive, a CD-R drive, a CD-RW drive, a DVD ROM, etc. The memory 415 may store, for example, processes 450 and / or data 440 formatted as one or more data structures. The disk 435 and / or memory 415 may store an operating system that controls and allocates resources of the computer 405.

[0111] Computer 405 can interact with, control, and / or be controlled by input / output (I / O) devices via input / output (I / O) controller 447, I / O interface 445, and input / output ports 420. Input / output devices include one or more displays 470, printers 472 (such as inkjet, laser, or 3D printers), and audio output devices 474 (such as speakers or headphones), text input devices 480 (such as keyboards), pointing and selection devices 482 (such as mice, trackballs, touchpads, touchscreens, joysticks, pointing sticks, stylus mice), audio input devices 484 (such as microphones), video input devices 486 (such as video and still cameras), video cards (not shown), disks 435, network devices 455, etc. Input / output ports 420 can include, for example, serial ports, parallel ports, and USB ports.

[0112] Computer 405 can operate in a networked environment and, as such, can be connected to network device 455 via I / O interface 445 and / or I / O port 420. Through network device 455, computer 405 can interact with network 460. Through network 460, computer 405 can be logically connected to remote computer 465. In one embodiment, computer 405 and other computers 465 can be configured to cooperate and function as a cloud network that provides computing services to other client computers. Networks with which computer 405 can interact include, but are not limited to, LANs, WANs, clouds, and other networks.

[0113] Definitions and Other Embodiments In another embodiment, the described methods and / or their equivalents may be implemented using computer-executable instructions. Thus, in one embodiment, a non-transitory computer-readable / storage medium is configured having stored thereon computer-executable instructions for an algorithm / executable application that, when executed by a machine, causes the machine (and / or associated components) to perform a method. Exemplary machines include, but are not limited to, processors, computers, servers operating in a cloud computing system, servers configured in a Software-as-a-Service (SaaS) architecture, smartphones, etc. In one embodiment, a computing device is implemented with one or more executable algorithms configured to perform any of the disclosed methods.

[0114] In one or more embodiments, the disclosed methods or their equivalents are performed by either computer hardware configured to perform the methods or by computer instructions embodied in modules stored on a non-transitory computer-readable medium, the instructions configured as an executable algorithm configured to perform the methods when executed by at least a processor of a computing device.

[0115] For simplicity of explanation, the illustrated methodologies in the figures are shown and described as a series of algorithmic blocks, but it should be understood that these methodologies are not limited by the order of the blocks. Some of the blocks may occur in a different order than shown and described and / or concurrently with other blocks. Also, not all illustrated blocks may be used to implement an example methodology. Blocks may be combined or separated into multiple actions / components. Furthermore, additional and / or alternative methodologies may employ additional actions not shown in the blocks.

[0116] The following contains definitions of selected terms employed herein. These definitions include various examples and / or forms of components that fall within the scope of the terms and that may be used to implement them. These examples are not intended to be limiting. Both singular and plural forms of terms may be within the scope of these definitions.

[0117] References to "one embodiment," "embodiment," "one example," "example," etc. indicate that the embodiment or example so described may include a particular feature, structure, characteristic, property, element, or limitation, but not all embodiments or examples necessarily include that particular feature, structure, feature, property, element, or limitation. Furthermore, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may.

[0118] AI: Artificial intelligence. API (application programming interface): Application programming interface.

[0119] ASIC (application specific integrated circuit): An integrated circuit for a specific application. BLR: Binary logistic regression.

[0120] CD (compact disk): Compact disc. CD-R (CD recordable): Recordable CD.

[0121] CD-RW (CD rewriteable): Rewritable CD. CPU (central processing unit): Central processing unit.

[0122] DNN: Deep Neural Network. DVD: Digital versatile disk and / or digital video disk.

[0123] ELR: Extended logistic regression. GPU: Graphics Processing Unit.

[0124] GUI: Graphical User Interface. HPC: High performance computing.

[0125] HTTP (hypertext transfer protocol): Hypertext Transfer Protocol. IAAS: Infrastructure as a Service.

[0126] IC: Information Component. I / O: Input / Output.

[0127] JSON: JavaScript Object Notation. LAN (local area network): Local area network.

[0128] MGPS: Multi-item Gamma Poisson contractile system. ML: Machine learning.

[0129] NAS: Network Attached Storage. NLG: Natural Language Generation.

[0130] NLP: Natural Language Processing. PAAS: Platform as a Service.

[0131] PCI (peripheral component interconnect): Peripheral Component Interconnect.

[0132] PRR: Proportional Reporting Ratio. RAM (random access memory): Random access memory.

[0133] DRAM (dynamic RAM): Dynamic RAM. SRAM (synchronous RAM): Synchronous RAM.

[0134] ReLU (rectified linear unit): Rectified linear unit. REST: Representational State Transfer.

[0135] RGPS: Regression adjusted gamma-Poisson contractor. ROM (read only memory): Read-only memory.

[0136] PROM (programmable ROM): Programmable ROM. EPROM (erasable PROM): Erasable PROM.

[0137] EEPROM (electrically erasable PROM): Electrically erasable PROM. SaaS: Software as a Service.

[0138] SOAP: Simple Object Access Protocol. SQL (structured query language): Structured query language.

[0139] TCP / IP: Transmission Control Protocol / Internet Protocol. USB (universal serial bus): Universal Serial Bus.

[0140] XML (extensible markup language): Extensible Markup Language. WAN (wide area network): Wide area network.

[0141] As used herein, a "data structure" is an organization of data in a computing system, stored in memory, a storage device, or other computerized system. A data structure may be, for example, any one of a data field, a data file, a data array, a data record, a database, a data table, a graph, a tree, a linked list, etc. A data structure may be formed from and contain many other data structures (e.g., a database contains many data records). According to other embodiments, other examples of data structures are possible as well.

[0142] As used herein, "computer-readable medium" or "computer storage medium" refers to a non-transitory medium that stores instructions and / or data that, when executed, are configured to perform one or more of the disclosed functions. In some embodiments, data may function as instructions. Computer-readable media may take forms including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical disks, magnetic disks, and the like. Volatile media may include, for example, semiconductor memory, dynamic memory, and the like. Common forms of computer-readable media may include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tape, other magnetic media, application-specific integrated circuits (ASICs), programmable logic devices, compact disks (CDs), other optical media, random access memory (RAM), read-only memory (ROM), memory chips or cards, memory sticks, solid-state storage devices (SSDs), flash drives, and other media with which a computer, processor, or other electronic device can function. Each type of media may include stored instructions of an algorithm that, when selected for implementation in an embodiment, is configured to perform one or more of the functions disclosed and / or claimed.

[0143] As used herein, "logic" refers to components implemented using computer or electrical hardware, non-transitory media having executable application or program module instructions stored thereon, and / or combinations thereof, to perform any of the functions or actions as disclosed herein and / or to cause functions or actions from other logic, methods, and / or systems to be performed as disclosed herein. Equivalent logic may include firmware, a microprocessor programmed with an algorithm, discrete logic (e.g., an ASIC), at least one circuit, analog circuit, digital circuit, programmed logic device, memory device containing algorithmic instructions, etc., any of which may be configured to perform one or more of the disclosed functions. In one embodiment, logic may include one or more gates, combinations of gates, or other circuit components configured to perform one or more of the disclosed functions. Where multiple logics are described, it may be possible for the multiple logics to be incorporated into one logic. Similarly, where a single logic is described, it may be possible for the single logic to be distributed among multiple logics. In one embodiment, one or more of these logics are corresponding structures associated with performing the disclosed and / or claimed functions. The choice of which type of logic to implement may be based on desired system requirements or specifications. For example, if faster speed is a consideration, hardware would be selected to implement the function. If lower cost is a consideration, stored instructions / executable applications would be selected to implement the function.

[0144] An "operable connection," or a connection by which entities are "operably connected," is a connection through which signals, physical communications, and / or logical communications may be sent and / or received. An operable connection may include a physical interface, an electrical interface, and / or a data interface. An operable connection may include different combinations of interfaces and / or connections sufficient to enable operable control. For example, two entities may be operably connected to communicate signals with each other directly or through one or more intermediary entities (e.g., a processor, an operating system, logic, a non-transitory computer-readable medium). Logical and / or physical communication channels may be used to create an operable connection.

[0145] A "user" as used herein includes, but is not limited to, one or more humans, one or more computers or other devices, or a combination thereof.

[0146] While the disclosed embodiments have been shown and described in considerable detail, it is not intended to limit, or in any way restrict, the scope of the appended claims to such detail. It is, of course, not possible to describe every conceivable combination of components or methodologies to describe various aspects of the subject matter. Accordingly, this disclosure is not limited to the specific details or illustrative examples shown and described. As such, this disclosure is intended to embrace changes, modifications, and variations that fall within the scope of the appended claims.

[0147] To the extent the term "includes" or "comprising" is used in the detailed description or claims, it is intended to be inclusive in the same manner as if the term "comprises" were interpreted when used as a transitional phrase in the claims.

[0148] To the extent the term "or" (e.g., A or B) is used in the detailed description or claims, it is intended to mean "A or B or both." If applicant intended to indicate "A or B only, but not both," the phrase "A or B only, but not both" would be used. Thus, use of the term "or" herein is inclusive, not exclusive.

Claims

1. 1. A computer-implemented method comprising: For multiple reports in the report database, generating, by an embedding layer of a neural signal detection network, a vector embedding for each of a set of one or more information objects included in the report, the set of one or more information objects including a target object, the method further comprising: representing the report using a representation layer of the neural signal detection network in a manner that describes correlations between the occurrence of a target event and the information objects and takes into account dependencies between the information objects that make up the report; modeling, by a logit layer of the neural signal detection network, a set of events including the target event based on the representation of the reports; determining a probability of occurrence for the target event given the set of modeled events; and identifying the presence of a signal by comparing a summary occurrence probability for the target event across the plurality of reports in the report database to a comparison reference probability.

2. 2. The computer-implemented method of claim 1, further comprising calculating a signal score from the summary occurrence probability and a comparison reference probability, the comparison reference probability being derived from the same set of reports used to create the signal score by recalculating further summary occurrence probabilities with the target object removed from each report used to calculate the summary occurrence probability.

3. 2. The computer-implemented method of claim 1, further comprising calculating a signal score from the summary occurrence probability and a comparison reference probability, the comparison reference probability being derived from the same set of reports used to create the signal score by recalculating a further summary occurrence probability from each report used to calculate the summary occurrence probability with an attention weight assigned to the target object for each report set to zero.

4. 4. The computer-implemented method of claim 1, claim 2, or claim 3, further comprising training the neural signal detection network by estimating parameters of the neural signal detection network by stochastic gradient descent using data log-likelihood as a loss function to be minimized.

5. further comprising the step of assigning an attention weight to each of the information objects; 5. A computer-implemented method as described in claim 1, claim 2, claim 3, or claim 4, wherein (i) a first attention weight assigned to a first information object known to cause the target event is greater than other attention weights assigned to other information objects not known to cause the target event, or (ii) a first attention weight assigned to a first information object known to reduce the likelihood of the target event is greater than other attention weights assigned to other information objects not known to reduce the likelihood of the target event.

6. 6. The computer-implemented method of claim 1, claim 2, claim 3, claim 4, or claim 5, wherein the target object is a target medication and the set of target events includes (i) adverse health events, or (ii) positive health events.

7. A program that causes at least a processor of a computer to execute a computer-implemented method described in any one of claims 1 to 6.

8. 1. A computing system comprising: a processor; a memory operatively connected to the processor; a non-transitory computer-readable medium operatively connected to the processor and memory and storing computer-executable instructions, the computer-executable instructions causing at least a processor of the computer to perform the computer-implemented method of any one of claims 1 to 6.

Images