Updating detection models and maintaining data confidentiality
The system efficiently updates analytical models across local nodes by using a monitoring, diagnostic, and evaluation module with a central module, ensuring secure and rapid distribution of model updates, addressing the challenges of manual updates and data confidentiality.
Patent Information
- Application Number
- JP2022517804
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-20
- Filing Date
- 2020-09-15
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2040-09-15
AI Technical Summary
Existing analytical models face challenges in rapid and secure updating across multiple local nodes while maintaining data confidentiality, as third-party operatives often evade detection, and manual updates are time-consuming.
A system comprising a monitoring, diagnostic, and evaluation module at each local node, with a central module (optional) for distributing model updates, ensuring data confidentiality by creating model updates locally and determining permission for application.
Enables rapid and secure distribution of model updates across local nodes, enhancing detection model accuracy and timeliness while preserving sensitive information.
Smart Images

Figure 0007578364000003 
Figure 0007578364000004 
Figure 0007578364000005
Abstract
Description
[Technical field]
[0001] This application relates generally to a system for evaluating and sharing detection models while maintaining data confidentiality, and methods for using such a system. [Background technology]
[0002] Analytical models for event detection are important for many different domains and industries. For example, various analytical models are used to detect financial fraud, aid in regulatory compliance, and many other complex data-driven problems. Many domains require up-to-date models for accurate and timely event detection. In some domains (e.g., many types of fraud), third-party operatives are actively trying to evade detection by current analytical models. Thus, a system is needed for updating detection models that allows model updates to be distributed across multiple local nodes of the system, analyzed, and implemented in a rapid manner. Furthermore, since manual creation of updated detection models can be a slow and time-consuming process, a system is needed that creates model updates for the nodes themselves from successful event detections and then distributes the created model updates to other parts of the system while maintaining data confidentiality of the location where the model updates were created. Summary of the Invention
[0003] One aspect of the present invention provides a system for updating detection models, the system comprising at least one local node having a monitoring module, a diagnostic module, an evaluation module, one or more current detection models, and system data generated by the current detection models, and a memory containing instructions, which are executed by at least one processor and are configured to perform the steps of receiving, by the monitoring module, model update information; determining, by the diagnostic module, a current detection model; determining, by the evaluation module, whether the model update information should be applied to the current detection model; determining, by the evaluation module, whether the local node is authorized to apply the model update information; and updating, by the evaluation module, the current detection model using the model update information.
[0004] In some embodiments, the system further comprises at least a second local node, a central module, with which the monitoring modules of each local node are in electronic communication, a database of all models available in the system in electronic communication with the central module, and a memory containing instructions, which are executed by the at least one processor, to generate model update information, the model update information generating step comprising: detecting, by the diagnostic module, a significant change in the system data; determining, by the diagnostic module, a list of all current detection models involved in the detection step; analyzing, by the diagnostic module, the system data involved in the detection step; generating, by the diagnostic module, the model update information; and transmitting, by the monitoring module, the model update information. the central module, a distributing step including analyzing a database of available models by the central module, determining a priority of the model update information by the central module, determining which local nodes should receive the model update information by the central module, and sending the model update information by the central module; and updating at least one local node, the distributing step including receiving the model update information by the at least one monitoring module, determining a current detection model by the at least one diagnostic module, determining by the at least one evaluation module whether the model update information should be applied to the current detection model, determining by the at least one evaluation module whether the local node has permission to apply the model update information, and updating the current detection model with the model update information by the at least one evaluation module.
[0005] According to another aspect of the present invention, a computer-implemented method is provided in a data processing system having a processor and a memory containing instructions, which, when executed by the processor, cause the processor to implement a system update detection model, the method including updating at least one local node, including receiving model update information by a monitoring module of at least one local node, determining by a diagnostic module of the local node a current detection model in use by the local node, determining by an evaluation module of the local node whether the model update information should be applied to the current detection model, determining by the evaluation module of the local node whether the local node is authorized to apply the model update information, and updating the current detection model using the model update information by the evaluation module of the local node.
[0006] According to another aspect of the present invention, there is provided a computer program product for updating a detection model, the computer program product comprising at least one computer readable storage medium having program instructions embodied thereon, the program instructions being executable by a processor to cause the processor to update local nodes by receiving, via at least one monitoring module, model update information; determining, via at least one diagnostic module, a current detection model; determining, via at least one evaluation module, whether the model update information should be applied to the current detection model; determining, via the at least one evaluation module, whether at least one local node is authorized to apply the model update information; and updating, via the at least one evaluation module, the current detection model using the model update information.
[0007] According to yet another aspect of the present invention, there is provided a computer program product for creating and updating detection models, the computer program product comprising at least one computer readable storage medium having program instructions embodied thereon, the program instructions being executable by a processor to cause the processor to: create model update information, the model update information including detecting, by a diagnostic module, a significant change in system data; determining, by the diagnostic module, a list of all current detection models involved in detecting the significant change; analyzing, by the diagnostic module, the system data involved in detecting the significant change; generating, by the diagnostic module, the model update information, the generated model update information not including any system data; generating, by a monitoring module, the model update information; and distributing, by a monitoring module, the model update information, receiving, by a central module, the model update information. and distributing, comprising: analyzing, by the central module, a database of available models; determining, by the central module, a priority of the model update information; determining, by the central module, which local nodes should receive the model update information; and sending, by the central module, the model update information to the at least one local node; and updating, by the at least one local node, comprising receiving, by the at least one monitoring module, determining, by the at least one diagnostic module, a current detection model; determining, by the at least one evaluation module, whether the model update information should be applied to the current detection model; determining, by the at least one evaluation module, whether the local node is authorized to apply the model update information; and updating, by the at least one evaluation module, the current detection model using the model update information.
[0008] According to yet another aspect of the present invention, there is provided a computer program product for creating and updating detection models, the computer program product comprising at least one computer readable storage medium having program instructions embodied thereon, the program instructions being executable by a processor and causing the processor to generate model update information, the program instructions including: detecting, by a first diagnostic module, a significant change in system data; determining, by the first diagnostic module, a list of all current detection models involved in detecting the significant change; analyzing, by the first diagnostic module, the system data involved in detecting the significant change; and generating, by the first diagnostic module, the model update information, the generated model update information. and updating at least a second local node, the generating including generating, by the first monitoring module, the model update information not including any system data, and sending, by the first monitoring module, the model update information; and updating at least a second local node, the updating including receiving, by the at least the second monitoring module, the model update information, determining, by the at least the second diagnostic module, a current detection model, determining, by the at least the second evaluation module, whether the model update information should be applied to the current detection model, determining, by the at least the second evaluation module, whether the local node is authorized to apply the model update information, and updating, by the at least the second evaluation module, the current detection model using the model update information.
[0009] Other features and advantages of the present disclosure will become apparent from the following detailed description of example embodiments that proceeds with reference to the accompanying drawings.
[0010] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings, in which: For the purpose of illustrating the disclosure, there are shown in the drawings presently preferred embodiments, it being understood, however, that the disclosure is not limited to the particular embodiments disclosed. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 illustrates a block diagram of an exemplary update system that includes a single local node. [Diagram 2] FIG. 1 illustrates a block diagram of an exemplary update system that includes multiple local nodes connected by a central module. [Diagram 3] FIG. 1 illustrates a block diagram of an exemplary update system that includes multiple directly connected local nodes. [Figure 4] FIG. 1 illustrates a flow chart of an exemplary method for updating an analysis system using an update system and manually created model update information. [Diagram 5] FIG. 1 illustrates a flow chart of an exemplary method in which local nodes are connected by a central module and update an analytical system using model updates created by one of the local nodes. [Figure 6] FIG. 13 illustrates a flow chart of an exemplary method for updating an analytical system using model update information created by one of the directly connected local nodes. [Figure 7] FIG. 1 illustrates a block diagram of an exemplary data processing system in which aspects of example embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] In the description and claims, the terms "a," "at least one of," and "one or more of" may be used in reference to certain features and elements of example embodiments. It should be understood that these terms and phrases are intended to state that there is at least one of a particular feature or element present in a particular example embodiment, but that there may be more than one. That is, these terms / phrases are not intended to limit the description or claims to a single feature / element present, nor are they intended to require that there be a plurality of such features / elements. To the contrary, these terms / phrases require only at least a single feature / element, and that there may be a plurality of such features / elements within the description and claims.
[0013] In addition, it should be understood that the following description uses multiple different examples of the various elements of the example embodiments to further explain the exemplary implementation of the example embodiments and to aid in understanding the mechanisms of the example embodiments. These examples are intended to be non-limiting and do not exhaust the various possibilities of implementing the mechanisms of the example embodiments. In view of this description, it will be apparent to those skilled in the art that there are many other alternative implementations of these various elements that can be utilized in addition to or in place of the examples provided herein without departing from the scope of the present disclosure.
[0014] The present disclosure may be a system, method, and / or computer program product, and may include a computer-readable storage medium including computer-readable program instructions for causing a processor to perform aspects of the present disclosure.
[0015] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable floppy disks, head disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combinations of these. As used herein, computer-readable storage media should not be construed as being, itself, ephemeral signals such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.
[0016] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network (e.g., the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network). The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage on the computer readable storage medium within each computing / processing device.
[0017] The computer readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Java(R)(TM), Smalltalk, C++, and traditional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present disclosure, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to customize the electronic circuitry by utilizing state information of the computer-readable program instructions.
[0018] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks included in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0019] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium in a specific manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams, and may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function.
[0020] Computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in the blocks of the flowcharts and / or block diagrams, thereby causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process.
[0021] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks included in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0022] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or in reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flowchart illustrations, as well as combinations of blocks included in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.
[0023] In overview, a cognitive system is a specialized computer system, or set of computer systems, configured using hardware logic and / or software logic (combined with hardware logic on which software runs) to emulate human cognitive functions. These cognitive systems apply human-like characteristics to communicating and manipulating ideas, and when combined with the inherent strengths of digital computing, can solve large-scale problems with high accuracy and flexibility. IBM® Watson™ is one example of such a cognitive system, which can process human-readable language and discern inferences between passages of text with human-like accuracy, at much higher speeds and in greater volume than humans. In general, such cognitive systems can perform the following functions:
[0024] [Table 1]
[0025] Embodiments herein relate to a system for updating analytical models across multiple local nodes. As used herein, an individual "local node" refers to software installed by an end user, such as an individual or a company. In some embodiments, a local node includes one computer system. In some embodiments, a local node includes multiple computer systems or servers controlled by an end user. In some embodiments, each local node in the system uses a set of current analytical models that are unique to that local node. In some embodiments, each local node in the system accesses and analyzes system data generated by one or more analytical models. This system data is unique to each local node and may include sensitive or confidential information.
[0026] As used herein, an individual "analytic model" or simply "model" is a software algorithm designed to detect a particular event using data analysis techniques. In some embodiments, the analytical model detects anomalies in the data. In some embodiments, the analytical model detects fraudulent events. In some embodiments, the data analysis techniques used by the analytical model include, but are not limited to, data pre-processing techniques, calculation of one or more statistical parameters, classification techniques such as statistical ratios based on classification or groups, calculation of probabilities, clustering of data and matching of data, regression analysis, and gap analysis. In some embodiments, the software of the local node includes one or more analytical models. In some embodiments, the software of the local node includes one or more analytical models and deterministic rules. In some embodiments, the software of the local node includes one or more analytical models for fraud detection. In some embodiments, the software of the local node includes one or more analytical models for compliance or non-compliance with regulations. In some embodiments, the software of the local node includes one or more models and deterministic rules for fraud detection. In some embodiments, the software of the local node includes one or more models and deterministic rules for compliance or non-compliance with regulations.
[0027] In some embodiments, the update system receives one or more model updates and pushes the updates to applicable local nodes. In some embodiments, the update system pushes the updates to all local nodes in the system. In some embodiments, the update system pushes the updates to only selected local nodes. In some embodiments, the update system determines which local nodes receive the model update push.
[0028] In some embodiments, each local node that receives a model update checks the update against a current model of the analysis system, and if applicable, the update system updates the current model. In some embodiments, the update system receives one or more manually created model updates. In some embodiments, the update system receives one or more model updates created by local nodes of the update system. In some embodiments, local nodes of the update system are connected by a central hub or central module that is not itself a local node. In some embodiments, local nodes of the update system are directly connected to each other, for example, as a distributed network.
[0029] In some embodiments, the update system, including any local nodes, is a standalone system that creates and pushes model updates to any software systems that use the analytical models, hi some embodiments, the update system itself is a component or subsystem of a larger analytical system (e.g., an analytical system for fraud detection).
[0030] 1 shows a block diagram representation of the components, outputs, and data flow of an exemplary single local node of an update system 100. The local node contains three main modules or subsystems: a monitoring module 101, a diagnostic module 102, and an evaluation module 103.
[0031] The monitoring module 101 monitors one or more factors to determine if a model update process is needed. In some embodiments, the monitoring module 101 checks the time since the last update process and initiates the update process if sufficient time has passed. In some embodiments, the monitoring module 101 initiates the update process if 6 hours, 12 hours, 1 day, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, 10 days, 15 days, 30 days, 1 month, 2 months, 3 months, 6 months, or 1 year have passed since the last update process. In some embodiments, the monitoring module 101 initiates the update process if it receives model update information pushed from a source outside the local node 100. For example, the monitoring module 101 may receive model update information pushed from a central module of the update system, from another local node, or directly from an update system administrator.
[0032] In some embodiments, the monitoring module 101 may initiate the update process when signaled by the diagnostic module 102. In some embodiments, the diagnostic module 102 may analyze the system data 104 and signal the monitoring module 101 to initiate the update process when one or more data thresholds are met. For example, the diagnostic module 102 may signal the monitoring module 101 when the diagnostic module's analysis of the system data 104 indicates an increase in event detection above a data threshold or a decrease in event detection below a data threshold. In some embodiments, the data threshold may be set manually, for example, by an end user. In some embodiments, the data threshold may be determined automatically by the diagnostic module 102, for example, when the event detection rate increases above a one-week moving average of the detection rate by a significant amount.
[0033] If an update process is initiated, the monitoring module 101 inquires about available model updates. In some embodiments, the monitoring module 101 queries a central module of the update system, another local node, or an update system administrator. In some embodiments, if an update process is initiated by a model update pushed from a source external to the local node 100, the monitoring module 101 does not inquire about additional available model updates. In some embodiments, if an update process is initiated by a model update pushed from a source external to the local node 100, the monitoring module 101 does inquire about additional available model updates.
[0034] When the monitoring module 101 has completed all available queries and received at least one model update, the monitoring module 101 passes the model update to the diagnostic module 102. The diagnostic module 102 compares the model update to a database 105 of current models available on the local node. In some embodiments, the diagnostic module 102 classifies the model update into current models 105, regardless of whether those current models 105 are actively in use. In some embodiments, the diagnostic module 102 classifies the model update into system data 104 generated by application of active current models 105.
[0035] When the diagnostic module 102 receives the model update information and at least compares the model update information to the database 105 of current models, the diagnostic module 102 passes the model update information and all available comparison and other analysis data to the evaluation module 103. The evaluation module 103 evaluates the model update information and determines whether the update information 106 should be applied. In some embodiments, the evaluation module 103 automatically applies the model update information and uses the model update information to change or modify the current model 105. In some embodiments, the evaluation module 103 analyzes the model update information and determines whether such a model already exists in the database 105 of current models. In some embodiments, the evaluation module 103 performs model updates on the associated system data 104 or associated category data generated by the diagnostic module 102 and determines whether the model update information provides the local node 100 with different system data than the current model 105 can generate. In some embodiments, the evaluation module 103 does not automatically apply any updates or perform any analysis unless authorized by an end user or administrator of the local node 100.
[0036] FIG. 2 illustrates a block diagram representation of the components, outputs, and data flow of an update system that includes multiple local nodes 200. The update system includes a central module 201 that connects all local nodes in the update system 200. FIG. 2 illustrates two local nodes, generally labeled 210 and 220. In some embodiments, there is no limit to the number of local nodes that may be present in the update system 200. It should be understood that the local nodes 210 and 220 are generally similar to the local nodes described in FIG. 1 and include monitoring modules 211, 221, diagnostic modules 212, 222, and evaluation modules 213, 223, respectively. Each local node further includes its own system data 214, 224, and current model databases 215, 225. It should be understood that each local node may include different system data and current models. In some embodiments, system data 214 may be the same as or similar to system data 224, or may be non-identical or similar to system data 224. In some embodiments, current model 215 may be the same as or similar to current model 225, or may be non-identical or similar to current model 225.
[0037] The central module 201 is not present in any of the local nodes, but in a separate location, such as a centralized management server. In some embodiments, the central module 201 can send and receive information to and from the monitoring modules 211, 221. In some embodiments, the central module 201 can send and receive information to and from any monitoring module in the update system. The central module 201 can access a master database 202 of models available to the update system. The database of available models 202 is a list of all possible analytical models currently present in the update system. In some embodiments, the database of current models (e.g., current models 215) in the individual nodes is equivalent to the dataset of available models 202. In some embodiments, the database of current models (e.g., current models 215) in the individual nodes is not equivalent to the dataset of available models 202, but includes at least one model in common with the database of available models 202.
[0038] In some embodiments, when a monitoring module in an individual node (eg, monitoring module 211) initiates a query about available model update information, the monitoring module communicates electronically with the central module 201.
[0039] In some embodiments, an individual node may communicate with one or more end users. In FIG. 2, for example, evaluation module 213 of node 210 may communicate with end user 217. In some embodiments, any module of an individual node may communicate with an end user. In some embodiments, an individual node may communicate with an end user to provide the end user with information regarding the update process. In some embodiments, an individual node may communicate with an end user to provide the end user with information regarding the results of the update (e.g., which models were updated). In some embodiments, an individual node may communicate with an end user to ask for permission from the end user before updating any models.
[0040] FIG. 3 illustrates another block diagram representation of components, outputs, and data flow of an update system including multiple local nodes 300. Update system 300 illustrates two local nodes generally labeled as 310 and 320. In some embodiments, there is no limit to the number of local nodes that may be present in update system 300. It should be understood that local nodes 310 and 320 are generally similar to the local nodes described in FIGS. 1 and 2 and include monitoring modules 311, 321, diagnostic modules 312, 322, and evaluation modules 313, 323, respectively. Each local node further includes its own system data 314, 324, and current model databases 315, 325. It should be understood that each local node may include different system data and current models. In some embodiments, system data 314 may be the same as or similar to system data 324, or may not be the same as or similar to system data 324. In some embodiments, current model 315 may be identical to or similar to current model 325 , or may be non-identical and dissimilar to current model 325 .
[0041] Unlike FIG. 2, update system 300 does not include any type of central module that connects all of the local nodes. Instead, each local node is directly connected to each other via a network. In some embodiments, each monitoring module is in electronic communication with every other monitoring module in update system 300. For example, as shown in FIG. 3, monitoring module 311 is in electronic communication with monitoring module 321.
[0042] In some embodiments, when an update process is initiated at an individual node, the monitoring module of that node queries other local nodes in the update system 300. For example, when an update process is initiated at local node 310, monitoring module 311 queries monitoring module 321 of local node 320. In some embodiments, when an update process is initiated at an individual node, the monitoring module of that node queries all other local nodes in the update system 300. In some embodiments, when an update process is initiated at an individual node, the monitoring module of that node queries only selected other nodes in the update system 300. In some embodiments, when an update process is initiated at an individual node, the monitoring module of that node queries only one other node in the update system 300.
[0043] In any embodiment herein, a system administrator may create an updated model and manually add it to the update system. For example, a system administrator may create an updated model and submit the model to the central module 201 as shown in FIG. 2. In another example, a system administrator may create an updated model and submit the model to the monitoring module 321 as shown in FIG. 3. In some embodiments, when an updated model is added to any update system described herein, an update process may be initiated across all or a portion of the nodes in the update system.
[0044] In any embodiment herein, any local node of the update system may generate model updates and automatically push them to other local nodes of the update system. In some embodiments, a local node generating its own model updates is advantageous because the update system can quickly respond to increases in fraud detection without end user or administrator involvement. For example, as shown in FIG. 2, the diagnostic module 212 analyzes the system data 214 and detects an increase in fraud detection greater than a pre-set threshold. The diagnostic module 212 then creates a list of one or more models that were used to detect the increase in fraud and analyzes the system data 214 to determine key characteristics and conditions of the nexus between the one or more models and the data. The diagnostic module 212 then removes from the models the system data 214 and all data unique to the local node 210. The monitoring module 211 then sends the model to the central module 201, which then determines whether the model is applicable to the update system 200 as a model update.
[0045] In any embodiment where a local node is generating model updates for the update system, it is important that the local node's unique system data is not shared with the central hub or other local nodes in the update system. In some embodiments, the diagnostic module that creates the model to be shared with the update system creates a new model that is independent of any unique system data from the local node. In some embodiments, the new model includes one or more of the following: one or more algorithms, metadata or high level aggregate statistics such as a creation date and time, a number of events detected over a certain period of time, a total transactional value of time, and one or more threshold points used to trigger the update. In some embodiments, the new model includes average ratio statistics for one or more data groups. In some embodiments, the new model can detect deviations from average ratio statistics for one or more data groups to determine future positive outcomes. In some embodiments, the new model includes one or more network or image graphics representing the one or more models. In some embodiments, the new model includes one or more network or image graphics representing the new model.
[0046] In any of the embodiments herein, the components of the update system may be stored in the same location, for example as software installed within an internal server system of an enterprise such as a bank, In some embodiments, some of the components of any of the update systems disclosed herein are stored in a different location, such as as part of a cloud-based service.
[0047] FIG. 4 illustrates a flow chart of an exemplary method for using manually created models to push model update information to an update system including a central module 400. In some embodiments, method 400 can be used in the update system illustrated in FIG. 2. First, a system administrator manually creates 401 a new model that will be used to update the system. In some embodiments, the new model includes one or more new or updated algorithms. In some embodiments, the new model includes information regarding the criteria required for use of the new model (e.g., type of business, amount of system data required, or type of detection performed by the model). In some embodiments, the new model includes priority information regarding how important the model is to the update system. For example, a new model that must be output to all local nodes is given the highest priority possible. In some embodiments, the priority information is classified as either low priority, medium priority, or high priority.
[0048] Next, the new model created by the system administrator is pushed to the update system, which receives the model (402). In some embodiments, a central module of the update system receives the model. After receiving the model (402), the central module updates the model database (403). For example, the central module 201 updates the available models database 202 in the update system 200 shown in FIG. 2.
[0049] The update system then determines which end users are applicable to the new model (404). In some embodiments, a central module has determined which end users are applicable. In some embodiments, the central module determines which end users are applicable to the model update by comparing criteria information in the new model to information about each end user, in addition to priority information in the new model. For example, if the model update for credit card fraud detection has a medium priority, the central module identifies which local nodes in the update system are involved in credit card fraud detection and then outputs the model to those identified local nodes (405). The model update is not output to any remaining local nodes, but a local node may receive the update when each of those remaining local nodes has begun the update process, e.g., if enough time has passed without an update to trigger a monitoring module. In another example, if the model update for credit card fraud detection has a high priority, the central module outputs the model to all local nodes (405). In another example, if a model update for credit card fraud detection has low priority, the central module may not immediately output the model to any local nodes, but instead wait for each local node to initiate the update process on its own.
[0050] After the model update information is transmitted from the central module, the model update information is received by at least one local node (411). In some embodiments, the model update information is received by multiple local nodes simultaneously. In some embodiments, the model update information is received by a monitoring module in any of the embodiments described herein.
[0051] In some embodiments, after the local node receives 411 the model update, the model update is not automatically installed. First, the local node queries the current model database to see if the model update replaces any existing models 412. Next, the local node determines 413 the relevance of the model update to the node. For example, at the local node 210 of the update system 200 shown in FIG. 2, the model update is received by the monitoring module 211 and then passed to the diagnostic module 212. The diagnostic module 212 first queries the current model database 215 and then determines the relevance of the model update to the local node 210. In some embodiments, the diagnostic module 212 terminates the update process after the relevance determining step 413. In some embodiments, if the diagnostic module 212 determines after the relevance determining step 413 that the model update is not needed by the local node, the diagnostic module 212 terminates the update process. In some embodiments, if the diagnostic module 212 determines after the determining relevance step 413 that the model update already exists on the local node, the diagnostic module 212 terminates the update process. In some embodiments, if the model update has a high priority, the diagnostic module 212 automatically bypasses the determining relevance step 413.
[0052] In some embodiments, after the local node determines that the model update is relevant or necessary, the local node determines whether it is authorized to apply the model update (414). In some embodiments, an evaluation module of the local node determines whether the local node is authorized to apply the model update. In some embodiments, the local node is not authorized to install the model update. In some embodiments, the local node is not automatically authorized to install the model update. In some embodiments, the local node must query or ask the end user for permission before installing the model update (416). For example, after the diagnostic module 212 determines that the model update is either relevant or that the model update has a high enough priority to bypass the determining relevance step 413, the model update is passed to the evaluation module 213. The evaluation module 213 then checks the update permission settings of the local node. In some embodiments, if the evaluation module 213 determines that it is not authorized to install the model update, the evaluation module 213 terminates the update process. In some embodiments, the evaluation module 213 queries the end user before installing the model update, for example, by issuing a prompt to the user or by sending an email or other communication to the end user.
[0053] After the local node determines that it has permission to install the model update, the local node installs 415 the model update. In some embodiments, the evaluation module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models into a current model database in the local node. In some embodiments, the model update replaces one or more models in a current model database in the local node. For example, after permission is established, the evaluation module 213 updates the current model database 215 with the model update.
[0054] In some embodiments, after the update 415 is completed, the local node creates 417 an output report. In some embodiments, the output report is shared with an end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report includes information about the model update information, such as the type of model that was updated, whether any old models were replaced, the date and time of the update, whether the new model is currently active, or any combination of these.
[0055] FIG. 5 illustrates a flow chart of an exemplary method for pushing model updates to an update system including a central module, where the model updates are automatically created from a local node in the update system 500. In some embodiments, the method 500 can be used in the update system illustrated in FIG. 2. Initially, a local node in the update system detects a change in results from an existing model (501). In some embodiments, the local node in the update system detects a change in fraud detection rate. In some embodiments, the change in fraud detection rate is an increase in fraud detections greater than a pre-set threshold. In some embodiments, the change in fraud detection rate is a significant increase or decrease in fraud over a particular period of time. In some embodiments, the local node in the update system detects a change in the magnitude of detected fraud. In some embodiments, the change in fraud magnitude is an increase in the value or monetary amount of detected fraud events greater than a pre-set threshold. In some embodiments, the change in fraud magnitude is a significant increase in the value or monetary amount of detected fraud events compared to a moving average of detected events. For example, a diagnostic module 212 as shown in FIG. 2 may analyze system data 214 to detect an increase in fraud detection rates that is more than a standard deviation away from a three-month moving average of the fraud detection rates.
[0056] After a change in results from an existing model is detected (501), the local node creates a list of all models involved in the detection (502). In some embodiments, the local node creates a list of all models directly involved in generating the event detected in step 501. In some embodiments, the local node creates a list of all models that were directly and indirectly involved in generating the event detected in step 501. In some embodiments, the local node creates a list of all models that were actively running when the event was detected in step 501. For example, the diagnostic module 212, which has access to both the system data 214 and the current models database 215, creates a list of all algorithmic models that were directly and indirectly involved in generating the fraudulent event previously detected in step 501.
[0057] After the local node creates (502) a list of models relevant to the detected change (501), the local node analyzes (503) the data involved in generating the event leading to the detected change. In some embodiments, the local node analyzes system data to determine features and conditions associated with the models listed in step 502 in generating the event detected in step 501. In some embodiments, the local node's analysis may include, but is not limited to, ordinary least squares, penalized regression, generalized additive models, quantile regression, logistic regression, and gated linear models. In some embodiments, the local node's analysis results in a transformed variant of one or more relevant models to reduce the complexity of those models. Examples include placing monotonicity constraints on nonlinear, non-monotonic models to fit the model around variable relationships known to be true, or utilizing monotonic neural networks in machine learning applications. In some embodiments, the associated visualization is a relevant but less complex model that approximates one or more applicable models, particularly machine learning models. Examples include surrogate models, locally interpretable model-agnostic explanations (LIME), maximum activation analysis, linear regression, and sensitivity analysis.
[0058] After the local node creates (502) a list of models and analyzes (503) the associated data, the local node may generate (504) model update features to be sent to other local nodes in the update system. In some embodiments, a diagnostic module of the local node generates (504) the model features. In some embodiments, the model update features are local node independent, i.e., the model update features are usable by any local node in the update system. Thus, all data unique to that local node is removed from the model update generated by the local node. In some embodiments, the model update features include one or more of the following: one or more algorithms, metadata or high level aggregate statistics such as a creation date and time, a number of events detected over a certain period of time, a total transaction time value, and one or more threshold points used to trigger an update. In some embodiments, the model update features include an average ratio statistic for one or more data groups. In some embodiments, the model update features may detect deviations from an average ratio statistic for one or more data groups to determine future positive outcomes. In some embodiments, the model update features include one or more network or pictorial graphics representing one or more models. In some embodiments, the model update features include one or more network or pictorial graphics representing the new model.
[0059] After the local node generates 504 the model features, the local node may output 505 the model update information. In some embodiments, the local node outputs the model update information to a central module of the update system, which receives 506 the model update information. For example, the monitoring module 211 of the local node 210 may receive the model update information from the diagnostic module 212, which may then send the model update information to the central module 201, which receives the model update information.
[0060] After receiving (506) the model update information from the local node, the central module of the update system queries (507) the model database. In some embodiments, the central module queries the model database to determine whether the model update information already exists. In some embodiments, the central module queries the model database to determine whether the model update information replaces an existing model in the database or is a new model to the database. In some embodiments, if the central module queries the model database and determines that the model update information can replace or modify an existing model in the database, the central module can apply the model information to the existing model. In some embodiments, the model information can include one or more of the following: the creation date and time of the model, the date and time the model was last updated, and the number of local nodes currently using the model. For example, when the central module 201 receives the model update information from the local node 210, it checks the model update information against the available model database 202. The central module 201 determines whether the model update information already exists in the available model database 202, and if so, the central module 201 applies the relevant information to all existing models.
[0061] After the update system central module queries the model database (507), the central module determines (508) a priority for the model update information. In some embodiments, a list of model update information priorities is created as high, medium, or low. In some embodiments, a list of model update information priorities is created on a numerical scale, for example, in the range of 1-10 or other common numerical range. In some embodiments, the central module determines the priority of the model update information by comparing the features of the model update information to a pre-determined scale. In some embodiments, the central module determines the priority of the model update information by comparing the features of the model update information to the model database. In some embodiments, the central module determines the priority of the model update information by comparing the features of the model update information to model information stored in an existing model database. In some embodiments, a priority rating is obtained from the comparison of the features of the model update information to the existing model information, and the priority rating is then converted to a priority.
[0062] For example, after the central module 201 of the update system 200 checks a model update for credit card fraud detection against the available model database 202, the central module 201 determines that a similar model already exists in the database and applies the information to the existing model. The central module 201 then compares the features of the model update with the existing model and calculates a priority rating. As a first example, the central module determines that the existing model for credit card fraud detection has not been updated for over a year, that the model update is a direct replacement of the existing model, and that the model update can improve the performance of detecting credit card fraud across a range of usage conditions. These differences result in a high priority rating, and the central module 201 converts this priority rating to a high priority. As a second example, the central module determines that the existing model for credit card fraud detection was recently updated, and that the model update is only expected to improve the performance of detecting credit card fraud that is known to be possessed by only a few end users with a sufficiently large user base. These differences result in a relatively low priority rating, which the central module 201 converts to a medium priority.
[0063] After determining the priority, the update system determines which end users are applicable to the new model (509). In some embodiments, the central module has determined which end users are applicable. In some embodiments, the central module determines which end users are applicable to the model update by comparing the features of the model update to information about each end user, in addition to the priority information of the new model. For example, if the model update for credit card fraud detection has a medium priority, the central module identifies which local nodes in the update system are involved in credit card fraud detection and then outputs the model to those identified local nodes (510). The model update is not output to any of the remaining local nodes, but the local node may receive the update when each of the remaining local nodes has started the update process, e.g., if enough time has passed without an update to trigger a monitoring module. In another example, if the model update for credit card fraud detection has a high priority, the central module outputs the model to all local nodes (510). In another example, if a model update for credit card fraud detection has low priority, the central module may not immediately output the model to any local nodes, but instead wait for each local node to initiate the update process on its own.
[0064] After the model update information is transmitted from the central module, the model update information is received by at least one local node (511). In some embodiments, the model update information is received by multiple local nodes simultaneously. In some embodiments, the model update information is received by a monitoring module in any of the embodiments described herein.
[0065] In some embodiments, after the local node receives the model update (511), the model update is not automatically installed. First, the local node queries the current model database to see if the model update replaces any existing models (512). Next, the local node determines the relevance of the model update to the node (513). For example, at the local node 210 of the update system 200 shown in FIG. 2, the model update is received by the monitoring module 211 and then passed to the diagnostic module 212. The diagnostic module 212 first queries the current model database 215 and then determines the relevance of the model update to the local node 210. In some embodiments, the diagnostic module 212 terminates the update process after the relevance determining step 513. In some embodiments, if the diagnostic module 212 determines after the relevance determining step 513 that the model update is not needed by the local node, the diagnostic module 212 terminates the update process. In some embodiments, if the diagnostic module 212 determines after determining relevance step 513 that the model update already exists on the local node, the diagnostic module 212 terminates the update process. In some embodiments, if the model update has a high priority, the diagnostic module 212 automatically bypasses determining relevance step 513.
[0066] In some embodiments, after the local node determines that the model update is relevant or necessary, the local node determines whether it is authorized to apply the model update (514). In some embodiments, an evaluation module of the local node determines whether the local node is authorized to apply the model update. In some embodiments, the local node is not authorized to install the model update. In some embodiments, the local node is not automatically authorized to install the model update. In some embodiments, the local node must query or ask the end user for permission before installing the model update (516). For example, after the diagnostic module 212 determines that the model update is either relevant or that the model update has a high enough priority to bypass the determining relevance step 513, the model update is passed to the evaluation module 213. The evaluation module 213 then checks the update permission settings of the local node. In some embodiments, if the evaluation module 213 determines that it is not authorized to install the model update, the evaluation module 213 terminates the update process. In some embodiments, the evaluation module 213 queries the end user before installing the model update, for example, by issuing a prompt to the user or by sending an email or other communication to the end user.
[0067] After the local node determines that it has permission to install the model update, the local node installs the model update (515). In some embodiments, the evaluation module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models into a current model database in the local node. In some embodiments, the model update replaces one or more models in a current model database in the local node. For example, after permission is established, the evaluation module 213 updates the current model database 215 with the model update.
[0068] In some embodiments, after the update 515 is completed, the local node creates an output report (517). In some embodiments, the output report is shared with an end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report includes information about the model update information, such as the type of model that was updated, whether any old models were replaced, the date and time of the update, whether the new model is currently active, or any combination of these.
[0069] FIG. 6 illustrates a flow chart of an exemplary method for pushing model updates to an update system that does not include a central module, where the model updates are automatically created from a local node in the update system 600. In some embodiments, the method 600 can be used in the update system illustrated in FIG. 3. Initially, a local node in the update system detects a change in results from an existing model (601). In some embodiments, the local node in the update system detects a change in fraud detection rate. In some embodiments, the change in fraud detection rate is an increase in fraud detections greater than a pre-set threshold. In some embodiments, the change in fraud detection rate is a significant increase or decrease in fraud over a particular period of time. In some embodiments, the local node in the update system detects a change in the magnitude of detected fraud. In some embodiments, the change in fraud magnitude is an increase in the value or monetary amount of detected fraud events greater than a pre-set threshold. In some embodiments, the change in fraud magnitude is a significant increase in the value or monetary amount of detected fraud events compared to a moving average of detected events. For example, a diagnostic module 312 as shown in FIG. 3 may analyze system data 314 to detect an increase in fraud detection rates that is more than a standard deviation away from a three-month moving average of the fraud detection rates.
[0070] After a change in results from an existing model is detected (601), the local node creates a list of all models involved in the detection (602). In some embodiments, the local node creates a list of all models directly involved in generating the event detected in step 601. In some embodiments, the local node creates a list of all models that were directly and indirectly involved in generating the event detected in step 601. In some embodiments, the local node creates a list of all models that were actively running when the event was detected in step 601. For example, the diagnostic module 312, which has access to both the system data 314 and the current models database 315, creates a list of all algorithmic models that were directly and indirectly involved in generating the fraudulent event previously detected in step 601.
[0071] After the local node creates (602) a list of models relevant to the detected change (601), the local node analyzes (603) the data involved in generating the event leading to the detected change. In some embodiments, the local node analyzes system data to determine features and conditions associated with the models listed in step 602 in generating the event detected in step 601. In some embodiments, the local node's analysis may include, but is not limited to, ordinary least squares, penalized regression, generalized additive models, quantile regression, logistic regression, and gated linear models. In some embodiments, the local node's analysis results in a transformed variant of one or more relevant models to reduce the complexity of those models. Examples include placing monotonicity constraints on nonlinear, non-monotonic models to fit the model around variable relationships known to be true, or utilizing monotonic neural networks in machine learning applications. In some embodiments, the associated visualization is a relevant but less complex model that approximates one or more applicable models, particularly machine learning models. Examples include surrogate models, locally interpretable model-agnostic explanations (LIME), maximum activation analysis, linear regression, and sensitivity analysis.
[0072] After the local node creates 602 a list of models and analyzes 603 the associated data, the local node may generate 604 model update features to be sent to other local nodes in the update system. In some embodiments, a diagnostic module of the local node generates 604 the model features. In some embodiments, the model update features are local node independent, i.e., the model update features are usable by any local node in the update system. Thus, all data unique to that local node is removed from the model update generated by the local node. In some embodiments, the model update features include one or more of the following: one or more algorithms, metadata or high level aggregate statistics such as a creation date and time, a number of events detected over a certain period of time, a total transaction time value, and one or more threshold points used to trigger an update. In some embodiments, the model update features include an average ratio statistic for one or more data groups. In some embodiments, the model update features may detect deviations from an average ratio statistic for one or more data groups to determine future positive outcomes. In some embodiments, the model update features include one or more network or pictorial graphics representing one or more models. In some embodiments, the model update features include one or more network or pictorial graphics representing the new model.
[0073] After the local node generates 604 features for the model, the local node may output 605 the model update information. In some embodiments, the local node outputs the model update information to at least one other local node in the update system, which receive the model update information (611). In some embodiments, the local node outputs the model update information to all other local nodes in the update system. For example, the monitoring module 311 of the local node 310 may receive the model update information from the diagnostic module 312, and then the monitoring module 311 may send the model update information to the other local nodes 320, which receive the model update information.
[0074] In some embodiments, after the local node receives 611 the model update, the model update is not automatically installed. First, the local node queries the current model database to see if the model update replaces any existing models 612. Next, the local node determines 613 the relevance of the model update to the node. For example, at the local node 310 of the update system 300 shown in FIG. 3, the model update is received by the monitoring module 311 and then passed to the diagnostic module 312. The diagnostic module 312 first queries the current model database 315 and then determines the relevance of the model update to the local node 310. In some embodiments, the diagnostic module 312 terminates the update process after the relevance determining step 613. In some embodiments, if the diagnostic module 312 determines after the relevance determining step 613 that the model update is not needed by the local node, the diagnostic module 312 terminates the update process. In some embodiments, if after the relevance determining step 613 the diagnostic module 312 determines that the model update information already exists on the local node, the diagnostic module 312 terminates the update process.
[0075] In some embodiments, after the local node determines that the model update is relevant or necessary, the local node determines whether it is authorized to apply the model update (614). In some embodiments, an evaluation module of the local node determines whether the local node is authorized to apply the model update. In some embodiments, the local node is not authorized to install the model update. In some embodiments, the local node is not automatically authorized to install the model update. In some embodiments, the local node must query or ask the end user for permission before installing the model update (616). For example, after the diagnostic module 312 determines that the model update is relevant, the model update is passed to the evaluation module 313. The evaluation module 313 then checks the update permission settings of the local node. In some embodiments, if the evaluation module 313 determines that it is not authorized to install the model update, the evaluation module 313 terminates the update process. In some embodiments, the evaluation module 313 queries the end user before installing the model update, for example, by issuing a prompt to the user or by sending an email or other communication to the end user.
[0076] After the local node determines that it has permission to install the model update, the local node installs 615 the model update. In some embodiments, the evaluation module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models into a current model database in the local node. In some embodiments, the model update replaces one or more models in a current model database in the local node. For example, after permission is established, the evaluation module 313 updates the current model database 315 with the model update.
[0077] In some embodiments, after the update 615 is completed, the local node creates an output report (617). In some embodiments, the output report is shared with an end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report includes information about the model update information, such as the type of model that was updated, whether any old models were replaced, the date and time of the update, whether the new model is currently active, or any combination of these.
[0078] In some embodiments, a user of any system disclosed herein can be one or more human users, such as those known as "human-in-the-loop" systems. In some embodiments, a user of any system disclosed herein can be a computer system, artificial intelligence (AI), cognitive or non-cognitive algorithms, etc.
[0079] The following table is a non-exclusive and non-exhaustive list of examples of using any of the systems and methods disclosed herein.
[0080] [Table 2]
[0081] 7 illustrates a block diagram of an exemplary data processing system 700 in which aspects of example embodiments may be implemented. Data processing system 700 is an example of a computer, such as a server or client, in which computer usable code or instructions for implementing the processes of any of the example embodiments of the present disclosure described herein may be located. In some embodiments, FIG. 7 represents a server computing device, such as a server, that implements the system for interpreting analytical results described herein.
[0082] In the illustrated example, data processing system 700 may employ a hub architecture including a north bridge and memory controller hub (NB / MCH) 701 and a south bridge and input / output (I / O) controller hub (SB / ICH) 702. A processing unit 703, a main memory 704, and a graphics processor 705 may be connected to the NB / MCH 701. The graphics processor 705 may be connected to the NB / MCH 701 via an accelerated graphics port (AGP). A network adapter 706 connects to the SB / ICH 702. An audio adapter 707, a keyboard and mouse adapter 708, a modem 709, a read only memory (ROM) 710, a hard disk drive (HDD) 711, an optical drive (CD or DVD) 712, a universal serial bus (USB) port and other communication ports 713, and a PCI / PCIe device 714 can be connected to the SB / ICH 702 through a bus system 716. The PCI / PCIe device 714 may include an Ethernet adapter, an add-in card, and a PC card for a notebook computer. The ROM 710 may be, for example, a flash basic input / output system (BIOS). The HDD 711 and the optical drive 712 may use an integrated drive electronics (IDE) or serial advanced technology attachment (SATA) interface. A super I / O (SIO) device 715 can be connected to the SB / ICH 702 .
[0083] An operating system may run on the processing unit 703. The operating system may coordinate and control various components within the data processing system 700. As a client, the operating system may be a commercially available operating system. An object-oriented programming system, such as the Java(R)(TM) programming system, may run in conjunction with the operating system to provide calls to the operating system from object-oriented programs or applications running on the data processing system 700. As a server, the data processing system 700 may be, for example, an IBM(R) eServer(TM) System p(R) running the Advanced Interactive Executive operating system or the LINUX(R) operating system. The data processing system 700 may be a symmetric multiprocessor (SMP) system that may include multiple processors in the processing unit 703. Alternatively, a single processor system may be employed.
[0084] Instructions for the operating system, object oriented programming system, and applications or programs are located in a storage device, such as HDD 711, and loaded into main memory 704 for execution by processing unit 703. The processes of embodiments of the medical record error detection system may be executed by processing unit 703 using computer usable program code, which may be located, for example, in a memory, such as main memory 704, ROM 710, or in one or more peripheral devices.
[0085] The bus system 716 may be comprised of one or more buses. The bus system 716 may be implemented using any type of communications fabric or architecture that can provide a transfer of data between different components or devices connected to the communications fabric or architecture. A communications unit, such as the modem 709 or the network adapter 706, may include one or more devices that can be used to transmit and receive data.
[0086] Those skilled in the art will appreciate that the hardware required to execute any of the systems and methods described herein may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, may be used in addition to or in place of the hardware shown. Additionally, any of the systems described herein may take the form of any of a number of different data processing systems, including, but not limited to, a client computing device, a server computing device, a tablet computer, a laptop computer, a telephone or other communications device, a personal digital assistant, and the like. In essence, any of the systems described herein may be any known or later developed data processing system, without architectural limitation.
[0087] The illustrated systems and methods are not exclusive. Other systems and processes may be obtained according to the principles of the embodiments described herein to achieve the same purpose. It should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the current designs may be implemented by those skilled in the art without departing from the scope of the embodiments. As described herein, the various systems, subsystems, agents, managers, and processes may be implemented using hardware or software components, or a combination thereof.
[0088] The present invention has been described with reference to exemplary embodiments, but is not limited to those exemplary embodiments. Those skilled in the art will appreciate that numerous changes and modifications may be made to the preferred embodiments of the present invention. It is therefore intended that the appended claims be construed to cover all such equivalent variations as falling within the scope of the present invention.
Claims
1. A computer program product for causing a computer to execute program instructions for updating a detection model, the program instructions being executable by a processor, the processor comprising: Creating model update information, the creating comprising: detecting, by a diagnostic module, a change in system data; determining, by the diagnostic module, a list of all current detection models involved in detecting the change; analyzing, by the diagnostic module, the system data involved in detecting the change; and generating, by the diagnostic module, model update information, wherein the generated model update information includes one or more elements selected from the group consisting of one or more algorithms, a generation date and time, a number of events detected over a particular time period, aggregate statistics, and one or more threshold points used to trigger an update of the model. creating said model update information generated by receiving, by at least one supervisory module, the model update information; determining, by at least one diagnostic module, a current detection model; determining, by at least one evaluation module, whether the model update should be applied to the current detection model; determining, by at least one evaluation module, whether at least one local node is authorized to apply the model update; and updating, by at least an evaluation module, the local node by using the model update information to update the current detection model.
2. 2. The computer program product of claim 1, wherein the program instructions are executable by the processor to further cause the processor to distribute the model update information to at least one local node.
3. The computer program product of claim 2 , wherein the model update information is distributed to at least two local nodes.
4. The computer program product of claim 2 , wherein the model updates are distributed by a central module to at least one local node.
5. The central module comprises: receiving, by the central module, the model update information; analyzing, by said central module, a database of available models; determining, by the central module, a priority of the model updates; determining, by the central module, which local nodes should receive the model update information; 5. The computer program product of claim 4, further comprising distributing, by the central module, the model update information to at least one local node by transmitting the model update information to at least one local node.
6. 6. The computer program product of claim 5, wherein the database of available models comprises models available for use by all local nodes.
7. 6. The computer program product of claim 5, wherein the priority is determined by comparing characteristics of the model update information with model information in an existing model database.
8. 3. The computer program product of claim 2, wherein the model updates are distributed to at least one local node by another local node.
9. The computer program product of claim 1, wherein generating the model update information occurs at a local node.
10. A computer program product as described in any one of claims 1 to 9, wherein creating the model update information includes transmitting the created model update information by the monitoring module.
11. A computer program according to any one of claims 1 to 10, wherein the change is a significant change.
12. A computer program as described in any one of claims 1 to 11, wherein the analyzing is performed by least squares methods, penalized regression, generalized additive models, quantile regression, logistic regression, and gated linear models.
13. 12. The computer program product of claim 11, wherein the created model update does not include any system data specific to the local node that created the model update.
14. 12. The computer program product of claim 11, wherein the monitoring module transmits the model update information to a central module external to the local node.
15. The computer program product of claim 11 , wherein the monitoring module transmits the model update information to a monitoring module of another local node.
16. 1. A system for updating a detection model, comprising: Monitoring module, Diagnostic modules, Evaluation module, One or more current detection models; and at least one local node having system data generated by said current detection model; and a memory containing instructions, the instructions being executable by at least one processor; Creating model update information, the creating comprising: detecting, by said diagnostic module, a change in system data; determining, by the diagnostic module, a list of all current detection models involved in detecting the change; analyzing, by the diagnostic module, the system data involved in detecting the change; and generating, by the diagnostic module, model update information, wherein the generated model update information includes one or more elements selected from the group consisting of one or more algorithms, a generation date and time, a number of events detected over a particular time period, aggregate statistics, and one or more threshold points used to trigger an update of the model. creating said model update information generated by receiving, by the monitor module, the model update information; determining, by the diagnostic module, the current detection model; determining, by the evaluation module, whether the model update should be applied to the current detection model; determining, by the evaluation module, whether the local node is authorized to apply the model update; and updating, by the evaluation module, the current detection model using the model update information.
17. The system of claim 16, wherein the system further comprises at least two local nodes.
18. 20. The system of claim 17, wherein said system further comprises a central module, said monitoring module of each local node being in electronic communication with said central module.
19. 20. The system of claim 18, wherein the system further comprises a database of all models available for the system in electronic communication with the central module.
20. The system of claim 18 , wherein the monitoring module is configured to receive model update information.
21. 21. The system of claim 20, wherein the monitoring module is capable of receiving model update information from a system administrator or a local node.
22. A system described in any one of claims 16 to 21, wherein the change is a significant change.
23. The system described in any one of claims 16 to 22, wherein the analyzing is performed by least squares, penalized regression, generalized additive models, quantile regression, logistic regression, and gated linear models.
24. The system of any one of claims 16 to 23, wherein the created model update does not include any system data specific to the local node that created the model update.
25. at least a second local node; a central module with which the monitoring modules of each local node are in electronic communication; a database of all models available for the system in electronic communication with the central module; and a memory containing instructions, the instructions being executable by the at least one processor; Distributing the model updates, receiving, by the central module, the model update information; analyzing, by the central module, the database of available models; determining, by the central module, a priority of the model updates; determining, by said central module, which local nodes should receive said model update information; the distributing step including a step of transmitting, by the central module, the model update information; updating at least one local node, receiving, by at least one supervisory module, model update information; determining, by at least one diagnostic module, the current detection model; determining, by at least one evaluation module, whether the model update should be applied to the current detection model; determining, by at least one evaluation module, whether the local node is authorized to apply the model update; and, by at least an evaluation module, updating the current detection model using the model update information.
26. 1. A computer implemented method in a data processing system having a processor and a memory containing instructions, the instructions being executed by the processor to cause the processor to implement a system update detection model, the method comprising: Creating model update information, the creating comprising: detecting, by a diagnostic module, a change in system data; determining, by the diagnostic module, a list of all current detection models involved in detecting the change; analyzing, by the diagnostic module, the system data involved in detecting the change; and generating, by the diagnostic module, model update information, wherein the generated model update information includes one or more elements selected from the group consisting of one or more algorithms, a generation date and time, a number of events detected over a particular time period, aggregate statistics, and one or more threshold points used to trigger an update of the model. creating said model update information generated by receiving, by a monitoring module of at least one local node, the model update information; determining, by a diagnostic module of the local node, a current detection model being used by the local node; determining, by an evaluation module of the local node, whether the model update should be applied to the current detection model; determining, by the evaluation module of the local node, whether the local node is authorized to apply the model update; and updating, by the evaluation module of the local node, the current detection model using the model update information.
27. 27. The method of claim 26, further comprising the step of updating at least two local nodes.
28. 28. The method of claim 27, further comprising distributing, by a central module, the model update information.
29. 29. The method of claim 28, further comprising the step of analyzing, by the central module, a database of available models.
30. 30. The method of claim 28, further comprising determining, by the central module, a priority of the model updates.
31. 30. The method of claim 28, further comprising the step of determining, by the central module, which local nodes should receive the model update information.
32. The method of claim 26, wherein the change is a significant change.
33. The method of any one of claims 26 to 32, wherein the analyzing is performed by least squares, penalized regression, generalized additive models, quantile regression, logistic regression, and gated linear models.
34. A method according to any one of claims 26 to 33, wherein the created model update does not include any system data specific to the local node that created the model update.
35. The step of distributing the model update information, comprising: receiving, by a central module, the model update information; - analyzing, by said central module, a database of available models; determining, by the central module, a priority of the model updates; determining, by said central module, which local nodes should receive said model update information; the distributing step including a step of transmitting, by the central module, the model update information to at least one local node; updating at least one local node, receiving, by at least one supervisory module, the model update information; determining, by at least one diagnostic module, the current detection model; determining, by at least one evaluation module, whether the model update should be applied to the current detection model; determining, by at least one evaluation module, whether the local node is authorized to apply the model update; The method of any one of claims 26 to 34, further comprising: updating, by at least an evaluation module, the current detection model using the model update information.
Citation Information
Patent Citations
Model update method and device
CN106610854A
Model data updating methods, model data updating devices and model data updating system
CN107229966A
Self-learning-based machine learning model updating method and system
CN108921301A
Systems, methods, and media for generating sanitized data, sanitizing anomaly detection models, and / or generating sanitized anomaly detection models
US20150326597A1
