Automated operational data management based on quality of service standards

Abstract

Managing operational data in a distributed processing system by monitoring the workload of the system to establish a current assessment of operational data movement between data sources and data targets, receiving historical information of prior data movements including prior movement instances that resulted in one or more quality of service criteria being compromised, determining that an upcoming operational data action will not meet a particular quality of service criterion based on the current assessment and the historical information, and responsively applying a data management optimization infrastructure (data backplane service) adapted to enhance the particular quality of service criterion as defined by the data sources and data targets. Predicting operational outcomes using a cognitive system trained with historical information including historical operational factors associated with historical operational outcomes of the distributed processing system relative to quality of service criteria.

Description

Field of the Invention

[0001] The present invention relates generally to computer systems and, more particularly, to methods of maintaining quality of service standards in distributed computing systems. Background Art

[0002] Over the years, computing systems have become significantly more complex. In the early days of computing, a single computer was used to handle all tasks related to a project. With the advent of more auxiliary systems and with the advent of network computing (especially the Internet), much of the computing world is moving towards distributed computing. Distributed computing or distributed systems are systems whose components are implemented at different locations (such as different networked computers). Examples include peer-to-peer networks, online games, telephony, and data management.

[0003] Data management is generally a technology concerned with the procurement, maintenance, and use of data (i.e., information). The data itself may be as simple as customer details (such as name and address) or more extensive as in the case of financial services such as financial crime investigation solutions. In such systems, the operational management of data is very complex. This challenge is particularly true in systems that use distributed functional processing architectures such as microservices. Microservices is a software development technique that allows applications to be constructed as a collection of loosely coupled services. The benefit of decomposing an application into different smaller services is increased modularity, making the application easier to understand, develop, test, and become more resilient to architectural erosion. Microservices can be considered to be discrete processes that communicate over a network, especially using technology-agnostic protocols such as the Hypertext Transfer Protocol (HTTP), to achieve any goal.

[0004] The nature of the specific microservices used depends largely on the application. For example, in a financial services fraud detection application, microservices may include: a receiving service that places transactions in a queue; an attachment processor service that checks for any attachments and, if present, sends them to another microservice, such as an optical character recognition service; a context creator service that analyzes the current transaction and associates it with any past transactions related to it; a decision execution engine that executes rules that have been set by the client to identify violations; an analysis engine that examines transactions and flags any outliers; a case manager service that decides whether to create a case for human follow-up based on any identified issues; and a notification manager that passes updates about the processing of each transaction back to the client's expense / procurement system.

[0005] As with all computing systems, it is important to be able to monitor distributed computing systems to ensure that they meet quality of service (QoS) requirements. QoS is a measure of the overall performance of a service (such as a telephone or computer network, or a cloud computing service), particularly as seen by the users of the network. In order to quantitatively measure quality of service, several related aspects of a network service are typically considered. As with microservices, the specific nature of the QoS requirements depends on the specific applications involved. QoS criteria can be set in a service level agreement, identifying, for example, response time requirements based on data type and contextual qualifications including time and data qualifications. Summary of the invention

[0006] In at least one embodiment, the present invention generally relates to a method for managing operational data in a distributed processing system, comprising: receiving definitions of data sources and data targets and service quality standards for the data sources and data targets; monitoring the operational workload of the distributed processing system to establish a current assessment of operational data movement between the data sources and the data targets; receiving historical information of previous operational data movements in the distributed processing system, which includes previous instances of operational data movements that caused one or more of the service quality standards to be compromised; determining from the current assessment and the historical information that an upcoming operational data action will not meet a specific service quality standard among the service quality standards; and in response to the determination, automatically applying a data management optimization infrastructure suitable for improving the specific service quality standard according to the definition. In an exemplary embodiment, a service level agreement model provides service quality standards, the service level agreement model includes data types, contextual qualifications, time / date qualifications, and response time requirements; a data movement execution history model provides historical information, the data movement execution history model includes service statistical history, service resource consumption, and service type execution prediction; a current system load model provides the monitoring of the operation workload, the current system load model includes resource utilization, current cluster size, and capacity assessment; a data type and QoS requirement model provides the definition and characteristics about the data type, the data type and QoS requirement model includes data type definition and data service quality definition; a process control mechanism controls the data management optimization infrastructure to generate worker threads on demand at a network location away from the process control mechanism to meet the service quality standards, the process control mechanism includes data optimization selection, data service, service feedback, and data movement scheduling. The response time requirement is based on the data type, contextual qualifications, and time / data qualifications. The determination can use a cognitive system trained with historical information, the cognitive system predicts the operation results based on the current assessment, and the operation results provide an indication that a specific service quality standard will not be met. The current assessment includes resource usage of resources of the distributed processing system, the capacity of the resources, and the response time of the resources. The data management optimization infrastructure includes multiple scalable data backplane services. In an exemplary application, the distributed processing system provides a fraud detection solution, wherein the data sources and data targets have data types including a large amount of customer information such as name, address, telephone number, social security number or tax identification number, transaction information such as transaction amount and transaction type (wire transfer, ACH, credit, debit), and case management data for tracking the aforementioned customer and financial information collection; service quality standards include resource allocation, data integrity specifications and service uptime; data backplane services include messaging interfaces, application program interfaces, streams or other data communication carriers.

[0007] The above and other objects, features, and advantages of various embodiments of the present invention will become apparent in the following detailed written description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention may be better understood by those skilled in the art, and its numerous objects, features and advantages of various embodiments may become apparent to those skilled in the art by referencing the accompanying drawings.

[0009] Figure 1 is a block diagram of a computer system programmed to perform automated operational data management as specified by quality of service standards in accordance with one implementation of the present invention;

[0010] Figure 2 is a diagram of a cloud computing environment according to one implementation of the present invention;

[0011] Figure 3 is a block diagram depicting functional modules of an operational data management system according to one embodiment of the present invention;

[0012] Figure 4 is a model diagram showing a solution for automating operational data management according to an implementation of the present invention, the solution using a service level agreement model, a data movement execution history model, a current system load model, and a data type and QoS requirement model;

[0013] Figure 5 According to one implementation of the present invention, Figure 3 A block diagram of a cognitive system for the operational results of an operational data management system;

[0014] Figure 6 is a diagram showing the logical flow of a data management process according to one embodiment of the present invention.

[0015] The use of the same reference numbers in different drawings indicates similar or identical items. DETAILED DESCRIPTION

[0016] While distributed computing and the use of microservices offer several advantages, they also present new problems for system designers. Historically, monolithic applications could run as a single large functional unit and access data that could be optimized for minimal data movement and replication. Data can easily exist in a single common data store, it can be distributed and accessed through federation mechanisms, and it can exist in multiple types of data structures (databases, file-based, etc.) while still allowing for a single data access API layer. In any of these cases, there has been an effort to not rely on data movement and redundancy, which can create data integrity, inconsistency, and timeliness issues.

[0017] All of these assumptions change with distributed computing systems. The difficulty is that in order to leverage segmented services, data must often be replicated. For example, in a financial services application, it is expected that many technology components will be leveraged to support a financial crime investigation solution. This may involve, for example, using component technology to correlate bank transaction data as belonging to the same person, understand personal network associations, or perform machine learning analysis on data to identify potential fraudulent behavior patterns. It is important to leverage existing services for these functions, but designers are often faced with the problem that these services need access to similar data records (customer information, transaction records, etc.), and they expect to get it in a specific format, reside in a specific schema, or use a specific data access interface to the desired data service. In those cases, the application creator does not have control over how the data is expected to be available and how to minimize data movement and replication. The problem becomes a problem for the entire solution provider to manage, and today it needs to be managed on a project-by-project basis because components can change, but the problem still exists.

[0018] Therefore, it is desirable to devise an improved method for managing data in such a distributed system. It would be further advantageous if the method could be automated to meet the quality of service (QoS) requirements specific to the particular system involved. In various embodiments of the present invention, these and other advantages are achieved by providing a solution that meets the needs of data movement, replication, and timeliness using a directed QoS approach to define how the system manages data movement throughout the system. The system allows application developers to define the source and destination of specific data elements, and provides QoS features that show how fast the data needs to be moved (timeliness), what the consistency goals are (e.g., ensured consistency, eventual consistency, etc.), how data is expected to be deleted, and where replicated data is created. The system can use a scalable infrastructure for data movement (including updates and deletions) so that QoS goals can be met through elastic scaling of the infrastructure.

[0019] Now refer to the drawings, specifically see Figure 1, depicts an embodiment 10 of a computer system that can implement automated operational data management according to the present invention. The computer system 10 is a symmetric multiprocessor (SMP) system having a plurality of processors 12a, 12b connected to a system bus 14. The system bus 14 is further connected to and in communication with a combined memory controller / host bridge (MC / HB) 16, which provides an interface to a system memory 18. The system memory 18 can be a local memory device, or alternatively can include multiple distributed memory devices, preferably dynamic random access memory (DRAM). There may be additional structures not shown in the memory hierarchy, such as onboard (L1) and second level (L2) or third level (L3) caches. The system memory 18 is loaded with one or more applications or software modules according to the present invention, including operating programs, data definitions, QoS standards, system monitors, data optimization, various backplane services, and cognitive systems for predicting operational results required to perform specific functions of the distributed system, all of which will be explained in further detail below. Although Figure 1 These various components are shown in a single memory 18, but it should be understood that some of these components may reside in other networked (remotely located) computer systems similar to or different from computer system 10. In particular, backplane services may be implemented at multiple network locations remote from data optimization.

[0020] MC / HB 16 also has an interface to peripheral component interconnect (PCI) express links 20a, 20b, and 20c. Each PCI Express (PCIe) link 20a, 20b is connected to a corresponding PCIe adapter 22a, 22b, and each PCIe adapter 22a, 22b is connected to a corresponding input / output (I / O) device 24a, 24b. MC / HB 16 may additionally have an interface to an I / O bus 26, which is connected to a switch (I / O fabric) 28. Switch 28 provides fan-out of the I / O bus to multiple PCI links 20d, 20e, 20f. These PCI links are connected to more PCIe adapters 22c, 22d, 22e, which in turn support more I / O devices 24c, 24d, 24e. I / O devices may include, but are not limited to, keyboards, graphic pointing devices (mouse), microphones, display devices, speakers, permanent storage devices (hard drives) or arrays of such storage devices, optical drives that receive optical disks 25 (an example of a computer-readable storage medium) such as CDs or DVDs, and network cards. Each PCIe adapter provides an interface between a PCI link and a corresponding I / O device. MC / HB 16 provides a low-latency path through which processors 12a, 12b can access PCI devices mapped anywhere within the bus memory or I / O address space. MC / HB 16 further provides a high-bandwidth path to allow PCI devices to access memory 18. Switch 28 can provide peer-to-peer communication between different endpoints, and if the data traffic does not involve cache-coherent memory transfers, the data traffic does not need to be forwarded to MC / HB 16. Switch 28 is shown as a separate logical component, but it can be integrated into MC / HB 16.

[0021] In this embodiment, PCI link 20c connects MC / HB 16 to service processor interface 30 to allow communication between I / O device 24a and service processor 32. Service processor 32 is connected to processors 12a, 12b via JTAG interface 34 and uses attention lines 36 that interrupt the operation of processors 12a, 12b. Service processor 32 may have its own local memory 38 and is connected to a read-only memory (ROM) 40 that stores various program instructions for system startup. Service processor 32 may also access a hardware operator panel 42 to provide system status and diagnostic information.

[0022] In alternative embodiments, computer system 10 may include modifications of these hardware components or their interconnections or additional components, and thus the depicted examples should not be interpreted as implying any architectural limitations with respect to the present invention. The present invention may also be implemented in an equivalent cloud computing network.

[0023] When the computer system 10 is initially powered on, the service processor 32 uses the JTAG interface 34 to interrogate the system (host) processors 12a, 12b and the MC / HB 16. After completing the interrogation, the service processor 32 obtains the inventory and topology of the computer system 10. The service processor 32 then performs different tests on the components of the computer system 10, such as built-in self-test (BIST), basic assurance test (BAT), and memory test. The service processor 32 reports any error information of the failures detected during the test to the operator panel 42. If a valid configuration of system resources is still possible after removing any components found to be faulty during the test, the computer system 10 is allowed to continue. The executable code is loaded into the memory 18, and the service processor 32 releases the host processors 12a, 12b to execute program code, such as an operating system (OS), which is used to start the application, specifically the automated operational data management application of the present invention, the results of which can be stored in the hard drive (I / O device 24) of the system. While the main processors 12a, 12b are executing program code, the service processor 32 may enter a monitoring and reporting mode to monitor and report any operating parameters or errors, such as cooling fan speed and operation, thermal sensors, power regulators, and recoverable and unrecoverable errors reported by any of the processors 12a, 12b, memory 18, and MC / HB 16. The service processor 32 may take further action based on the type of error or defined thresholds.

[0024] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0025] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as punch card or protruding structure in the groove with instructions recorded thereon, and any suitable combination of the above. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses passing through fiber optic cables) or electrical signals emitted by wires.

[0026] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0027] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional process programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can be personalized to perform computer-readable program instructions by utilizing the state information of the computer-readable program instructions to perform the electronic circuit, so as to perform various aspects of the present invention.

[0028] The present invention will be described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0029] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a manufactured product containing instructions for implementing aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0030] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in or in multiple boxes in the flowchart and / or block diagram.

[0031] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of the possible implementations of the system, method and computer program product according to various embodiments of the present invention. To this end, each box in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0032] The computer system 10 executes program instructions for automating a data management process that uses novel optimization techniques to manage data in a distributed system. Thus, a program embodying the present invention may additionally include conventional aspects of different data management tools, and these details will become apparent to those skilled in the art when referring to the present disclosure. Some of those tools may involve cloud computing. It should be understood that although the present disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to a cloud computing environment. On the contrary, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0033] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include different features, service models, and deployment models.

[0034] Features may include, but are not limited to, on-demand services, extensive network access, resource pools, rapid elasticity, and measured services. On-demand self-service refers to the ability of cloud consumers to automatically provision computing capabilities (such as server time and network storage) on demand unilaterally without requiring human interaction with the provider of the service. Extensive network access refers to the ability to be available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptops, and personal digital assistants, etc.). Resource pooling occurs when the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers generally do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity means the ability to quickly and elastically provision capacity, automatically scaling down and releasing quickly to scale up quickly in some cases. For consumers, the capacity available for provisioning generally appears to be unlimited and can be purchased in any quantity at any time. Measured services are the ability of cloud systems to automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to providers and consumers of the utilized services.

[0035] Service models may include, but are not limited to, software as a service, platform as a service, and infrastructure as a service. Software as a Service (SaaS) refers to the ability to use the provider's applications running on the cloud infrastructure provided to consumers. Applications can be accessed from different client devices through thin client interfaces such as web browsers. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS) refers to the ability to be deployed on a cloud infrastructure provided to consumers, which is an application created or acquired by consumers using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but have control over deployed applications and possible application hosting environment configurations. Infrastructure as a Service (IaaS) refers to the ability to provide consumers with processing, storage, networks, and other basic computing resources that consumers can deploy and run arbitrary software (including operating systems and applications). Consumers do not manage or control the underlying cloud infrastructure, but have control over operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0036] Deployment models may include, but are not limited to, private cloud, community cloud, public cloud, and hybrid cloud. A private cloud refers to a cloud infrastructure that is operated only for an organization. It may be managed by the organization or a third party and may exist on-premises or off-premises. A community cloud has a cloud infrastructure shared by several organizations and supports a specific community that shares concerns (e.g., missions, security requirements, policies, and compliance considerations). It may be managed by an organization or a third party and may exist on-premises or off-premises. In a public cloud, the cloud infrastructure is available to the general public or a large industry group and is owned by an organization that sells cloud services. The cloud infrastructure of a hybrid cloud is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and applications to be portable (e.g., cloud bursting for load balancing between clouds).

[0037] Cloud computing environments can be service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure consisting of a network of interconnected nodes. Figure 2 An illustrative cloud computing environment 50 is depicted in . As shown, the cloud computing environment 50 includes one or more cloud computing nodes 52 in a network 56, and local computing devices used by cloud consumers (e.g., personal digital assistants (PDAs) or cellular phones 54a, desktop computers 54b, laptop computers 54c, and / or automotive computer systems 54d) can communicate with the cloud computing nodes 52. The nodes 52 can also communicate with each other. They can be grouped physically or virtually in one or more networks (not shown), such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services for which cloud consumers do not need to maintain resources on local computing devices. It should be understood that Figure 2 The types of computing devices 54a-54d shown are intended to be illustrative only, and computing nodes 52 and cloud computing environment 50 may communicate with any type of computerized device (eg, using a web browser) over any type of network and / or network-addressable connection.

[0038] See now Figure 3, depicts one embodiment of an operational data management system 60 constructed in accordance with the present invention. The operational data management system 60 generally includes an operational function or module 62, a system monitoring function or module 64, and a data movement optimization function or module 66. The operational module 60 includes data engineering (information engineering) 68, a service level agreement 70, and a collection 72 of data type / object characteristics and addresses. Data engineering 68 comprises the main body of the operational module 62 and involves an architectural approach for planning, analyzing, designing, and implementing applications. Its specific functions will vary depending on the specific application. A service level agreement 70 is a contract between a service provider and one or more clients. The service provider and the client agree on specific aspects of the service such as quality, availability, and liability. A service level agreement may contain a number of service performance indicators for corresponding service level targets. For the financial services example, service level agreement indicators may include service availability, cost tradeoffs, and support response time. The service level agreement 70 embodied in the operational module 62 reflects the quantitative values ​​in the contract. Data type / object characteristics and addresses are used to describe relevant data characteristics of an application suitable for data engineering 68 or service level agreement 70. Characteristics may include basic characteristics of data types as well as more advanced characteristics (data structures, classes, etc.). For the financial services example, specific characteristics may include structured and unstructured data, graph data, and large amounts of information requiring big data implementation, where data sources and data targets have data types such as name, address, phone number, social security number or tax ID or other identification number. An address is a network location (physical or virtual) where data is to be stored and retrieved from, and may have different forms depending on the protocol used (HTTP, TCP / IP, etc.).

[0039] The system monitor module 64 tracks the current and past operational performance of the components of the distributed data system. The system monitor module 64 receives current information from different hardware tools 76, such as central processing units (CPUs), disk drives or other permanent (non-volatile) memory, volatile memory (i.e., RAM), clusters of these or other resources, etc. This information may include any parameters associated with the device, such as performance usage, allocation, power consumption, resource availability, etc. This information is used to build the current system capacity 78. The system monitor 64 also includes historical operational performance information 80, i.e., the history of device usage and capacity associated with different parameters, such as time periods (e.g., peak usage times), specific clients, or specific services.

[0040] The data movement optimization module 66 includes a separate monitor 82 that tracks data movement within the distributed data system, particularly in relation to specifications provided by the system designer, as well as other data performance factors, such as consistency and timeliness. This information may also be provided to the historical operational performance 80 of the system monitor module 64. The data movement optimization module may then call data synchronization services 84 as needed based on current data performance factors, as further explained below. These services may include, for example, file system replication, messaging, database access, transport protocols, etc. Service workers 86 are optimized accordingly as needed.

[0041] In an exemplary implementation, the present invention uses different models to provide input for optimizing data management. Figure 4 As shown, the process control mechanism 90 receives input from a service level agreement model 92, a data movement execution history model 94, a current system load model 96, and a data type and QoS requirement model 98. Any one or all of these features can be embodied in the computer system 10. The service level agreement model 92 provides definitions of data sources, targets, and respective related QoS standards for storage. The model defines how data is laid out and managed throughout the system. Generally speaking, application developers will only create this definition, and then the data will be automatically moved and managed by the system at runtime, so solution developers will not need to focus on that part of the system operation, but can focus on domain values, and can also flexibly adopt data architecture changes without infrastructure coding investments in this area. In an exemplary implementation, the service level agreement model 92 includes a list of data types, contextual qualifications, time / date qualifications, and response time requirements.

[0042] The data movement execution history model 94 reflects the historical operation of the data movement request, and the processing control mechanism 90 can use the data movement request to learn from the past operation in order to predict the current operation results and make an informed assessment of how much resources should be applied to the work effort to meet (comply with) the standards. Figure 5As further explained, the data movement execution history model 94 can use machine learning and prediction techniques to ensure that the system operates at an optimal threshold. In an exemplary implementation, the data movement execution history model 94 includes service statistics history, service resource consumption, and service type execution prediction. The current system load model 96 tracks the existing workload of the system. This information is needed to understand the current capacity of the system and the ability to meet the QoS standards of the upcoming data movement operations. A heavily loaded system may require more backplane operation threads to complete the work compared to a lightly loaded system with available resources. Of course, this may change in real time, so it is necessary to actively monitor and adjust. In an exemplary implementation, the current system load model 96 includes resource utilization, current cluster size, and capacity assessment. The data type and QoS requirement model 98 describes the characteristics of various data types and how to implement QoS definitions. For example, real-time consistency with low latency timeliness requirements will be different in implementation between relational databases, distributed file systems, block storage, etc. In an exemplary implementation, the data type and QoS requirement model 98 includes data type definitions and data QoS definitions.

[0043] The process control mechanism 90 instructs the elastic microservice 100 (data backplane) to call a worker thread to meet the needs of the data to be moved. The elastic microservice 100 constitutes a scalable infrastructure that actually moves (updates or deletes) data across heterogeneous system components and technologies. These systems are conventional and have a variety of well-known and expected behavioral characteristics. The data backplane service is a mechanism for data communication in a data architecture. Apache's Kafka is an example. Kafka is an open source stream processing software platform that provides a unified, high-throughput, low-latency communication mechanism for processing real-time data feeds. The data backplane service of the fraud detection solution may include such messaging interfaces, application program interfaces (APIs), streams, or other tools for data communication. Elastic scaling refers to the ability to dynamically expand or shrink the infrastructure based on demand, that is, to increase or decrease resources such as physical disk space, memory, CPU, etc. based on the needs of the application at a specific moment.

[0044] In an exemplary implementation, the process control mechanism 90 includes data optimization selection, data services, service feedback, and data movement scheduling. The data optimization selection receives response time requirements from the service level agreement model 92 and selects the order of services to be processed by the data service. The data service receives data type and QoS definitions from the data type and QoS requirement model 98 and determines which backplane services are suitable for a particular data type. The data service can then command the data movement scheduling to initiate the necessary data backplane services. The data backplane service provides feedback to the service feedback of the process control mechanism 90, which can also update the service statistical history in the data movement execution history model 94.

[0045] In a preferred implementation, the predictive function of the data management system is embodied in a novel cognitive system. A cognitive system (sometimes referred to as deep learning, deep thinking, or deep question-answering) is a form of artificial intelligence that uses machine learning and problem solving. Although there are alternative designs, cognitive systems generally use neural networks. Neural networks can be of various types. A feedforward neural network is an artificial neural network in which the connections between units do not form a loop. A feedforward neural network is the earliest designed and simplest type of artificial neural network. In a feedforward neural network, information moves only in one direction, from the input node forward, through the hidden node (if any), to the output node. There are no loops or loops in the network. Therefore, it is different from a recursive neural network. A recursive neural network is a class of artificial neural networks in which the connections between units form a directed loop. This creates an internal state of the network that enables it to exhibit dynamic temporal behavior. Unlike a feedforward neural network, a recursive neural network can use its internal memory to process arbitrary input sequences. A convolutional neural network is a specific type of feedforward neural network based on animal visual perception, and is therefore particularly useful in processing image data. A convolutional neural network is similar to an ordinary neural network, but consists of neurons with learnable weights and biases.

[0046] There are many alternatives to neural networks for machine learning, such as support vector machines (SVMs). SVMs essentially construct a multidimensional mathematical space based on training examples and provide boundaries in that space that allow for binary classification of inputs, e.g., into "good" versus "bad" answers. Another approach involves Bayesian networks, which represent a set of variables as a directed acyclic graph. The network is then used to compute probabilistic relationships between the variables. The cognitive system is not limited to using a single approach, i.e., it can incorporate any number of these machine learning algorithms.

[0047] The modern implementation of AI is IBM Watson TMCognitive technologies that apply advanced natural language processing, information retrieval, knowledge representation, automated reasoning, and machine learning techniques to the field of open domain question answering. Such cognitive systems can rely on existing documents (corpora) and analyze them in different ways to extract answers relevant to queries such as people, locations, organizations, and specific objects, or to identify positive and negative sentiment. Different techniques can be used to analyze natural language, identify sources, find and generate hypotheses, find and score evidence, and merge and rank hypotheses. Models that score and rank answers can be trained based on large sets of question (input) and answer (output) pairs. The more algorithms that independently find the same answer, the more likely the answer is correct, resulting in an overall score or confidence level.

[0048] Figure 5 How a novel cognitive system 120 can be trained and applied according to one embodiment of the present invention is shown. The predictive functionality of the cognitive system 120 is based on historical information used as training data 122. In this example, the cognitive system is used to provide ongoing operational results for a financial services application that provides a fraud detection solution. The training data 122 thus constitutes previous examples of operational factors in different situations, with actual results regarding QoS requirements. For example, the training data 122 may include time domain information (time of day, day of week, day of month or other calendar date, etc.) reflecting peak usage times or lulls in activity, snapshots of resource availability, specific customers being served (or just the number of customers), transaction load (i.e., how many transactions have been recently requested or are currently being processed), and network traffic on communication lines used by the operating system. Each data point in the training data 122 may include this and other information related to data movement parameters (of all data types) compared to QoS requirements. In other words, the data points will provide input factors that lead to a given data management state in which some QoS requirements are met and other QoS requirements are not met (historical operational results). The training teaches the cognitive system 120 the likelihood that certain QoS requirements will not be met for a given operational situation. Historical operational factors may be updated with service feedback from the data backplane service.

[0049] Once the cognitive system 120 has been so trained, it can be used by the operational data management system to predict likely behavior based on current factors. Current system operational factors 124 are fed to the cognitive system 120; these factors will include the same types of inputs as the training data (time, resources, etc.). Using specific machine learning algorithms implemented in the cognitive system 120, the predicted operational results can be forwarded to the data movement optimization 126 of the data management system. Further to this fraud detection example, the cognitive system can provide an indication that any or all computing, resource, data allocation, or service uptime requirements may be compromised now or in the near future. The data movement optimization 126 can then prioritize the data backplane services required to more effectively handle these deficiencies based on the specific QoS requirements that are compromised. In one implementation, the predicted operational results can be assigned different values ​​indicating the likelihood of QoS failure in quantitative form (based on the confidence value generated by the cognitive system that a given QoS standard will not be met), and the data movement optimization 126 can prioritize the services associated with the requirements indicated as most likely to fail, that is, calling those services first and / or providing those services more to QoS standards with higher likelihood of failure.

[0050] See also Figure 6 The present invention may be further understood by referring to the flowchart of FIG. 1 , which shows the logical flow for a data management process 150 according to one implementation. The process 150, which may be executed on any computer system (including computer system 10 or a distributed system), begins by receiving data definitions of sources and targets and service quality standards 152 for those sources and targets. These definitions may be provided by application developers according to the specific encoding and variables used. The data management system continuously monitors the operational workload 154. The monitored factors may include, for example, resource usage, capacity, and response time. Thus, a current assessment 156 is established for the ongoing operational data flow. The current assessment is used to predict the upcoming operational results 158. These results may be identified by the cognitive system using historical information 160 about past operational data movements. The predicted results allow data movement optimization to identify which QoS standards are at risk 162. Some QoS standards may have greater risks and are therefore given priority in resource allocation accordingly. Data movement optimization can then apply appropriate optimization infrastructure to improve the identified QoS standards 164. The optimization infrastructure (data backplane service) generates worker threads on demand to meet the QoS goals to be achieved. As long as operation 166 is continued, the process iteratively returns to block 154 to continue monitoring.

[0051] Thus, the present invention combines workload monitoring and subsequent predictive scaling to provide excellent flexibility for the data backplane to meet QoS standards, so that the data backplane can dynamically adapt to the needs of the application at a particular moment. Although the present invention has been described with reference to specific embodiments, this description is not meant to be interpreted as limiting. Various modifications of the disclosed embodiments and alternative embodiments of the present invention will become apparent to those skilled in the art after reference to the description of the present invention. Therefore, it is expected that such modifications may be made without departing from the spirit or scope of the present invention as defined in the appended claims.

Claims

1. A method for managing operational data in a distributed processing system, comprising: receiving definitions of data sources and data destinations and quality of service standards for the data sources and data destinations; monitoring the operational workload of the distributed processing system to establish a current assessment of operational data movement between data sources and data targets; receiving historical information of previous operational data movements in the distributed processing system, including previous instances of operational data movements that resulted in one or more of the quality of service metrics being compromised; determining from the current evaluation and the historical information that an upcoming operational data action will not satisfy a particular one of the quality of service criteria; as well as responsive to said determination, automatically applying a data management optimization infrastructure adapted to enhance a particular quality of service criterion according to said definition; Among them, the process control mechanism controls the data management optimization infrastructure to generate worker threads on demand at a network location far away from the process control mechanism to meet the service quality standards. The process control mechanism includes data optimization selection, data service, service feedback and data movement scheduling.

2. The method according to claim 1, wherein: The service level agreement model provides service quality standards. The service level agreement model includes data type, context qualification, time / date qualification and response time requirements; The data movement execution history model provides historical information. The data movement execution history model includes service statistical history, service resource consumption, and service type execution prediction; A current system load model provides monitoring of the operational workload, the current system load model including resource utilization, current cluster size, and capacity assessment; as well as The data type and QoS requirement model provides the definition and characteristics about the data type, and the data type and QoS requirement model includes the data type definition and the data service quality definition.

3. The method according to claim 2, wherein: Response time requirements are based on data type, contextual qualifications, and time / data qualifications.

4. The method according to claim 1, wherein: The determination uses a cognitive system trained with historical information that predicts an operational outcome based on a current assessment that provides an indication that a particular quality of service criterion will not be met.

5. The method according to claim 1, wherein: Current assessments include resource usage of resources of the distributed processing system, capacity of the resources, and response time of the resources.

6. The method according to claim 1, wherein: The data management optimization infrastructure includes multiple scalable data backplane services.

7. The method according to claim 6, wherein: Distributed processing systems provide fraud detection solutions; The data sources and data targets have data types including customer name, customer address, customer telephone number, customer identification number, transaction information including transaction amount and transaction type, and case management data for tracking collection of operational data; Quality of service standards include resource allocation, data integrity specifications, and service uptime; as well as Data backplane services include messaging interfaces, application programming interfaces, and streams.

8. A computer system comprising: one or more processors to process program instructions; a memory device connected to the one or more processors; as well as Program instructions resident in the memory device for managing data operations in a distributed processing system by: receiving definitions of data sources and data destinations and quality of service standards for the data sources and data destinations; monitoring the operational workload of the distributed processing system to establish a current assessment of operational data movement between data sources and data targets; receiving historical information of previous operational data movements in the distributed processing system, including previous instances of operational data movements that resulted in one or more of the quality of service metrics being compromised; determining from the current evaluation and the historical information that an upcoming operational data action will not satisfy a particular one of the quality of service criteria; as well as responsive to said determination, automatically applying a data management optimization infrastructure adapted to enhance a particular quality of service criterion according to said definition; Among them, the process control mechanism controls the data management optimization infrastructure to generate worker threads on demand at a network location far away from the process control mechanism to meet the service quality standards. The process control mechanism includes data optimization selection, data service, service feedback and data movement scheduling.

9. The computer system of claim 8, wherein: The service level agreement model provides service quality standards. The service level agreement model includes data type, context qualification, time / date qualification and response time requirements; The data movement execution history model provides historical information. The data movement execution history model includes service statistical history, service resource consumption, and service type execution prediction; A current system load model provides monitoring of the operational workload, the current system load model including resource utilization, current cluster size, and capacity assessment; as well as The data type and QoS requirement model provides the definition and characteristics about the data type, and the data type and QoS requirement model includes the data type definition and the data service quality definition.

10. The computer system according to claim 8, wherein: Response time requirements are based on data type, contextual qualifications, and time / data qualifications.

11. The computer system according to claim 8, wherein: The determination uses a cognitive system trained with historical information that predicts an operational outcome based on a current assessment that provides an indication that a particular quality of service criterion will not be met.

12. The computer system according to claim 8, wherein: Current assessments include resource usage of resources of the distributed processing system, capacity of the resources, and response time of the resources.

13. The computer system according to claim 8, wherein: The data management optimization infrastructure includes multiple scalable data backplane services.

14. The computer system of claim 8, wherein: Distributed processing systems provide fraud detection solutions; The data sources and data targets have data types including customer name, customer address, customer telephone number, customer identification number, transaction information including transaction amount and transaction type, and case management data for tracking collection of operational data; Quality of service standards include resource allocation, data integrity specifications, and service uptime; Data backplane services include messaging interfaces, application programming interfaces, and streams.

15. A computer program product comprising: computer readable storage medium; as well as The program instructions residing in the storage medium are used to manage the operation data in the distributed processing system by: receiving definitions of data sources and data destinations and quality of service standards for the data sources and data destinations; monitoring the operational workload of the distributed processing system to establish a current assessment of operational data movement between data sources and data targets; receiving historical information of previous operational data movements in the distributed processing system, including previous instances of operational data movements that resulted in one or more of the quality of service metrics being compromised; determining from the current evaluation and the historical information that an upcoming operational data action will not satisfy a particular one of the quality of service criteria; as well as responsive to said determination, automatically applying a data management optimization infrastructure adapted to enhance a particular quality of service criterion according to said definition; Among them, the process control mechanism controls the data management optimization infrastructure to generate worker threads on demand at a network location far away from the process control mechanism to meet the service quality standards. The process control mechanism includes data optimization selection, data service, service feedback and data movement scheduling.

16. The computer program product of claim 15, wherein: The service level agreement model provides service quality standards. The service level agreement model includes data type, context qualification, time / date qualification and response time requirements; The data movement execution history model provides historical information. The data movement execution history model includes service statistical history, service resource consumption, and service type execution prediction; A current system load model provides monitoring of the operational workload, the current system load model including resource utilization, current cluster size, and capacity assessment; as well as The data type and QoS requirement model provides the definition and characteristics about the data type, and the data type and QoS requirement model includes the data type definition and the data service quality definition.

17. The computer program product of claim 15, wherein: Response time requirements are based on data type, contextual qualifications, and time / data qualifications.

18. The computer program product of claim 15, wherein: The determination uses a cognitive system trained with historical information that predicts an operational outcome based on a current assessment that provides an indication that a particular quality of service criterion will not be met.

19. The computer program product of claim 15, wherein: Current assessments include resource usage of resources of the distributed processing system, capacity of the resources, and response time of the resources.

20. The computer program product of claim 15, wherein: The data management optimization infrastructure includes multiple scalable data backplane services.

Images