Direct data loading of records generated by middleware

Abstract

Direct data loading of records generated by middleware A computer-implemented method is presented for reducing middleware data loading overhead and facilitating direct data loading into a database management system (DBMS). The method includes, during data loading, sending external writes to a memory-based DBMS if a corresponding internal write is for the vertex, exporting all external writes to a disk-based DBMS as an export file, and sending external reads for the vertex to the in-memory DBMS if the middleware requests the data. At the end of data loading, the method further includes generating a file for raw data for the disk-based DBMS from the export file and sending the generated raw file to the disk-based DBMS.

Description

[Technical Field]

[0001] The present invention relates generally to database management systems, and more particularly to direct data loading of middleware-generated records into database management systems. [Background technology]

[0002] The ability to operate rapidly and decisively in today's increasingly competitive marketplace is critical to an organization's success. The amount of information available to businesses is growing rapidly and is often overwhelming. Those organizations that effectively and efficiently manage these large amounts of data and use the information to make business decisions will realize a competitive advantage in the marketplace. Such a competitive advantage can be achieved by using database management systems (DBMSs), which store large amounts of data that support diverse workloads and heterogeneous applications. DBMSs can incorporate strategies that promote keeping data highly available, beneficial to business transaction processing and decision making.

[0003] A DBMS is a database program that uses standard methods for cataloging, retrieving, and querying data. It manages input data, organizes that data, and provides ways in which the data can be modified or extracted by users or other programs. It provides a query language and report writer that allow users to interactively query a relational database. These essential components give users access to all management information as needed. The DBMS applies entered text to the database as criteria to identify and report records in the database that meet the criteria. Entering text into a DBMS field requires the user to understand the DBMS and how it represents data in the database. For example, for a given search term, the user must know which fields in the DBMS are appropriate for searching that term. For a search query with multiple search terms, the user must be familiar with the multiple fields in the DBMS and how those fields interact to limit or define the search. The user must also know the hierarchical structure between database tables and the keys that link the tables together.

[0004] A common goal of a DBMS is to provide high performance in terms of transaction throughput and transaction latency to minimize hardware costs, wait times, and increase the number of transactions per unit of time. Even with significant investments in hardware, achieving the desired performance is often expensive or even impossible. Another common goal of a DBMS is to reduce the complexity associated with the application development process, thus saving time and money as well as reducing the risk of errors.

[0005] However, traditional DBMSs are built on the assumption that data loading is a "one-time transaction." Data loading is considered an offline process on the critical path, where users define the schema and load the majority of the data "one go," before submitting any queries. When this architectural design assumption is combined with explosive data growth, it results in data loading emerging as a bottleneck in the DBMS's data analysis pipeline.

[0006] Nevertheless, some analyses require data loading into the DBMS beforehand. Middleware can be beneficial if the DBMS provides a simple application programming interface (API) for the analyses. However, middleware can also present overhead when data loading is performed. Thus, data loading in the DBMS can cause a bottleneck. Therefore, a method is needed to reduce data loading in the DBMS. Summary of the Invention

[0007] According to one embodiment, a method is provided for reducing middleware data loading overhead and facilitating direct data loading into a database management system (DBMS). The method includes, during data loading, sending external writes to a memory-based DBMS if a corresponding internal write is for the vertex, exporting all external writes to a disk-based DBMS as an export file, and sending external reads for the vertex to the in-memory DBMS when the middleware needs the data. At the end of data loading, the method further includes generating a file for raw data for the disk-based DBMS from the export file and sending the generated raw file to the disk-based DBMS.

[0008] According to another embodiment, a system is provided for reducing middleware data loading overhead and facilitating direct data loading into a database management system (DBMS). The system includes a memory and one or more processors in communication with the memory, the one or more processors configured to: during data loading, send external writes to the memory-based DBMS if corresponding internal writes are for a vertex; export all external writes to the disk-based DBMS as export files; and send external reads for the vertex to the in-memory DBMS when the middleware needs the data. The system further generates files for raw data in the disk-based DBMS from the export files at the end of data loading, and sends the generated raw files to the disk-based DBMS.

[0009] According to yet another embodiment, a non-transitory computer-readable storage medium having a computer-readable program for reducing middleware data loading overhead and facilitating direct data loading into a database management system (DBMS) is presented. During data loading, the non-transitory computer-readable storage medium performs the following steps: sending external writes to a memory-based DBMS if a corresponding internal write is for a vertex; exporting all external writes to a disk-based DBMS as an export file; and sending external reads for the vertex to the in-memory DBMS when the middleware needs the data. At the end of data loading, the non-transitory computer-readable storage medium performs the following steps: generating a file for raw data for the disk-based DBMS from the export file; and sending the generated raw file to the disk-based DBMS.

[0010] According to one embodiment, a method is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), including receiving an internal write in an export extension of the middleware, determining whether the internal write is for a vertex, sending the internal write to an in-memory DBMS if the internal write is for the vertex, and appending the internal write to a recovery file.

[0011] In one preferred embodiment, if an internal write is not associated with a vertex, the internal write is appended to the export file.

[0012] According to another embodiment, a method is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), the method including receiving an internal read in an export extension of the middleware, sending the internal read to an in-memory DBMS to receive a result, determining whether the result contains records, and if the result does not contain records, sending the internal read to a disk-based DBMS.

[0013] In one preferred embodiment, if the result contains a record, the result is returned to the caller.

[0014] Advantages of the present invention include reducing data loading via middleware. Benefits of the present invention further include more efficient central processing unit (CPU) utilization and more efficient input / output (I / O) bandwidth utilization. Data loading is an upfront investment that a DBMS must make to be able to support efficient query execution. Given the amount of data collected by current applications, minimizing data loading overhead is important to prevent data loading from becoming a bottleneck in the data analysis pipeline. This results in higher storage capacity, faster processing, and better transcription speeds for unstructured data. Additional advantages include higher quality, reduced cost, better coverage, faster performance, fewer application errors, and fewer data errors.

[0015] In one preferred embodiment, during data loading, sending the external writes further includes exporting the internal writes as a recovery file.

[0016] In another preferred embodiment, at the start of data loading, if a recovery file exists, external writes to a memory-based DBMS are generated from the recovery file, and the generated external writes are sent to the memory-based DBMS.

[0017] In yet another preferred embodiment, during data loading, sending external reads further includes sending external reads to a disk-based DBMS if the in-memory DBMS has nothing to fetch.

[0018] In yet another preferred embodiment, the middleware export extension supports both disk-based and memory-based DBMSs.

[0019] In yet another preferred embodiment, the middleware export extension handles internal writes and internal reads.

[0020] In yet another preferred embodiment, if the internal write is not for a vertex, the internal write is appended to the export file.

[0021] It should be noted that exemplary embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, a person skilled in the art will infer from the above and following description that, unless otherwise notified, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different types of subject matter, in particular between features of method-type claims and features of apparatus-type claims, is also considered to be described within this specification.

[0022] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0023] The present invention is provided in more detail in the following description of preferred embodiments with reference to the following drawings.

[0024] [Figure 1] 1 is an exemplary middleware architecture for direct data loading of middleware-generated records into a database management system (DBMS), according to one embodiment of the present invention.

[0025] [Figure 2] 10 is an exemplary flowchart illustrating a methodology when receiving an internal write, according to one embodiment of the present invention.

[0026] [Figure 3] 10 is an exemplary flowchart illustrating a methodology when receiving an internal read, according to one embodiment of the present invention.

[0027] [Figure 4] 10 is an exemplary flowchart illustrating a methodology for initiating initialization of an export extension according to one embodiment of the present invention.

[0028] [Figure 5] FIG. 2 is a diagram of an exemplary middleware according to one embodiment of the present invention.

[0029] [Figure 6] FIG. 2 is an exemplary diagram illustrating a request / response cycle according to one embodiment of the present invention.

[0030] [Figure 7] 1 is an exemplary JanusGraph architecture according to one embodiment of the present invention.

[0031] [Figure 8] 1 is an exemplary processing system according to one embodiment of the present invention.

[0032] [Figure 9] FIG. 1 is a block / flow diagram of an exemplary cloud computing environment in accordance with an embodiment of the present invention.

[0033] [Figure 10] FIG. 2 is a schematic diagram of exemplary abstraction model layers according to one embodiment of the present invention.

[0034] [Figure 11] FIG. 2 is a block / flow diagram of a method for applying the middleware architecture of FIG. 1 to Internet of Things (IoT) systems / devices / infrastructure, according to one embodiment of the present invention.

[0035] [Figure 12]FIG. 2 is a block / flow diagram of an exemplary IoT sensor used to collect data / information associated with the middleware architecture of FIG. 1 in accordance with one embodiment of the present invention.

[0036] [Figure 13] FIG. 2 is a block / flow diagram of an exemplary method for data loading of middleware-generated records directly into a DBMS, according to one embodiment of the present invention.

[0037] Throughout the drawings, the same or similar reference numbers represent the same or similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0038] Embodiments of the present invention provide a method and device for using direct data loading of records generated by middleware into a database management system (DBMS). A DBMS is a collection of interrelated data and a set of programs for storing and accessing the interrelated data in a simple and effective manner. Database systems are developed for large amounts of data. When dealing with large amounts of data, there are two things that need to be optimized: data storage and data retrieval. Data loading is the process that involves obtaining transformed data and loading the transformed data, where users can access the transformed data. Data loading has been considered a "one-time transaction," an offline process in the critical path of query execution. DBMS architectures are adapted to this assumption. Nevertheless, today, the rate at which data is generated and collected has invalidated the "one-off" assumption and turned data loading into a bottleneck in the data analysis pipeline.

[0039] Exemplary embodiments of the present invention advantageously contemplate that for a DBMS, the middleware converts an internal representation of an update (internal write) into one or more update commands (external writes). Also, internal writes for vertices can be advantageously identified during data loading, and the middleware can advantageously support disk-based and memory-based DBMSs, with disk-based DBMSs advantageously providing a way to directly append files for their raw data.

[0040] Exemplary embodiments of the present invention advantageously use a middleware extension that manages both disk-based and memory-based DBMSs. During data loading, the extension sends external writes to the memory-based DBMS if the corresponding internal write is for a vertex, exports the internal writes as files (recovery files), exports all external writes to the disk-based DBMS as files (export files), and sends external reads for the vertices to the in-memory DBMS if the middleware needs / requires the data. In another option, the extension sends external reads to the disk-based DBMS if the in-memory DBMS has nothing to fetch. At the end of data loading, the extension or other component advantageously generates files for the raw data in the disk-based DBMS from the export files and sends them to the disk-based DBMS. In another embodiment, at the beginning of data loading, if a recovery file exists, the extension advantageously generates external writes to the memory-based DBMS from files and sends them to the memory-based DBMS.

[0041] The present invention is described with respect to a given example architecture; however, it should be understood that other architectures, structures, substrate materials, and process features and steps / blocks may vary within the scope of the present invention. Note that for clarity, certain features may not be shown in every drawing. This is not intended to be construed as limiting any particular embodiment, or example, or the scope of the claims.

[0042] FIG. 1 is an exemplary middleware architecture for direct data loading of middleware-generated records into a database management system (DBMS) according to one embodiment of the present invention.

[0043] The middleware architecture 10 includes a data loading application 12, in which the vertices and edges of the graph are loaded into TinkerPop TM Graphs are structures made up of vertices and edges. Both vertices and edges can have any number of key / value pairs called properties. Vertices represent separate objects, such as people, places, or events. Edges represent relationships between vertices. For example, a person might know another person, attend an event, or have recently been to a particular place, or some combination thereof. Properties represent non-relational information about vertices and edges. Example properties include vertices with names and ages, and edges with timestamps or weights, or a combination thereof. Such graphs are also known as property graphs, which are implemented in Apache TinkerPop. TM It is the basic data structure of Apache TinkerPop. TM is a graph computing framework for both graph databases (OLTP) and graph analytical systems (OLAP).

[0044] TinkerPop TMThe API 13 may be integrated with JanusGraph 20, a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across multi-machine clusters. JanusGraph 20 includes a JanusGraph Core 22, a first storage manager SPI 23, and a second storage manager SPI 40. The JanusGraph Core 22 provides internal reads and writes to the storage manager SPI 23. The storage manager 23 advantageously includes an export extension 24. The export extension 24 receives and processes internal writes and internal reads. The export extension 24 is a component that supports the SPI 23, 40. The export extension 24 coordinates multiple extensions that support the SPI 23, 40 with the ability to preload and export data. Because the export extension 24 accesses other extensions using only the SPI 23, 40, no modifications to JanusGraph 20 are required.

[0045] The export extension 24 receives the recovery file 30, and during data loading, the extension sends external writes to the in-memory based DBMS 44 if their corresponding internal writes are for a vertex. If the internal write is for a vertex, the internal write is sent to the in-memory DBMS extension 44. In a first option, if the internal write is for a vertex, the internal write is advantageously exported as a recovery file 32. In a second option, if the internal write is not for a vertex, the method advantageously exports all external writes as export files 34 to the disk-based DBMS extension 42, and sends external reads for the vertex to the in-memory DBMS via the disk-based DBMS driver 46 if the middleware needs / requests it.

[0046] In-memory databases 44 are based on database management systems that store their data collections directly in the working memory of one or more computers. The use of random access memory (RAM) has the important advantage that in-memory databases have significantly faster access speeds. The stored data is then available very quickly when needed. The greatest advantage of using an in-memory database is the significantly higher access speed that results from using RAM. This also leads to faster data analysis. However, it is not just reduced fetch times that optimize data analysis. In-memory DBs advantageously allow the evaluation of structured and unstructured data from any system.

[0047] At the end of the data loading, the extension 24 advantageously generates a raw data file 52 for a disk-based DBMS 56 from the export file 34. The extension 24 advantageously transmits the raw data file 52 to the disk-based DBMS 56 via a disk-based DBMS tool 54, thereby providing a method for directly adding the raw data file with the disk-based DBMS. A converter 50 advantageously converts the export file 34 to the raw data file 52.

[0048] FIG. 2 is an exemplary flowchart illustrating a methodology when receiving an internal write, according to one embodiment of the present invention.

[0049] In block 102, an internal write is received.

[0050] In block 104, it is determined whether an internal write is associated with the vertex.

[0051] If no, the process proceeds to block 110. If yes, the process proceeds to block 106.

[0052] In block 106, the internal write is sent to the in-memory DBMS extension.

[0053] In block 108, if the internal write is for a vertex, the internal write is appended to the recovery file.

[0054] In block 110, if the internal write is not for a vertex, the internal write is appended to the export file.

[0055] FIG. 3 is an exemplary flowchart illustrating a methodology when receiving an internal read, according to one embodiment of the present invention.

[0056] In block 152, an internal read is received.

[0057] In block 154, the internal read is sent to the in-memory DBMS extension and the results are received.

[0058] At block 156, it is determined whether the result contains the record.

[0059] If yes, the process proceeds to block 160. If no, the process proceeds to block 158.

[0060] In block 158, the internal read is sent to the disk-based DBMS extension and the results are received.

[0061] In block 160, the results are returned to the caller.

[0062] FIG. 4 is an exemplary flowchart illustrating a methodology for initiating initialization of an export extension according to one embodiment of the present invention.

[0063] With respect to appending internal writes to the recovery file or block 108, the following occurs.

[0064] At block 182, initialization of the export extension begins.

[0065] At block 184, it is determined whether a recovery file exists.

[0066] If yes, the process proceeds to block 190. If no, the process proceeds to block 186.

[0067] In block 186, all internal writes are sent to the in-memory DBMS extension.

[0068] In block 188, all internal writes are copied to the new recovery file.

[0069] At block 190, the initialization of the export extension is completed.

[0070] FIG. 5 is a diagram of an exemplary middleware according to one embodiment of the present invention.

[0071] Middleware can be described as glue that combines two separate and already-existing software programs. Generally, middleware can be a variety of different and specifically designed software. Middleware is a type of software that makes it easy for software developers to implement communication and input / output between two programs, so that software developers can focus on the specific purpose of their application. Acting almost as an extension to existing operating systems, middleware helps integrate software between applications and services.

[0072] Middleware can be used to connect any two pieces of software. It works by allowing data to be passed between the two. One example of the use of middleware is when middleware is used to connect a database system with a web server, allowing a user to request data from the database using a form displayed in a web browser. Middleware is software that provides common services and capabilities to applications outside of those provided by the operating system. Data management, application services, messaging, authentication, and API management are all commonly handled by middleware. Middleware also helps developers build applications more efficiently. It acts like connective tissue between applications, data, and users. Middleware can make it cost-effective to develop and run applications at scale for organizations with multi-cloud, containerized environments.

[0073] In middleware 200, middleware 210 advantageously connects systems of engagement 220 to systems of record 240. Systems of engagement 220 may include, but are not limited to, public cloud systems 222, mobile systems 224, Internet of Things (IoT) systems 226, social media systems 228, affiliates 230, and websites 232. Systems of record may include, but are not limited to, private cloud 242, customer relationship management (CRM) systems 244, servers 246, databases 248, back-office processes 250, and application programming interface (API) services 252.

[0074] 11 and 12 below illustrate one practical application relating to an IoT system 226 that advantageously uses middleware 210 to interface with a system of record 240.

[0075] FIG. 6 is an exemplary diagram illustrating a request / response cycle according to one embodiment of the present invention.

[0076] At 260, requests 262 and responses 264 to application 270 are processed by middleware 272. "Middleware" 272 is a function that handles every request 262 before it is processed by any specific pass operations and handles every response before returning it. Middleware 272 gets each request 262 that arrives at application 270. Middleware 272 may then do something with the request 262 or execute any required code. Middleware 272 then passes the request 262 (by any pass operation) to be processed by the rest of application 270. Middleware 272 then gets the response 264 generated by application 270 (by any pass operation). Middleware 272 may then do something with the response 264 or execute any required code. Middleware 272 then returns the response 264.

[0077] FIG. 7 is an exemplary JanusGraph architecture according to one embodiment of the present invention.

[0078] JanusGraph is a scalable, transactional property graph database. A property graph is a mapping between entities (called vertices) with their relationships (called edges). Property graph queries can traverse multiple edges and vertices to reveal relationships between entities. JanusGraph is thus a scalable graph database that has the advantage of being optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across multi-machine clusters.

[0079] JanusGraph architecture 300 includes an application 302 hosted on JanusGraph 310 and a Gremlin Graph Computer 330. Gremlin Graph Computer 330 communicates with a big data platform 335.

[0080] The storage backend 340 is pluggable and supports at least Cassandra, HBase, BerkeleyDB, Google BigTable, and in-memory storage options. The storage backend 340 is where the data is actually stored. Given its flexibility in handling multiple database engines, the storage backend 340 has the advantage of allowing users to choose an option that is already deployed or in which the user may have expertise. There is only one storage backend 340.

[0081] Next, for the external index backend 350, at least Elasticsearch, Solr, and Lucene are supported. The external index backend 350 is optional but required for indexing over multiple properties, full-text and string search, and geo-mapping. Again, there can be only one external index backend 350.

[0082] Storage backend 340, 350, followed by TinkerPop in the Gremlin box 310 TM API 321 represents how users can interact with the graph. It is commonly known as the Gremlin Console and is part of TinkerPop TM This is an example of an application that calls API 321, which is a command line interface used to interact with JanusGraph.

[0083] Finally, 310 represents the JanusGraph Server. This runs a script named gremlin_server.sh. The Gremlin Server runs on Apache TinkerPop TM JanusGraph is part of the Gremlin Server project. JanusGraph has the advantage of essentially acting as a plug-in for Gremlin Server, telling it how and where to store graph data. JanusGraph Server 310 advantageously includes an administrative API 320, an internal API layer 322, a database layer 324, a storage and index interface layer 326, and an OLAP input / output (I / O) interface 328.

[0084] Gremlin is Titan's query language used to retrieve data from and modify data in a graph. Gremlin is a path-oriented language that succinctly expresses complex graph traversal and mutation operations. Gremlin is a functional language whereby traversal operators are chained to form path-like expressions. Gremlin works for both online transaction processing (OLTP)-based graph databases and online analytical processing (OLAP)-based graph processors. Gremlin's automaton and functional language foundations allow Gremlin to naturally support imperative and declarative queries, host language agnosticism, user-defined domain-specific languages, extensible compilers / optimizers, single- and multi-machine execution models, and hybrid depth- and breadth-first evaluation.

[0085] In summary, exemplary embodiments of the present invention advantageously assume that for a DBMS, the middleware converts an internal representation of an update (internal write) into one or more update commands (external writes). Also, internal writes for vertices may be advantageously identified during data loading, and the middleware may advantageously support disk-based and memory-based DBMSs, with disk-based DBMSs advantageously providing a way to directly add files for their raw data. Exemplary embodiments of the present invention advantageously use a middleware extension that manages both disk-based and memory-based DBMSs. During data loading, the extension sends external writes to the memory-based DBMS if the corresponding internal write is for a vertex, exports the internal writes as files (recovery files), exports all external writes to the disk-based DBMS as files (export files), and sends external reads for the vertices to the in-memory DBMS when the middleware needs the data. In another option, if the in-memory DBMS does not fetch anything, the extension sends external reads to the disk-based DBMS. At the end of data loading, an extension or other component generates files for the raw data in the disk-based DBMS from the export file and sends them to the disk-based DBMS. In another embodiment, at the beginning of data loading, if a recovery file exists, an extension generates external writes from the file to the memory-based DBMS and sends them to the memory-based DBMS.

[0086] Advantages of the present invention include reducing data loading through middleware. Advantages of the present invention further include more efficient CPU utilization and more efficient I / O bandwidth utilization. Data loading is an upfront investment that a DBMS must make to be able to support efficient query execution. Given the volume of data collected by applications, it is important to minimize data loading overhead to prevent data loading from becoming a bottleneck in the data analysis pipeline. This results in higher storage capacity, faster processing, and better transcription speeds for unstructured data. Additional advantages include higher quality, reduced cost, better coverage, faster performance, fewer application errors, and fewer data errors.

[0087] FIG. 8 is an exemplary processing system for processing streaming algorithms according to an embodiment of the present invention.

[0088] 8, which illustrates a hardware configuration of a computing system 600 according to one embodiment of the present invention. As can be seen, the hardware configuration includes at least one processor or central processing unit (CPU) 611. The CPU 611 is interconnected via a system bus 612 to a random access memory (RAM) 614, a read-only memory (ROM) 616, an input / output (I / O) adapter 618 (for connecting peripheral devices such as a disk unit 621 and a tape drive 640 to the bus 612), a user interface adapter 622 (for connecting a keyboard 624, a mouse 626, speakers 628, a microphone 632, and / or other user interface devices to the bus 612), a communications adapter 634 for connecting the system 600 to a data processing network, the Internet, an intranet, a local area network (LAN), etc., and a display adapter 636 for connecting the bus 612 to a display device 638 and / or a printer 639 (e.g., a digital printer, etc.).

[0089] FIG. 9 is a block / flow diagram of an exemplary cloud computing environment in accordance with one embodiment of the present invention.

[0090] FIG. 9 is a block / flow diagram of an exemplary cloud computing environment in accordance with one embodiment of the present invention.

[0091] Although the present invention includes detailed descriptions relating to cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0092] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0093] Characteristics include: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, as needed automatically, without requiring human interaction with the service provider. Wide Network Access: This capacity is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. While consumers generally have no control or knowledge over the exact location of the provided resources, there is location independence in that it may be possible to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Capacity can be rapidly and elastically provisioned, in some cases automatically, to rapidly scale out, and rapidly released to rapidly scale in. Consumers often perceive unlimited capacity available for provisioning, and can purchase any amount at any time. Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0094] The service model is as follows: Software as a Service (SaaS): The consumer is offered the ability to use a provider's application running on a cloud infrastructure. The application is accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the deployed applications and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources, upon which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and, in some cases, limited control of selected networking components (e.g., host firewalls).

[0095] Deployment models are as follows: Private Cloud: The cloud infrastructure operates solely for an organization. It may be managed by that organization or a third party and may exist on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may exist on-premises or off-premises. Public Cloud: The cloud infrastructure is made available to the general public or large industry associations and is owned by an organization that sells cloud services. Hybrid Cloud: The cloud infrastructure remains a unique entity but is a composition of two or more clouds (private, community, or public) joined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0096] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0097] Referring now to FIG. 9 , an exemplary cloud computing environment 750 is shown to enable use cases of the present invention. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 754A, a desktop computer 754B, a laptop computer 754C, or an automobile computer system 754N, or any combination thereof, may communicate. The nodes 710 can communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or any combination thereof, as described above. This enables the cloud computing environment 750 to provide infrastructure, platform, or software as a service, or any combination thereof, without the need for the cloud consumer to maintain resources on their local computing device. It should be understood that the types of computing devices 754A-N shown in FIG. 9 are intended to be illustrative only, and that computing node 710 and cloud computing environment 750 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0098] 10 is a schematic diagram of exemplary abstract model layers according to one embodiment of the present invention. It should be understood in advance that the components, layers, and functions illustrated in FIG. 10 are intended to be exemplary only, and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0099] Hardware and software layer 860 includes hardware and software components. Example hardware components include mainframe 861, RISC (reduced instruction set computer) architecture-based servers 862, servers 863, blade servers 864, storage devices 865, and networks and networking components 866. In some embodiments, software components include network application server software 867 and database software 868.

[0100] The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 871, virtual storage 872, virtual networks including virtual private networks 873, virtual applications and operating systems 874, and virtual clients 875.

[0101] In one example, management layer 880 may provide the functions described below. Resource provisioning 881 provides dynamic procurement of computing and other resources used to execute tasks within the cloud computing environment. Metering and pricing 882 tracks costs as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 883 provides access to the cloud computing environment to consumers and system administrators. Service level management 884 allocates and manages cloud computing resources to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 885 provides proactive coordination and procurement of anticipated future cloud computing resource needs according to SLAs.

[0102] Workload tier 890 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 891, software development and lifecycle management 892, virtual classroom instruction delivery 893, data analytics processing 894, transaction processing 895, and direct data loading 896.

[0103] FIG. 11 is a block / flow diagram of a method for applying the middleware architecture of FIG. 1 to Internet of Things (IoT) systems / devices / infrastructure, according to one embodiment of the present invention.

[0104] According to some embodiments of the invention, the network is implemented using IoT methodologies. For example, direct data loading 902 may be incorporated into, for example, wearable, implantable, or ingestible electronic devices and Internet of Things (IoT) sensors. Wearable, implantable, or ingestible devices may include at least health and wellness monitoring devices and fitness devices. Wearable, implantable, or ingestible devices may further include at least implantable devices, smart watches, head-mounted devices, security and prevention devices, and gaming and lifestyle devices. IoT sensors may be incorporated into at least home automation applications, automotive applications, user interface applications, lifestyle or entertainment applications or a combination thereof, city or infrastructure applications or a combination thereof, toys, healthcare, fitness, retail, tags or trackers or a combination thereof, platforms and components, etc. The direct data loading 902 described herein may be incorporated into any type of electronic device for any type of use or application or operation.

[0105] IoT systems allow users to achieve deeper automation, analysis, and integration within a system. IoT improves the reach and precision of these areas. IoT utilizes existing and emerging technologies for sensing, networking, and robotics. IoT features include artificial intelligence, connectivity, sensors, active engagement, and the use of small devices. In various embodiments, the direct data loading 902 of the present invention can be incorporated into a variety of different devices or systems, or combinations thereof. For example, the direct data loading 902 can be incorporated into a wearable or portable electronic device 904. The wearable / portable electronic device 904 can be included in an implantable device 940, such as smart clothing 943. The wearable / portable device 904 can include a smart watch 942 and smart jewelry 945. The wearable / portable device 904 may further include a fitness monitoring device 944, a health and wellness monitoring device 946, a head-mounted device 948 (e.g., smart glasses 949), a security and prevention system 950, a gaming and lifestyle device 952, a smartphone / tablet 954, a media player 956, or a computer / computing device 958, or a combination thereof.

[0106] The direct data loading 902 of the present invention may further be incorporated into Internet of Things (IoT) sensors 906 for various applications such as home automation 920, automobile 922, user interface 924, lifestyle or entertainment or a combination thereof 926, city or infrastructure or a combination thereof 928, retail 910, tags or trackers or a combination thereof 912, platforms and components 914, toys 930, or healthcare 932 or a combination thereof, and fitness 934. The IoT sensors 906 may use direct data loading 902. Of course, one skilled in the art may envision incorporating such direct data loading 902 into any type of electronic device for any type of application without being limited to those described herein.

[0107] FIG. 12 is a block / flow diagram of an exemplary IoT sensor used to collect data / information related to the middleware architecture of FIG. 1 in accordance with one embodiment of the present invention.

[0108] The IoT loses its distinction without sensors. IoT sensors act as the archetypes that transform the IoT from a standard passive network of devices into an active system that can be integrated into the real world.

[0109] IoT sensors 906 may use direct data loading 902 to continuously transmit information / data in real time over a network 908 to any type of distributed system. Exemplary IoT sensors 906 may include, but are not limited to, position / presence / proximity sensors 1002, motion / speed sensors 1004, displacement sensors 1006 such as acceleration / tilt sensors 1007, temperature sensors 1008, humidity / moisture sensors 1010, and flow sensors 1011, acoustic / sound / vibration sensors 1012, chemical / gas sensors 1014, force / load / torque / strain / pressure sensors 1016, or electrical / magnetic sensors 1018, or combinations thereof. One skilled in the art may contemplate using any combination of such sensors to collect data / information from a distributed system for further processing. One skilled in the art may contemplate using other types of IoT sensors, such as, but not limited to, magnetometers, gyroscopes, image sensors, optical sensors, radio frequency identification (RFID) sensors, or microflow sensors, or combinations thereof. The IoT sensor may also include an energy module, a power management module, an RF module, and a sensing module, where the RF module manages their signal processing, communication through WiFi, ZigBee, Bluetooth, wireless transceiver, duplexer, etc.

[0110] FIG. 13 is a block / flow diagram of an exemplary method for learning relationships between multiple event types according to one embodiment of the present invention.

[0111] In block 1302, during data loading, send external writes to a memory-based database management system (DBMS) if the corresponding internal write is for the vertex, export all external writes to a disk-based DBMS as a file (export file), and transmit external reads for the vertex to the in-memory DBMS when the middleware needs the data.

[0112] At the end of the data loading, in block 1304, a file for the raw data of the disk-based DBMS is generated from the export file, and the generated file is transmitted to the disk-based DBMS.

[0113] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0114] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0115] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0116] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0117] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams or combinations thereof, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0119] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational blocks / steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0120] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function(s). In some alternative implementations, the functions described in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, depending on the functionality involved, or the blocks may possibly be executed in the reverse order. It should also be noted that each block of a block diagram or flowchart diagram, or combination thereof, and combinations of blocks in block diagrams or flowchart diagrams, or combinations thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0121] References in the specification to "one embodiment" or "one embodiment" of the present principles, as well as other variations thereof, mean that the particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," as well as any other variations thereof, in various places throughout this specification are not necessarily all referring to the same embodiment.

[0122] It should be understood that the use of any of the following terms " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B), for example, "A / B," "A and / or B," and "at least one of A and B." As a further example, "A, B, or C, or a combination thereof," and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to many of the items listed, as would be readily apparent to one of ordinary skill in this and related arts.

[0123] Having described preferred embodiments of a system and method for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS) (intended as illustrative and not limiting), it should be noted that modifications and variations can be made by those skilled in the art in light of the above teachings. It should therefore be understood that changes can be made in the particular embodiments described that are within the scope of the invention as outlined by the appended claims. Having thus described aspects of the invention with the detail and specificity required by the patent laws, what is claimed and desired to be protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A computer-implemented method executed on a processor, the computer-implemented method comprising: sending an external write corresponding to the received internal write to a memory-based database management system (DBMS) if the received internal write is for a vertex; exporting all said external writes to a disk-based DBMS as an export file; sending the middleware external reads for vertices to the memory-based DBMS when the middleware requests data; generating a file for raw data of the disk-based DBMS from the export file; sending the generated file for the raw data to the disk-based DBMS; A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein sending the external writes further comprises exporting the internal writes as a recovery file used to send the external writes to the memory-based DBMS upon initiation of data loading.

3. At the start of data loading, if the recovery file exists, generating the external writes to the memory-based DBMS from the recovery file; The computer-implemented method of claim 2 , further comprising sending the generated external writes to the memory-based DBMS.

4. 4. The computer-implemented method of claim 1, wherein sending the external read further comprises sending the external read to the disk-based DBMS if the memory-based DBMS has nothing to fetch.

5. The computer-implemented method of claim 1 , wherein the middleware export extension supports both disk-based and memory-based DBMSs.

6. The computer-implemented method of claim 1 , wherein the middleware export extension handles the internal writes and internal reads.

7. The computer-implemented method of claim 1 , further comprising exporting to the export file an external write corresponding to a received internal write if the internal write is not for a vertex.

8. A computer-readable program that executes on a processor, the computer-readable program, when executed on the processor, causing a computer to: sending an external write corresponding to the received internal write to a memory-based database management system (DBMS) if the received internal write is for a vertex; exporting all said external writes to a disk-based DBMS as an export file; sending the middleware external reads for vertices to the memory-based DBMS when the middleware requests data; generating a file for raw data of the disk-based DBMS from the export file; sending the generated raw data file to the disk-based DBMS; A computer-readable program that causes the computer to execute the following:

9. 9. The computer readable program of claim 8, wherein the step of sending the external writes further comprises the step of exporting the internal writes as a recovery file used to send the external writes to the memory-based DBMS at the start of data loading.

10. At the start of data loading, if the recovery file exists, generating the external writes to the memory-based DBMS from the recovery file; 10. The computer readable program of claim 9, further comprising transmitting the generated external write to the memory-based DBMS.

11. 11. The computer readable program of claim 8, wherein the step of sending the external read further comprises the step of sending the external read to the disk-based DBMS if the memory-based DBMS has nothing to fetch.

12. 12. The computer readable program of claim 8, wherein the middleware export extension supports both disk-based and memory-based DBMSs.

13. The computer readable program of claim 8 , wherein the middleware export extension processes the internal writes and internal reads.

14. 14. The computer readable program of claim 8, further comprising: if a received internal write is not for a vertex, exporting to the export file an external write that corresponds to the internal write.

15. 1. A system comprising: Memory and one or more processors in communication with the memory, If the received internal write is for a vertex, sending an external write corresponding to the internal write to a memory-based database management system (DBMS); exporting all said external writes to a disk-based DBMS as an export file; sending the middleware external reads for vertices to the memory-based DBMS when the middleware requests data; generating a file for raw data of the disk-based DBMS from the export file; Sending the generated raw data file to the disk-based DBMS one or more processors configured to A system comprising:

16. 16. The system of claim 15, wherein sending the external writes further comprises exporting the internal writes as a recovery file used to send the external writes to the memory-based DBMS at the start of data loading.

17. If the recovery file exists, generating the external writes to the memory-based DBMS from the recovery file; The system of claim 16 , further comprising: transmitting the generated external writes to the memory-based DBMS.

18. 18. The system of claim 15, wherein sending the external read further comprises sending the external read to the disk-based DBMS if the memory-based DBMS has nothing to fetch.

19. 19. The system of claim 15, wherein the middleware export extension supports both disk-based and memory-based DBMSs.

20. 20. The system of claim 15, wherein the middleware export extension handles the internal writes and internal reads.

21. 21. The system of claim 15, wherein if a received internal write is not for a vertex, an external write corresponding to the internal write is exported to the export file.

22. 1. A computer-implemented method executed on a processor, the computer-implemented method comprising: receiving an internal write in an export extension of the middleware; determining whether the internal write is for a vertex; sending an external write corresponding to the internal write to an in-memory database management system (DBMS) if the internal write is for a vertex; appending the internal write to a recovery file used to send the external write to the in-memory DBMS at the start of data loading, if the internal write is for a vertex; exporting all said external writes to a disk-based DBMS as an export file; generating a file for raw data of the disk-based DBMS from the export file; sending the generated file for the raw data to the disk-based DBMS; A computer-implemented method comprising:

23. 23. The computer-implemented method of claim 22, further comprising appending the internal write to an export file if the internal write is not for a vertex.

24. 1. A computer-implemented method executed on a processor, the computer-implemented method comprising: In the middleware export extension, receiving an internal write; sending an external write corresponding to the internal write to an in-memory database management system (DBMS) to update the in-memory DBMS; receiving an internal read; sending an external read corresponding to the internal read to the in-memory DBMS and receiving a result; determining whether the results include a record; If the result contains a record, the result is returned to the caller; if the result does not contain any records, sending the external read corresponding to the internal read to a disk-based DBMS; A computer-implemented method comprising:

Images