Direct data loading of records generated by middleware

Abstract

A computer-implemented method for reducing data load overhead of middleware to facilitate direct data loading to a database management system (DBMS) is presented. The method includes sending external writes to a memory-based DBMS if the corresponding internal writes are for vertices, exporting all external writes to a disk-based DBMS as an export file, and sending external reads for vertices to the in-memory DBMS if the middleware requests data, upon data loading. The method also includes generating a file of raw data for the disk-based DBMS from the export file upon the end of data loading, and sending the generated raw file to the disk-based DBMS.

Description

Background Technology

[0001] This invention relates generally to database management systems, and more specifically to loading and routing data of records generated by middleware into a database management system.

[0002] In today's increasingly competitive markets, the ability to act quickly and decisively is crucial to an organization's success. Businesses have access to a rapidly increasing amount of information, often overwhelming them. Organizations that effectively and efficiently manage this massive amount of data and use this information to make business decisions will gain a competitive advantage in the market. This competitive advantage can be achieved by using a Database Management System (DBMS), which stores large amounts of data to support diverse workloads and heterogeneous applications. DBMSs benefit business transaction processing and decision-making and can be incorporated into strategies that promote the high availability of data.

[0003] A DBMS is a database program that uses standard methods for cataloging, retrieving, and running queries on data. A DBMS manages incoming data, organizes it, and provides a way for data to be modified or retrieved by users or other programs. A DBMS provides a query language and report writer that allows users to interactively query the relational database. These essential components enable users to access all administrative information as needed. The DBMS applies the entered text to the database as criteria to identify and report records in the database that meet those criteria. Entering text into fields in the DBMS requires the user to understand the DBMS and how it represents the data in the database. For example, for a given search term, the user must know which field in the DBMS is appropriate for searching that term. For search queries with multiple search terms, the user must be familiar with multiple fields in the DBMS and how these fields interact to limit or otherwise define the search. Furthermore, the user must understand the hierarchical structure between database tables and the keys used to link tables together.

[0004] A common goal of DBMSs is to deliver high performance in terms of transaction throughput and transaction latency to minimize hardware costs, latency, and increase the number of transactions per unit of time. Even with significant hardware investments, achieving the desired performance is often costly and sometimes impossible. Another common goal of DBMSs is to reduce the complexity of the application development process, thereby saving time and money, and reducing the risk of errors.

[0005] However, traditional DBMSs are built on the assumption that data loading is a "one-off transaction." Data loading is considered an offline process outside the critical path, where the user-defined schema loads most of the data "all at once" before any queries are submitted. When this architectural design assumption is combined with explosive data growth, the result is that data loading becomes the bottleneck of the DBMS's data analysis pipeline.

[0006] However, some analyses require data to be pre-loaded into the DBMS. Middleware is useful if the DBMS provides a simple application programming interface (API) for analysis. However, middleware also incurs overhead when data loading occurs. Therefore, data loading within the DBMS can become a bottleneck. Thus, it is necessary to implement methods to reduce data loading within the DBMS. Summary of the Invention

[0007] According to an embodiment, a method is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS). The method includes: during data loading, if the corresponding internal write is vertex-based, sending the external write to a memory-based DBMS; exporting all external writes to a disk-based DBMS as an export file; and if the middleware requires data, sending vertex-based external reads to the memory-based DBMS. The method further includes, at the end of data loading, generating a file of raw data for the disk-based DBMS from the export file, and sending the generated raw file to the disk-based DBMS.

[0008] According to another embodiment, a system is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS). The system includes a memory and one or more processors communicating with the memory, the processors being configured to, during data loading, send an external write to the memory-based DBMS if the corresponding internal write is vertex-based, export all external writes to the disk-based DBMS as an exported file, and send an external read to the memory-based DBMS if the middleware requires data. At the end of data loading, the system further generates a file of raw data for the disk-based DBMS from the exported file and sends the generated raw file to the disk-based DBMS.

[0009] According to another embodiment, a non-transitory computer-readable storage medium is proposed, comprising a computer-readable program for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS). The non-transitory computer-readable storage medium performs the following steps: during data loading, if the corresponding internal write is for a vertex, an external write is sent to a memory-based DBMS; all external writes are exported to a disk-based DBMS as an exported file; and if the middleware requires data, an external read for a vertex is sent to the memory-based DBMS. The non-transitory computer-readable storage medium performs the following steps: at the end of data loading, a file of raw data for the disk-based DBMS is generated from the exported file, and the generated raw file is sent to the disk-based DBMS.

[0010] According to an embodiment, a method is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS). The method includes: receiving an internal write in an exported extension of the middleware; determining whether the internal write is vertex-oriented; when the internal write is vertex-oriented, sending the internal write to the in-memory DBMS; and appending the internal write to a recovery file.

[0011] In a preferred aspect, if the internal write is not directed to a vertex, the internal write is appended to the exported file.

[0012] According to another embodiment, a method is provided for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS). The method includes: receiving an internal read in an exported extension of the middleware; sending the internal read to a memory-based DBMS to receive a result; determining whether the result includes records; and if the result does not contain records, sending the internal read to a disk-based DBMS.

[0013] In a preferred embodiment, if the result includes records, the result is returned to the caller.

[0014] The advantages of this invention include reduced data loading via middleware. Further advantages include more efficient use of the central processing unit (CPU) and more efficient use of input / output (I / O) bandwidth. Data loading is an upfront investment that a DBMS must undertake to support efficient query execution. Given the volume of data collected by applications today, it is important to minimize the overhead of data loading to prevent it from becoming a bottleneck in the data analysis pipeline. This results in greater storage capacity for unstructured data, faster processing, and better transmission speeds. Further advantages include higher quality, lower cost, clearer scope, faster performance, fewer application errors, and fewer data errors.

[0015] In a preferred aspect, the sending of the external write step, concurrent with data loading, further includes exporting the internal write as a recovery file.

[0016] In another preferred aspect, at the start of data loading, if the recovery file exists, an external write to the memory-based DBMS is generated from the recovery file, and the generated external write is sent to the memory-based DBMS.

[0017] In another preferred aspect, the step of sending an external read during data loading further includes sending the external read to a disk-based DBMS if the memory DBMS has not retrieved any external read.

[0018] In another preferred aspect, the middleware's export extensions support both disk-based and storage-based DBMSs.

[0019] In another preferred aspect, the middleware's export extension handles both internal writes and internal reads.

[0020] In another preferred aspect, if the internal write is not vertex-specific, the internal write is appended to the exported file.

[0021] It should be noted that exemplary embodiments are described with reference to different subjects. Specifically, some embodiments are described with reference to method type claims, while other embodiments are described with reference to apparatus type claims. However, those skilled in the art will conclude from the above and below description that, unless otherwise indicated, any combination of features relating to different subjects (in particular, between features of method type claims and features of apparatus type claims) is also considered to be described in this document, except for any combination of features belonging to one type of subject matter.

[0022] These and other features and advantages will become apparent from the following detailed description of exemplary embodiments, which are read in conjunction with the accompanying drawings. Attached Figure Description

[0023] The present invention will be described in detail below with reference to the following drawings, in which:

[0024] Figure 1 This is an exemplary middleware architecture according to an embodiment of the present invention for loading middleware-generated record data directly into a database management system (DBMS);

[0025] Figure 2 This is an exemplary flowchart illustrating a method for receiving internal writes according to an embodiment of the present invention;

[0026] Figure 3This is an exemplary flowchart illustrating a method for receiving internal reads according to an embodiment of the present invention;

[0027] Figure 4 This is an exemplary flowchart illustrating a method for initiating the initialization of exporting an extension according to an embodiment of the present invention;

[0028] Figure 5 This is an exemplary middleware diagram according to an embodiment of the present invention;

[0029] Figure 6 This is an example diagram illustrating a request / response loop according to an embodiment of the present invention;

[0030] Figure 7 This is an exemplary JanusGraph architecture according to an embodiment of the present invention;

[0031] Figure 8 This is an exemplary processing system according to an embodiment of the present invention;

[0032] Figure 9 This is a block diagram / flowchart of an exemplary cloud computing environment according to an embodiment of the present invention;

[0033] Figure 10 This is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention;

[0034] Figure 11 It is a method for using according to an embodiment of the present invention to Figure 1 A block diagram / flowchart illustrating how middleware architecture can be used in conjunction with Internet of Things (IoT) systems / devices / infrastructure.

[0035] Figure 12 This is for collecting and according to an embodiment of the present invention. Figure 1 A block diagram / flowchart of an example IoT sensor for data / information related to the middleware architecture; and

[0036] Figure 13 This is a block / flowchart of an exemplary method for loading middleware-generated record data directly into a DBMS according to an embodiment of the present invention.

[0037] Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0038] According to embodiments of the present invention, methods and apparatus are provided for directly loading records generated by middleware into a database management system (DBMS). A DBMS is a collection of interconnected data sets and programs used to store and access this data in an easy and efficient manner. Database systems are developed for handling large volumes of data. When handling large volumes of data, two things need to be optimized: data storage and data retrieval. Data loading is the process of obtaining and loading transformed data, which is then accessible to users. Data loading is considered a "one-time transaction," an offline process outside the critical path of query execution. The architecture of a DBMS conforms to this assumption. However, the current rate at which data is generated and collected has rendered the "one-time" assumption invalid and made data loading a bottleneck in data analysis pipelines.

[0039] Exemplary embodiments of the invention advantageously assume that the middleware translates the internal representation of updates (internal writes) into one or more commands for updates (external writes) to the DBMS. Furthermore, internal writes to vertices can be advantageously identified during data loading, and the middleware can advantageously support both disk-based and memory-based DBMSs, with disk-based DBMSs advantageously providing a way to directly add files to their original data.

[0040] Exemplary embodiments of the invention advantageously employ an extension for middleware that simultaneously manages a disk-based DBMS and a memory-based DBMS. During data loading, if the corresponding internal write is vertex-based, the extension sends external writes to the memory-based DBMS, exports the internal writes as a file (recovery file), exports all external writes to the disk-based DBMS as a file (export file), and sends vertex-based external reads to the memory-based DBMS if the middleware requests / requests data. Alternatively, if the memory-based DBMS does not retrieve anything, the extension sends external reads to the disk-based DBMS. At the end of data loading, the extension or other components advantageously generate a file of the original data for the disk-based DBMS from the export file and send it to the disk-based DBMS. In another embodiment, at the beginning of data loading, if the recovery file exits, the extension advantageously generates external writes from the file to the memory-based DBMS and sends them to the memory-based DBMS.

[0041] It should be understood that the invention will be described based on the given illustrative framework; however, other frameworks, structures, substrate materials, and process features and steps / blocks may be varied within the scope of the invention. It should be noted that, for clarity, certain features may not be shown in all the figures. This is not intended to be construed as limiting the scope of any particular embodiment, illustration, or claims.

[0042] Figure 1 This is an exemplary middleware architecture according to an embodiment of the present invention for loading middleware-generated record data directly into a database management system (DBMS).

[0043] Middleware architecture 10 includes a data loading application 12, where the vertices and edges of the graph are fed into TinkerPop. TM In API 13, a graph is a structure composed of vertices and edges. Vertices and edges can each have any number of key / value pairs called attributes. Vertices represent discrete objects such as people, places, or events. Edges represent relationships between vertices. For example, a person might know another person, have participated in an event, and / or have recently visited a specific place. Attributes express non-relational information about vertices and edges. Example attributes include vertices with names and ages, and edges with timestamps and / or weights. (The previously mentioned...) Figure 1 It is called a property graph, and it is an Apache TinkerPop graph. TM The fundamental data structure of Apache TinkerPop. TM It is a graph computing framework for graph databases (OLTP) and graph analysis systems (OLAP).

[0044] TinkerPop TM API 13 can be integrated with JanusGraph 20. JanusGraph 20 is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across multi-machine clusters. JanusGraph 20 includes JanusGraph Core 22, a first storage manager SPI 23, and a second storage manager SPI 40. JanusGraph Core 22 provides internal reads and writes to storage manager SPI 23. Storage manager 23 advantageously includes exported extension 24. Exported extension 24 receives and processes internal writes and internal reads. Exported extension 24 is the component that underpins SPI 23 and 40. Exported extension 24 coordinates multiple extensions supporting SPI 23 and 40, with the ability to preload and export data. Because exported extension 24 only uses SPI 23 and 40 to access other extensions, no modifications to JanusGraph 20 are required.

[0045] Export extension 24 receives recovery file 30, and when data is loaded into the extension, if its corresponding internal write at the time of data loading is vertex-based, the external write is sent to the memory-based DBMS 44. If the internal write is vertex-based, the internal write is sent to the memory DBMS extension 44. In the first option, when the internal write is vertex-based, the internal write is advantageously exported as recovery file 32. In the second option, when the internal write is not vertex-based, the method advantageously exports all external writes to the disk-based DBMS extension 42 as export file 34, and if the middleware requests / requests it, vertex-based external reads are sent to the memory DBMS via the disk-based DBMS driver 46.

[0046] In-memory databases (44) are based on database management systems that store their data sets directly in the working memory of one or more computers. A key advantage of using random access memory (RAM) is the significantly faster access speed of in-memory databases. The stored data can then be retrieved very quickly when needed. The greatest advantage of using in-memory databases is the significantly higher access speed resulting from the use of RAM. This also leads to faster data analysis. However, optimizing data analysis involves more than just reducing retrieval time. In-memory databases advantageously enable the evaluation of structured and unstructured data from any system.

[0047] At the end of data loading, extension 24 advantageously generates a file 52 of the raw data for the disk-based DBMS 56 from the exported file 34. Extension 24 advantageously sends the raw file 52 to the disk-based DBMS 56 via the disk-based DBMS tool 54, thereby providing a way to directly add files to the raw data using the disk-based DBMS. Converter 50 advantageously converts the exported file 34 into the raw file 52.

[0048] Figure 2 This is an exemplary flowchart illustrating a method for receiving internal writes according to an embodiment of the present invention.

[0049] In box 102, receive internal writes.

[0050] In box 104, determine whether the internal write is related to the vertex.

[0051] If no, the process proceeds to box 110. If yes, the process proceeds to box 106.

[0052] In box 106, internal writes are sent to the memory DBMS extension.

[0053] In box 108, when the internal write is directed to a vertex, the internal write is appended to the recovery file.

[0054] In box 110, when the internal write is not directed to a vertex, the internal write is appended to the exported file.

[0055] Figure 3 This is an exemplary flowchart illustrating a method for receiving internal reads according to an embodiment of the present invention.

[0056] At box 152, receive internal reads.

[0057] In box 154, the internal read is sent to the memory DBMS extension and the result is received.

[0058] In box 156, determine whether the results include records.

[0059] If yes, the process proceeds to box 160. If no, the process proceeds to box 158.

[0060] In box 158, internal reads are sent to a disk-based DBMS extension and the results are received.

[0061] In box 160, the result is returned to the caller.

[0062] Figure 4 This is an exemplary flowchart illustrating a method for initiating the initialization of an export extension according to an embodiment of the present invention.

[0063] To append internal writes to the recovery file or block 108, perform the following steps.

[0064] In box 182, the initialization of the export extension begins.

[0065] In box 184, confirm whether the recovery file exists.

[0066] If yes, the process proceeds to box 190. If no, the process proceeds to box 186.

[0067] At box 186, all internal writes are sent to the memory DBMS extension.

[0068] In box 188, copy all internal writes to the new recovery file.

[0069] In box 190, the initialization of the export extension ends.

[0070] Figure 5 This is an exemplary middleware diagram according to an embodiment of the present invention.

[0071] Middleware can be described as a glue that combines two separate, existing software programs. Typically, middleware can be a variety of different, specially designed software. Middleware is a type of software that makes it easier for software developers to implement communication and input / output between two programs, allowing them to focus on the specific purpose of their application. As an extension of existing operating systems, middleware facilitates the integration of software between applications and services.

[0072] Middleware can be used to connect any two pieces of software. Middleware works by allowing data to be passed between them. An example of using middleware is connecting a database system to a web server. This allows users to request data from the database using a form displayed in a web browser. Middleware is software that provides common services and functionality to applications outside of those provided by the operating system. Data management, application services, messaging, authentication, and API management are all typically handled by middleware. Furthermore, middleware helps developers build applications more efficiently. Middleware acts like a connective tissue between applications, data, and users. For organizations with multi-cloud and containerized environments, middleware can make developing and running applications at scale cost-effective.

[0073] In middleware diagram 200, middleware 210 advantageously connects participating system 220 to recording system 240. Participating system 220 may include, but is not limited to, public cloud system 222, mobile terminal system 224, Internet of Things (IoT) system 226, social media system 228, affiliates 230, and website 232. Recording system may include, but is not limited to, private cloud 242, customer relationship management (CRM) 244, server 246, database 248, background process 250, and application programming interface (API) service 252.

[0074] The following Figure 11 and 12 A practical application related to IoT system 226 will be shown, which advantageously uses middleware 210 to connect to recording system 240.

[0075] Figure 6 This is an exemplary diagram illustrating a request / response loop according to an embodiment of the present invention.

[0076] In Figure 260, requests 262 and responses 264 to application 270 are processed by middleware 272. Middleware 272 is a function that works with each request 262 before it is processed by any particular path operation, and also with each response before it is returned. Middleware 272 receives each request 262 that arrives at application 270. Middleware 272 can then do something or run any required code on that request 262. Middleware 272 then passes the request 262 to be processed by the remainder of application 270 (via some path operation). Middleware 272 then receives the response 264 generated by application 270 (via some path operation). Middleware 272 can then do something or run any required code on that response 264. Middleware 272 then returns response 264.

[0077] Figure 7 This is an exemplary JanusGraph architecture according to an embodiment of the present invention.

[0078] JanusGraph is a scalable transactional property graph database. A property graph is a mapping between entities (called vertices) through their relationships (called edges). Property graph queries can traverse multiple edges and vertices to help clarify the relationships between entities. Therefore, JanusGraph is a scalable graph database optimized for storing and querying graphs containing tens of billions of vertices and edges distributed across multi-machine clusters.

[0079] The JanusGraph architecture 300 includes applications 302 that are fed into the JanusGraph 310 and the Gremlin graphics computer 330. The Gremlin graphics computer 330 communicates with the big data platform 335.

[0080] Storage Backend 340 is pluggable and supports at least Cassandra, HBase, BerkeleyDB, Google BigTable, and in-memory storage options. Storage Backend 340 is where the actual data is stored. Given its flexibility in working with many database engines, Storage Backend 340 has the advantage of allowing users to choose options that may have already been deployed by the user or that they possess expertise in. Only one Storage Backend 340 exists.

[0081] Next, for the external index backend 350, at least Elasticsearch, Solr, and Lucene are supported. The external index backend 350 is optional, but required for indexing multiple attributes, full-text and string searches, and geomapping. Similarly, only one external index backend 350 can exist.

[0082] After storage backends 340 and 350, in TinkerPop in Gremlin box 310 TM API 321 describes how users interact with the graphical interface. It is often referred to as the Gremlin console and is the interface that calls TinkerPop. TM This is an example of an application based on API 321. It's a command-line interface for interacting with JanusGraph.

[0083] Finally, 310 represents the JanusGraph server. This snippet is run using a script named gremlin_server.sh. The Gremlin server is Apache TinkerPop. TM As part of the project, JanusGraph essentially functions as a Gremlin server plugin, instructing the Gremlin server on how and where to store graph data. The JanusGraph server 310 advantageously includes a management API 320, an internal API layer 322, a database layer 324, a storage and indexing interface layer 326, and an OLAP input / output (I / O) interface 328.

[0084] Gremlin is Titan's query language for retrieving and modifying data in graphs. It's a path-oriented language that concisely expresses complex graph traversal and mutation operations. Gremlin is also a functional language where traversal operators are chained together to form path-like expressions. Gremlin is used for graph databases based on Online Transaction Processing (OLTP) and graph processors based on Online Analytical Processing (OLAP). Gremlin's automata and functional language foundations enable it to naturally support imperative and declarative queries, host language diagnostics, user-defined domain-specific languages, scalable compilers / optimizers, single-machine and multi-machine execution models, and mixed depth-and-breadth-first evaluation.

[0085] In summary, exemplary embodiments of the invention advantageously assume that the middleware translates the internal representation of updates (internal writes) into one or more commands for updates (external writes) to the DBMS. Furthermore, internal writes to vertices can be advantageously identified during data loading, the middleware can advantageously support both disk-based and memory-based DBMSs, and the disk-based DBMS advantageously provides a way to directly add files to its raw data. Exemplary embodiments of the invention advantageously employ an extension of the middleware that manages both disk-based and memory-based DBMSs. During data loading, if the internal write corresponding to the extension is vertex-based, the extension sends external writes to the memory-based DBMS, exports the internal writes as files (recover files), exports all external writes to the disk-based DBMS as files (export files), and sends external reads for vertices to the memory-based DBMS if the middleware needs data. In another option, if the memory-based DBMS does not retrieve any content, the extension sends external reads to the disk-based DBMS. At the end of data loading, the extension or another component generates files of the raw data for the disk-based DBMS from the exported files and sends them to the disk-based DBMS. In another embodiment, at the start of data loading, if the recovery file exits, the extender writes the files from the external storage to the storage-based DBMS and sends them to the storage-based DBMS.

[0086] The advantages of this invention include reduced data loading via middleware. Further advantages include higher CPU utilization and more efficient I / O bandwidth utilization. Data loading is an upfront investment that a DBMS must undertake to support efficient query execution. Considering the amount of data collected by applications, minimizing data loading overhead to prevent it from becoming a bottleneck in the data analysis pipeline is crucial. This results in higher storage capacity, faster processing, and better transmission speeds for unstructured data. Other advantages include higher quality, lower cost, clearer scope, faster performance, fewer application errors, and fewer data errors.

[0087] Figure 8 This is an exemplary processing system for processing stream algorithms according to embodiments of the present invention.

[0088] See now Figure 8This figure illustrates a hardware configuration of a computing system 600 according to an embodiment of the present invention. As can be seen, the hardware configuration has at least one processor or central processing unit (CPU) 611. The CPU 611 is interconnected via a system bus 612 to random access memory (RAM) 614, read-only memory (ROM) 616, input / output (I / O) adapter 618 (for connecting peripheral devices such as disk unit 621 and tape drive 640 to bus 612), user interface adapter 622 (for connecting keyboard 624, mouse 626, speaker 628, microphone 632 and / or other user interface devices to bus 612), communication adapter 634 for connecting system 600 to a data processing network, Internet, intranet, local area network (LAN), etc.; and display adapter 636 for connecting bus 612 to display device 638 and / or printer 639 (e.g., digital printer, etc.).

[0089] Figure 9 This is a block diagram / flowchart of an exemplary cloud computing environment according to one embodiment of the present invention.

[0090] Figure 9 This is a block diagram / flowchart of an exemplary cloud computing environment according to one embodiment of the present invention.

[0091] It should be understood that although this invention includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0092] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0093] The features are as follows:

[0094] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0095] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0096] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0097] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0098] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0099] The service model is as follows:

[0100] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. The applications can be accessed from different client devices via a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0101] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0102] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0103] The deployment model is as follows:

[0104] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0105] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0106] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0107] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0108] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0109] See now Figure 9 This illustration depicts an illustrative cloud computing environment 750 used to implement the present invention. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 that can communicate with local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 754A, desktop computers 754B, laptop computers 754C, and / or automotive computer systems 754N. The nodes 710 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 750 to provide infrastructure, platform, and / or software as a service without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 9 The types of computing devices 754A-N shown are intended to be illustrative only, and computing node 710 and cloud computing environment 750 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0110] Figure 10 This is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. It should be understood in advance that... Figure 10 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0111] The hardware and software layer 860 includes hardware and software components. Examples of hardware components include: a mainframe 861; a server 862 based on a RISC (Reduced Instruction Set Computer) architecture; a server 863; a blade server 864; a storage device 865; and a network and network components 866. In some embodiments, the software components include network application server software 867 and database software 868.

[0112] The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 871; virtual storage 872; virtual network 873, including virtual private network; virtual application and operating system 874; and virtual client 875.

[0113] In one example, management layer 880 can provide the following functionalities: Resource Provisioning 881 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 882 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 883 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 884 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 885 provides pre-scheduling and procurement of cloud resources, anticipating future requirements for those resources according to the SLA.

[0114] The workload layer 890 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 891; software development and lifecycle management 892; virtual classroom education delivery 893; data analytics and processing 894; transaction processing 895; and direct data loading 896.

[0115] Figure 11 It is a method for using according to an embodiment of the present invention to Figure 1 A block diagram / flowchart illustrating how middleware architecture can be used in conjunction with Internet of Things (IoT) systems / devices / infrastructure.

[0116] According to some embodiments of the present invention, networks are implemented using IoT methods. For example, direct data loading 902 can be incorporated into, for example, wearable, implantable, or ingestible electronic devices and Internet of Things (IoT) sensors. Wearable, implantable, or ingestible devices may include at least health and health monitoring devices and fitness devices. Wearable, implantable, or ingestible devices may further include at least implantable devices, smartwatches, head-mounted devices, security and prevention devices, and gaming and lifestyle devices. IoT sensors can be incorporated into at least home automation applications, automotive applications, user interface applications, lifestyle and / or entertainment applications, urban and / or infrastructure applications, toys, healthcare, fitness, retail tags and / or trackers, platforms and components, etc. The direct data loading 902 described herein can be incorporated into any type of electronic device for any type of use or application or operation.

[0117] IoT systems allow users to achieve deeper automation, analytics, and integration within systems. IoT improves coverage and accuracy in these areas. IoT leverages existing and emerging technologies for sensing, networking, and robotics. IoT is characterized by artificial intelligence, connectivity, sensors, proactive engagement, and the use of small devices. In various embodiments, the direct data loading 902 of the present invention can be incorporated into a variety of different devices and / or systems. For example, direct data loading 902 can be incorporated into a wearable or portable electronic device 904. The wearable / portable electronic device 904 may include an implantable device 940, such as smart clothing 943. The wearable / portable device 904 may include a smartwatch 942 and smart jewelry 945. The wearable / portable device 904 may further include a fitness monitoring device 944, a health and wellness monitoring device 946, a head-mounted device 948 (e.g., smart glasses 949), a security and prevention system 950, a gaming and lifestyle device 952, a smartphone / tablet 954, a media player 956, and / or a computer / computing device 958.

[0118] The direct data loading 902 of this invention can be further incorporated into Internet of Things (IoT) sensors 906 for various applications, such as home automation 920, automotive 922, user interface 924, lifestyle and / or entertainment 926, urban and / or infrastructure 928, retail 910, tags and / or trackers 912, platforms and components 914, toys 930 and / or healthcare 932, and fitness 934. IoT network sensors 906 may employ direct data loading 902. Of course, those skilled in the art will anticipate incorporating such direct data loading 902 into any type of electronic device for any type of application, not limited to those described herein.

[0119] Figure 12This is for collecting and according to an embodiment of the present invention. Figure 1 A block diagram / flowchart of an exemplary IoT sensor for data / information related to the middleware architecture.

[0120] Without sensors, the Internet of Things (IoT) loses its essence. IoT network sensors are instruments used to define how IoT transforms from a standard passive network of devices into an active system capable of real-world integration.

[0121] IoT network sensor 906 can continuously and in real-time transmit information / data to any type of distributed system via network 908 using direct data loading 902. Exemplary IoT sensor 906 may include, but is not limited to, a location / presence / proximity sensor 1002, a motion / velocity sensor 1004, a displacement sensor 1006 (such as an acceleration / tilt sensor 1007), a temperature sensor 1008, a humidity / moisture sensor 1010, a flow sensor 1011, an acoustic / sound / vibration sensor 1012, a chemical / gas sensor 1014, a force / load / torque / strain / pressure sensor 1016, and / or an electro / magnetic sensor 1018. Those skilled in the art can envision using any combination of such sensors to collect data / information from a distributed system for further processing. Those skilled in the art can envision using other types of IoT sensors, such as, but not limited to, magnetometers, gyroscopes, image sensors, light sensors, radio frequency identification (RFID) sensors, and / or microflow sensors. The IoT network sensor may also include an energy module, a power management module, an RF module, and a sensing module. The RF module, through its signal processing, WiFi, Management communications such as radio transceivers and duplexers.

[0122] Figure 13 This is a block diagram / flowchart of an exemplary method for learning the relationship between multiple event types according to an embodiment of the present invention.

[0123] At box 1302, during data loading, if the corresponding internal write is for a vertex, the external write is sent to the memory-based database management system (DBMS), all external writes are exported to the disk-based DBMS as a file (export file), and if the middleware needs data, the external read for a vertex is sent to the memory DBMS.

[0124] At box 1304, when the data loading is complete, a file of raw data for the disk-based DBMS is generated from the exported file, and the generated file is transferred to the disk-based DBMS.

[0125] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0126] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0127] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0128] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0129] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0130] These computer-readable program instructions may be provided to at least one processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks or modules of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks or modules of a flowchart and / or block diagram.

[0131] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operation blocks / steps to be executed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more blocks or modules of a flowchart and / or block diagram.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0133] References to this principle in the specification as "one embodiment" or "embodiment" and other variations thereof mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment of this principle. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification, as well as any other variations, do not necessarily refer to the same embodiment.

[0134] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to include selecting only the first listed item (A), or only the second listed item (B), or selecting both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed items (A and C), or only the second and third listed items (B and C), or all three options (A, B, and C). It will be apparent to those skilled in the art that this can be extended for many of the listed items.

[0135] Preferred embodiments of systems and methods for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS) (which are intended to be illustrative and not restrictive) have been described. It should be noted that modifications and variations can be made by those skilled in the art based on the foregoing teachings. Therefore, it should be understood that changes may be made to the specific embodiments described within the scope of the invention as outlined in the appended claims. Various aspects of the invention, having the details and features required by patent law, have thus been described, and the claimed and desired protection by a patent certificate is set forth in the claims.

Claims

1. A computer-implemented method, executed on a processor, for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), the method comprising: During data loading: If the corresponding internal write is for the vertices of the graph representing the data, the external write is sent to the memory-based DBMS; All external writes are exported to a disk-based DBMS as an export file; as well as If the middleware requests data, it will send an external read of the vertex to the memory DBMS; as well as When data loading is complete: A file containing the raw data for the disk-based DBMS is generated from the exported file; as well as The generated raw file is sent to the disk-based DBMS.

2. The method according to claim 1, wherein, During data loading, the external write step further includes exporting the internal write as a recovery file.

3. The method according to claim 2, wherein, If a recovery file exists at the start of data loading: External writes to the memory-based DBMS are generated from the recovery file; and The generated external writes are sent to a memory-based DBMS.

4. The method according to claim 1, wherein, During data loading, the external read step further includes: if the memory DBMS does not acquire any content, then sending the external read to the disk-based DBMS.

5. The method according to claim 1, wherein, The middleware's export extensions support both disk-based and memory-based DBMSs.

6. The method according to claim 1, wherein, The middleware's export extension handles the internal writes and internal reads.

7. The method according to claim 1, wherein, If the internal write is not directed to a vertex, then the internal write is appended to the exported file.

8. A non-transitory computer-readable storage medium comprising a computer-readable program executing on a processor for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), wherein, When the computer-readable program is executed on the processor, the computer-readable program causes the computer to perform the following steps: During data loading: If the corresponding internal write is for the vertices of the graph representing the data, the external write is sent to the memory-based DBMS; All external writes are exported to a disk-based DBMS as an export file; as well as If the middleware requests data, it will send an external read of the vertex to the memory DBMS; as well as When data loading is complete: A file containing the raw data for the disk-based DBMS is generated from the exported file; as well as The generated raw file is sent to the disk-based DBMS.

9. The non-transitory computer-readable storage medium according to claim 8, wherein, During data loading, the external write step further includes exporting the internal write as a recovery file.

10. The non-transitory computer-readable storage medium according to claim 9, wherein, If a recovery file exists at the start of data loading: External writes to the memory-based DBMS are generated from the recovery file; and The generated external writes are sent to a memory-based DBMS.

11. The non-transitory computer-readable storage medium according to claim 8, wherein, During data loading, the external read step further includes: if the memory DBMS does not acquire any content, then sending the external read to the disk-based DBMS.

12. The non-transitory computer-readable storage medium of claim 8, wherein the middleware's exported extensions support both disk-based DBMS and memory-based DBMS.

13. The non-transitory computer-readable storage medium according to claim 8, wherein, The middleware's export extension handles the internal writes and internal reads.

14. The non-transitory computer-readable storage medium according to claim 8, wherein, If the internal write is not directed to a vertex, then the internal write is appended to the exported file.

15. A system for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), the system comprising: Memory; as well as One or more processors communicating with the memory are configured to: During data loading: If the corresponding internal write is for the vertices of the graph representing the data, the external write is sent to the memory-based DBMS; All external writes are exported to a disk-based DBMS as an export file; as well as If the middleware requests data, it will send an external read of the vertex to the memory DBMS; as well as When data loading is complete: A file containing the raw data for the disk-based DBMS is generated from the exported file; as well as The generated raw file is sent to the disk-based DBMS.

16. The system according to claim 15, wherein, During data loading, the external write step further includes exporting the internal write as a recovery file.

17. The system according to claim 16, wherein, If a recovery file exists at the start of data loading: External writes to the memory-based DBMS are generated from the recovery file; and The generated external writes are sent to a memory-based DBMS.

18. The system according to claim 15, wherein, During data loading, the external read step further includes: if the memory DBMS does not acquire any content, then sending the external read to the disk-based DBMS.

19. The system according to claim 15, wherein, The middleware's export extensions support both disk-based and memory-based DBMSs.

20. The system according to claim 15, wherein, The middleware's export extension handles the internal writes and internal reads.

21. The system according to claim 15, wherein, If the internal write is not directed to a vertex, then the internal write is appended to the exported file.

22. A computer-implemented method, executed on a processor, for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), the method comprising: The middleware's exported extension receives internal writes; Determine whether the internal write is directed to the vertices of the graph representing the data; When the internal write is directed to a vertex, the internal write is sent to the memory DBMS; as well as The internal data is appended to the recovery file.

23. The method according to claim 22, wherein, If the internal write is not directed to a vertex, then the internal write is appended to the exported file.

24. A computer-implemented method, executed on a processor, for reducing middleware data loading overhead to facilitate direct data loading into a database management system (DBMS), the method comprising: In the exported extensions of the middleware: Receive internal writes; The external write corresponding to the internal write is sent to the memory DBMS to update the memory DBMS; Receive internal reads; The external read corresponding to the internal read is sent to the memory DBMS to receive the result; Determine whether the result includes records; If the result includes records, then the result is returned to the caller; as well as If the result is not recorded, the internal read is sent to the disk-based DBMS.

Images