Ingestion system for distributed graph databases

Abstract

The disclosed technology is capable of reading ingestion configuration data of a client among multiple clients targeting a graph database; converting the ingestion configuration data from a declarative representation to a graphical representation of the ingestion configuration data; storing the graphical representation of the ingestion configuration data in the graph database; and providing the graphical representation of the ingestion configuration data to a data service; wherein the data service includes one or more of the following: (i) a physical group of at least one computing device configured to store at least a portion of the graph database, (ii) a logical group of at least one computing device configured to store at least a portion of the graph database, or (iii) a combination of (i) and (ii).

Description

Technical Field

[0001] The technical field covered by this disclosure is data ingestion for distributed graph databases. Background Technology

[0002] A distributed data processing system provides a software framework for large-scale distributed storage and processing of data. This distributed software framework can store portions of a database across many different computers on a network. The distributed data processing system coordinates data creation, reading, writing, updating, and deleting operations, query operations, and computations across the computer network.

[0003] Indexes are used to quickly locate data in a database. Just as a book's index identifies the page where a specific word is printed, an index in the database context identifies a specific logical or physical storage location where a specific data item is stored in the database. Indexes are created by the computer that performs the build process. When data is added to or deleted from the database, or when data in the database is updated, the index needs to be rebuilt accordingly. If the data in the database changes frequently, the index may need to be rebuilt more frequently. In a distributed system, database changes and index rebuilds can be performed on a single computer and then propagated across a network of computers within the distributed system.

[0004] There are different types of distributed database architectures, including graph and non-graph architectures. Examples of non-graph architectures include relational databases and key-value stores. In a relational database, data is organized according to a schema that includes tables and keys. Tables store data based on a logical arrangement of rows and columns. Keys are used to join data stored in different tables together. Structured Query Language (SQL) is used to construct and execute queries, as well as to create, update, and manipulate tables and keys. The results of queries from a relational database can include data retrieved from different tables within the relational database.

[0005] Graph databases organize data using graph data structures that consist of multiple interconnected graph primitives. Examples of graph primitives include nodes, edges, and predicates, where nodes store data, edges create relationships between two nodes, and predicates are semantic labels assigned to edges that define or describe the type of relationship that exists between the nodes connected by the edges. Nodes in a graph database can correspond to tables outside the graph database, subsets of tables, or sets of rows and columns from different tables. Another example of a graph primitive is a specific part of a graph or subgraph, such as a particular pair of nodes plus the edges connecting them. Graph query languages ​​are used to construct and execute queries against graph databases, as well as to create, update, and manipulate components of graph databases. Using graph databases, query results can include subsets of the graphs stored in the graph database; for example, one or more graph primitives.

[0006] Distributed systems can be organized based on the arrangement of computers on a network designed for query throughput, response time, reliability, scalability, or performance. For example, a distributed system can include multiple data centers, each residing in a different physical, geographical location, and each containing a complete copy of the database. Data centers can receive updated copies of the database through a replication process. Each data center can include multiple clusters of server computers, and each cluster or server computer can include one or more machines. In a distributed system, portions of the database or index can be logically fragmented. Different fragments can be stored on the same machine or different machines within the distributed system.

[0007] The solutions described in this section are feasible solutions, but not necessarily previously conceived or adopted solutions. Therefore, unless otherwise stated, none of the solutions described in this section should be assumed to be prior art simply because they are included in this section. Attached Figure Description

[0008] In the attached diagram:

[0009] Figure 1 This is a block diagram illustrating at least one embodiment of a computing system that can implement various aspects of the present disclosure.

[0010] Figure 2A It can be used to achieve Figure 1 A flowchart of a process that is part of a computing system.

[0011] Figure 2B The diagram shows that it can be made by Figure 1 An example of an applicable and enabled ingestion configuration created or evaluated as part of a computing system.

[0012] Figure 3A and Figure 3B The diagram shows that it can be made by Figure 1 A block diagram of various examples of ingestion configurations created or evaluated as part of a computing system.

[0013] Figure 4 It can be used to achieve Figure 1 A flowchart of a process that is part of a computing system.

[0014] Figure 5 It can be used to achieve Figure 1 A flowchart of a process that is part of a computing system.

[0015] Figure 6 This is a block diagram illustrating an embodiment of a hardware system that can be used to implement... Figure 1All aspects of the computing system. Detailed Implementation

[0016] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent, however, that the invention may be practiced without these specific details. In other instances, well-known structures and devices are illustrated in block diagram form to avoid unnecessarily obscuring the invention.

[0017] Overview

[0018] Web-based software applications often store and process massive amounts of data. For example, connected network systems (such as social media applications and applications based on them) may create, store, query, and perform complex operations on billions of data records every day.

[0019] Web-based software applications often include search functionality, allowing users to input search queries to search for and retrieve data records that match their search criteria. Examples of search criteria are keywords, such as entity names, job titles, or skills. Examples of data records include user profiles, job postings, news feeds, and comments. In applications that use or are based on social graphs, complex operations may be required to determine the correct set of query results. For example, the system may need several operations to determine the first, second, or third-level connections of entities, especially when entities have a large number of connections. As another example, determining which entity connections match a specific search query may require the system to perform a complex set of operations, especially when the query contains multiple search criteria. To improve the efficiency of performing these and other operations and to increase the speed of retrieving accurate query results from the database, a distributed graph database design can be used, where portions of the graph database and / or its indexes are stored on multiple different machines across a network. Storing data on multiple machines across a network improves parallelism and throughput and prevents bottlenecks.

[0020] One technical challenge is ensuring that the appropriate portions of the graph database index (or data input) are properly configured to be allocated to the appropriate set of machines.

[0021] One existing approach to addressing this challenge involves deployment planning. In this approach, human administrators create deployment plans by writing computer programming code that specifies how portions of the graph database are created and updated on a particular cluster or machine. In existing approaches, these deployment plans are written using imperative programming languages. Therefore, the deployment plan explicitly lists the commands or steps that must be executed. The deployment plan for the cluster typically needs to be propagated to every individual machine in the cluster.

[0022] The drawback of this existing approach is that changes to the deployment plan require manual editing of the code within the deployment plan. Since this code can be quite complex, there is a high risk of errors during the manual editing process.

[0023] Similarly, in existing solutions, the deployment plan is specific to a particular machine or cluster. Therefore, once edited, the updated deployment plan needs to be propagated to all applicable machines or clusters. In distributed systems comprising a large number of machines (e.g., hundreds or thousands), updating the deployment plan can be a very time-consuming and labor-intensive process.

[0024] Furthermore, different clusters or machines may have different deployment plans. Therefore, to avoid errors, the propagation of deployment plan changes requires detailed and complete knowledge of all machines or clusters to which the changes are applied or not. Errors in the propagation of deployment plans can lead to, for example, loading incorrect data into the graph database or building incorrect indexes on specific machines or clusters. In short, existing deployment configuration schemes have proven difficult to manage and are subject to a high risk of costly errors.

[0025] As described in more detail below, the disclosed techniques improve upon existing solutions by replacing error-prone and difficult-to-maintain complex, manually coded deployment plans with graph-based ingestion of configuration metadata evaluated and applied by a query evaluator of a graph database.

[0026] According to the disclosed techniques, loading or updating data from one or more sources of a truth data source into a portion of a graph database can be referred to as ingestion. Data can be consumed or updated in a distributed graph database by establishing an ingestion pipeline between a machine, cluster, or data center where at least a portion of the graph database resides and a source in the truth data source. An ingestion configuration establishes the ingestion pipeline and defines how the ingestion pipeline operates. Unlike existing schemes that are tied to a specific machine or cluster, the disclosed techniques allow ingestion configurations to be established independently of any particular machine or cluster. Alternatively, an ingestion configuration created using the disclosed techniques can be associated with a different portion of a source in the truth data source that writes data to the graph database.

[0027] According to the disclosed technology, the ingestion configuration is directly encoded as graph data into the graph database so that when an image of the graph database is built or distributed to a specific machine, cluster, or data center, the ingestion configuration is included in the image and thus also distributed as part of the image to the same specific machine, cluster, or data center. Because the image contains its own ingestion configuration, the image can be referred to as a self-describing image.

[0028] Another term used in this document for "image" is "index." In the context of the disclosed techniques, an image is an index that facilitates the retrieval of data stored in one or more source data repositories. An image, as used herein, may contain portions of data that have been ingested into a graph database from one or more source data repositories. For example, an image can be defined as a collection of ingested data. Within an image, the ingested data is arranged according to the graph database structure, in a manner that allows for partial queries on the machines or clusters on which the image resides.

[0029] In this context, the computer-executed process of transforming a source of truth data into a graph structure can be referred to as image building. Data transformation is necessary because the format of the ingested data may be incompatible with the graph database. An already built image can be copied from one physical or logical storage location to another. For example, an image can be copied from one data center, cluster, or machine to another. An image can be modified by writing new graph primitives to the graph database and then rebuilding the image. As used herein, loading can refer to loading an image from non-volatile memory to volatile memory, while serving an image can refer to making the loaded image available for queries via network traffic. Serving an image may involve, for example, changing the value of a flag variable to indicate that the image is ready to accept queries from the network.

[0030] Self-describing images ensure the traceability of ingestion configurations across machines or clusters. Similarly, because the self-describing image can be queried, the ingestion configuration can be easily identified and evaluated to determine its suitability for a particular data center, cluster, or machine before loading, writing, or serving the image. In this way, the disclosed technology supports and facilitates the early detection of ingestion configuration errors and mismatches; thereby preventing ingestion configuration errors from propagating to many other machines, including, for example, machines used to serve real-time network services.

[0031] The disclosed technical implementation provides an ingestion configuration service configured, at least in part, based on a declarative query language. The declarative query language allows users to express which data to retrieve, thereby allowing the query processing engine to retrieve said data seamlessly. The declarative query language works in a more general way and involves giving broad descriptions of what tasks need to be accomplished, rather than details about how to accomplish those tasks. Examples of declarative query languages ​​include prolog and SQL. The use of declarative languages ​​simplifies the process of creating, modifying, and maintaining ingestion configurations because it allows the ingestion configuration to be described according to the desired results without explicitly listing the commands or steps that must be performed. As mentioned above, ingestion configurations can initially be created independently of any particular machine or cluster. During or after the initial creation, the ingestion configuration can be applied to or enabled for one or more specific machines or clusters using simple declarative statements also encoded in the graph database.

[0032] As used herein, a node may refer to a node or vertex in a graph of a graph database, rather than a computer in a computer network. A data center, as used herein, may refer to the physical, geographic location of one or more machines or clusters. A cluster, as used herein, may refer to a logical grouping of one or more machines within a data center, or a logical grouping that is independent of the physical location of the machines. A machine, as used herein, may refer to a physical machine, such as a server computer, or a virtual machine running on a server computer. The advantages and benefits described above are provided as examples. The disclosed techniques are not limited to these examples and may provide additional or different advantages and benefits compared to those described above.

[0033] Exemplary use cases

[0034] The disclosed techniques can be described with reference to exemplary use cases for creating, modifying, and managing ingestion configurations for distributed graph databases in the context of web applications; for example, social graph applications such as professional social networking applications.

[0035] Other use cases

[0036] The disclosed techniques are not limited to social graph applications, but can be used more generally to create, modify, and manage ingestion configurations for graph databases. The disclosed techniques can be used by many different types of web applications, where graph-based ingestion configurations can improve data consistency, reliability, or performance, such as any application where graph databases can be frequently queried and / or frequently updated.

[0037] Exemplary computing system

[0038] Figure 1The illustration shows a computing system in which the features described in this document can be implemented. Figure 1 In one embodiment, the computing system 100 includes a user system 110, a network 120, a graph database 130, a distributed data service 140, a set of source data repositories 150 (which are truth sources), a query service 160, an application software system 170, and an ingestion configuration service 180.

[0039] User system 110 includes at least one computing device, such as a personal computing device, server, mobile computing device, or smart home appliance. User system 110 includes at least one software application installed on the computing device or accessible via a network, including user interface 112. For example, user interface 112 may be or include the front-end portion of application software system 170.

[0040] User interface 112 is any type of user interface as described above. User interface 112 can be used to input search queries and view or otherwise perceive output including data retrieved by query service 160 and / or generated by application software system 170. For example, user interface 112 may include a graphical user interface and / or a conversational voice / voice interface, which includes mechanisms for inputting search queries and viewing query results and / or other digital content. Examples of user interface 112 include web browsers, command-line interfaces, and mobile apps. User interface 112 as used herein may include an application programming interface (API).

[0041] Graph database 130 is a graph database containing data that can be retrieved and / or manipulated by query service 160 to generate query results. In some implementations, a portion of graph database 130 may be part of another system, or accessible by or through another system, such as distributed data service 140. Data is ingested into graph database 130 and stored in graph 132. Graph 132 includes each of a plurality of clients for graph database 130. C i client C i Sub-graphics 136. Client C i Sub-graphic 136 contains client C i Ingest configuration metadata 134.

[0042] Client C i The ingestion configuration metadata 134 is configured to facilitate the ingestion of data from one or more source data repositories 150 into graph 132 using the scheme described herein. Client Ci An exemplary implementation of ingesting configuration metadata 134 is shown in the accompanying figures below and described in more detail below.

[0043] As used herein, a client may correspond to a logical group of data stored in one or more source data repositories 150. For example, a client may correspond to a search criterion for a specific type of query or a query that can be executed by query service 160 to retrieve a specific query result set to be delivered to application software system 170. Thus, the client's corresponding logical group of data needs to be searchable via graph database 130. The client's corresponding logical group of data includes one or more datasets.

[0044] For example, "connection" might be the name of a client. A connection client might have multiple datasets, including a "member-to-member connection" dataset, a "member-to-company connection" dataset, and a "member-to-follower connection" dataset. Each dataset of a client can consume data from one or more source data repositories 150. For example, a dataset might consume data only from offline sources, only from nearline sources, or from both offline and nearline sources. Distributed data service 140 stores portions of graph database 130 in a distributed manner. Distributed data service 140 includes multiple data services. D n 142, of which, D This indicates a tag or label that identifies a specific data service, and n It is a positive integer. As used in this article, data service can refer to a physical or geographical grouping of machines, a logical grouping of machines, or a single machine. For example, data service D n 142 can be a data center, cluster, cluster group, or machine. Data service. D n 142 can store a complete copy of the graphics database 130 or only one or more portions of the graphics database 130.

[0045] Each data service D n 142 includes the corresponding image manager M n 144, and stores one or more self-describing images. I i,n 146. image manager M n 144 management is stored in the data service D n Image 146 on 142. For example, image manager. M n144 Performs image copying, building, loading, writing, servicing, and deletion operations. Image Manager M n 144 is also being tracked and replicated to the data service. D n Version 142 of the image, and in data services D n Versions of images built, loaded, written, serviced, and deleted on 142. Image Manager. M n 144 Local storage about data services D n 142 received and in data service D n Information about the image used on 142, such as for data services. D n Status data for 142.

[0046] Self-describing image I i,n 146 contains data for the graphics database 130. i Each client in a client group C i Image of sub-graphic 136, client C i Sub-graphics 136 includes the client C i Ingest configuration metadata 134, among which, i It is a positive integer. Therefore, the self-describing map I i,n 146 can contain clients for each client of the graphics database 130. C i Subgraph 136. Alternatively, self-describing image. I i,n 146 can contain a subset of clients that are only for all clients of the graphics database 130. C i Subgraph 136.

[0047] Source data repository 150 stores the sources of truth data consumed by graph database 130. Source data repository 150 includes one or more offline storage facilities. O x 152 and one or more nearline storage 154 N y ,in, O and N It is a tag or label that identifies a specific source data repository 150, and x and yIt is a positive integer. As used in this article, i , n , x and y Each element can have the same value or different values, and i , n , x and y The value of any one of them can change over time.

[0048] Offline storage O x Example 152 is a source of truth data storage implemented using a relational database. Nearline storage 154 N y An example is a real-time event stream, such as a Kafka event stream. Each source data repository in source data repository 150 stores data according to a schema different from the graph schema of graph database 130. For example, source data repository 150 can store data according to a non-graph schema such as a relational schema.

[0049] The source data repository 150 may reside on at least one persistent and / or volatile storage device, which may reside within the same local network as at least one other device of the computing system 100 and / or in a network remote relative to at least one other device of the computing system 100. Therefore, although described as being included within the computing system 100, a portion of the source data repository 150 may be a part of the computing system 100 or accessible by the computing system 100 via a network (such as network 120).

[0050] Query service 160 interprets and executes queries against graph database 130. Query service 160 also performs various logical operations on graph database 130. For example, query service 160 can create, update, or delete graph primitives from graph database 130. Part of query service 160 may be part of, or accessed by, another system, such as distributed data service 140, application software system 170, or ingestion configuration service 180. Query service 160 can be implemented using a declarative query language.

[0051] Application software system 170 is any type of application software system that includes or utilizes the functionality provided by query service 160. Examples of application software system 170 include, but are not limited to, network-connected software such as social media platforms, and systems that may or may not be based on network-connected software, such as general search engines, job search software, recruiter search software, sales assistance software, advertising software, learning and education software, or any combination of any of the foregoing.

[0052] Ingestion Configuration Service 180 is an interface to Query Service 160 that facilitates the creation and modification of ingestion configurations for one or more client Ci, as described in more detail below. Ingestion Configuration Service 180 is an API or includes a suite of APIs. Ingestion Configuration Service 180 can be implemented using a declarative programming language.

[0053] Although not specifically shown, it should be understood that any of the user system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 includes an interface embodied as computer programming code stored in computer memory. When executed, this computer programming code enables the computing device to support bidirectional communication with any of the following using a communication coupling mechanism: user system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, or ingestion configuration service 180. Examples of communication coupling mechanisms include network interfaces, inter-process communication (IPC) interfaces, and application programming interfaces (APIs).

[0054] The client portion of application software system 170 can operate within user system 110, for example, as a plugin or widget in the graphical user interface of the software application, or as a web browser executing user interface 112. In an embodiment, the web browser can transmit HTTP requests over a network (e.g., the Internet) in response to receiving user input through a user interface provided by the web application and displayed through the web browser. The server running query service 160 and / or the server portion of application software system 170 can receive input, perform at least one operation using the input, and return output using HTTP responses received and processed by the web browser.

[0055] Each of user system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 is implemented using at least one computing device communicatively coupled to electronic communication network 120. Any one of user system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 can be bidirectionally communicatively coupled to network 120. User system 100 and one or more different user systems (not shown) can be bidirectionally communicatively coupled to application software system 170, query service 160, or ingestion configuration service 180.

[0056] A typical user of user system 110 can be an administrator of graph database 130, an administrator of distributed data service 140, an administrator of application software system 170, or an end user of application software system 170. User system 110 is configured to communicate bidirectionally with any one of graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 via network 120.

[0057] The features and functions of user system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 are implemented using computer software, hardware, or a combination of software and hardware, and may include automatic functions, data structures, and combinations of digital data, which are schematically represented in the figure. User system 110, graph database 130, distributed data service 140, source data repository 150, query service 160, application software system 170, and ingestion configuration service 180 are shown as... Figure 1 The diagrams depict individual components for ease of discussion, but this does not imply the need to separate these components. The illustrated systems, services, and data storage (or their functions) can be divided into any number of physical systems comprising a single physical computer system, and capable of communicating with each other in any suitable manner.

[0058] Network 120 can be implemented on any medium or mechanism that facilitates the exchange of data, signals, and / or instructions between various components of computing system 100. Examples of network 120 include, but are not limited to: local area network (LAN), wide area network (WAN), Ethernet or Internet, or at least one terrestrial, satellite, or wireless link, or any combination of any number of different network and / or communication links.

[0059] It should be understood that the computing system 100 is merely one example of an implementation of the technology disclosed herein. Although described for ease of discussion, reference may be made to... Figure 1 The system may be referred to as "System 100," but the disclosed techniques may be implemented using other suitable configurations of hardware and software components. Similarly, the specific embodiments shown in the following figures and described below are provided by way of example only, and this disclosure is not limited to these exemplary embodiments.

[0060] Exemplary operation of ingesting configuration services

[0061] Figure 2A This is a simplified flowchart illustrating embodiments of the operation and components of a computing system capable of performing various aspects of the disclosed technology. (As shown in...) Figure 2AThe operations of stream 200 shown can be implemented using processor-executable instructions stored in computer memory. To provide a clear example, Figure 2A The operation is described as being performed by computing system 100, but other embodiments may use other systems, devices or implemented technologies.

[0062] exist Figure 2A In this context, event services 202 are exchanged between application software system 170 and query service 160. Event services 202 can include online, offline, and / or nearline communication of data and / or instructions over a network. For example, event services 202 can include data record updates, activity logs, queries, and / or query results. Query service 160 converts event services 202 received from application software system 170 into a format that can be read and processed by graph database 130 (i.e., event services 204), and converts event services 204 received from graph database 130 into a format that can be read and processed by application software system 170 (i.e., event services 202).

[0063] Data storage for the graph database 130 is distributed across the distributed data service 140. The query service 160 manages the ingestion of data from the source data repository 150 to the distributed data service 140. In doing so, the query service 160 encodes the ingestion configuration created by the ingestion configuration service 180 into the graph database 130. Figure 2A The diagram illustrates the specific client. C i The creation and modification of ingestion configurations and the transfer of an image containing that ingestion configuration to a specific data service of distributed data service 140. D n Example 142. Although the creation and modification of the ingestion configuration should occur during the load time, such creation and modification can occur while query service 160 is processing event services 202 and 204, or at other times.

[0064] To create client-side C i The ingestion configuration service 180 receives client information from user system 110. C i API call 208 (in) Figure 2A Step 1 in the process). Client C i API call 208 can be issued by a human administrator or by an automated process or organization. (Client) C i API call 208 contains one or more parameters indicating the required ingestion configuration. These can be included in the client. C iExamples of arguments in API call 208 include a unique client identifier and arguments containing ingestion configuration and offline targeting. O x Storage 152 and nearline storage 154 N y The path name of the file where the information is stored. Offline. O x Storage 152 is an example of a Hadoop DFS path containing offline data files (to be ingested). These "offline data files" are snapshots of the primary truth source relational database tables. Nearline Storage 154 N y The example is the name of a Kafka topic, with 154 nearline storage. N y Receive real-time update streams from a Kafka topic. Ingest configuration service 180 from the client. C i API call 208 requested the ingestion configuration to be read. C i Intake configuration 210, and will C i Ingest configuration 210 is transmitted to query service 160 ( Figure 2A Step 2).

[0065] Ingestion configuration is metadata, specific to a particular client. C i The metadata includes data identifying one or more source data repositories, instructions for connecting to the identified source data repository(s) 150, transformation logic, and metadata. The transformation logic included in the ingestion configuration provides a way to transform input data 216 stored in the source data repository 150 into data that can be stored in the graph database 130, based on a pattern 224 of the source data repository 150. C i Instructions for sub-graphic 136.

[0066] Ingestion configuration may also include identifying one or more data services to which the ingestion configuration is applied. D n One or more tags or labels. For example, ingestion configuration can specify that a particular offline configuration should only apply to a specific data service. D n This ingestion configuration service allows administrators to selectively customize ingestion configurations for different data services without having to write explicit code to implement customized ingestion configurations. For example, production clusters and test machines can each have configurations for the same client. C i Different intake configurations, and these differences canC i The input configuration 210 is indicated and encoded in the graphics database 130.

[0067] Ingestion configurations include priority data, such as those indicating the ingestion priority order across multiple source data repositories. Priority data can specify, for example, that ingestion from an offline data store should precede ingestion from a nearline data store, or that ingestion from a first nearline data store should precede ingestion from a second nearline data store. Another example of metadata that can be included in an ingestion configuration is a historical date / timestamp, which indicates the starting point for consuming input data from a nearline source (e.g., going back from that date / timestamp and consuming nearline updates to ensure data consistency). Yet another example of metadata that can be included in an ingestion configuration is a flag name that facilitates the collection and aggregation of metrics. Additional examples of ingestion configurations are available in... Figure 3A and Figure 3B As shown below.

[0068] Query service 160 reads C i Intake configuration 210, will C i Ingestion configuration 210 is converted into one or more C i Graphic primitive 212, and one or more C i Graphic primitive 212 is stored on the client. C i In sub-graphic 136 ( Figure 2A Step 3). In Figure 3A It shows C i An example of graphic primitive 212 is described below.

[0069] Simultaneously or later, the ingestion configuration service 180 receives data services. D n API call 218 ( Figure 2A Step 4). Data services D n API call 218 can be issued by a human administrator or by an automated process or organization. Data Services D n API call 218 contains instructions for a specific data service. D n Enable specific C i Take in one or more variables of configuration 210. See below for reference. Figure 2B This describes the difference between applicable intake and enabled intake. (Included in data services) Dn Examples of variables in API call 218 include data services. D n Identifiers and client identifiers C i The data service will be accessed via configuration service 180. D n API call 218 requested the read of the ingestion configuration modification to enable it. D n In the intake configuration 220, and will be enabled. D n The ingestion configuration 220 is transmitted to the query service 160 ( Figure 2A Step 5).

[0070] Query service 160 read enabled D n Intake configuration 220 will be enabled. D n The intake configuration 220 is converted into one or more D n Graphic primitive 212, and the one or more of the above D n Graphic primitive 212 is stored on the client. C i In sub-graphic 136 ( Figure 2A Step 6). D n An example of graphic primitive 212 is in Figure 3B As shown below.

[0071] During ingestion, query service 160 reads input data 216 and pattern 224 from one or more source data repositories 150, converts input data 216 into a graphical representation of input data 216, and stores the graphical representation of input data 216 in graphical database 130.

[0072] Since the ingestion configuration is encoded in the graph database 130, therefore when targeting data services... D n When building an image, the ingestion configuration is included in the image. Therefore, when targeting data services... D n When building an image, it is possible to query and extract ingestion configurations from the image, as well as identify its associated images and (one or more) client data, and then process this data in the data service. D n The data is stored locally in state data 222. This allows data services... D n The relative state data 222 is used to assess the image that it may receive from other sources.

[0073] For example, if the image is copied from another location to a data service D n Then data services D n It can query and retrieve the ingestion configuration for the image and compare it with its own state data 222. If the ingestion configuration for the image does not match the state data 222, the data service... D n The system can determine the next appropriate action, which could be rejecting the image or determining that the image is acceptable and can be served by data services. D n Serving with zero or more modifications.

[0074] Examples of ingestion configurations that can be applied and enabled

[0075] Client C i Different configurations can be applied to different data services, or there can be configurations that apply to a specific subset of services. The "applies to" attribute establishes this relationship. For example, C ix It is a client C applicable to services D1 and D2. i The configuration, and C iy It is only applicable to client C of service D3. i The configuration. However, C ix It's possible that services may not actually be enabled on either D1 or D2. The "Enabled" element tracks the services that are actually enabled for the applicable configuration. The set of "Enabled" services for a client configuration is a subset of the set of "Applies to" services for the client configuration. For example, although configuration C... ix This applies to services D1 and D2, but C ix It may only be enabled on D1.

[0076] Figure 2B The diagram shows that it can be made by Figure 1 This is an example of a selectively adjustable ingestion configuration 250 created or evaluated as part of a computing system. Configuration 250 includes applicable and enabled ingestion for each of three tags: tag 1, tag 2, and tag 3. Each tag in tags 1, 2, and 3 corresponds to a physical, geographical, and / or logical grouping of machines. For example, any tag in tags 1, 2, and 3 could correspond to a data center, cluster, grouping of clusters, or machine. Configuration 250 includes only 3 tags as an example; any configuration can have any number of tags.

[0077] For illustrative purposes, assume that each of the tags 1, 2, and 3 corresponds to a different data center. In configuration 250, each data center has a different set of applicable ingestions, but in other configurations, some or all data centers may have the same set of applicable ingestions. Each applicable ingestion corresponds to a different client. C i For example, each applicable ingestion 252, 254, 256, 258 is specified for a specific client. C i The ingestion configuration is specified, and the ingestion configuration is specified to apply to all machines associated with tag 1.

[0078] For example, by adding "Applies to" graphic primitives C i Graphic primitive 212 enables the ingestion configuration to be applied to a specific marker. Applicability information can be included in the initial version of the ingestion configuration, for example, as... C i Take in part of configuration 210, or you can add it later, for example, to enable it. D n The intake configuration 220 can be applied to all tags. C i The ingestion configuration 210 is referred to as the basic configuration. Once the basic configuration for all applicable tags is encoded in the graph database 130, including "applicable" information in the basic configuration maximizes its flexibility. For example, if an administrator later wishes to enable ingestion on a specific machine, they can do so by simply building or copying an image that includes the basic configuration and adding enabling graph primitives via the ingestion configuration service 180 that indicate which machines ingestion should be enabled.

[0079] In configuration 250, ingestions 252, 254, and 258 have been enabled for flag 1. As a result, when an image is written to flag 1, only ingestions 252, 254, and 258 will be included in the image. Similarly, although ingestions 260, 262, 264, and 266 are already included in the applicable ingestion set for flag 2, only ingestions 262 and 264 are included in the enabled ingestion set. For flag 3, ingestions 268, 270, 272, and 274 are included in the applicable ingestion set, but only ingestions 268 and 274 are included in the enabled ingestion set.

[0080] As in Figure 2BAs shown, only enabled ingestions are written into the image for a specific label. Therefore, the image built for label 1 only includes ingestions 252, 254, and 258. If an attempt is made to copy the image built for label 1 to label 3, a mismatch occurs because ingestions 252, 254, and 258 are not yet enabled for label 3. Similarly, if an attempt is made to copy the image built for label 2 to label 1, a mismatch occurs because ingestions 262 and 264 are not yet enabled for label 1. For example, label 3 could be an experimental node, while ingestions 268 and 274 could be small samples of a larger dataset. If label 1 is a production node that requires the complete dataset, then using an image built for label 3 would result in an incomplete dataset in the production system. In this way, the disclosed technique helps ensure data consistency and prevent costly ingestion errors.

[0081] Graph-based ingestion configuration example

[0082] Figure 3A and Figure 3B The diagram shows that it can be made by Figure 1 A block diagram illustrating an example of an ingestion configuration created or evaluated as part of a computational system.

[0083] As described above, the disclosed technology is capable of representing an ingested configuration as a logical graph of at least a portion of a graph database. The graphical representation of the ingested configuration data for a graph database client includes specifications for ingesting data from at least one source data repository into the graph database.

[0084] As in Figure 3A and Figure 3B The graphical representation shown, which ingests configuration data, includes at least two nodes, at least one edge, and at least one predicate. The at least two nodes include at least an offline configuration node and / or a near-line configuration node. Edges indicate the logical relationship between the two nodes. Predicates include labels for the logical relationship between the two nodes. These labels relate to specifications for ingesting data from at least one source data repository into the graph database.

[0085] Specifications for ingesting data from at least one source data repository into a graph database include ingestion configuration data for one or more offline data sources for the client and / or ingestion configuration data for one or more nearline data sources for the client.

[0086] Ingestion configuration data for offline or near-line data sources includes location data, such as path names or KAFKA topic names, metadata, and transformation logic, identifying the physical or logical location of at least one source data repository for the client. Metadata can specify the priority order for writing data to the graph database for the client. Transformation logic can include instructions for transforming input data stored in one or more source data repositories into a graphical representation of the input data. Ingestion configuration data is written in declarative form.

[0087] The graphical representation of the ingested configuration data may also include applicable data service data that identifies at least one data service to which the ingested configuration data is applicable, and enabled data service data that identifies at least one data service for which the ingested configuration data is enabled. When an image is built for a data service, the logical graph is included in the image, such that the graphical representation of the ingested configuration is included in the image stored at the data service. This is why the image is self-describing.

[0088] Figure 3A and Figure 3B The diagram illustrates a graphical representation of the ingestion configuration, which can be created and encoded in the graphical database 130 by the query service 160.

[0089] Figure 3A The diagram illustrates ingestion configuration 300. Ingestion configuration 300 comprises four nodes: ingestion configuration 302, offline configuration 304, nearline configuration 306, and a data service. D n 308. Ingestion configuration 300 includes three sides: a first side between ingestion configuration 302 and offline configuration 304, a second side between ingestion configuration 302 and nearline configuration 306, and a third side between ingestion configuration 302 and data service. D n The third edge between 308. Each edge has a predicate describing the relationship between the two connected edges: 310 is included between ingestion configuration 302 and offline configuration 304, 312 is included between ingestion configuration 302 and nearline configuration 306, and 312 is included between ingestion configuration 302 and data service. D n The `applicable_to` option (which can be applied) between 308 and 304. This can be achieved by, for example, removing offline configuration (304), removing nearline configuration (306), or removing data services. D n 308 or add something like a data service D n 308 However, another node used for different data services and the applicable_to edge modify the ingestion configuration 302. These modifications utilize the state described in a declarative language, and the query evaluator of the graph database modifies 302.

[0090] Figure 3B The diagram illustrates ingestion configuration 350. Except for the inclusion of additional nodes and edges, ingestion configuration 350 is identical to ingestion configuration 300. More specifically, ingestion configuration 350 has added new nodes and data sub-services. D m 318 and the newly enabled edge 316. In Figure 3B In the example, data sub-service D m It is a data service D n A subset. This indicates that ingestion configuration 302 is only for the data subservice. D m Enabled on (one or more) machines, not for data services. D n Enabled on all machines.

[0091] Exemplary process for creating an intake configuration

[0092] Figure 4 It can be used to achieve Figure 1 The flowchart is a part of the process of the computing system. More specifically, process 400 is an example of a process that can be used by, for example, a query service 160 coordinated with ingestion configuration service 180, to create an ingestion configuration and encode the ingestion configuration into a graph database.

[0093] In operation 402, process 400 reads ingestion configuration data for clients in a client set for the graph database. To do this, operation 402 may read one or more arguments of an API call. Reading ingestion configuration data may include reading declarative representations of ingestion configuration data for offline data sources and / or ingestion configuration data for nearline data sources for clients.

[0094] Reading ingestion configuration data may include reading location data that identifies the physical or logical location of the data source for the client, metadata specifying the priority order for the client to write data to the graph database, and a declarative representation of any combination of any of the following: location data that identifies the physical or logical location of the data source for the client; metadata specifying the priority order for the client to write data to the graph database; and transformation logic for converting input data stored in the data source into a graph representation of the input data. Reading ingestion configuration data may also include reading applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable and / or a declarative representation identifying enabled data service data for at least one data service to which the ingestion configuration data is enabled.

[0095] The reading of ingestion configuration data for the client can be performed using an application programming interface (API) configured at least in part according to a declarative query language. In operation 404, process 400 creates a graphical representation of the ingestion configuration read in operation 402. To this end, operation 402 can parse the ingestion configuration data into graphical primitives.

[0096] Creating a graphical representation of ingestion configuration data for a client may include converting the ingestion configuration data into graphical primitives, the graphical primitives including at least two nodes, at least one edge, and at least one predicate; wherein the edge of the at least one edge indicates a logical relationship between two of the at least two nodes, the predicate of the at least one predicate includes a label for the logical relationship between the two nodes, and the label relates to the ingestion configuration, and the ingestion configuration includes specifications for ingesting input data from a non-graphical data source into a graphical database.

[0097] In operation 406, process 400 stores a graphical representation of the ingestion configuration data created by operation 404 in a graphical database. To this end, operation 406 can encode the graphical representation of the ingestion configuration in a client-side sub-graph of the graphical database.

[0098] In operation 408, process 400 modifies the graphical representation of the ingested configuration data to include enabling data services. To do this, operation 408 may add new edges to the graphical representation of the ingested configuration data and store the modified graphical representation of the ingested configuration data in a graphical database. Operation 408 may include modifying the graphical representation of the ingested configuration data to indicate that the ingested configuration data is enabled for at least one data service, or modifying the graphical representation of the ingested configuration data to indicate that the ingested configuration data is not enabled for at least one data service.

[0099] Following operation 408, process 400 can provide the data service with a graphical representation of the ingestion configuration data for the client; for example, in response to a request from the data service. Following operation 408, the data service can create an image containing a graphical representation of the ingestion configuration data for the client.

[0100] Process 400 may include receiving ingestion configuration data from at least one first device, creating a graphical representation using at least one second device, and storing the graphical representation in a graphical database on at least one third device.

[0101] Exemplary process for detecting ingestion configuration mismatch

[0102] Figure 5 It can be used to achieve Figure 1The flowchart is a part of a process in a computing system. More specifically, process 500 is an example of a process that can be performed by a data service to evaluate ingested configuration data before building, loading, writing, or serving an image at the data service.

[0103] In operation 502, process 500 queries a graph database to determine an ingestion configuration for a client. Operation 502 may include querying an image of the graph database to determine, for a client among a plurality of clients of the graph database, ingestion configuration data applicable to a plurality of data services serving the graph database, wherein the client may correspond to at least one dataset arranged according to a non-graph or graph pattern.

[0104] In operation 504, process 500 reads status data for the data service to determine, for the client, ingestion configuration data enabled for the data service. Operation 504 may include reading ingestion configuration data for local storage of an image that has previously been built, loaded, written to, or serviced on the data service.

[0105] In operation 506, process 500 compares the ingestion configuration data enabled for the data service as determined in operation 504 with the ingestion configuration data applicable to the data service as determined in operation 502. Operation 506 may include evaluating ingestion configuration data for offline data sources for clients and / or evaluating ingestion configuration data for nearline data sources for clients.

[0106] For example, operation 506 may include comparing applicable offline configuration data with enabled offline configuration data and / or comparing applicable nearline configuration data with enabled nearline configuration data. As another example, operation 506 may include evaluating location data that identifies the physical or logical location of a non-graphical data source for the client and / or evaluating metadata that specifies the priority order for the client to write data to the graphics database.

[0107] In operation 508, process 500 determines whether there is a mismatch between the ingestion configuration data enabled for the data service and the ingestion configuration data applicable to the data service. Examples of mismatches include offline configurations applicable to the data service but not enabled for the data service and / or nearline configurations applicable to the data service but not enabled for the data service.

[0108] Detecting a mismatch between ingestion configuration data enabled for a data service and ingestion configuration data applicable to the data service may include determining that offline ingestion is applicable to the data service but is not enabled for the data service, or that nearline ingestion is applicable to the data service but is not enabled for the data service, or that the metadata contained in the ingestion configuration data enabled for the data service does not match the metadata contained in the ingestion configuration data applicable to the data service, or any combination of the foregoing.

[0109] If operation 508 detects a mismatch, process 500 proceeds to operation 510. In operation 510, process 500 modifies the data service operation based on the detected mismatch. The operation to modify the data service based on the detected mismatch may include building an image on the data service, writing the image to a network-accessible storage location, deleting the image from the data service, copying the image from another storage location to the data service, or any combination of any of the foregoing.

[0110] If no mismatch is detected in operation 508, process 500 proceeds to operation 512. In operation 512, process 500 continues with operations for data service without modifying the existing operations. For example, process 500 may continue to load, write, or service the image containing the ingestion configuration data identified in operation 502.

[0111] Exemplary hardware architecture

[0112] According to one embodiment, the techniques described herein are implemented by at least one dedicated computing device. The dedicated computing device may be hardwired to execute the techniques, or may include digital electronic devices, such as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) persistently programmed to execute the techniques, or may include at least one general-purpose hardware processor programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such a dedicated computing device may also combine custom hardwired logic, ASIC, or FPGA with custom programming to implement these techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a mobile computing device, a wearable device, a networked device, or any other device incorporating hardwired and / or program logic to implement the techniques.

[0113] For example, Figure 6 This is a block diagram illustrating a computer system 600 on which embodiments of the present invention may be implemented. The computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled to the bus 602 for processing information. The hardware processor 604 may be, for example, a general-purpose microprocessor.

[0114] Computer system 600 also includes main memory 606, such as random access memory (RAM) or other dynamic storage device, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. When such instructions are stored in a non-transitory computer-readable storage medium accessible to processor 604, computer system 600 is rendered as a dedicated machine customized to perform the operations specified in the instructions.

[0115] The computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to the bus 602 for storing static information and instructions for the processor 604. A storage device 610 (e.g., a magnetic disk or optical disk) is provided and coupled to the bus 602 for storing information and instructions.

[0116] Computer system 600 may be coupled via bus 602 to output device 612, such as a display, like a liquid crystal display (LCD) or touchscreen display, for displaying information to a computer user, or a speaker, haptic device, or other form of output device. Input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selection to processor 604. Another type of user input device is cursor control 616, such as a mouse, trackball, or arrow keys, for communicating directional information and command selection to processor 604 and for controlling cursor movement on display 612. This input device typically has two degrees of freedom on two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify position in a plane.

[0117] Computer system 600 may implement the techniques described herein using custom hardwired logic, at least one ASIC or FPGA, firmware and / or program logic (which, in conjunction with the computer system, enables computer system 600, or is referred to as programming it as a special-purpose machine). According to one embodiment, computer system 600 performs the techniques described herein in response to processor 604 executing at least one sequence of instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0118] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, etc.

[0119] Storage media differ from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the information transmission between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires with bus 602. Transmission media can also take the form of sound waves or light waves, such as those generated in radio waves and infrared data communication.

[0120] Various forms of media may be involved in transmitting at least one sequence of instructions to processor 604 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 600 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 602. Bus 602 transmits the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received in main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0121] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupling to network link 620 connected to local network 622. For example, communication interface 618 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide data communication connectivity to a corresponding type of telephone line. As another example, communication interface 618 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 618 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0122] Network link 620 typically provides data communication to other data devices via at least one network. For example, network link 620 may provide a connection to host computer 624 or to data devices operated by Internet Service Provider (ISP) 626 via local network 622. ISP 626, in turn, provides data communication services via a global packet data communication network commonly referred to as the "Internet" 628. Both local network 622 and Internet 628 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals transmitting digital data to and from computer system 600 via various networks, as well as signals on network link 620 and via communication interface 618, are exemplary forms of transmission media.

[0123] Computer system 600 can send messages and receive data, including program code, via a network, network link 620, and communication interface 618. In the Internet example, server 530 can transmit application request code via Internet 628, ISP 626, local network 622, and communication interface 618. Received code can be executed by processor 604 upon receipt and / or stored in storage device 610 or other non-volatile storage device for later execution.

[0124] Additional examples

[0125] The following provides illustrative examples of the techniques disclosed herein. Embodiments of the techniques may include any of the examples or combinations described below.

[0126] In Example 1, a method includes: reading ingestion configuration data for a client among a plurality of clients for a graph database; converting the ingestion configuration data for the client from a declarative representation to a graphical representation of the ingestion configuration data for the client; storing the graphical representation of the ingestion configuration data for the client in the graph database; and providing the graphical representation of the ingestion configuration data for the client to at least one data service; wherein the at least one data service includes one or more of the following: (i) a physical group of at least one computing device configured to store at least a portion of the graph database, (ii) a logical group of at least one computing device configured to store at least a portion of the graph database, or (iii) a combination of (i) and (ii).

[0127] In Example 2, which includes the subject of Example 1, reading ingestion configuration data includes reading one or more declarative representations of: (i) ingestion configuration data for an offline data source of the client, or (ii) ingestion configuration data for a nearline data source of the client, or (iii) a combination of (i) and (ii). In Example 3, which includes the subject of Example 2 or Example 1, reading ingestion configuration data includes reading one or more declarative representations of: (i) location data identifying the physical or logical location of a data source for the client, (ii) metadata specifying the priority order for writing data to the graph database for the client, or (iii) transformation logic for converting input data stored in the data source into a graph representation of the input data, or (iv) a combination of at least two of (i), (ii), and (iii). In Example 4 of a topic that includes any one of Examples 1-3, reading ingestion configuration data includes reading a declarative representation of one or more of the following: (i) applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable, (ii) enabled data service data that identifies at least one data service for which the ingestion configuration data is enabled, or (iii) a combination of (i) and (ii). In Example 5 of a topic that includes any one of Examples 1-4, creating the graphical representation of the ingestion configuration data for the client includes converting the ingestion configuration data into graphical primitives, the graphical primitives including at least two nodes, at least one edge, and at least one predicate; wherein the edge of the at least one edge indicates a logical relationship between two of the at least two nodes.

[0128] In Example 6, which includes the subject of Example 5, the predicate in the at least one predicate includes a label for the logical relationship between the two nodes, and the label relates to an ingestion configuration, and the ingestion configuration includes a specification for ingesting input data from a non-graphical data source into the graphical database.

[0129] In Example 7 of a topic that includes any one of Examples 1-6, the method further includes: reading ingestion configuration data for the client using an application programming interface (API) configured at least in part according to a declarative query language. In Example 8 of a topic that includes any one of Examples 1-7, the method further includes: (i) modifying the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is enabled for the at least one data service, or (ii) modifying the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is not enabled for the at least one data service.

[0130] In Example 9, a method for evaluating an image of a graph database includes: querying the image of the graph database to determine, for a client among a plurality of clients of the graph database, ingestion configuration data applicable to a data service among a plurality of data services serving the graph database; wherein the client corresponds to at least one dataset arranged according to a non-graph mode; reading state data for the data service to determine, for the client, ingestion configuration data enabled for the data service; comparing the ingestion configuration data enabled for the data service with ingestion configuration data applicable to the data service; detecting a mismatch between the ingestion configuration data enabled for the data service and the ingestion configuration data applicable to the data service; and modifying the operation of the data service based on the detected mismatch; wherein the data service includes one or more of the following: (i) a physical group of at least one computing device configured to store at least a portion of the graph database, (ii) a logical group of at least one computing device configured to store at least a portion of the graph database, or (iii) a combination of (i) and (ii).

[0131] In Example 10, which includes the subject of Example 9, comparing the ingestion configuration data enabled for the data service with ingestion configuration data applicable to the data service includes one or more of the following: (i) evaluating ingestion configuration data for an offline data source for the client, (ii) evaluating ingestion configuration data for a nearline data source for the client, or (iii) a combination of (i) and (ii).

[0132] In Example 11, which includes any of Examples 9-10, comparing the ingestion configuration data enabled for the data service with the ingestion configuration data applicable to the data service includes one or more of the following: (i) evaluating location data that identifies the physical or logical location of a non-graphical data source for the client, (ii) evaluating metadata that specifies the priority order for writing data to the graph database for the client, or (iii) a combination of (i) and (ii).

[0133] In Example 12, which includes any of Examples 9-11, detecting a mismatch between the ingestion configuration data enabled for the data service and the ingestion configuration data applicable to the data service includes: determining one or more of the following: (i) offline ingestion is applicable to the data service but not enabled for the data service; (ii) nearline ingestion is applicable to the data service but not enabled for the data service; (iii) metadata contained in the ingestion configuration data enabled for the data service does not match metadata contained in the ingestion configuration data applicable to the data service; or (iv) at least two of (i), (ii), and (iii).

[0134] In Example 13, which includes any of Examples 9-12, the operation of modifying the data service based on the detected non-match includes one or more of the following: (i) building an image on the data service, (ii) writing the image to a network-accessible storage location, (iii) deleting the image from the data service, (iv) copying the image to the data service from another storage location, or (v) at least two of (i), (ii), (iii) and (iv).

[0135] In Example 14, a system includes: at least one processor; and at least one computer memory operatively coupled to the at least one processor; wherein the at least one computer memory is configured according to a logical graph of at least a portion of a graph database; the logical graph includes a graphical representation of ingesting configuration data; the graphical representation includes: a specification for ingesting data from at least one source data repository into the graph database for a client among a plurality of clients of the graph database.

[0136] In Example 15, which includes the subject of Example 14, the graphical representation of the ingested configuration data includes at least two nodes, at least one edge, and at least one predicate, and the at least two nodes include one or more of the following: (i) an offline configuration node, (ii) a near-line configuration node, or (iii) a combination of (i) and (ii).

[0137] In Example 16, which includes any of Examples 14-15, the graphical representation of the ingested configuration data includes at least two nodes, at least one edge, and at least one predicate; wherein the edge of the at least one edge indicates a logical relationship between two of the at least two nodes, the predicate of the at least one predicate includes a label for the logical relationship between the two nodes, and the label relates to a specification for ingesting data from the at least one source data repository into the graphical database.

[0138] In Example 17, which includes any one of Examples 14-16, the graphical representation of the ingestion configuration data includes one or more of the following: (i) ingestion configuration data for an offline data source of the client, (ii) ingestion configuration data for a nearline data source of the client, or (iii) a combination of (i) and (ii).

[0139] In Example 18, which includes any one of Examples 14-17, the graphical representation of the ingested configuration data includes one or more of the following: (i) location data identifying the physical or logical location of the at least one source data repository for the client; (ii) metadata specifying the priority order for writing data to the graphical database for the client; (iii) transformation logic for converting input data stored in the at least one source data repository into a graphical representation of the input data; or (iv) at least two of (i), (ii), and (iii).

[0140] In Example 19, which includes any one of Examples 14-18, the graphical representation of the ingestion configuration data includes one or more of the following: (i) applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable, (ii) enabled data service data that identifies at least one data service for which the ingestion configuration data is enabled, or (iii) a combination of (i) and (ii).

[0141] In Example 20, which includes the subject of Example 19, the logical graph is contained in an image of at least a portion of the graph database, and the image is stored in a data service of the at least one data service.

[0142] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive. The unique and exclusive indicator of the scope of the invention, and what the applicant wishes to be within the scope of the invention, is the literal and equivalent scope of the set of claims issued in this application, issued in a particular form, including any subsequent amendments. Any definitions of terms incorporated herein may govern the meaning of those terms as used in the claims. Any limitations, elements, characteristics, features, advantages, or attributes not expressly recited in the claims should not in any way limit the scope of the claims. This specification and drawings are to be regarded as illustrative rather than restrictive.

[0143] Terms such as “computer-generated” and “computer-selected” as used herein may refer to the result of one or more processors, such as a server computer, a network of server computers, a client computer, or a combination of client and server computers, executing one or more computer program instructions.

[0144] As used herein, “online” can refer to a specific characteristic of a system based on a connected network. For example, many connected network-based systems are accessible to users via a connection to a public network, such as the Internet. However, an “online” system may perform certain operations even when it is offline. Therefore, referring to a system as “online” does not mean that such a system is always online, or that the system needs to be online for the disclosed technology to be operational.

[0145] As used herein, the terms “include” and “comprise” (and variations thereof, such as “including,” “includes,” “comprising,” “comprises,” “comprised,” etc.) are intended to include, rather than exclude, other features, components, integers, or steps.

[0146] Various features of this disclosure have been described using process steps. The function / process of a given process step may be performed in different ways and by different systems or system modules. Furthermore, a given process step may be divided into multiple steps and / or multiple steps may be combined into a single step. Additionally, the order of the steps may be changed without departing from the scope of this disclosure.

[0147] It will be understood that the embodiments disclosed and defined in this specification are extended to alternative combinations of various features mentioned or apparent from the text or drawings. These different combinations constitute various alternative aspects of the embodiments.

Claims

1. A method for ingesting data into a graph database, the method comprising: Read ingestion configuration data for one of a plurality of clients targeting the graph database, wherein the client corresponds to a logical grouping of data stored in at least one source data repository; The ingestion configuration data for the client is transformed from a declarative representation into a graphical representation of the ingestion configuration data for the client. The graphical representation of the ingested configuration data for the client is encoded into the graphical database to achieve the creation of a self-describing image of the graphical database; The ingestion configuration for at least one data service in a distributed data service is evaluated using the self-describing image of the graph database through the following operations: Compare the ingestion configuration data enabled for the at least one data service with ingestion configuration data applicable to the at least one data service; Detecting a mismatch between the ingestion configuration data enabled for the at least one data service and the ingestion configuration data applicable to the at least one data service; and Based on the detected mismatch, modify the operation of the at least one data service; The at least one data service includes one or more of the following: (i) a physical group of at least one computing device configured to store at least a portion of the graphics database; (ii) a logical group of at least one computing device configured to store at least a portion of the graphics database; or (iii) a combination of (i) and (ii). The method is performed by at least one computing device.

2. The method according to claim 1, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) ingestion configuration data for the client’s offline data source, or (ii) ingestion configuration data for the client’s nearline data source, or (iii) a combination of (i) and (ii).

3. The method according to claim 1, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) location data identifying the physical or logical location of the data source for the client, (ii) metadata specifying the priority order for writing data to the graph database for the client, or (iii) transformation logic for converting input data stored in the data source into a graph representation of the input data, or (iv) a combination of at least two of (i), (ii) and (iii).

4. The method according to claim 1, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable, (ii) enabled data service data that identifies at least one data service for which the ingestion configuration data is enabled, or (iii) a combination of (i) and (ii).

5. The method according to claim 1, wherein, Creating a graphical representation of the ingestion configuration data for the client includes: converting the ingestion configuration data into graphical primitives, wherein the graphical primitives include at least two nodes, at least one edge, and at least one predicate; wherein the edge of the at least one edge indicates the logical relationship between two of the at least two nodes.

6. The method according to claim 5, wherein, The predicate in the at least one predicate includes a label for the logical relationship between the two nodes, and the label relates to an ingestion configuration, which includes a specification for ingesting input data from a non-graphical data source into the graphical database.

7. The method according to claim 1, further comprising: Use an application programming interface (API) configured at least in part according to a declarative query language to read ingestion configuration data for the client.

8. The method according to claim 1, further comprising: (i) Modify the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is enabled for the at least one data service, or (ii) Modify the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is not enabled for the at least one data service.

9. The method according to claim 1, further comprising: The self-describing image of the graph database is queried to determine, for a client among a plurality of clients of the graph database, ingestion configuration data applicable to a plurality of data services serving the graph database, wherein the client corresponds to at least one dataset arranged according to a non-graph pattern; and Read the status data for the data service to determine the ingestion configuration data enabled for the data service for the client.

10. The method according to claim 9, wherein, Comparing the ingestion configuration data enabled for the data service with ingestion configuration data applicable to the data service includes one or more of the following: (i) evaluating ingestion configuration data for an offline data source for the client, (ii) evaluating ingestion configuration data for a nearline data source for the client, or (iii) a combination of (i) and (ii).

11. The method according to claim 9, wherein, Comparing the ingestion configuration data enabled for the data service with the ingestion configuration data applicable to the data service includes one or more of the following: (i) evaluating location data that identifies the physical or logical location of a non-graphical data source for the client, (ii) evaluating metadata that specifies the priority order for writing data to the graph database for the client, or (iii) a combination of (i) and (ii).

12. The method according to claim 9, wherein, Detecting the mismatch between the ingestion configuration data enabled for the data service and the ingestion configuration data applicable to the data service includes determining one or more of the following: (i) offline ingestion is applicable to the data service but not enabled for the data service, (ii) nearline ingestion is applicable to the data service but not enabled for the data service, (iii) metadata included in the ingestion configuration data enabled for the data service does not match metadata included in the ingestion configuration data applicable to the data service, or (iv) at least two of (i), (ii) and (iii).

13. The method according to claim 9, wherein, The operation of modifying the data service based on the detected non-match includes one or more of the following: (i) building an image on the data service, (ii) writing the image to a network-accessible storage location, (iii) deleting the image from the data service, (iv) copying the image to the data service from another storage location, or (v) at least two of (i), (ii), (iii) and (iv).

14. A system for ingesting data into a graph database, the system comprising: At least one processor; At least one computer memory storing executable instructions that, when executed by the at least one processor, cause the at least one processor to perform the following steps; Read ingestion configuration data for one of a plurality of clients targeting the graph database, wherein the client corresponds to a logical grouping of data stored in at least one source data repository; The ingestion configuration data for the client is transformed from a declarative representation into a graphical representation of the ingestion configuration data for the client. The graphical representation of the ingested configuration data for the client is encoded into the graphical database to achieve the creation of a self-describing image of the graphical database; The ingestion configuration for at least one data service in a distributed data service is evaluated using the self-describing image of the graph database through the following operations: Compare the ingestion configuration data enabled for the at least one data service with ingestion configuration data applicable to the at least one data service; Detecting a mismatch between the ingestion configuration data enabled for the at least one data service and the ingestion configuration data applicable to the at least one data service; and Based on the detected mismatch, modify the operation of the at least one data service; The at least one data service includes one or more of the following: (i) a physical group of at least one computing device configured to store at least a portion of the graphics database, (ii) a logical group of at least one computing device configured to store at least a portion of the graphics database, or (iii) a combination of (i) and (ii).

15. The system of claim 14 further includes reading ingestion configuration data for the client using an application programming interface (API) configured at least in part according to a declarative query language.

16. The system according to claim 14, wherein, Creating the graphical representation of the ingestion configuration data for the client includes converting the ingestion configuration data into graphical primitives, each graphical primitive comprising at least two nodes, at least one edge, and at least one predicate; wherein the edge of the at least one edge indicates a logical relationship between two of the at least two nodes, the predicate of the at least one predicate includes a label for the logical relationship between the two nodes, and the label relates to a specification for ingesting data from the at least one source data repository into the graphical database.

17. The system according to claim 14, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) ingestion configuration data for the client’s offline data source, or (ii) ingestion configuration data for the client’s nearline data source, or (iii) a combination of (i) and (ii).

18. The system according to claim 14, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) location data identifying the physical or logical location of the source data repository for the client, (ii) metadata specifying the priority order for writing data to the graph database for the client, (iii) transformation logic for converting input data stored in the source data repository into a graph representation of the input data, or (iv) a combination of at least two of (i), (ii), and (iii).

19. The system according to claim 14, wherein, Reading ingestion configuration data includes reading one or more of the following declarative representations: (i) applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable, (ii) enabled data service data that identifies at least one data service for which the ingestion configuration data is enabled, or (iii) a combination of (i) and (ii).

20. The system of claim 19, further comprising: (i) Modify the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is enabled for the at least one data service, or (ii) Modify the graphical representation of the ingestion configuration data to indicate that the ingestion configuration data is not enabled for the at least one data service.

Images