SYSTEM AND METHOD FOR ACCESSING DATA ENTITIES MANAGED BY A DATA PROCESSING SYSTEM

Abstract

A technique for obtaining information about data entity instances managed by a data processing system using at least one data store, the technique including: obtaining a query including a first portion including information for identifying an instance of a first data entity stored in the at least one data store and a second portion indicating at least one attribute of the first data entity, generating a plurality of executable queries from the query including a first set of one or more executable queries and a second set of one or more executable queries, including using the first portion to generate the first set of executable queries for identifying the instance of the first data entity and using the second portion to generate the second set of executable queries for obtaining attribute values ​​of the instance of the first data entity, and executing the plurality of executable queries to obtain results of the query.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 143,895, filed January 31, 2021, and entitled "SYSTEMS AND METHODS FOR ACCESSING DATA ENTITIES MANAGED BY A DATA PROCESSING SYSTEM."

[0002] Field Aspects of the present disclosure relate to techniques for managing access to (eg, query, create, update, or delete) data managed by a data processing system using data entities and data entity instances. [Background technology]

[0003] background Modern data processing systems manage vast amounts of data (e.g., millions, billions, or even trillions of data records) and how this data can be accessed. A data processing system may provide multiple interfaces to access the data it manages. For example, a data processing system may provide a graphical user interface (GUI) that allows a user to perform operations (e.g., query, update, delete, create) on data managed by the data processing system. As another example, a data processing system may provide an application programming interface (API) that allows software programs to perform operations on data managed by the data processing system. Summary of the Invention [Means for solving the problem]

[0004] overview Some embodiments provide a method, executed by a data processing system, for computationally efficiently obtaining a first attribute value of a first instance of a data entity in response to a query, the method comprising: obtaining, through an application programming interface (API) of the data processing system, a query for accessing data, the query including a first portion including information for identifying at least a first instance of the data entity stored in at least one data store and a second portion indicating at least one attribute of the data entity for which a value is to be obtained; and generating a plurality of executable queries, the plurality of executable queries including a first set of one or more executable queries and a second set of one or more executable queries, wherein the first portion of the query is used to obtain at least one attribute value of the first instance of the data entity. generating a first set of executable queries to identify at least first instances of data entities stored in at least one data store; generating a second set of executable queries using a second portion of the queries to obtain attribute values of the at least first instances of data entities identified using the first set of executable queries, the attribute values including the first attribute value; obtaining results of the queries provided via the API by executing the plurality of executable queries, the results including at least the first attribute value of the first instances of data entities; and outputting at least some of the results including the at least the first attribute value of the first instances of data entities.

[0005] Some embodiments provide a data processing system including at least one computer hardware processor and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method of obtaining a first attribute value of a first instance of a data entity in response to a query, the method comprising: obtaining, through an application programming interface (API) of the data processing system, a query for accessing data, the query including a first portion including information for identifying the at least first instance of the data entity stored in the at least one data store and a second portion indicating at least one attribute of the data entity for which a value is to be obtained; and obtaining one or more of the first set of first attribute values. generating a plurality of executable queries including a plurality of executable queries and a second set of one or more executable queries, wherein the generating includes using a first portion of the queries to generate a first set of executable queries for identifying at least a first instance of a data entity stored in at least one data store; generating using a second portion of the queries to generate a second set of executable queries for retrieving attribute values of the at least a first instance of the data entity identified using the first set of executable queries, the attribute values including a first attribute value; obtaining results of the queries provided via the API by executing the plurality of executable queries, the results including at least a first attribute value of the first instance of the data entity; and outputting at least some of the results including the at least a first attribute value of the first instance of the data entity.

[0006] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for obtaining first attribute values of first instances of data entities in response to a query, the method comprising: obtaining, through an application programming interface (API) of a data processing system, a query for accessing data, the query including a first portion including information for identifying at least first instances of data entities stored in at least one data store and a second portion indicating at least one attribute of the data entities for which values are to be obtained; and executing a first set of one or more executable queries and a second set of one or more execution queries. generating a plurality of executable queries, the plurality of executable queries including: generating a first set of executable queries using a first portion of the queries to identify at least a first instance of a data entity stored in at least one data store; generating a second set of executable queries using a second portion of the queries to obtain attribute values of at least a first instance of the data entity identified using the first set of executable queries, the attribute values including a first attribute value; obtaining results of the queries provided via the API by executing the plurality of executable queries, the results including at least a first attribute value of the first instance of the data entity; and outputting at least some of the results including the at least a first attribute value of the first instance of the data entity.

[0007] In some embodiments, the first and second sets of executable queries include executable Structured Query Language (SQL) queries.

[0008] In some embodiments, the first portion includes information for identifying one or more instances of each of a plurality of data entities, the plurality of data entities including a data entity, and generating the first set of executable queries includes, for each particular data entity of the plurality of data entities, generating a respective executable query for identifying an instance of the data entity stored in the at least one data store.

[0009] In some embodiments, the at least one attribute includes a plurality of attributes of the data entity, and generating the second set of executable queries includes, for each particular attribute of at least some of the plurality of attributes, generating a respective executable query to obtain a value of the particular attribute of the instance of the data entity identified using the first set of executable queries.

[0010] In some embodiments, the plurality of attributes includes a plurality of dynamic attributes, and generating the second set of executable queries includes grouping the plurality of dynamic attributes into groups of dynamic attributes, each group having at least one of the plurality of dynamic attributes, and generating, for each group of the plurality of dynamic attributes, a respective executable query to obtain values of attributes in the group for instances of the data entity.

[0011] In some embodiments, grouping the plurality of dynamic attributes is performed based on the type of the dynamic attribute. In some embodiments, grouping the plurality of dynamic attributes is performed based on whether the dynamic attributes are stored in the same table in at least one data store. In some embodiments, grouping the plurality of dynamic attributes is performed based on whether the dynamic attributes inherit their values along a common inheritance path.

[0012] In some embodiments, executing the plurality of executable queries includes executing a first set of executable queries to obtain information identifying at least a first instance of a data entity, and after executing the first set of executable queries, executing a second set of executable queries using the information identifying the at least a first instance of the data entity.

[0013] In some embodiments, generating the second set of executable queries is performed after executing the first set of executable queries, hi some embodiments, generating the second set of executable queries is performed before executing the first set of executable queries.

[0014] In some embodiments, the at least one attribute includes a group of multiple dynamic attributes of the data entity, the second set of executable queries includes a plurality of executable queries respectively corresponding to the groups of the multiple dynamic attributes, and executing the second set of executable queries includes executing each of the plurality of executable queries to obtain a respective plurality of dynamic attribute value results.

[0015] In some embodiments, the method further includes, after executing the second set of executable queries, combining the dynamic value attribute results to obtain a result of the query.

[0016] In some embodiments, the second portion of the query includes an inline view, and generating the second set of executable queries includes generating the executable queries using information specified in the inline view. In some embodiments, the inline view includes a table query. In some embodiments, the table query is for performing an aggregation function. In some embodiments, the inline view specifies inner and outer join keys.

[0017] In some embodiments, the at least one attribute of the data entity includes at least one static attribute and at least one dynamic attribute.

[0018] In some embodiments, at least one attribute of the data entity includes a first dynamic attribute, the first dynamic attribute configured to inherit its value from an attribute of a second data entity.

[0019] In some embodiments, the second portion of the query indicates a hierarchy of attributes including at least one attribute. In some embodiments, the second portion of the query indicates at least a portion of the hierarchy of attributes using nesting.

[0020] In some embodiments, the second part of the query uses recursion to indicate at least a portion of the hierarchy of attributes.

[0021] In some embodiments, outputting at least some of the results includes formatting at least some of the results in a hierarchical format corresponding to a hierarchy of the attributes, and outputting at least some of the results in the hierarchical format.

[0022] In some embodiments, outputting at least some of the results comprises outputting at least some of the results in a hierarchical format. In some embodiments, outputting at least some of the results comprises outputting the results in a non-tabular format.

[0023] In some embodiments, a first portion of the query is transformed through a series of transformations that augment the information in the first portion with information managed by the data processing system to generate a first set of executable queries configured to be executed against at least one data store, and a second portion of the query is transformed through a series of transformations that augment the information in the second portion with information managed by the data processing system to generate a second set of executable queries configured to be executed against at least one data store.

[0024] In some embodiments, generating the first set of executable queries includes generating the first executable query at least in part by generating an intermediate query representation using a first portion of the query, augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information, and / or permission information to obtain an augmented intermediate query representation, and generating the first executable query using the augmented intermediate query representation. In some embodiments, generating the first executable query includes generating an executable SQL query. In some embodiments, generating the intermediate query representation includes generating an abstract syntax tree.

[0025] In some embodiments, generating the second set of executable queries includes generating the second executable query, at least in part, by generating an intermediate query representation using the second portion of the query, augmenting the intermediate query representation with the unique identifier information, the dynamic attribute reference information, the view type information, and / or the permission information to obtain an augmented intermediate query representation, and generating the second executable query using the augmented intermediate query representation.

[0026] In some embodiments, executing the plurality of executable queries includes executing a first set of executable queries to obtain information identifying the plurality of data entity instances, and generating a second set of one or more executable queries is performed using a second portion of the queries and the information identifying the plurality of data entity instances.

[0027] In some embodiments, the at least one attribute includes multiple attributes, each attribute of the multiple attributes configured to inherit its value from one or more attributes of the same other data entity instance, and the method further includes grouping, by the data processing system, the multiple attributes into a single group; generating, by the data processing system, a single executable query for the single group; and executing, by the data processing system, the single executable query to cause the data processing system to obtain attribute values for the multiple attributes.

[0028] In some embodiments, the at least one attribute includes a plurality of attributes, each attribute of the plurality of attributes configured to inherit its value from one or more attributes of another data entity instance, and the method further includes grouping, by the data processing system, the plurality of attributes into one or more groups using the grouping criteria and generating a separate executable query for each of the groups, the separate executable query, when executed by the data processing system, causing the data processing system to obtain attribute values for the attributes of the respective group.

[0029] In some embodiments, the grouping criterion is to group attributes that share a common inheritance path into a single group, such that a single one of the separate executable queries is used to access the inherited values of these grouped attributes.

[0030] In some embodiments, at least one data store is configured to store information defining a plurality of data entity instances and associated attribute values and relationships between the different data entity instances, and the at least one data store is configured to store information describing data stored in a distributed database of a distributed network of computing systems communicatively connected to the data processing system using the plurality of data entity instances.

[0031] In some embodiments, at least one data store is configured to store a data entity instance of a plurality of data entity instances for each of a plurality of data sets stored by a distributed database of the distributed computing system, for each of a plurality of software applications configured to be executed by any system portion of the distributed computing system, or for each or multiple system portions of the distributed computing system.

[0032] In some embodiments, the query is a declarative query.

[0033] Some embodiments provide a method for obtaining information about data entity instances managed by a data processing system using at least one data store using a query, the method including: obtaining, using at least one computer hardware processor, a query, the query including a first portion including information for identifying at least one data entity instance stored in the at least one data store and a second portion indicating at least one attribute of the at least one data entity; generating from the query a plurality of executable queries, the plurality of executable queries including a first set of one or more executable queries and a second set of one or more executable queries, the plurality of executable queries including using the first portion of the query to generate the first set of one or more executable queries for identifying the at least one data entity instance stored in the at least one data store and using the second portion of the query to generate the second set of one or more executable queries for obtaining attribute values of the at least one data entity instance identified using the first set of executable queries; executing the plurality of executable queries to obtain results of the query; and outputting the results.

[0034] Some embodiments provide a data processing system configured to manage a plurality of data entity instances using at least one data store, the data processing system including at least one computer hardware processor and at least one non-transitory computer readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method comprising obtaining a query, the query including a first portion including information for identifying an instance of at least one data entity stored in the at least one data store, and a second portion including information for identifying at least one instance of the at least one data entity. and a second portion indicating one or more attributes of the at least one data entity instance identified using the first set of executable queries; generating from the query a plurality of executable queries including a first set of one or more executable queries and a second set of one or more executable queries, the plurality of executable queries including using the first portion of the query to generate the first set of one or more executable queries for identifying at least one data entity instance stored in the at least one data store; and using the second portion of the query to generate the second set of one or more executable queries for obtaining attribute values of the at least one data entity instance identified using the first set of executable queries; executing the plurality of executable queries to obtain results of the queries; and outputting the results.

[0035] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method comprising: obtaining a query, the query including a first portion including information for identifying an instance of at least one data entity stored in at least one data store, and a second portion indicating at least one attribute of the at least one data entity; and generating a first set of one or more executable queries from the query. generating a plurality of executable queries comprising the first set of one or more executable queries and a second set of one or more executable queries, the plurality of executable queries including using a first portion of the queries to generate a first set of one or more executable queries for identifying instances of at least one data entity stored in the at least one data store, and using a second portion of the queries to generate a second set of one or more executable queries for retrieving attribute values of the instances of the at least one data entity identified using the first set of executable queries; executing the plurality of executable queries to obtain results of the queries; and outputting the results.

[0036] Some embodiments provide a method for obtaining information about data entity instances managed by a data processing system using a query using at least one data store, the method including: obtaining, using at least one computer hardware processor, a query, the query including a first portion including information for identifying at least one data entity instance stored in the at least one data store and a second portion indicating at least one attribute of the at least one data entity; using the first portion of the query to generate a first set of one or more executable queries for identifying the at least one data entity instance stored in the at least one data store; executing the first set of one or more executable queries to obtain information identifying the first plurality of data entity instances; using the second portion of the query and the information identifying the first plurality of data entity instances to generate a second set of one or more executable queries for obtaining attribute values of the first plurality of data entity instances; and executing the second set of one or more executable queries to obtain the attribute values of the first plurality of data entity instances.

[0037] Some embodiments provide a method that includes using at least one computer hardware processor to obtain a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for retrieving the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the query; and outputting the results in a hierarchical format corresponding to the attribute hierarchy.

[0038] Some embodiments provide a data processing system including at least one computer hardware processor and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method including: obtaining a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store, and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for retrieving the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the queries; and outputting the results in a hierarchical format corresponding to the attribute hierarchy.

[0039] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method including: obtaining a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store, and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for retrieving the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the queries; and outputting the results in a hierarchical format corresponding to the attribute hierarchy.

[0040] Some embodiments provide a method that includes using at least one computer hardware processor to obtain a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for retrieving the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain query results; and outputting the results.

[0041] Some embodiments provide a data processing system including at least one computer hardware processor and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method including: obtaining a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store, and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for obtaining the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the queries; and outputting the results.

[0042] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method including: obtaining a query, the query including a first portion including information for identifying at least one data entity instance stored in at least one data store, and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for obtaining the at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the queries; and outputting the results.

[0043] Some embodiments provide a method for obtaining information about data entity instances managed by a data processing system using queries using at least one data store, the method including: obtaining, using at least one computer hardware processor, a table query; generating from the table query a set of one or more executable queries for accessing data stored in the at least one data store using the one or more data entity instances, the set including generating an intermediate query representation of the table query; augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information and / or permission information to obtain an augmented intermediate query representation; generating a set of one or more executable queries using the augmented intermediate query representation; executing the set of executable queries to obtain results of the table query; and outputting at least some of the results.

[0044] The above is a non-limiting summary of the invention, which is defined by the appended claims.

[0045] BRIEF DESCRIPTION OF THE DRAWINGS Various aspects and embodiments are described with reference to the following figures. It should be understood that the figures are not necessarily drawn to scale. Features that appear in more than one figure are designated by the same or similar reference numerals in all figures in which they appear. [Brief explanation of the drawings]

[0046] [Figure 1A] 1 illustrates an example environment in which a data processing system may operate in accordance with some embodiments of the technology described herein. [Figure 1B] 1B further illustrates the example environment of FIG. 1A in which a data processing system may operate, according to some embodiments of the technology described herein. [Figure 1C] FIG. 1 illustrates that in some applications, data processing systems may be configured to manage millions or billions of data entity instances according to some embodiments of the techniques described herein. [Figure 1D] 1 is a block diagram illustrating exemplary aspects of a data processing system 105 and an entity application programming interface (API) 120 integrated into the data processing system 105, according to some embodiments of the technology described herein. [Figure 1E] FIG. 1 illustrates an example process 101 for generating query results in response to a declarative query for obtaining information about instances of one or more data entities, according to some embodiments of the technology described herein. [Figure 1F] 1A shows an illustrative example of a declarative query being processed by the example process 101 shown in FIG. 1E, in accordance with some embodiments of the technology described herein. [Figure 2A] 1 shows an illustrative diagram of a data entity having one or more static attributes and / or one or more dynamic attributes, in accordance with some embodiments of the techniques described herein. [Figure 2B] 1 shows an illustrative diagram of "BizTerm" data entities in accordance with some embodiments of the technology described herein. [Figure 2C] 1 shows an illustrative diagram of "BizTerm" data entities in accordance with some embodiments of the technology described herein. [Figure 2D] 2D is a table illustrating properties of at least some of the attributes of the data entity shown in FIG. 2C, including attributes, whether static or dynamic, according to some embodiments of the technology described herein. [Figure 2E] FIG. 2 illustrates aspects of how static and dynamic attribute values are stored by a data processing system in accordance with some embodiments of the techniques described herein. [Figure 2F] 1 is a screenshot of an exemplary graphical user interface illustrating visualization of data entity instances and their attribute values, according to some embodiments of the technology described herein. [Figure 2G] FIG. 2 illustrates relationships between data entities according to an example entity model, in accordance with some embodiments of the techniques described herein. [Figure 3] 3 is a flowchart of an example process 300 for generating query results in response to a declarative query for obtaining information about instances of one or more data entities, according to some embodiments of the technology described herein. [Figure 4A] FIG. 4 illustrates an example declarative query 402 including an identification portion and an enrichment portion, according to some embodiments of the technology described herein. [Figure 4B] 4B illustrates example results generated by processing the declarative query shown in FIG. 4A in accordance with some embodiments of the techniques described herein. [Figure 4C] FIG. 4 is a diagram of another example declarative query 410 in accordance with some embodiments of the techniques described herein. [Figure 4D]4 is a diagram of a declarative query 420 that specifies a hierarchy of attributes whose values should be returned in response to processing of the declarative query, according to some embodiments of the technology described herein. [Figure 4E] 4D illustrates example results returned in response to processing the declarative query 420 shown in FIG. 4D, according to some embodiments of the technology described herein. [Figure 4F] FIG. 4D is a diagram of a variation of the declarative query 420 shown in FIG. 4D designed to reduce the number of related data entity instance results returned when the query is processed, according to some embodiments of the technology described herein. [Figure 5A] FIG. 5 illustrates an example declarative query 502 including an identification portion and an enrichment portion, according to some embodiments of the techniques described herein. [Figure 5B] 5B illustrates an executable SQL query that may be generated by a data processing system by processing the declarative query 502 shown in FIG. 5A in accordance with some embodiments of the techniques described herein. [Figure 5C] 5B illustrates an identifying portion of the declarative query 502 shown in FIG. 5A, an example of an executable SQL query that may be generated using the identifying portion of the declarative query 502, and an illustration of at least a portion of results that may be obtained by executing the executable SQL query, in accordance with some embodiments of the techniques described herein. [Figure 5D] 5B illustrates an enrichment portion of the declarative query 502 shown in FIG. 5A, an example of an executable SQL query that may be generated using the enrichment portion of the declarative query 502, and an illustration of at least a portion of results that may be obtained by executing the executable SQL query, according to some embodiments of the techniques described herein. [Figure 5E] 5B illustrates an enrichment portion of the declarative query 502 shown in FIG. 5A, an example of another executable SQL query that may be generated using the enrichment portion of the declarative query 502, and an illustration of at least a portion of results that may be obtained by executing the executable SQL query, according to some embodiments of the techniques described herein. [Figure 5F] 5B illustrates an enrichment portion of the declarative query 502 shown in FIG. 5A, an example of another executable SQL query that may be generated using the enrichment portion of the declarative query 502, and an illustration of at least a portion of results that may be obtained by executing the executable SQL query, according to some embodiments of the techniques described herein. [Figure 5G] FIG. 1 illustrates pivoting of SQL query results outside of SQL, in accordance with some embodiments of the techniques described herein. [Figure 6A] FIG. 1 illustrates an enrichment portion of a declarative query and each executable SQL generated for each attribute identified in the enrichment portion, according to some embodiments of the techniques described herein. [Figure 6B] FIG. 10 illustrates a set of executable SQL queries that may be generated for dynamic attributes having the same type, according to some embodiments of the techniques described herein. [Figure 6C-1] FIG. 6 illustrates a declarative query 610 having an enrichment portion that specifies multiple dynamic attributes, some of which are of a common type, in accordance with some embodiments of the techniques described herein. [Figure 6C-2] FIG. 6 illustrates a declarative query 610 having an enrichment portion that specifies multiple dynamic attributes, some of which are of a common type, in accordance with some embodiments of the techniques described herein. [Figure 6C-3] FIG. 6 illustrates a declarative query 610 having an enrichment portion that specifies multiple dynamic attributes, some of which are of a common type, in accordance with some embodiments of the techniques described herein. [Figure 6D] FIG. 10 illustrates an example of an SQL query for retrieving values of dynamic attributes xRegion and xLineOfBusiness of the same type, according to some embodiments of the technology described herein. [Figure 6E] FIG. 10 illustrates how different dynamic attributes of a data entity may inherit values from respective attributes of different data entities, according to some embodiments of the techniques described herein. [Figure 6F]FIG. 10 illustrates an example SQL query for accessing values of inherited dynamic attributes that share a common inheritance path, in accordance with some embodiments of the technology described herein. [Figure 7A] 7 is a flowchart of an example process 700 for generating one or more database-specific executable SQL queries from at least a portion of a declarative query, according to some embodiments of the technology described herein. [Figure 7B] 1 illustrates a series of transformations performed on at least a portion of a declarative query to generate one or more database-specific executable SQL queries, according to some embodiments of the technology described herein. [Figure 7C] 1 illustrates a series of transformations performed on at least a portion of a declarative query to generate one or more database-specific executable SQL queries, according to some embodiments of the technology described herein. [Figure 7D] 1 illustrates a series of transformations performed on at least a portion of a declarative query to generate one or more database-specific executable SQL queries, according to some embodiments of the technology described herein. [Figure 7E] 1 illustrates a series of transformations performed on at least a portion of a declarative query to generate one or more database-specific executable SQL queries, according to some embodiments of the technology described herein. [Figure 7F] 1 illustrates a series of transformations performed on at least a portion of a declarative query to generate one or more database-specific executable SQL queries, according to some embodiments of the technology described herein. [Figure 8A] 1 illustrates an example instance of a data entity having a hierarchical structure, in accordance with some embodiments of the techniques described herein. [Figure 8B] 8 illustrates an example declarative query 805 for searching hierarchical data using recursion, according to some embodiments of the technology described herein. [Figure 8C-1]8C illustrates exemplary results obtained by a data processing system by processing the example declarative query 805 shown in FIG. 8B. [Figure 8C-2] 8C illustrates exemplary results obtained by a data processing system by processing the example declarative query 805 shown in FIG. 8B. [Figure 8C-3] 8C illustrates exemplary results obtained by a data processing system by processing the example declarative query 805 shown in FIG. 8B. [Figure 8D] 8 illustrates another example declarative query 810 for searching hierarchical data using recursion, according to some embodiments of the technology described herein. [Figure 8E] 8B illustrates exemplary results obtained by a data processing system by processing the example declarative query 810 shown in FIG. 8D. [Figure 9A] 1 illustrates a portion of an entity relationship diagram in accordance with some embodiments of the techniques described herein. [Figure 9B] 1 illustrates a pseudo-code specification of a declarative query for searching hierarchical data, according to some embodiments of the technology described herein. [Figure 9C] 9C illustrates a JavaScript Object Notation (JSON) implementation of the pseudo-code declarative query specification shown in FIG. 9B, in accordance with some embodiments of the techniques described herein. [Figure 9D] 9D shows an example diagram of results that may be obtained by processing the JSON declarative query of FIG. 9C according to some embodiments of the techniques described herein. [Figure 9E] 10 illustrates a pseudo-code specification of another declarative query for searching hierarchical data, in accordance with some embodiments of the technology described herein. [Figure 9F-1] 9E illustrates a JSON implementation of the pseudo-code declarative query specification shown in FIG. 9E, in accordance with some embodiments of the techniques described herein. [Figure 9F-2] 9E illustrates a JSON implementation of the pseudo-code declarative query specification shown in FIG. 9E, in accordance with some embodiments of the techniques described herein. [Figure 9G]9F-1 and 9F-2 show example diagrams of results that may be obtained by processing the JSON declarative queries of FIGS. 9F-1 and 9F-2, according to some embodiments of the techniques described herein. [Figure 10A] 1 illustrates an example declarative table query in accordance with some embodiments of the techniques described herein. [Figure 10B] 10B illustrates example results of the declarative table query shown in FIG. 10A in accordance with some embodiments of the techniques described herein. [Figure 10C] 10 illustrates another example declarative table query in accordance with some embodiments of the techniques described herein. [Figure 10D] 10D illustrates example results of the declarative table query shown in FIG. 10C, according to some embodiments of the techniques described herein. [Figure 10E] 1 illustrates an in-line view of an exemplary declarative query in accordance with some embodiments of the technology described herein. [Figure 10F] 1 illustrates an in-line view of an exemplary declarative query in accordance with some embodiments of the technology described herein. [Figure 10G-1] 10F illustrates an example result of executing the declarative query of FIG. 10E in accordance with some embodiments of the techniques described herein. [Figure 10G-2] 10F illustrates an example result of executing the declarative query of FIG. 10E in accordance with some embodiments of the techniques described herein. [Figure 10H-1] 10F illustrates an example result of executing the declarative query of FIG. 10F according to some embodiments of the techniques described herein. [Figure 10H-2] 10F illustrates an example result of executing the declarative query of FIG. 10F according to some embodiments of the techniques described herein. [Figure 10H-3] 10F illustrates an example result of executing the declarative query of FIG. 10F according to some embodiments of the techniques described herein. [Figure 11] FIG. 1 is a block diagram of an exemplary computing system environment that may be used to implement some embodiments of the technology described herein. DETAILED DESCRIPTION OF THE INVENTION

[0047] Detailed Description The inventors have developed a new technique that allows efficient access (e.g., query, create, update, and delete) of data managed by a data processing system using data entities and their instances. As described herein, in some embodiments, a data processing system may manage data using data entities, which may be used to organize data using an object-oriented paradigm. Similar to how object-oriented programming involves classes and their instances, a data processing system may be configured with definitions of data entities, and the data processing system may manage data using instances of data entities and data entity definitions. An instance of a data entity may be referred to herein as a "data entity instance." A data entity may include multiple attributes, and an attribute may take on different values, such as numbers, strings, or references to other data entities, when instantiated. The technique developed by the inventors allows efficient access to data entity instances managed by a data processing system.

[0048] In particular, the inventors have developed an interface that provides efficient access to data entity instances managed by a data processing system, which interface can be incorporated into an application programming interface (API). Because the interface / API is for providing access to data entity instances, it is sometimes referred to herein as an "entity" API. The entity API provides users and computer programs with a declarative language to specify desired operations to be performed, such as querying a data processing system for data entity instances that meet one or more criteria, creating new data entity instances, updating values of data entity instances, and deleting existing data entity instances. The entity API accepts input that indicates the desired operations and is specified in a declarative language (e.g., JavaScript Object Notation (JSON) or any other suitable format), generates executable statements from the input (e.g., executable SQL queries if the desired operation is to query a data processing system about data entity instances and the data entity instances are stored in a relational database that supports SQL), and executes the executable statements to perform the desired operations.

[0049] The inventors have recognized that data processing systems may be configured to manage millions or billions of entity instances. For example, as described in more detail below with reference to FIG. 1A , the techniques described herein may, in some embodiments, be used for metadata management in an enterprise setting, whereby data entity instances store information about individual datasets (e.g., tables, transactions, documents, data records, etc.) stored across a globally distributed information technology (IT) infrastructure including many databases, data warehouses, data lakes, etc. In this regard, a data entity instance may store information about the corresponding dataset, such as, for example, when the dataset was created, where it is stored, its size, the identities of users who are authorized to edit the dataset, information identifying which application programs use the dataset, the sensitivity level of the data, etc. Large organizations (e.g., financial institutions such as banks or credit card companies, public utilities such as telephone companies or electric companies, etc.) typically manage millions or billions of such datasets, and therefore there may be millions or billions of data entity instances storing information about such datasets managed by a data processing system. In this application, the data processing system stores information about other data (sometimes called "metadata"), and so this application can be referred to as "metadata management." However, it should be understood that the techniques described herein are not limited to data processing systems used for metadata management, but can be applied to any data processing system that uses data entities and data entity instances to manage data, whether the managed data is metadata or any other type of data.

[0050] Regardless of the type of data managed by a data processing system using data entity instances, the above examples make clear that in many practical applications, the data processing system can be expected to manage millions or even billions of data entity instances. Accordingly, the inventors have designed and developed an entity API to provide efficient access to data entity instances managed by a data processing system, such that access to the data entity instances is provided quickly and without delay, preferably while minimizing the computational load on the data processing system.

[0051] To this end, a beneficial feature of the Entity API is that input (e.g., a declarative statement, e.g., a declarative entity) provided to the Entity API is processed in two parts: identifying data entities (or instances thereof) of interest and accessing attribute values of the identified data entities. These parts are sometimes referred to herein as an “identification” part and an “enrichment” part. For example, a declarative statement for retrieving information about data entity instances, which may be referred to as a “declarative query,” may include two parts: an identification part (e.g., identification part 404 of declarative query 402 in FIG. 4A ) that identifies the data entity instances of interest (e.g., all instances of data entity “BizTerm” in the example of FIG. 4A ) and an enrichment part (e.g., enrichment part 406 of declarative query 402 in the example of FIG. 4A ) that indicates attributes whose values should be retrieved in the identified data entity instances (e.g., attributes “Name” and “Description” in the example of FIG. 4A ). Also, during a first stage of processing the declarative query, its identification portion is processed to generate one or more executable (e.g., SQL) queries that, when executed against a data store in the data processing system that stores the data entity instances, return information that identifies the data entity instances of interest (e.g., unique identifiers). Concurrently with or subsequent to the first stage, during a second stage of processing the declarative query, its enrichment portion is processed to generate one or more other executable (e.g., SQL) queries that obtain values of attributes indicated as being of interest in the enrichment portion. At this stage, different executable queries may be generated for different attributes or groups of attributes. The results of all executable queries are then combined and output (e.g., to a user or to the computer program that entered the declarative query).

[0052] It should be understood that in some embodiments, input to the Entity API may be a declarative statement, which may be referred to as a "declarative query" if the statement is to retrieve information from the Entity API. However, the Entity API is not limited to receiving only declarative queries as input, but may be configured to receive any suitable type of query (whether declarative in the sense described herein or not) or statement (whether declarative in the sense described herein or not) as input.

[0053] There are many technical advantages provided by the above-described staged architecture for specifying and executing input queries (e.g., declarative queries) and / or statements (e.g., declarative statements). First, by separating "identification" from "enrichment," processing can be performed using a "streaming" architecture, whereby only some of the data entities identified during the "identification" stage are enriched with their attribute values, and additional data entity instances are enriched only after they are requested. This is described in more detail below in the section titled "Batch and Streaming Query Execution, Scoping Returned Results." This streaming architecture provides fast access to data entity instances without the delay of waiting for enrichment of all data entity instances, and reduces the computational burden on the data processing system because enrichment of all data entity instances is not necessarily required in some cases.

[0054] Second, it should be understood that the attribute values of a single data entity instance are not necessarily stored in a single row in one table, but instead may be stored across multiple different tables within a data processing system. For example, as described herein, a data entity may have static attributes and dynamic attributes, and the dynamic attributes may be stored in different tables using name-value pairs. Generating multiple executable queries is made possible by the tiered architecture, allowing more efficient access to these attribute values. For example, as described in more detail below in the section titled "Accessing Attribute Values for a Group of Attributes," the enrichment portion of a declarative query may be processed by first grouping attributes that have the same type and / or are stored in the same table and generating one executable query for each group of attributes, thereby substantially reducing the amount of computation performed when retrieving attribute values because (1) each table is queried once instead of multiple times, and (2) unlike traditional SQL queries that join all tables that record attribute values in the underlying database (which in practical applications may require joining dozens of tables in the underlying database), attribute values are retrieved using separate executable queries and can be joined outside the underlying database (e.g., through pivoting, as described in more detail below in the section titled "Pivoting"). The tiered architecture provides many other efficiencies and advantages described herein, including those described below in the section titled "Optimizing the Execution of Enrichment Queries."

[0055] Another advantageous feature of an entity API is that it allows a user or computer program to access (e.g., query, create, update, delete) data entities independently of how the data entity instances are stored by the data processing system. Traditional data processing systems require a user to query the system to learn how the data being queried is stored. For example, to write a query, a user querying a traditional data processing system needs to know the type of database system (e.g., relational database, object-oriented database, NoSQL database, and the database's manufacturer or vendor) that the data processing system uses to store the data and how that database system stores the data being queried (e.g., the names of the table or tables that store the values of interest). Without this information, the user would not be able to write the query at all. Furthermore, the user may want to know the exact version and vendor of the database system so that they can optimize access to the data of interest.

[0056] Conversely, queries enabled by the Entity API may be specified independently of how the data entity instances being queried are stored by the data processing system. In this way, the exact same queries can be used to query data entity instances regardless of how they are stored within the data processing system, using a relational database, an object-oriented database, a NoSQL database, a cache, a search index (e.g., as described in more detail below in the section entitled "Searching Data Using One or More Search Indexes"), or in any other way.

[0057] As described herein, in some embodiments, data entity attributes may be static or dynamic, meaning that their values are stored using different mechanisms (e.g., as described herein, including those described with reference to FIG. 2E). However, queries (e.g., declarative queries) enabled by the entity API (and particularly the enrichment portion of the query that may identify attributes for which values should be returned) may be specified independently of whether the attributes of interest are implemented as static or dynamic variables.

[0058] In some embodiments, to process statements (e.g., queries, e.g., declarative queries, etc.) that do not include information about how the underlying data entity instances are stored in a data processing system, the Entity API is configured to transform the input statements through a series of transformations to arrive at executable statements (e.g., executable SQL queries) that can be executed against a database system that stores the data entity instances and the values of their attributes. The statements provided as input to the Entity API are “declarative” in the sense that they would not be executable against a database system that stores the data entity instances if they were not declarative statements, and they “declare” the actions desired to be performed (e.g., by “declaring” which data entity instances and which of their attributes to return, update, create, etc.), but are not strictly “executable” themselves, as they do not depend on how the data entity instances are stored. Thus, in some embodiments, the declarative statements (e.g., declarative queries) are stored with the desired data entity instances and attribute values, and may not include any low-level optimizations that can be applied when executing the statements against the database system that stores the data entity instances.

[0059] In this sense, unlike SQL statements, the Entity API's declarative statements (e.g., declarative queries) are not directly executable by an RDBMS (e.g., ORACLE, DB2, POSTGRESS, SQL SERVER, etc.) or other data storage system, but rather are translated into one or more executable statements, which are then executed by the RDBMS or other storage system. While portions of the Entity API statements may be specified using SQL, as described herein, there are other portions of the Entity API statements that are not specified using SQL and are first processed by the Entity API as part of the overall statement to translate the Entity API statements into one or more executable (e.g., SQL or other type) statements (e.g., executable SQL queries or other types of executable queries). Furthermore, unlike SQL, the Entity API statements are intended to specify operations to be performed on data entity instances—a higher-level language in which statements can be translated into executable SQL (or other types of queries), but the statements are not themselves SQL queries.

[0060] Thus, in some embodiments, a declarative statement may be transformed through a series of transformations that augment information in the declarative statement with information managed internally by the data processing system and entity API to generate executable statements (e.g., executable SQL statements, e.g., SQL queries) that can be executed against a database or other system used by the data processing system to store data entity instances and the values of their attributes. Aspects of these transformations are described herein, including below in the section referred to as "Generating Executable Queries" and referring to Figures 7A-7F.

[0061] As noted above, some data entity attributes may reference other data entities, and as a result, data entity instances may be related to one another. Another advantageous feature of the Entity API is that a single declarative query may be used to retrieve information about multiple related data entities. In some embodiments, the declarative query may explicitly specify the hierarchy of related attributes whose values should be retrieved (sometimes referred to herein as "nesting"), or may do so using recursion.

[0062] Thus, in some embodiments, a declarative query may specify a hierarchy of attributes (some of which are attributes of different related data entities), and values of attributes in the hierarchy may be retrieved by processing the declarative query. In some embodiments, the retrieved attribute values may be output in a hierarchical format that corresponds to the hierarchical structure of the hierarchy of attributes specified in the declarative query. For example, the hierarchical format may have a nesting structure that is the same as the nesting structure of attributes in the hierarchy of attributes specified in the declarative query. Aspects of hierarchical queries and hierarchical output of results are described herein, including below in the section entitled "Searching Hierarchical Data." As described herein, hierarchical queries enabled by the Entity API enable a single declarative query to be used to retrieve results across multiple different data entity hierarchies and perform aggregation operations on them.

[0063] Accordingly, some embodiments provide a method for obtaining information about data entity instances managed by a data processing system using at least one data store using a declarative query, the method comprising: obtaining, using at least one computer hardware processor, a query (e.g., a declarative query) including a first portion (sometimes referred to herein as an "identification portion") containing information for identifying at least one data entity instance stored in the at least one data store, and a second portion (sometimes referred to herein as an "enrichment portion") indicating at least one attribute of the at least one data entity; and extracting from the query one or more of the first set of data entity instances. generating a plurality of executable queries (e.g., executable SQL queries) including a first set of executable queries and a second set of executable queries, the second set including: (1) using a first portion of the queries to generate a first set of one or more executable queries for identifying instances of at least one data entity stored in at least one data store; and (2) using a second portion of the queries to generate a second set of one or more executable queries for retrieving attribute values of the instances of the at least one data entity identified using the first set of executable queries; executing the plurality of executable queries to obtain results of the queries; and outputting at least some (e.g., all) of the results.

[0064] In some embodiments, the at least one data entity includes a plurality of data entities, and generating the first set of executable queries includes, for each data entity of the plurality of data entities, generating a respective executable (e.g., SQL) query to identify instances of the data entity stored in the at least one data store.

[0065] In some embodiments, the at least one data entity comprises a first data entity, the at least one attribute comprises a first plurality of attributes of the first data entity, and generating the second set of executable queries comprises, for each attribute of at least some of the first plurality of attributes, generating a respective executable (e.g., SQL) query that obtains a value of the attribute of instances of the at least one data entity identified using the first set of executable queries.

[0066] As described herein, in some embodiments, a data entity may have multiple attributes that may be grouped, and one respective executable query may be generated for each group of attributes, thereby significantly improving overall performance. Thus, in some embodiments, the first plurality of attributes includes multiple dynamic attributes, and generating the second set of executable queries includes (1) grouping the multiple dynamic attributes into dynamic attribute groups, each group having at least one of the multiple dynamic attributes, and (2) generating a respective executable query for each of the multiple groups of dynamic attributes to obtain attribute values in the group for an instance of the first data entity.

[0067] In some embodiments, grouping multiple dynamic attributes is performed based on the type of the dynamic attribute, whether the dynamic attributes are stored in the same table in at least one data store, and / or whether the dynamic attributes inherit their values along a common inheritance path.

[0068] In some embodiments, executing the plurality of executable queries includes (1) executing a first set of executable queries to obtain data entity instance results (e.g., satisfying criteria specified in the identifying portion of the input query); and (2) after executing the first set of executable queries, executing a second set of executable queries to obtain attribute value results for the data entity instances identified by executing the first set of executable queries. In some embodiments, the second set of executable queries is generated before the first set of executable queries is executed. In some embodiments, the second set of executable queries is generated after the first set of executable queries is executed.

[0069] In some embodiments, the at least one attribute includes a group of multiple dynamic attributes of a first data entity of the at least one data entity, the second set of executable queries includes a plurality of executable queries respectively corresponding to the groups of the multiple dynamic attributes, and executing the second set of executable queries includes executing each of the plurality of executable queries to obtain a plurality of dynamic attribute value results. In some embodiments, after executing the second set of executable queries, the plurality of dynamic attribute value results are combined to obtain the attribute value results.

[0070] The Entity API supports inline views, as described herein, including those described below in the section titled "Inline Views." In some embodiments, the second portion of the query includes an inline view, and generating the second set of one or more executable queries includes generating the executable queries using information specified in the inline view. In some embodiments, the inline view may include a table query (e.g., a declarative table query). The table query may be for performing an aggregation function. In some embodiments, the inline view may specify inner and outer join keys.

[0071] In some embodiments, the at least one attribute of the first data entity includes a first plurality of attributes of the first data entity, the first plurality of attributes including at least one static attribute and at least one dynamic attribute. In some embodiments, the at least one attribute of the first data entity includes a first dynamic attribute configured to inherit a value from a second data entity.

[0072] In some embodiments, the second portion of the query may indicate a hierarchy of attributes including at least one attribute. This indication may be done using nesting, recursion, or both. In some embodiments, outputting the results includes formatting the results in a hierarchical format corresponding to the hierarchy of the attributes and outputting the results in the hierarchical format. Such a format may be non-tabular.

[0073] In some embodiments, generating the first set of executable queries (e.g., for identifying data of interest) includes, at least in part, generating a first executable query by: (1) generating an intermediate query representation (e.g., an abstract syntax tree) using a first portion of the query; (2) augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information, and / or permission information to obtain an augmented intermediate query representation; and (3) generating the first executable query using the augmented intermediate query representation.

[0074] In some embodiments, generating the second set of executable queries (e.g., for accessing values of the data entity instances identified using the first set of executable queries) includes generating the second executable queries, at least in part, by (1) generating an intermediate query representation using the second portion of the query, (2) augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information, and / or permission information to obtain an augmented intermediate query representation, and (3) generating the second executable query using the augmented intermediate query representation.

[0075] Some embodiments provide a method that includes using at least one computer hardware processor to obtain a query (e.g., a declarative query), the query including a first portion including information for identifying at least one data entity instance stored in at least one data store and a second portion indicating an attribute hierarchy including at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for retrieving at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the query; and outputting at least some (e.g., all) of the results in a hierarchical format corresponding to the attribute hierarchy.

[0076] Some embodiments provide a method that includes using at least one computer hardware processor to obtain a query (e.g., a declarative query), the query including a first portion including information for identifying at least one data entity instance stored in at least one data store and a second portion indicating at least one attribute of the at least one data entity; generating from the query a plurality of executable queries for obtaining at least one data entity instance and attribute values of the at least one data entity instance from the at least one data store; executing the plurality of executable queries to obtain results of the queries; and outputting at least some (e.g., all) of the results.

[0077] It should be understood that the techniques described herein can be implemented in any of numerous ways, and that the techniques are not limited to any particular manner of implementation. Example implementation details are provided herein for illustrative purposes only. Furthermore, the techniques disclosed herein can be used individually or in any suitable combination, and aspects of the techniques described herein are not limited to the use of any particular technique or combination of techniques.

[0078] Exemplary Data Processing Systems and Applications As mentioned above, the inventors have recognized that data processing systems can be configured to manage millions or billions of data entity instances. For example, the techniques described herein can be used for metadata management in an enterprise setting, whereby data entity instances store information about individual datasets (e.g., tables, transactions, documents, data records, etc.) stored across a globally distributed information technology (IT) infrastructure including many databases, data warehouses, data lakes, etc. As mentioned above, in this regard, a data entity instance may store information about the corresponding dataset, such as when the dataset was created, where it is stored, its size, the identities of users who are authorized to edit the dataset, information identifying which application programs use the dataset, the sensitivity level of the data, etc. Large organizations (e.g., financial institutions such as banks or credit card companies, public utilities such as telephone companies or electric companies, etc.) typically manage millions or billions of such datasets, and therefore there may be millions or billions of data entity instances storing information about such datasets managed by a data processing system.

[0079] 1A illustrates an example environment in which a data processing system according to some embodiments of the techniques described herein may be used for metadata management. It should be understood that the techniques described herein are not limited to data processing systems used for metadata management, but are applicable to any data processing system that uses data entities and data entity instances to manage data, whether the managed data is metadata or any other type of data.

[0080] 1A illustrates an enterprise system that includes systems 140, 142, 144 distributed across multiple different geographic locations (e.g., different cities, countries, continents, etc.), each storing vast amounts of data (e.g., in one or more database systems, data warehouses, data lakes, etc.). For example, systems 140, 142, 144 may be components of the enterprise system of a global bank, with system 140 located in the United States, system 142 located in Brazil, and system 144 located in Europe.

[0081] As shown in the example embodiment of FIG. 1A , each of systems 140, 142, and 144 includes a respective set of computing devices. System 140 includes server 140A and database system 140B. System 142 includes server 142A and database system 142B. System 144 includes server 144A and database system 144B. During operation of the enterprise system, each of systems 140, 142, and 144 may generate and / or store large amounts of data (e.g., terabytes of data). For example, the enterprise system may be a credit card company system, with each of systems 140, 142, and 144 generating and / or storing transaction data, credit scores, and / or any other suitable data. In another example, the enterprise system may be a bank system, with each of systems 140, 142, and 144 generating and / or storing bank records, loans, account holders, and / or any other suitable data. In another example, the enterprise systems may be telephone company systems, with each of the systems 140, 142, 144 generating and / or storing telephone calls, text messages, data usage, and / or any other suitable data.

[0082] In some embodiments, database systems 140B, 142B, 144B may be configured to store data (e.g., of an enterprise system). Each of database systems 140B, 142B, 144B may include a database, a data warehouse, a data lake, and / or any other database system. Database systems 140B, 142B, 144B may be of any suitable type, either the same type or different types. For example, each of these systems may include one or more relational database systems (e.g., ORACLE, SQL SERVER, etc.). As another example, in some embodiments, each of these systems may include one or more other types of database systems (e.g., non-relational (e.g., NoSQL) database systems, multiple file systems, or any other suitable type of database system).

[0083] In the example embodiment of FIG. 1A , data processing system 105 may be configured to store information 107 that describes data stored in enterprise systems 140, 142, and 144. In this sense, information 107 may be considered metadata. The metadata may include any of many types of information about the data stored in enterprise systems 140, 142, and 144. For example, the metadata may include information about the systems (e.g., servers 140A, 142A, and 144A) that process the data, the software applications running on the enterprise systems used to process the data, and / or the application's rules for storing the data. In another example, the metadata may include information about data throughout enterprise software systems, such as how the data was generated, the size of the data, a description of the data, which users have permission to read, update, create, delete, or perform any other action on the data, and / or any other suitable information about the data.

[0084] In some embodiments, a data processing system may manage metadata using data entity instances and data entity definitions. For example, data processing system 105 may store a data entity instance for each of multiple datasets (e.g., tables) stored by an enterprise system. Each such data entity instance may store information about the dataset (e.g., when the dataset was created or updated, where the dataset is stored, the size of the dataset, identities of users who are authorized to read, edit, delete, or perform any other suitable actions on the dataset, information identifying which software applications use the dataset, information identifying the sensitivity level of the data in the dataset, and / or any other suitable metadata). As another example, data processing system 105 may store a data entity instance for each column of a table in an enterprise system. Each such data entity may store information about the column (e.g., the meaning of values in the column, who has authority to read, write, update, and / or delete values in the column, the range of allowed values for entries in the column, and / or any other suitable metadata). As yet another example, data processing system 105 may store a data entity instance for each of multiple software applications configured to be executed by any system or device portion of an enterprise system. Such a data entity instance may store information about the software application (e.g., which data sets the software application processes, where the application places its output, a description of the application's functionality, the application's version, the application's dependencies on data and / or other applications, where the application's executable files can be found, and / or any other suitable metadata). As yet another example, data processing system 105 may store a data entity instance for each of multiple system portions of an enterprise system.

[0085] As can be readily appreciated from the above, in such a metadata management scenario, the data processing system 105 may manage millions or even billions of such data entity instances, which is why it is important that querying, creating, updating, deleting or performing any other suitable operation on data entity instances is performed as efficiently as possible.

[0086] FIG. 1B further illustrates the example environment of FIG. 1A in which a data processing system may operate, according to some embodiments of the techniques described herein.

[0087] 1B, data processing system 105 may be configured to manage data using data entities, at least in part, by using entity model 160, which includes data entity definitions and data entity relationships. In some embodiments, instances of data entities defined by entity model 160 may be stored in data persistence layer 150. Thus, at least some or all of the data associated with data entity instances may be stored in data persistence layer 150. Aspects of entity model 160 and persistence layer 150 are described in more detail herein, including with reference to FIG. 1D.

[0088] 1B , in some embodiments, entity API 120 provides access to data entity instances managed by data processing system 105 and persisted in data persistence layer 150. For example, entity API 120 exposes functionality that allows data entity instances to be created, updated, deleted, and queried. In some embodiments, the data processing system may be configured such that data entity instances managed by data processing system 105 can be accessed (e.g., created, updated, deleted, and / or queried) only through entity API 120, as indicated by the boundary defining entity API 120 that also encompasses data persistence layer 150 in FIG. 1B . The functionality exposed by entity API can be accessed through any of many types of entity API interfaces 110.

[0089] In some embodiments, declarative statements (e.g., declarative queries) for performing operations on one or more data entity instances managed by data persistence layer 150 may be provided as input to entity API 120. The declarative statements may be converted into one or more executable queries using input conversion module 130. The queries are then executed and the results are provided to output conversion module 165, which may further process the results (e.g., combine, format, organize, etc.) before providing the results to the user or computer program that provided the declarative statements. Aspects of entity API interface 110, entity API 120, input conversion module 130, and output conversion module 164 are described in more detail herein, including with reference to FIG. 1D .

[0090] With respect to metadata management, in some embodiments, the interface 110 may be configured to generate a graphical user interface (GUI) that allows a user to access information 107 about data stored in the systems 140, 142, and 144. The GUI allows a user to (1) request information about data entity instances stored by the data processing systems and (2) view information about data entity instances stored by the data processing systems. In some embodiments, the GUI may allow a user to access stored information 107 (e.g., metadata) about data stored by the systems 140, 142, and 144. For example, the GUI may allow a user 102 to track data being generated in an enterprise software system (e.g., quality measures and other characteristics of the data). In another example, the GUI may allow a user 102 to visualize lineage information. Lineage information may include information about relationships between different data entity instances. Aspects of pedigree information are described in U.S. Pat. No. 10,489,384, entitled "SYSTEM AND METHODS FOR DETERMINING RELATIONSHIPS AMONG DATA ELEMENTS," which is incorporated herein by reference in its entirety.

[0091] To further illustrate the concept of metadata management for a data processing system, Figure 1B shows two example data entity instances that store information about each data in database system 144B. Database system 144B stores table 146, which contains information about customers (e.g., of a bank). Columns in table 146 include "identifier," "name," "credit score," and "score date." Data processing system 105 stores metadata about table 146, and in this example accomplishes this in data persistence layer 150 by storing an instance 147 of a "dataset" data entity that stores information about table 146.

[0092] In this example, the data entity instance 147 stores values for attributes including the attributes "Type" (indicating the type of information in the table), "Business Manager" (indicating the person responsible for the data in the table), "Number of Entries" (indicating the number of entries in the table), "Private" (indicating a privacy setting), "Storage Size" (indicating the amount of memory the table occupies), and "Data ID." In some embodiments, the "Dataset" data entity instance 147 may store values for other attributes in addition to or instead of those shown in FIG. 1B.

[0093] In applications where a data entity instance includes metadata about data (e.g., information about a table), in some embodiments, the data entity instance may include information that can be used to identify and / or access the data. As shown in the example of FIG. 1B , a “data ID” attribute identifies the data (e.g., a table) that the information in “dataset” data entity instance 147 describes. For example, the value of “data ID” may be an identifier for table 146. In some embodiments, the value of “data ID” may enable a user to navigate to table 146. For example, the value of “data ID” may be a hyperlink that navigates to table 146 in database 144B. In some embodiments, a data entity instance does not itself store how to access a table, but a data processing system may store information that associates the data entity instance with information that can be used to identify and / or access the data itself. For example, the data processing system may store such information in one or more tables (e.g., in data persistence layer 150) or in any suitable manner.

[0094] As shown in the example of FIG. 1B , data persistence layer 150 also stores a "Credit Score" data entity instance 149. The "Credit Score" data entity instance 149 may be an instance of the "BizTerm" data entity 220 shown in FIGS. 2C and 2E . The "Credit Score" data entity instance 149 includes values for the attributes "Type," "Description," "Business Owner," "Valid Lower Bound," "Valid Upper Bound," "Private," and "Data ID." As indicated by the arrow between the "Data ID" attribute and the "Credit Score" column of table 146, the "Credit Score" data entity instance 158 describes the data in the "Credit Score" column of table 146. As shown in the example of FIG. 1B , the "Data ID" attribute indicates the data that the information in the "Dataset" data entity instance 149 describes. For example, the "Data ID" value may be an identifier for the "Credit Score" column in table 146. In some embodiments, the "Data ID" value may allow a user to access the data stored in the "Credit Score" column of table 146. For example, the value of "Data ID" may be a hyperlink to a user that can be used to access the data in the "Credit Score" column.

[0095] FIG. 1C is a diagram illustrating how, in some applications, a data processing system may be configured to manage millions or billions of data entity instances, according to some embodiments of the techniques described herein. As shown in the example embodiment of FIG. 1C , in some embodiments, data persistence layer 150 stores a large number (e.g., thousands, millions, or billions) of data entity instances that store information about each component of the enterprise system of FIG. 1A (e.g., a dataset, a software application, a system component of the enterprise system, or any other component). Arrows from each data entity instance in FIG. 1C indicate the component that the information in the data entity instance describes. A component may be a dataset, an application (e.g., one or more computer programs), a system (e.g., a database system), and / or other component of the enterprise system. For example, a data entity instance may store information about data (e.g., a table) in the enterprise system. In another example, a data entity instance may store information about an application of the enterprise system. In yet another example, a data entity instance may store information about a user of the enterprise system.

[0096] As shown in the example embodiment of FIG. 1C , metadata stored in data entity instances in data persistence layer 150 may be used to provide visualizations 109 to users 102. Visualizations 109 may include visualizations of lineage information associated with data entity instances. Lineage information may include information about relationships between different data entity instances. Aspects of lineage information are described in U.S. Pat. No. 10,489,384, entitled “SYSTEMS AND METHODS FOR DETERMINING RELATIONSHIPS AMONG DATA ELEMENTS,” which is incorporated herein by reference in its entirety.

[0097] FIG. 1D is a block diagram 100 illustrating aspects of an example data processing system 105 and an entity application programming interface (API) 120 integrated with the data processing system 105, in accordance with some embodiments of the technology described herein.

[0098] In some embodiments, data processing system 105 may be configured to manage data using data entities and their instances. Data processing system 105 may be configured to manage data using data entities at least in part by using entity model 160, which includes data entity definitions 162 and data entity relationships 164.

[0099] A data processing system may be configured to manage data using any suitable number of data entities. For example, entity model 160 may define tens, hundreds, thousands, or tens of thousands of data entities, as may be the case in an enterprise computing environment.

[0100] In some embodiments, data entity definition 162 includes a definition or specification of each data entity supported by the data processing system. A data entity definition may define the structure of a data entity. To do so, a data entity definition may define the attributes of the data entity. An entity attribute may be scalar, taking on values such as strings and numbers. Alternatively, an entity attribute may reference one or more other data entities. A data entity definition may be referred to as a "data entity class" or a "data entity specification," or may simply be referred to herein as a "data entity." Data entity definitions are described in more detail herein, including below with reference to Figures 2A-2G.

[0101] In some embodiments, data entity relationships 164 define relationships between data entities. For example, data entity relationships 164 may define whether a particular data entity can have an attribute that references another data entity (such an attribute may be referred to herein as a "reference" attribute, in the sense that its value references another data entity instance).

[0102] In some embodiments, an attribute of one data entity may be configured to inherit or take its value from an attribute of another data entity. Different attributes of the same data entity may be configured to inherit values from different data entities. In some embodiments, inheritance configuration may be achieved using reference attributes. For example, one attribute (a reference attribute) may indicate the data entity instance from which a second attribute should inherit a value. For example, data entity "D1" may have two attributes "A1" and "A2", where A2 is a reference attribute and the value of attribute "A2" may identify another data entity D2 that should inherit the value of attribute "A1". Changing the value of attribute "A2" to a different data entity "D3" changes the data entity from which attribute "A1" inherits a value.

[0103] Thus, the concept of attribute inheritance described herein differs from "inheritance," as that term is used in connection with object-oriented programming. In object-oriented programming, inheritance refers to a child class inheriting properties (e.g., variables, function definitions, and function implementations) from a base class. This enables code reuse and software extensibility through public classes and interfaces. Conversely, "inheritance," as described herein, refers to the situation in which a data entity's attributes can be configured to take, or "inherit," values from, the attributes of another data entity. The configuration refers to the data entity from which the attribute inherits data—it is not fixed, as with a base class in an object-oriented setting. Because inheritance configuration is not at the attribute level, different attributes of the same underlying data entity can be configured to take, or "inherit," values from different attributes of multiple other data entities. This provides great flexibility in how attribute values can be set, providing a level of indirection not available in object-oriented programming. Indeed, not only can a single data entity be configured to inherit attribute values from multiple other data entities, but, as noted above, that configuration can be changed at the level of a single data entity instance.

[0104] As can be appreciated from the above, inheritance for a data entity can be configured at the attribute level. To this end, in some embodiments, a data processing system may store an "inheritance configuration" for each of at least some (e.g., all) attributes of a data entity. The "inheritance configuration" may indicate whether the attribute inherits values from another data entity attribute, and if so, from which.

[0105] Conversely, conventional object-oriented systems do not have the attribute-level inheritance construct described herein. As noted above, conventional object-oriented systems do not support inheritance of values at the attribute level of a data entity instance. Rather, object-oriented inheritance is at the class level, such that a data entity instance attribute inherits all values from a single parent data entity. Further aspects of inheritance are described herein, including with reference to Figures 6E and 6F.

[0106] In some embodiments, entity model 160 (e.g., data entity definitions 162 and data entity relationships 164) may be stored partially or completely in data persistence layer 150. In other embodiments, entity model 160 may be stored in any other suitable storage, as aspects of the technology described herein are not limited in this respect.

[0107] In some embodiments, instances of data entities defined by entity model 160 may be stored in data persistence layer 150. Accordingly, at least some or all of the data associated with a data entity instance may be stored in data persistence layer 150. Non-limiting examples of data associated with a data entity instance include values of any attributes of the data entity instance, identification information of the data entity instance (e.g., a unique identifier of the data entity instance that allows data processing system 105 to uniquely identify the data entity instance), a definition of the data entity, information indicating one or more other data entities that have attributes from which the data entity instance is configured to inherit values, permission information indicating users that have permission to create, read, write, and / or update the data entity instance, and / or any other suitable data. Some or all of these types of data may be stored in data persistence layer 150 for each of one or more of the data entity instances. In the exemplary embodiment of FIG. 1D , data persistence layer 150 includes one or more data stores 152 for storing data entity instances (data associated with the data entity instances). In some embodiments, data store 152 may include a relational database system such that data associated with data entity instances is stored in tables of the relational database system. For example, data store 152 may include an ORACLE database system, a MICROSOFT SQL SERVER database system, a POSTGRESQL database system, an IBM DB2 database system, and / or any other suitable relational database system, as aspects of the disclosure described herein are not limited in this respect. However, data store 152 is not limited to including only relational database systems and may be configured to store data in any suitable manner.For example, data store 152 may include an object-oriented database, a distributed database, a NoSQL database (e.g., a MONGODB database), one or more text files, or any other suitable database and / or system for managing data. As another example, in some embodiments, data store 152 may include one or more search indexes (e.g., to optimize data retrieval), a cache (e.g., to pre-fetch small data sets that are repeatedly accessed, such as reference data used to classify, categorize, or interpret other data), or any suitable combination thereof. Aspects of search indexes are described herein, including below in the section entitled "Searching Data Using One or More Search Indexes."

[0108] An advantage of the Entity API architecture, as described herein, is that calls to the Entity API are storage agnostic. In this manner, a data processing system (e.g., data processing system 105) may be implemented using any suitable storage solution (such as an RDBMS, a file, an object-oriented database, a search index, a cache, any suitable combination thereof, etc.), and calls to the Entity API do not need to be adapted to the storage solution employed in a particular deployment or implementation.

[0109] More generally, each data store 152 may include one or more storage devices that store data in one or more formats of any suitable type. For example, the storage device portion of a data store may store data using one or more database tables, spreadsheet files, plain text files, and / or files in any other suitable format (e.g., mainframe native format). The storage devices may be of any suitable type and may include one or more servers, one or more database systems, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other device configured to electronically store data. In embodiments in which a data store includes multiple storage devices, the storage devices may be located in one physical location (e.g., within one building) or distributed across multiple physical locations (e.g., multiple buildings, different cities, states, or countries). The storage devices may be configured to communicate with each other using one or more networks of any suitable type, and aspects of the technology described herein are not limited in this respect.

[0110] Data persistence layer 150 may be configured to store any suitable number of data entity instances. As discussed above, data processing systems may be configured to support tens, hundreds, thousands, or tens of thousands of data entities. In enterprise computing environments, data persistence layer 150 may be configured to store thousands, millions, or even billions of data entity instances. For example, data persistence layer 150 may store at least 10,000 data entity instances, at least 50,000 data entity instances, at least 100,000 data entity instances, at least 500,000 data entity instances, at least 1,000,000 data entity instances, at least 5 million data entity instances, at least 10 million data entity instances, at least 50 million data entity instances, at least 100 million data entity instances, at least 500 million data entity instances, at least 1 billion data entity instances, at least 5 billion data entity instances, between 100,000 and 5 million data entity instances, between 1 million and 500 million data entities, between 1 million and 5 billion data entities, or any other range of data entity instances within these ranges.

[0111] In some embodiments, entity API 120 provides access to data entity instances managed by data processing system 105 and persisted in data persistence layer 150. For example, entity API 120 exposes functions that allow for the creation, update, deletion, and query of data entity instances. In some embodiments, data processing system 105 may be configured such that data entity instances managed by data processing system 105 may be accessed (e.g., created, updated, deleted, and / or queried) only through entity API 120, as indicated by the bold border defining entity API 120 in FIG. 1D . In such embodiments, entity API 120 may provide a “fence” around data entity instances.

[0112] The functionality exposed by the entity API may be accessed through any of many types of entity API interfaces 110. For example, a query for information associated with a data entity instance may be provided (e.g., by one or more users) through one or more graphical user interfaces 112, and query results may be presented to the users through the graphical user interfaces 112. As another example, a query for information associated with a data entity instance may be provided programmatically through a software interface, such as a scripting interface 114 (e.g., a web-based program written using a scripting language), a graph-based computer program 116, and / or any other third-party software. In addition to retrieving information, any of these types of interfaces may also be used to create, delete, or otherwise update data entity instances (e.g., by updating the values of entity instance attributes, changing where attributes inherit values, adding new attributes to a data entity instance, removing attributes from a data entity instance, etc.). It should be understood that the above examples of entity API interfaces are illustrative and that other types of interfaces may be used to access functionality exposed by the entity API 120.

[0113] With respect to graph-based computer programs 116, in some embodiments, graph-based computer programs may be developed as dataflow graphs. Dataflow graphs may include components called "nodes" or "vertices," which represent data processing operations to be performed on input data, and links between the components, which represent data flow. Techniques for executing computations encoded by dataflow graphs are described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference in its entirety. An environment for developing computer programs as dataflow graphs is described in U.S. Patent Application Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference in its entirety.

[0114] In some embodiments, entity API 120 enables obtaining information about data entity instances through the use of declarative queries. A declarative query may specify (1) a data entity instance of interest and (2) attributes whose values should be inherited by the data entity of interest. A declarative query is not an executable query that can be executed by one or more database portions of data persistence layer 150. Rather, entity API 120 may be configured to process a declarative query to use information used by the declarative query to generate a set of one or more executable queries (different from the declarative query) that, when executed by a data processing system against one or more data stores 152 in data persistence layer 150, can be used to retrieve information of interest as specified by the declarative query. The one or more executable queries generated using the information specified by the declarative query may be executable SQL queries or any other suitable type of executable query.

[0115] In some embodiments, declarative queries may be specified using JavaScript Object Notation (JSON), however, in other embodiments, they may be specified in any other suitable format (e.g., using any suitable markup language), as aspects of the technology described herein are not limited in this respect.

[0116] In some embodiments, a declarative query may include two parts: a so-called identification part and a so-called enrichment part. The identification part may include information for identifying an instance of at least one data entity. The enrichment part may include information identifying attributes of at least one data entity that are of interest.

[0117] In some embodiments, data processing system 105 may be configured to process such declarative queries by (1) using an identification portion of the declarative query to identify at least one data entity instance, and (2) using an enrichment portion of the declarative query to obtain attribute values of the identified data entity instance. For example, in some embodiments, data processing system 105 may (1) use the identification portion of the declarative query to generate a first set of one or more executable (e.g., SQL) queries, (2) use the enrichment portion of the declarative query to generate a second set of one or more executable (e.g., SQL) queries, and (3) execute the first set of executable queries to One or more identified The declarative query may be configured to process the declarative query by (4) obtaining results indicative of the data entity instances, (5) executing a second set of executable queries to obtain results indicative of attribute values of the identified data entity instances, (6) combining the results obtained from executing the first and second sets of queries, and (7) outputting the results, optionally applying formatting to the results (e.g., hierarchical formatting).

[0118] 1D , a declarative query 115 may be provided as input to entity API 120 from one of entity API interfaces 110. Within entity API 120, input translation module 130 may generate multiple executable queries 135 from declarative query 115. Declarative query 115 may include an identification portion and an enrichment portion, and multiple executable queries 135 may include one or more executable queries generated from the identification portion of declarative query 115 and one or more executable queries generated from the enrichment portion of declarative query 115.

[0119] 1D, the input transformation modules include an entity instance identification module 132, an entity instance enrichment module 134, and a query transformation module 136. These modules may be configured to generate an executable query 135 from the declarative query 115.

[0120] For example, in some embodiments, entity instance identification module 132 may parse the identification portion of declarative query 115 and generate an intermediate representation of the query to identify data entity instances (e.g., as an abstract syntax tree). Query transformation module 136 may also (1) transform the intermediate representation of the query with a series of one or more transformations to augment the representation with additional information, and (2) generate an executable query from the transformed intermediate representation. This process is described in more detail herein in a preferred exemplary format with reference to FIG. 3 and in FIGS. 7A-7F.

[0121] As another example, in some embodiments, the entity instance enrichment module 134 may parse the enrichment portion of the declarative query 115 to generate intermediate representations of one or more queries (and often multiple queries) for retrieving attribute values of the identified data entity instances. The query transformation module 136 may then (1) transform each intermediate query representation with a series of one or more transformations to augment the representation with additional information, and (2) generate executable queries from each transformed intermediate representation. This process is described in more detail herein, including with reference to Figures 3 and 7A-7F.

[0122] After executable query 135 is generated, data processing system 105 may execute executable query 135 by, for example, using data access module 140 to invoke query 135 in one or more data stores 152 to retrieve data 137 from data persistence layer 150. Data 137 may include two types of results: (1) results from executing one or more executable queries to identify data entity instances; these results include information indicative of one or more identified data entity instances (e.g., identifiers of the identified entity instances); and (2) results from executing one or more executable queries to retrieve attribute values; these results include attribute values of the identified data entity instances. Data 137 may be provided to output transformation module 165, which may process data 137 and then output it as query results 155 to one or more of entity API interfaces 110.

[0123] 1D , the output transformation module 165 includes a data compositing module 163 and a hierarchy output module 167. In some embodiments, the data compositing module 163 may be configured to combine results in the data 137 to generate the query results 155. As described above, the data 137 may include (1) results including information indicative of one or more identified data entity instances and (2) results including attribute values of the identified data entity instances. The data compositing module 163 may combine these results (e.g., using pivoting or in any other manner) such that the query results 155 associate each identified data entity instance with the value of its attributes. Aspects of combining results of executing identification queries and enrichment queries are described herein, including with reference to FIG. 3 and in the sections entitled "Declarative Queries: Identification and Enrichment" and "Optimizing the Execution of Enrichment Queries."

[0124] 1D , output transformation module 165 also includes hierarchical output module 167. In some embodiments, hierarchical output module 167 may be configured to format data 137 for hierarchical presentation (e.g., using tabbing and / or nesting of text, using nested containers in a graphical user interface, using nested tags in a markup language, etc.). In some embodiments, the hierarchical structure of the results may match the hierarchical structure of the declarative query (e.g., the hierarchical structure of the attributes identified in the enrichment portion of the declarative query). Aspects of hierarchical querying and presentation of results are described herein, including with reference to FIGS. 4D-4F, 8A-8E, and 9A-9G.

[0125] 1D focuses in part on how data processing system 105 executes declarative queries, it should be understood that data processing system 105 is not limited to executing only declarative queries. Indeed, entity API 120 is also configured to process declarative statements (e.g., using module 138) to create, delete, or otherwise update data entity instances persisted to layer 150. Aspects of creating, deleting, and updating data entity instances are described herein, including in a section entitled "Additional Entity API Operations."

[0126] Another illustration of how a data processing system 105 may be configured to process declarative queries is shown in FIG. 1E, which is a diagram of an example process 101 for generating query results in response to a declarative query 115 for obtaining information about instances of one or more data entities, according to some embodiments of the technology described herein.

[0127] 1E , declarative query 115 having entity instance identification portion 115a and entity instance enrichment portion 115b may be processed by input transformation module 130 of data processing system 105. As a result, one or more entity instance identification executable queries 135a may be generated using entity instance identification portion 115a of declarative query 115. Entity instance identification executable queries 135a may be executed by the data processing system to obtain entity instance identification results 137a from one or more data stores in data persistence layer 150.

[0128] The entity instance enrichment portion 115b may be used to generate one or more entity instance enrichment executable queries 135b for accessing attribute values of entity instances identified by the results 137a. As indicated by arrow 131, in this illustrative example, when the enrichment executable query 135b is executed, the query may reference the results 137a of executing the identification query such that attribute values of only the data entity instances identified by the results 137a (these are instances identified using the identification portion of the declarative query 115) are retrieved. For example, the enrichment executable query 135b may include references to one or more tables and / or variables (e.g., see table “BizTermID0” in the example of FIGS. 5C-5F) for storing information identifying the data entity instances identified by using the query 135a. After the entity instance identification executable query 135a is executed, the results may be stored in the referenced tables and / or variables and may be accessed by the enrichment executable query 135b when these queries are executed.

[0129] In some embodiments, the entity instance identifying executable query 135a may be executed before the entity instance enrichment query 135b is generated. In some such embodiments, because data entity instance identifiers are available, these identifiers may (but need not) be inserted directly into the enrichment query 135b. In this implementation, the executable enrichment query is generated based on both the content of the entity instance enrichment portion and the results of executing the identifying executable query 135a. However, in some embodiments, including the embodiments of FIGS. 1E and 5C-5F, the identification query 135a does not need to be executed before the enrichment query 135b is generated because the enrichment query 135b may reference one or more tables and / or variables (or other storage) in which the data entity instance identifiers will be stored (after the identification query 135a is executed), and the references (e.g., table and / or variable names) may be included in the enrichment query 135b before the entity instance identifying query 135a is executed. Thus, in some implementations, the enrichment query may be generated before or after the identification query is performed, as aspects of the technology described herein are not limited in this respect.

[0130] After query 135a is executed, entity instance enrichment executable query 135b may be executed to obtain entity instance enrichment results 137b from data store 152 in data persistence layer 150. As described herein, data store 152 may include a relational database or any other suitable type of data storage solution, examples of which are provided herein (e.g., with reference to data store 152) as aspects of the technology described herein are not limited in this respect.

[0131] Results 137a and / or 137b may then be processed for output using output transformation module 165 to obtain query results 155, which may then be output. Various aspects of the stages of process 101 are described in further detail herein, including below with reference to FIG. 3.

[0132] It should be understood that entity instance identification executable query 135a and entity instance enrichment query 135b obtain different information from data persistence layer 150. In particular, query 135a may be used to identify data entity instances that meet the criteria set forth in entity instance identification portion 115a and return information identifying such data entity instances. However, the results to query 135a generally do not include values of data entity instance attributes. Rather, the values of the data entity instance attributes are obtained by processing entity instance enrichment executable query 135b. It should also be understood that the data obtained by queries 135a and 135b may be stored in different tables or other data structures in data store 152. For example, identifiers of data entity instances may be stored separately from values of the instance's dynamic attributes, which may be stored as name-value pairs.

[0133] 1F shows an illustrative example of a declarative query 170 being processed by the example process 101 shown in FIG. 1E, in accordance with some embodiments of the techniques described herein. The declarative query 170 includes an entity instance identification portion 170a and an entity instance enrichment portion 170b.

[0134] Identification portion 170a indicates that instances of "BizTerm" data entities whose names begin with "A" should be identified. Enrichment portion 170b indicates that for each data entity instance identified using identification portion 170a, values for several attributes should be searched, including "Name" and "Accountable Party" - the ovals indicate that more attributes may be specified in enrichment portion 170b, but in this example only two are shown for clarity of presentation.

[0135] Input transformation module 130 then processes entity instance identification portion 170a to generate entity instance identification executable SQL query 172a, which, when executed by a data processing system, retrieves entity instance ID results 174a from data persistence layer 150. Entity instance ID results 174a include identifiers of data entity instances (shown in table 175) that satisfy the conditions set forth in entity instance identification portion 170a. While table 175 has only seven rows, this is for clarity of presentation only; any suitable number of data entity instances (e.g., tens, hundreds, thousands, millions, etc.) may be identified in response to execution of an entity instance identification query (e.g., query 172a in this example or any other entity instance identification query described herein).

[0136] Input translation module 130 also processes entity instance enrichment portion 170b to generate entity instance enrichment executable SQL query 172b, which, when executed by the data processing system, retrieves entity instance enrichment results 174b from data persistence layer 150. As indicated by arrow 171, in this illustrative example, when enrichment executable query 172b is executed, the query may reference results 174a of executing identification query 172a so that attribute values of only the data entity instances identified by results 174a are obtained. For example, enrichment executable query 172b may include references to tables and / or other variables for storing information identifying the data entity instances identified using query 172a. After entity instance identification executable query 172a is executed, the results may be stored in tables and / or other variables and accessed by enrichment executable query 172b when these queries are executed.

[0137] In some embodiments, the entity instance identification executable query 172a may be executed before the entity instance enrichment query 172b is generated. In some such embodiments, the data entity instance identifiers are available, so that they can be inserted directly into the enrichment query 172b, if necessary. In this implementation, the executable enrichment query 172b is generated based on both the content of the entity instance enrichment portion and the results of executing the identification executable query 172a. However, as described herein, the identification query 172a does not need to be executed before the enrichment query 172b is generated because the enrichment query 172b may reference one or more tables and / or variables in which the data entity instance identifiers will be stored (after the identification query 172a is executed), and the references (e.g., table and / or variable names) may be included in the enrichment query 172b before the entity instance identification query 172a is executed.

[0138] 1F , entity instance enrichment results 174b include values of attributes specified in entity enrichment portion 170b from data persistence layer 150. For example, table 176a includes values of the “name” attribute of the instances of data entities identified in entity instance ID result 174a. As another example, table 176b includes values of the “accountable party” attribute of the instances of data entities identified in entity instance ID result 174a. If entity instance enrichment portion 170b specified additional attributes (as indicated by ovals in 170b), entity instance enrichment results 174b could include values of those additional attributes (as indicated by ovals in 174b).

[0139] As described herein, in some embodiments, different executable SQL queries may be used to search for values for different groups of one or more attributes. In the example of FIG. 1F, executable SQL queries 172b includes at least two executable SQL queries—a first executable SQL query for obtaining values for the “Name” attribute and a second executable SQL query (which, unlike the first executable SQL query, is executed separately) for obtaining values for the “Accountable Party” attribute. This is why, in this example, the results obtained by executing these two queries are shown in separate tables 176a and 176b.

[0140] After the entity enrichment results 176b are obtained, the results are processed by output transformation module 165 to generate query results 155, which include table 178, which in this example groups all retrieved attribute values into specific data within the table rows for that entity instance. As described herein, the query results output in response to processing an entity API query need not be a table, but in some embodiments may have any other suitable format (e.g., a hierarchical format), examples of which are provided herein. The query results are shown as rows of table 178 for clarity of presentation of this example in FIG. 1F.

[0141] In the example of FIG. 1F, output transformation module 165 is shown receiving both entity instance ID results 174a and entity instance enrichment results 174b and generating query results from both of these inputs. A similar architecture is shown in FIG. 1E. Entity instance ID results 174a may include additional information about each entity instance ID beyond just its identifier (e.g., the name of the entity instance, the class of the entity, a name to display when generating a GUI referencing the entity, one or more other identifiers, and / or any other suitable information). Entity instance enrichment results may include values of one or more attributes of the identified instances. Query results may include some or all of these types of information.

[0142] In some embodiments, query results may be generated directly from entity instance enrichment results, e.g., without using entity instance ID results 174a. For example, if only entity instance identifiers are to be returned without other information about each entity instance (e.g., other information that may be part of entity instance ID results 174a) and enrichment results 174b organize attribute values using entity instance IDs (e.g., as shown in the first column of tables 176a and 176b in FIG. 1F), the enrichment results may be used to generate query results (e.g., table 178) directly, e.g., without using entity instance ID results 174a.

[0143] Data entities, static and dynamic attributes, and relationships between entities In some embodiments, data managed by a data processing system may be organized using data entities. Data entities may be used to organize data using an object-oriented paradigm. Similar to how object-oriented programming includes its classes and instances, definitions of data entities may be configured in a data processing system, and the data processing system may manage data using instances of data entities and data entity definitions.

[0144] In some embodiments, a data entity definition defines the structure of a data entity. A data entity definition may define the attributes (or variables) of a data entity. Entity attributes can take scalar values, such as strings and numbers. For example, the "BizTerm" data entity 220 shown in FIG. 2C has a string-valued attribute named "Definition." Entity attributes may reference other data entities. Such entity attributes can be called "reference attributes." For example, as shown in FIG. 2G, the BizTerm group attribute of data entity 220 references the BizTerm group data entity. Other examples of entity attribute types are also provided herein.

[0145] In some embodiments, a data entity definition may specify one or more additional pieces of information for an attribute of a data entity. For example, a data entity definition may specify an attribute type. As another example, a data entity definition may specify the manner in which an attribute's value is stored by a data processing system (e.g., whether the attribute value is stored in a row or in a column). As yet another example, a data entity definition may specify whether an attribute of a data entity inherits a value (e.g., from an attribute of another data entity). In some implementations, a data entity definition may specify a particular value (e.g., a default value) for a particular attribute, in which case all instances of the data entity have the particular attribute set to the particular value. In other cases, the value of a data entity instance attribute may vary from instance to instance of the data entity.

[0146] In some embodiments, a data entity may include one or more static attributes and / or one or more dynamic attributes. For example, Figure 2A shows an illustrative diagram of a data entity 200 having a plurality of static attributes 202, including static attributes 203a and 203b, and a plurality of dynamic attributes 204, including dynamic attributes 205a and 205b. As another example, Figure 2B shows an illustrative diagram of a data entity 210 named "BizTerm" having a plurality of static attributes 212, including static attributes 212a and 212b, and a dynamic attribute 214, including dynamic attributes 214a and 214b. A data entity may have any suitable number of static attributes (including zero) and any suitable number of dynamic attributes (including zero), and aspects of the technology described herein are not limited in this respect.

[0147] 2C shows another illustrative diagram of a "BizTerm" data entity 220 with static attributes 222 ("Name" and "Description") and dynamic attributes 224 ("Legacy URL" and "Legacy Modified Date"). Attributes 225 are referential attributes in that they reference other data entities. For example, the BizTerm Group attribute of data entity 220 references a BizTerm Group data entity, the PII Classification attribute points to a PII Classification data entity, and the Data Element attribute points to a collection of DataElem data entities.

[0148] 2D is a table 230 illustrating properties of some of the attributes of the data entity shown in FIG. 2C, including whether the attributes are static or dynamic, in accordance with some embodiments of the techniques described herein. In particular, table 230 shows that the "Name," "Description," and "Legacy URL" attributes have string values, while the "Legacy Modification Date" takes a value of type Date. Table 230 shows that "Name" and "Description" are static variables, while "Legacy URL" and "Legacy Modification Date" are dynamic variables.

[0149] In some embodiments, static and dynamic attributes may be stored differently by a data processing system. For example, the values of static attributes may be stored in columns of one or more database tables. Conversely, the values of dynamic attributes may be stored as name-value pairs and thus stored in separate rows of one or more database tables.

[0150] FIG. 2E illustrates how static and dynamic attribute values may be stored by a data management system, according to some embodiments of the techniques described herein. In some embodiments, to store static attribute values, each entity instance may be represented as a row in a database table (or multiple database tables), and each static attribute of the instance may be represented as a column in the table. For example, as shown in FIG. 2E, static attribute values for an instance of data entity 220 are stored in static attribute storage 235 (which may include one or more tables), specifically table 240. Table 240 includes a column for each static attribute, the rows of which correspond to different instances of "BizTerm" data entity 220. Thus, the table has a column called "Name" for storing values of the "Name" attribute of different instances of data entity 220 (e.g., instances 1001 and 1002 in this example), and a column called "Description" for storing values of the "Description" attribute of different instances of data entity 220. In some embodiments, table 240 may be defined by a database administrator using a data definition language.

[0151] On the other hand, in some embodiments, dynamic attribute values may be stored using name-value pairs. In particular, each database record may store the attribute values for a particular entity instance. For example, as shown in FIG. 2E, dynamic attribute values for an instance of data entity 220 are stored in dynamic storage 245 (which may include one or more tables) using tables 250 and 260. In this example, table 250 assigns each dynamic attribute a unique identifier, and table 260 stores a row for each dynamic attribute value—the dynamic attribute value is stored together with the attribute's unique identifier (e.g., 1, 2, ... in this example) and the data entity instance identifier (e.g., 1001, 1002, ... in this example).

[0152] Another example of a data entity instance 265 is shown in FIG. 2F, which is a screenshot of an exemplary graphical user interface showing a visualization of the data entity instance and values of at least some of its attributes. Data entity instance 265 is an instance of the "BizTerm" data entity. In this example, data entity instance 265 is named "Credit Score." Data entity instance 265 has a number of static attributes (e.g., "Name," "Definition," and "Type") and dynamic attributes. In this example, the dynamic attributes are grouped into attributes for storing various types of information related to data governance in a banking application. For example, data entity instance 265 has governance-related dynamic attributes including "Business Owner," "Governance Group," "Risk Data Domain," "Business Line," "Custodian," "Subject Matter Expert," etc. As another example, the "Credit Score" data entity instance has privacy- and security-related attributes including "Confidentiality," "PII Classification," and "Security Coverage." The screenshot of FIG. 2F shows example values for some of these attributes of data entity instance 265.

[0153] As can be seen from the above examples, attributes can be considered to be of different types depending on the type of value they take. Examples of different types of attributes are described below by way of example and not limitation. For example, an attribute can take a scalar value such as a number, a string, a date, a time, or a datetime. Attributes of this type can be referred to as "extended attributes" in some of the examples described herein. In some embodiments, an attribute can have multiple values and take a set of values, each of which can be a number, a string, a date, a time, or a datetime.

[0154] As another example, an attribute may be discrete, thereby taking on values selected from a discrete set of values. Such a set may be referred to as an "enumeration." For example, an attribute may be of type "classification," whereby the values are labels selected from a set of labels. This allows tagging of data with desired labels. As one specific example, certain data may be classified as or containing personally identifiable information (PII), using values such as "yes" or "no." In the example of FIG. 2F, the PII classification dynamic attribute is of type "classification," as in the attribute "governance group." As another example, an attribute may take on values that represent an individual user or a group of users. In the example of FIG. 2F, the "business owner" attribute may be of type "accountable party," and its values may represent a user or a group of users. In the example of FIG. 2F, "business owner" takes on the value "Wade L. Register." In some embodiments, one or more (e.g., all) of the values in the desired set of values may be references to another data entity instance. All of these example attributes are discrete.

[0155] As another example of a discrete attribute, an attribute may take on values within an ordered set of values. For example, the values within the ordered set may be ordered according to a hierarchy. As one specific example, an attribute may store values related to geographic regions, and the values may be selected from a hierarchical list of values (e.g., "United States," "Delaware," or "Wilmington"). An attribute taking on values within an ordered set of values may be of type "Hierarchy." In the example of FIG. 2F, the "Risk Data Domain," "Business Line," and "Region" attributes are dynamic attributes of type "Hierarchy." When accessing the value of an attribute of type "Hierarchy," some embodiments may return the specific value that the variable is taking on along with other related values. For example, as shown in FIG. 2F, the value for "Risk Data Domain" is "Credit Risk," which is displayed in FIG. 2F along with the related value "Corporate Risk," which is a value related to "Credit Risk" in the hierarchy of values for "Risk Data Domain" because "Credit Risk" is of type "Corporate Risk." In some embodiments, one or more (e.g., all) of the values within the ordered set of values may be references to another data entity instance.

[0156] As another example, an attribute may be a reference attribute, whose value may be a reference to another data entity (e.g., a reference to an instance of another data entity). In some embodiments, an attribute may have multiple values, taking on a set of values, each of which may be a reference to a data entity.

[0157] As another example, in some embodiments, an attribute may be of type "file attachment," and its value may identify a file of interest (e.g., a document, a report, a configuration file, a spreadsheet, etc.) In the example of Figure 2F, the "design document" attribute is of type "attachment."

[0158] In some embodiments, values of dynamic attributes of the same type may be stored in the same underlying relational database table, thereby allowing optimizations when retrieving values of these attributes. For example, as described below in the section entitled "Optimizing Enrichment Query Execution," a single executable SQL query can be used to retrieve values of multiple dynamic variables of the same type.

[0159] As can be appreciated from the above, storing a data entity instance includes storing many different types of data because an entity instance may have different types of attributes. Furthermore, a data entity instance may have static and dynamic variables, and variables may be stored in different ways by data processing systems. Thus, a single data entity instance may be stored using rows and / or columns of one or more tables in a database system (e.g., a relational database system, an object-oriented database system, etc.). The tables may be stored in one or more computer-readable storage media.

[0160] As described above with reference to entity model 160 of data processing system 105, a data processing system may be configured with data entities that relate to each other through the use of reference attributes, whereby an attribute of one data entity may reference another data entity. A configuration of a data processing system may include entity model 160, which includes data entity definitions 162 and information specifying relationships 164 between the data entities. Aspects of such an entity model may be visualized in an entity relationship diagram (ERD), such as the illustrated ERD shown in FIG. 2G, which is a diagram 270 illustrating relationships between data entities according to an example entity model that may be configured in a data processing system, according to some embodiments of the technology described herein.

[0161] In the example of FIG. 2G , the “BizTerm” data entity 272 has the attributes “Name,” “Description,” “PII Classification,” “Legacy URL,” “Legacy Modification Date,” and “Design Document,” which can take on various values. One or more of these attributes may be static (e.g., “Name” and “Description”). One or more of these attributes may be dynamic (e.g., “PII Classification ID,” “Legacy URL,” “Legacy Modification Date,” etc.). The “BizTerm” data entity also has the reference attributes “BizTerm Group,” “PII Classification,” and “Data Element,” which reference other data entities—the BizTerm Group data entity 274, the PII Classification data entity 278, and the Data Element entity 276, respectively. The BizTerm Group attribute may reference one or more BizTerm Group data entity instances. The PII Classification attribute may reference zero or one PII Classification data entity instance. The Data Element attribute may point to zero, one, or more Data Element entities.

[0162] The Data Element entity 276 includes an attribute "Name" and two reference attributes "BizTerm" (which references the parent BizTerm) and "Dataset" which reference one or more instances of the Dataset entity 280. The Dataset entity 280 also includes a reference attribute "Application" which points to one or more instances of the Application data entity 282.

[0163] It should be understood that the entity model shown in Figure 2G is exemplary and non-limiting. Any suitable data entity model having any suitable data entities and relationships between the data entities may be configured in a data processing system, and aspects of the technology described herein are not limited to any particular entity model.

[0164] Finding information about data entity instances using declarative queries FIG. 3 is a flowchart of an example process 300 for generating query results in response to a query (e.g., a declarative query) for obtaining information about instances of one or more data entities, according to some embodiments of the techniques described herein.

[0165] Process 300 may be performed by any suitable computing device. Process 300 may be performed by any suitable software running on a data processing system (e.g., data processing system 105). For example, with reference to data processing system 105 shown in FIG. 1D , process 300 may be performed, at least in part, by using input transformation module 130 to perform operation 304 (including operations 304a and 304b) and output transformation module 160 to perform operation 308. As another example, data access module 140 may be configured to perform operation 306 in some embodiments.

[0166] In the example environment of FIG. 3 , process 300 begins at operation 302, where a query (e.g., a declarative query) is obtained. The query may include a first portion that identifies an instance of a data entity (the “identification portion” of the query) and a second portion that indicates at least one attribute of the data entity for which values are to be searched (the “enrichment portion” of the query). In some embodiments, the query may be specified using JSON. However, in other embodiments, the query may be specified in any other suitable format, and aspects of the technology described herein are not limited in this respect. Example queries (e.g., declarative queries) are provided herein.

[0167] The query may be obtained from any suitable source. For example, in some embodiments, the query may be obtained through one of the entity API interfaces 110 described with reference to FIG. 1D. As one example, the query may be written by a user in a graphical user interface. As another example, the query may be generated programmatically using a scripting language, a graph-based computer program, a third-party application, or any other suitable software.

[0168] After the query is obtained in operation 302, process 300 proceeds to operation 304, where one or more executable queries are generated from the query obtained in operation 302. In some embodiments, the executable queries may be executable SQL queries.

[0169] In some embodiments, operation 304 may be performed in two stages. First, in operation 304a, one or more executable queries (e.g., executable SQL queries) are generated from the identification portion of the query ("identification queries"). Next, in operation 304b, one or more executable queries (e.g., executable SQL queries) are generated from the enrichment portion of the query ("enrichment queries"). The executable identification queries and enrichment queries generated in operations 304a and 304b, respectively, may be generated in any suitable manner, including any of the methods described in the section below entitled "Generating Executable Queries." First, the identification portion of the query may be transformed through a series of transformations that augment the information in the first portion with information managed by the data processing system to generate a first set of one or more executable queries configured to be executed against a data store of the at least one data store used by the data processing system to store data entity instances, and the second, enrichment portion of the query may be transformed through a series of transformations that augment the information in the second portion with information managed by the data processing system to generate a second set of one or more executable queries configured to be executed against a data store of the at least one data store used by the data processing system to store attribute values.

[0170] In some embodiments, in operation 304a, the identified portion of the query may be used to generate a single executable (SQL) query according to process 700 described herein with reference to Figures 7A-7F.

[0171] In some embodiments, operation 304b may generate one or more executable SQL queries using the enrichment portion of the query according to process 700 described herein with reference to Figures 7A-7F. Whether multiple executable SQL queries are generated using the enrichment portion depends on the number of attributes identified in the enrichment portion. If only one attribute is identified, one executable SQL query may be sufficient. However, in some embodiments, if multiple attributes are identified in the enrichment portion, the enrichment portion may be used to generate multiple executable SQL queries.

[0172] In particular, in some embodiments, when multiple attributes are identified in the enrichment portion of a query, the attributes may be grouped into groups, with one executable SQL generated for each group of attributes. In some embodiments, the attributes may be divided into groups as follows: (1) static attributes of a data entity may be grouped into a single group; (2) dynamic attributes may be grouped into one or more groups based on their type and / or the way they are stored in the entity model (e.g., extended attributes, classifications, hierarchies, accountable parties, and attachments in the example entity model shown in FIG. 2G); and (3) one group for each referential dynamic attribute (i.e., a dynamic attribute that references another data entity instance) or inline view in the enrichment portion. Aspects of grouping dynamic attributes into groups are described herein, including below in the section entitled "Accessing Attribute Values for a Group of Attributes."

[0173] In some embodiments, for each group of static or dynamic attributes, a respective executable query (e.g., an executable SQL query) may be generated using process 700 described with reference to FIG. 7A. For any reference attributes that reference instances of a second data entity, the same overall approach may be applied recursively: (1) the required static and dynamic attributes of the other data entity instances are grouped, (2) an executable SQL query is generated for each generated group (e.g., using process 700), and (3) if the second data entity instance includes a reference attribute that references an instance of a third data entity, (3a) its required static and dynamic variables are grouped, and (3b) an executable SQL query is generated for each generated group (e.g., using process 700). If the third data entity instance has a reference attribute that references an instance of a fourth data entity, the recursive grouping of static and dynamic attributes into groups and generating executable queries for each group may continue.

[0174] In some embodiments, the enrichment portion of a query may include an inline view that includes another query whose results are a list of instances of one or more data entities. attribute value or a declarative entity query that includes a declarative table query (aspects of declarative table queries are described herein, including in the section below entitled "Additional Aspects of the Entity API"). For example, the enrichment portion of declarative query 920 includes two inline views: inline view 922 that includes a declarative entity query and inline view 924 that includes a declarative table query. In this manner, declarative queries may be nested. If the enrichment portion includes inline views, the declarative query in the inline views may be processed in a similar manner as described above, i.e., grouping static and dynamic attributes and generating respective executable (e.g., SQL) queries for each group, with further recursive processing applied to any referenced data entity queries or further nested inline views.

[0175] After the executable instances and enrichment queries are generated in operation 304, process 300 proceeds to operation 306, where the executable queries are executed to obtain query results, which may be combined and output in operation 308.

[0176] In some embodiments, performing operation 306 includes first executing the identification query generated in operation 304a to obtain information identifying one or more data entity instances. This information is stored in one or more tables and / or variables. After the identification query is executed, operation 306 may include executing the enrichment query generated in operation 304b to obtain attribute values of the identified data entity instances. Thus, the enrichment query generated in operation 304b may reference the information identifying the data entity instances obtained by executing the identification query generated in operation 304a. The reference may be to tables and / or variables that store the information, data entity instances.

[0177] Also, as described herein, in some embodiments, in a variation of the illustrated process 300, the enrichment queries may be generated after the identification query is executed. In such embodiments, the process may proceed by (1) generating one or more executable identification queries from the identification portion of the declarative query, (2) executing the one or more identification queries to obtain information identifying one or more data entity instances, (3) generating one or more executable enrichment queries (optionally using the information identifying the data entity instances) (obtained in the second step), and (4) executing the one or more enrichment queries to obtain attribute values of the identified data entity instances. Thus, it should be understood that the enrichment queries may be generated before or after the identification query is executed, and aspects of the technology described herein are not limited in this respect.

[0178] In some embodiments, the query results obtained in operation 306 may include (1) an identifier for each data entity instance identified by executing the executable query obtained using the identification portion of the query, and (2) values of attributes obtained by executing one or more executable queries obtained using the enrichment portion of the query. Such query results may be combined by placing the data entity instance identifiers in a table and, because attribute values are also associated with the identifiers of each entity instance, placing the dynamic attribute values in the same table (which, in effect, performs a pivot operation, as described below). Aspects of combining query results for generated executable queries are further described herein, including in sections entitled "Declarative Queries: Identification and Enrichment" and "Optimizing the Execution of Enriched Queries."

[0179] The combined query results may be output in any of a number of ways. For example, in some embodiments, the results may be stored in memory or one or more other suitable computer-readable storage media, allowing the results to be subsequently accessed. In some embodiments, if the query was received from a software program (e.g., a script, a graph-based computer program, third-party software, etc.), the results may be provided to the software program. In some embodiments, if the query was entered by a user via a graphical user interface, the results may be displayed to the user through the graphical user interface.

[0180] The combined query results may be output in any suitable format. For example, the query results may be output in text format, JSON, or any other suitable form. In some embodiments, the query results may be formatted for hierarchical presentation (e.g., using tabbing and / or nesting of text, using nested containers in a graphical user interface, using nested tags in a markup language, etc.). In some embodiments, the hierarchical structure of the results may match the hierarchical structure of the query (e.g., the hierarchical structure of the attributes identified in the enrichment portion of a declarative query). Aspects of hierarchical querying and presentation of results are described herein, including with reference to Figures 4D-4F, 8A-8E, and 9A-9G.

[0181] Declarative Queries: Identification and Enrichment 4A is a diagram illustrating an example declarative query 402 including an identification portion 404 and an enrichment portion 406, in accordance with some embodiments of the technology described herein. The declarative query 402 is specified in JSON format in this example. However, it should be understood that declarative queries may be used in any other suitable format, and aspects of the technology described herein are not limited to using only the JSON format to specify declarative queries.

[0182] As described herein, the identification portion of a declarative query includes information for identifying one or more data entity instances, which may be stored by a data processing system (e.g., using data persistence layer 150 described with reference to FIG. 1D ). The enrichment portion of a declarative query includes information identifying one or more attributes of one or more data entities for which values should be obtained (for the identified data entity instances) in response to processing the declarative query. In this manner, processing a declarative query includes (1) processing the identification portion to select a data entity instance, and (2) processing the enrichment portion to obtain attribute values of the selected data entity instances. If the value of the attribute is a reference to one or more other data entity instances, processing the enrichment portion may further include using the reference value to obtain the value of the attribute of the other data entity instance or to obtain the value of the attribute of yet another data entity instance related to the other data entity instance, as described herein.

[0183] In the example of Figure 4A, the declarative query 402 is to retrieve information about any data entity instances whose "Entity Class" is "BizTerm." This is indicated in the second line of the identification section 404, as shown in Figure 4A. The enrichment section 406 also specifies that for each data entity instance identified using the identification section 404, values for the "Name" and "Description" attributes should be obtained.

[0184] 4B illustrates an exemplary query result 422 generated by processing the declarative query 402 of FIG. 4A . In this example, the query result 422 generated in response to processing the declarative query 402 includes information about a single data entity instance. (This is a condensed example for clarity; in practice, information about many data entity instances may be returned.) The information about the single data entity instance includes (1) identifying information about the data entity instance and (2) values of the (single instance of) data entity identified in the enrichment portion 406 of the declarative query 402 (i.e., values of the attributes “Name” and “Description,” which in this example are “Primary Account Number” and a null value, respectively). If additional attributes were identified in the enrichment portion 406, values of those attributes would also be included in the query result 422.

[0185] Also, as shown in FIG. 4B , the identification information for the data entity instance (portion of the query result 422) includes multiple fields with corresponding values, including, for example, a field for indicating the entity class of the data entity instance ("Entity Class") with a corresponding value "BizTerm" (consistent with the identification portion 404 of the declarative query 402), a field for indicating the internal name of the data entity ("Distinguished Name") with a corresponding value "Primary Account Number", a field for indicating the name of the data entity instance to be used for display ("Display Name") with a corresponding value "Primary Account Number", a field for indicating the entity type ("Entity Type") with a corresponding value "Business Team", and a field for indicating the alphanumeric (or numeric or alphabetic) identifier of the data entity instance ("Composite ID") with a corresponding value "55.1171".

[0186] As can be seen from Figures 4A and 4B, declarative query 402 is a query because it is a declarative statement for retrieving information about data entity instances managed by a data processing system. A declarative query for obtaining information about data entities can be referred to as an "entity query" (see, e.g., line 1 of identification portion 404). As described herein, embodiments of the technology described herein are not limited to simply retrieving information about data entities (e.g., data entity instances, data entity definitions, associated configuration information, etc.), but can also be used to update, delete, or create such information in a data processing system. In such cases, declarative statements can be used to perform such actions. Examples of such declarative statements (a declarative query is one example of a declarative statement) and descriptions of how such statements are processed by a data processing system are described herein, including in the section below entitled "Additional Entity Operations."

[0187] In the simple example of FIG. 4A , the identification portion 404 specifies that instances of data entities having a particular class should be identified. However, the identification portion may specify which data entity instances should be selected in other, potentially more complex, ways. For example, the identification portion may specify that data entity instances having a particular attribute value should be selected. As one specific example, the identification portion 411 of the declarative query 410 shown in FIG. 4C indicates at line 412 that information should be retrieved about data entity instances having a class "BizTerm" and whose "name" attribute is a string containing the value "account." As another example, the identification portion may specify that data entity instances related to other data entities having a particular attribute value should be selected.

[0188] More generally, in some embodiments, the conditions for selecting a data entity instance may depend on any data stored and / or accessible by the data processing system, including any data associated with the data entity instance itself, global parameters and / or variables, the configuration of the data processing system, calculated environment variable values, and / or any other suitable data stored and / or accessible by the data processing system. In some embodiments, one or more such conditions may be explicitly stated (e.g., using JSON) in the identification portion of the declarative query, examples of which are provided herein. In some embodiments, one or more such conditions may be imposed by the data processing system when processing the declarative query. For example, the data processing system may provide user information about only those data entity instances that the user has permission to access. Such conditions may, in some embodiments, be automatically imposed without having to be explicitly written as part of the declarative query.

[0189] Thus, in some embodiments, the condition for selecting a data entity instance depends on data associated with the data entity instance (e.g., attribute values of the data entity instance, attribute values of related data entity instances, a definition of the data entity, permissions associated with access to the data entity, the version of the data stored in the data, etc.). Additionally or alternatively, the condition for selecting a data entity instance may depend on other information. For example, the condition may specify a numerical limit on the number of results to return (e.g., see line 414 of declarative query 410 shown in FIG. 4C , which limits the number of results returned to 5,000). A limit on the number of results returned can also be set in an enrichment portion of a declarative query (e.g., see enrichment portion 415 shown in FIG. 4C , which limits the number of results returned in response to a “select” statement to 500). As another example, the condition may specify that only data entity instances with a relevance to which a user of the data processing system (e.g., the logged-in user) has access are to be selected.

[0190] As yet another example, the conditions may specify selecting data entity instances based on their history (e.g., when they were created, when they were last updated, whether they were updated within a specified time period, whether they were created and / or edited by a particular user or group of users, etc.).

[0191] As yet another example, a condition may specify selecting a data entity instance based on a workflow status. In some embodiments, a data processing system may be configured to manage changes to a data entity instance using a workflow management system such that at least some changes made by a user to any data (e.g., attribute values) associated with the data entity instance must be approved by one or more other users. The workflow management system may track which changes need to be approved and which changes have been approved. Thus, a data entity instance may be associated with a workflow state in which all previously submitted changes have been approved (sometimes referred to as a "published" or "approved" workflow state), a workflow state in which one or more previously submitted changes have not yet been approved (sometimes referred to as a "pending" or "pending approval" workflow state), or possibly any other workflow state. More complex workflow management systems may have other states depending on the manner in which change management is implemented. Aspects of how a workflow management system may be implemented are described in U.S. Patent Application Publication No. 2020 / 0234242, filed January 22, 2020, and entitled "Finite State Machines for Implementing Workflows for Data Objects Managed by a Data Processing System," which is incorporated herein by reference in its entirety.

[0192] However, regardless of the details of how the workflow management system is implemented, in some embodiments, data entities may be associated with corresponding workflow statuses (e.g., workflow states), and conditions for identifying data entity instances (e.g., explicitly stated in the identification portion of the declarative query and / or implicitly applied by the data processing system when processing the declarative query) may specify selecting data entity instances based on their workflow states. For example, the conditions may specify selecting data entity instances having a particular workflow state (e.g., “published,” “pending approval,” etc.), selecting data entity instances that have been updated within a threshold time period or within a specified range (e.g., moved from the “pending approval” state to the “approved” state), or selecting data entity instances based on workflow status in any other suitable manner; aspects of the technology described herein are not limited in this respect. For example, as shown in FIG. 4C , declarative query 410 uses a “query view type” parameter 418 to indicate that only “published” data entity instances should be selected when identification portion 411 is processed (and thus any data entity instances associated with changes for which approval is still pending will not be selected).

[0193] In some embodiments, a data entity instance may be "versioned" in that when a change is made to a data entity instance, the changed data entity instance is stored as a new version of that data entity instance, and the previous version of that data entity instance is persisted. In some such embodiments, the identification portion of the declarative query may be used to select the latest version of the data entity, one or more previous versions of the data entity that are persisted by the data processing system, a value indicative of changes between the current version and one or more previous versions of the data entity, and / or any other suitable information about the current and / or persisted data entity instance.

[0194] As can be seen from the above, there is a rich set of conditions that can be included as part of the identification portion of a declarative query to indicate which data entity instances should be selected. The techniques developed by the inventors provide various ways of expressing such conditions. For example, in some embodiments, the identification portion of a declarative query may specify the data entity instances to be selected using a Structured Query Language (SQL) query. As another example, in some embodiments, the identification portion of a declarative query may specify the data entity instances to be selected using a so-called "filter expression" syntax. As yet another example, in some embodiments, the identification portion of a declarative query may specify the data entity instances to be selected using alphanumeric (or numeric or alphabetic) identifiers associated with the data entity instances. This third option may be applicable if those alphanumeric identifiers are already accessible (e.g., after creating or updating the data entity instances). As yet another example, as described below in the section entitled "Searching Data Using One or More Search Indexes," in some embodiments in which a data processing system uses one or more search indexes to access data entity instances, the identification portion of a declarative query may specify the data instances to be selected using a search string.

[0195] In some embodiments, the identification portion of a declarative query may specify a data entity instance using an SQL query. For example, the SQL query may use any suitable syntax and / or commands that conform to the ANSI SQL standard (e.g., any past, current, or future version). In some embodiments, the SQL query may conform to any vendor-specific SQL implementation that is consistent with the ANSI SQL standard (e.g., an implementation from MySQL, Oracle, PostgreSQL, Teradata, etc.). In some embodiments, the SQL query may use a strict subset (i.e., some, but not all, syntax and / or commands) of the ANSI SQL standard. The SQL query in Table 1 is an example of an SQL query that can be included in the identification portion of a declarative query.

[0196] [Table 1]

[0197] Additional examples of SQL queries used in the identification portion of a declarative query are provided herein and included in Figures 4D, 9C, and 9F-1 and 9F-2.

[0198] In some embodiments, the identification portion of a declarative query may specify the data entity instances to select using so-called "filter expression" syntax. In some embodiments, the filter expression may include an SQL "WHERE" clause. Furthermore, unlike SQL queries that conform to the ANSI SQL standard, the filter expression may include a so-called "entity model" path, which allows values to be accessed using strings that conform to the entity model using dot notation, as sometimes used in object-oriented programming languages. For example, dot notation may be used to traverse data entity relationships within a data entity model (e.g., according to relationships in the ERD shown in FIG. 2G). An example of selecting data entity instances using filter expression syntax is shown in Table 2 below. This filter expression is used to select instances of "BizTerm" data entities whose names begin with the letter "A" and belong to a business term group called "Corporate Business Glossary." In this example, "BizTermGroup.Name" is the entity model path.

[0199] [Table 2]

[0200] Additional examples of filter expressions used in the identification portion of a declarative query are provided herein, including in Figures 4A, 4C, 5A, and 8B.

[0201] As described herein, data entities may be related to one another. For example, attributes of a data entity instance may themselves reference other data entity instances (e.g., the value of a data attribute may be a reference to another data entity instance). One or more of those other data entity instances may also have attributes that reference other data entity instances, and so on. As a result, a data entity instance may have an associated hierarchy of attributes, and in some embodiments, a single declarative query may be used to retrieve values for some or all of the attributes in such a hierarchy. In this sense, the declarative query may be considered hierarchical.

[0202] Thus, in some embodiments, a declarative query may cycle through related entities and their attributes, allowing a large number of related data to be searched in a single request. To this end, in some embodiments, the enrichment portion of a declarative query includes information that identifies one or more attributes of related data entities of interest, such that values may be obtained by processing the declarative query (e.g., of data entity instances identified using the identification portion of the same declarative query).

[0203] The enrichment portion may indicate the attributes of interest in any of a number of ways. For example, in some embodiments, the indication may be made explicit in the enrichment portion of the declarative query, e.g., using nesting as described herein, including with reference to Figures 4D-4F and 9A-9G. As another example, in some embodiments, the indication may be made by not explicitly stating the hierarchy of attributes in the declarative query, but instead relying on recursion as described herein, including with reference to Figures 8A-8C.

[0204] An example of a hierarchical declarative query is shown in Figure 4D, which is a diagram of a declarative query 420 that specifies a hierarchy of attributes whose values should be returned in response to processing the declarative query, according to some embodiments of the technology described herein. Declarative query 420 includes an identification portion 421 that includes an SQL query to select instances of the data entity "technology group" whose "name" attribute takes the value "cust_dwh".

[0205] Declarative query 420 also includes an enrichment portion 422 that uses nesting to specify a hierarchy of attributes whose values should be searched for. In this example, an instance of a "technology group" data entity may reference a list of one or more "dataset" data entity instances, and enrichment portion 422 indicates that for each identified "technology group" data entity instance (referenced by the particular "technology group" data entity instance), the identity and name of each particular dataset data entity instance should be searched for.

[0206] FIG. 4E illustrates an example result 430 returned in response to processing the declarative query 420 shown in FIG. 4D. In particular, result 430 includes information about nested entity instances. In particular, result 430 includes information about a single instance of a "Technology Group" data entity, including (1) identifying information 431, (2) the value of its name attribute 432, and (3) information about two "Dataset" data entity instances referenced by the single instance of the "Technology Group" data entity. The information about the two "Dataset" instances is shown in nested results 434 and 436. For each "Dataset" instance, the returned information includes identifying information about the instance (e.g., entity class, distinguished name, display name, entity type, and composite ID value) and a "Name" attribute value. In this example, the values of the "Name" attribute in results 434 and 436 are "CUSTOMER_INFO" and "DEPOSIT_ACCOUNT," respectively.

[0207] In some embodiments, results generated by processing a declarative query may be formatted in a hierarchical format corresponding to the hierarchy of attributes specified in the declarative query. For example, as can be seen from the examples of FIGS. 4D and 4E , the hierarchical structure of results 430 shown in FIG. 4E matches (and thus corresponds to) the nested hierarchical structure shown in enrichment portion 422 of declarative query 420 in FIG. 4D . In this manner, declarative queries are used not only to retrieve data from one or more data stores (e.g., from data persistence layer 150), but also to format the data appropriately for output. For example, data retrieved by processing a declarative query may be stored in one or more tables in a relational database. However, after being retrieved, the hierarchical structure of the declarative query may be used (e.g., by hierarchical output module 167 described with reference to FIG. 1D ) to reorganize and output the data in a hierarchical format consistent with the hierarchical format of the enrichment portion of the declarative query.

[0208] In some embodiments, it may be desirable not to return all related data entity instances and associated values. For example, there may simply be too many related data entity instances. In this case, the results may be limited by imposing a numerical limit on the number of results returned (e.g., as shown in FIG. 4C ). As another example, the results may be filtered by using a filter expression or SQL statement within a nested structure in the enrichment portion of the declarative query. For example, as shown in FIG. 4F , the declarative query 420 of FIG. 4D may be augmented with a filter expression 442 to reduce the number of “Dataset” entity instances by returning only those instances where Application.Name='Feed' to obtain declarative query 440.

[0209] As described herein, a declarative query includes an identification portion and an enrichment portion, and to process a declarative query, (1) the identification portion is translated into one or more executable SQL queries (sometimes referred to herein as an “identification query” or “identification queries”), (2) the enrichment portion is translated into one or more executable SQL queries, (3) all executable SQL queries are executed to obtain respective results (sometimes referred to herein as an “enrichment query” or “enrichment queries”), and the results are (4) combined and, optionally, formatted (e.g., hierarchically) to generate a response to the declarative query. Figures 5A-5F illustrate some aspects of this process.

[0210] 5A illustrates an example declarative query 502 including an identification portion and an enrichment portion, in accordance with some embodiments of the techniques described herein. The declarative query 502 includes an identification portion 504 and an enrichment portion 506. The identification portion 504 indicates that, when the identification portion 504 is processed, instances of "BizTerm" data entities with names beginning with "A" should be identified. The enrichment portion 506 indicates that, for each data entity instance identified using the identification portion 504, information about the "name," "description," "business term type," and "data element" attributes should be retrieved. In this example, "name" and "description" are static attributes, and "business term type" and "data element" are reference attributes.

[0211] 5B illustrates executable SQL queries that may be generated by a data processing system by processing the declarative query 502 shown in FIG. 5A in accordance with some embodiments of the techniques described herein. In particular, as shown in FIG. 5B, the identification portion 504 may be processed to generate an executable SQL query 510 for identifying “BizTerm” data entity instances having names starting with “A.” Separately, the enrichment portion 506 may be processed to generate (1) an executable SQL query 512 for obtaining the value of the name attribute, (2) an executable SQL query 514 for obtaining the value of the description attribute, (3) an executable SQL query 516 for obtaining the value of the “business term type” attribute, and (4) an executable SQL query 518 for obtaining the value of the “data element” attribute. The results obtained by executing the SQL query 510 may include (e.g., numeric) identifiers of the identified data entity instances, and these identifiers may be used to combine the results of executing all of the executable SQL queries 510, 512, 514, 516, and 518. As described herein, in some embodiments, the executable SQL query generated using the enrichment portion may be executed (and optionally generated) using the results of executing an executable SQL query (e.g., query 510 in this example) to identify data entity instances of interest (in this example, data entity instances having names starting with "A").

[0212] In this example, the value of each attribute is obtained using a single executable SQL query; however, in some embodiments, any other suitable combination of two or more SQL queries or executable SQL statements may be used to obtain one or more values of an attribute, and it should be understood that aspects of the technology described herein are not limited in this respect.

[0213] It should also be appreciated that in some embodiments, a single executable SQL query may be used to retrieve values for multiple different attributes. This may provide efficiencies in processing declarative queries (e.g., when different attribute values are stored in the same table, are of the same type, are stored in the same storage medium, etc.), thereby reducing the amount of computational resources (e.g., processor cycles, memory, network bandwidth) used to process the query. Examples of using a single executable SQL query to retrieve values for multiple different attributes are provided herein, including the discussion with reference to Figures 5D and 6A-6F.

[0214] Returning to the example of Figures 5A and 5B, and to provide further detail about this example, consider Figure 5C, which shows an illustrative example of an executable SQL query 510 generated using the identification portion 504 of the declarative query 502. The executable SQL query 510 may be generated in any suitable manner, including, for example, as described herein, including with reference to Figures 7A-7C. The executable SQL query 510 may be an executable, database-specific SQL query for retrieving data from a particular type of relational database. The results of executing this example query are shown in table 520, which shows the identifiers of the "BizTerm" data entity instances (i.e., 1024, 1043, 1053, 1087, 1103, 1008, 1197) identified by processing the executable SQL query 510.

[0215] Figure 5D shows an illustrative example of an executable SQL query 530 generated using the enrichment portion 506 of the declarative query 502 that, when executed, retrieves the values of the "name" and "description" attributes of each of the "BizTerm" data entity instances identified using the executable query 510 and having the identifiers shown in Figure 5C. The generated values are shown in table 532, which has a row for each data entity instance identifier and a column for each of these two attributes. In the executable SQL query 530, "Tqt.BizTermID0" refers to the "BizTerm" data entity instance retrieved as a result of executing the SQL query 510 shown in Figure 5C.

[0216] Although separate executable SQL queries 512 and 514 could be used to obtain the values of the "Name" and "Description" attributes, in the example of FIG. 5D a single SQL query 530 is used because, in the example, values for both of these attributes are stored in the same table. As described herein, including below with reference to FIGS. 6A-6F, in some embodiments where attribute values are stored in the same table (or there are other efficiencies gained from using a single query rather than multiple queries to access data), a single executable SQL query may be used to access the attribute values instead of multiple SQL queries. However, separate queries could be used as well, and examples of such queries are shown in Tables 3 and 4 below.

[0217] [Table 3]

[0218] [Table 4]

[0219] 5E shows an illustrative example of an executable SQL query 516 generated using the enrichment portion 506 of the declarative query 502 that, when executed, retrieves the values of the business term type attributes for each of the "BizTerm" data entity instances identified using the executable query 510 and having the identifiers shown in FIG. 5C. The results of executing the executable SQL query 516 are shown in table 536. Table 536 includes a row for each entity instance identifier shown in FIG. 5C. The business term type attributes reference other data entity instances, and the values of at least some of their attributes are shown in columns of table 536.

[0220] Figure 5F shows an illustrative example of an executable SQL query 518 generated using the enrichment portion 506 of the declarative query 502 that, when executed, retrieves the values of the data element attributes of each of the "BizTerm" data entity instances identified using the executable query 510 and having the identifiers shown in Figure 5C. The results of executing the executable query 518 are shown in table 538. Table 538 includes a row for each entity instance identifier shown in Figure 5C. The data element attributes reference other data entity instances, and the values of at least some of the attributes are shown in columns of table 538.

[0221] 5C-5F , in some embodiments, each piece of data requested in the enrichment portion 506 is associated with a respective identifier that indicates the associated data entity instance, and thus, data resulting from processing the various executable SQL queries may be combined using the data entity instance identifier 520. The data may be combined in any suitable manner. For example, the data may be combined into a single table. As another example, the data may be formatted for a hierarchical presentation. As yet another example, the data may be written to a file or other database system in any suitable format, and aspects of the technology described herein are not limited in this respect.

[0222] Optimizing enrichment query execution Temp Tables and Common Table Expressions In some embodiments, the results generated by executing the identification and enrichment SQL queries may be combined using a table. For example, in some embodiments, the data entity instance identifiers (e.g., data instance identifiers 520 described above) obtained by executing the identification SQL query may be inserted into a temporary table, and the temporary table may then be joined with at least some (e.g., all) of the data obtained using the enrichment SQL query. Additionally, the data in the generated table (after the join) may be formatted in any suitable manner and output in any of the methods described herein.

[0223] In some embodiments, when accessing information about only a single data entity and its direct properties (e.g., information about data entities other than those referenced by the single data entity is not requested), instead of using separate identification and enrichment queries, a common table expression may be used to obtain the requested data and the results may be combined using a temporary table. An example of such a common table expression is shown below in Table 5.

[0224] [Table 5]

[0225] pivot As described above, after the different executable SQL queries generated from the enrichment portion of the declarative query are processed to obtain respective results, these results can be combined. Recall that different executable enrichment queries can be used to access values stored in different database tables. A conventional approach to accessing values stored in different tables and then combining those values is to (1) join multiple tables in a database (e.g., by joining the tables using an SQL "join" command) to obtain the joined table, and (2) read the desired portion of the joined table. However, the inventors have recognized that such an approach is highly inefficient (in terms of processing and memory requirements) in the present context because data entity instances have many dynamic attributes, many of which are stored in different tables. As a result, processing many declarative queries requires joining dozens (e.g., at least 10, at least 20, at least 30, 10 to 50) of tables each time the query is executed, which is highly inefficient. As an example, it is not uncommon for a data entity to have at least 20 dynamic attributes, which would require performing at least as many joins as the values of the dynamic attributes are joined with a database that stores a table of dynamic attribute values.

[0226] An example of the complexity of performing a pivot using SQL is shown in Table 5 below. Table 5 contains SQL statements for accessing variables for multiple attributes (in this example, of type "extended attribute"). Each attribute requires a left outer join, so the SQL for only two such attributes is shown in Table 6 below. Generating and executing such code is highly inefficient, as it requires building numerous intermediate tables in the database that store the attribute values. This takes a long time (and therefore the query takes longer to run) and consumes a significant amount of memory.

[0227] [Table 6]

[0228] On the other hand, processing the enrichment part of the declarative query by generating and executing several different executable SQL queries with different attributes allows combining attribute values after they have been accessed from the database storing them, which avoids the need to perform pivots in SQL, for example, by joining multiple tables in the database where the tables are stored.

[0229] Thus, in some embodiments, attribute values obtained by processing different executable SQL enrichment queries may be combined by processing these data after they are accessed from the databases in which they are persisted. In some embodiments, the combination can be performed by pivoting the data using software written in Java or any other programming language, without invoking any SQL statements (e.g., SQL commands to join tables) for processing by the database system from which the data was accessed. Simply put, the results can be combined externally from any SQL processing that would have been performed within the database system using traditional methods. When accessing values for multiple dynamic variables, the result of each query includes rows of name-value pairs, which can become columns in a table that stores attribute values for data entity instances.

[0230] 5G is a diagram illustrating pivoting of SQL query results without using SQL, according to some embodiments of the techniques described herein. As shown in FIG. 5G, results 542 and 544 obtained from running different enrichment queries may be combined by pivoting 546 using Java software or any other software program (but not SQL commands, and without joining tables storing attributes) to obtain result 548. In this example, the results obtained from each query include many rows, which the Java software turns into columns of entity instances, as shown in result 548.

[0231] Accessing attribute values for a group of attributes As described herein, the inventors have recognized that declarative queries may be processed more efficiently if fewer executable queries (e.g., executable SQL queries or any other suitable type of executable query) are used to obtain attribute values. For example, if particular attribute values are stored in a common table, it is more efficient to execute a single query against the same table to retrieve the attribute values rather than executing multiple queries against a database that stores the table. An example of this is described above with reference to FIG. 5C. Further examples are described below, including with reference to FIGS. 6A-6F.

[0232] Both Figures 6A and 6B show the enrichment portion 600 of a declarative query. In this example, the enrichment portion indicates that information about four attributes (Attributes #1-4) is to be retrieved. Figure 6A shows that a separate executable SQL query is generated for each of the four attributes. In particular, executable SQL queries 602, 604, 606, and 608 are generated to retrieve the values of Attributes #1, #2, #3, and #4, respectively. However, in this example, Attributes #2 and #3 are of the same type (Type B), meaning that it may be more efficient to access these attributes using a single query rather than multiple queries, for example, because the values of these attributes are stored in the same underlying table. For example, Attributes #2 and #3 may be both of type "extended attribute" or type "classification." These two attribute types, as well as other attribute types, are described herein, including in the discussion with reference to Figures 2A and 2B. As shown in Figure 6B, a single executable SQL query 605 may be generated and used to retrieve the values of Attributes #2 and #3, rather than separate executable SQL queries 604 and 606.

[0233] Thus, in some embodiments, processing the enrichment portion of the declarative query to generate executable SQL queries to obtain information about the attributes identified in the enrichment portion includes (1) grouping the attributes in the enrichment portion into groups and (2) generating a respective executable query for each of the plurality of groups.

[0234] In some embodiments, attributes having the same type may be grouped into a common group. For example, in some embodiments, dynamic attributes of type "extension attributes" (identified in the enrichment section) may be grouped together, and a single executable SQL query may be generated to obtain information about the attributes in this group. As another example, in some embodiments, dynamic attributes of type "classification" (identified in the enrichment section) may be grouped together, and a single executable SQL query may be generated to obtain information about the attributes in this group. As another example, in some embodiments, dynamic attributes of type "hierarchy" (identified in the enrichment section) may be grouped together, and a single executable SQL query may be generated to obtain information about the attributes in this group. As yet another example, in some embodiments, dynamic attributes of type "accountable party" (identified in the enrichment section) may be grouped together, and a single executable SQL query may be generated to obtain information about the attributes in this group.

[0235] For example, as shown in Figure 6C-1, a declarative query 610 includes an identification portion 612 and an enrichment portion 614 and indicates that information about various types of attributes should be accessed, including information about a "name" static attribute 620 and the dynamic attributes: "x Sensitivity," "x Group," "x Area," "x Business Line," "x Additional Definitions," "x Legacy System URL," "x SME," and "x Administrator." While several of these dynamic attributes are of the same type, Figure 6C-1 shows that a single executable SQL query is generated for each dynamic attribute.

[0236] However, in some embodiments, the data processing system may determine that some of these dynamic attributes are of the same type. For example, as shown in FIG. 6C-2, the data processing system may determine that attributes "x Confidentiality" and "x Group" have type "Classification" and may be grouped in group 622, attributes "x Area" and "x Business Line" are type "Hierarchy" and may be grouped in group 624, attributes "x Additional Definitions" and "x Legacy System URL" are type "Extended Attribute" and may be grouped in group 626, and attributes "x SME" and "x Administrator" are type "Accountable Party" and may be grouped in group 628. Also, a single executable SQL query may be generated for each group of dynamic attributes having the same type. A single executable SQL query may be generated for all static attributes (here, there is only one static attribute "Name" in group 620).

[0237] 6C-3, (1) a single executable SQL query 623 may be generated for a group 622 of attributes of type "Classification" that includes the "x Sensitivity" and "x Group" dynamic attributes, (2) a single executable SQL query 625 may be generated for a group 624 of attributes of type "Hierarchy" that includes the "x Area" and "x Business Line" dynamic attributes, (3) a single executable SQL query 627 may be generated for a group 626 of attributes of type "Extended Attributes" that includes the "x Additional Definitions" and "x Legacy System URL" dynamic attributes, and (4) a single executable SQL query 629 may be generated for a group 628 of attributes of type "Accountable Party" that includes the "x SME" and "x Administrator" dynamic attributes. In addition, a single executable SQL query 621 may be generated for a group 621 of static attributes.

[0238] As an example, FIG. 6D shows an exemplary single executable SQL query 630 for retrieving information about the “x Region” and “x Business Line” hierarchies—both hierarchies represented by respective identifiers 632 and 634 (hierarchy IDs) as shown in FIG. 6D—demonstrating that information about both of these dynamic attributes can be retrieved using the same executable SQL query.

[0239] As another example, in some embodiments, attributes whose values are stored in the same table may be grouped into a common group, and a single executable SQL query may be generated to retrieve the attributes in that group. As another example, in some embodiments, attributes whose values are stored on the same device and / or the same computer-readable storage medium may be grouped into a common group, and a single executable SQL query may be generated to retrieve the attributes in that group.

[0240] As yet another example, in some embodiments, dynamic variables that share a common hierarchical path may be grouped together, as will now be described in more detail with reference to Figures 6E and 6F.

[0241] In some embodiments, a dynamic attribute of a data entity may inherit its value from a corresponding attribute of another data entity. For example, as shown in FIG. 6E , data entity 650 may include static attribute 652 and dynamic attribute 654. Among dynamic attributes 654, attribute "A, B, C" 656 is configured to inherit its value from a corresponding dynamic attribute 664 of data entity 660, as indicated by common inheritance path 657. That is, attributes of a data entity or data entity instance share a common inheritance path if the attribute is configured to inherit its value from an attribute of another data entity or data entity instance of the same attribute. Attributes "X," "Y," and "Z" 658 are configured to inherit values for corresponding dynamic attribute 674 of data entity 670. Thus, in response to a query for the value of dynamic attribute 654 of data entity 650, the values of attribute 664 and attribute 674 need to be obtained. Because variables A, B, and C are inherited along a common inheritance path 657 (e.g., from the same data entity 660), a single executable SQL query may be used to access the inherited values of these variables. Query 682 in Figure 6F is an example of such a query. Similarly, because variables X, Y, and Z are inherited along a common inheritance path (e.g., from the same data entity 670), a single executable SQL query may be used to access the inherited values of these variables. Query 684 in Figure 6F is an example of such a query.

[0242] Batch and streaming query execution, limiting the scope of returned results As described herein, the number of data entity instances returned in response to processing a declarative query can be large. For example, the number of data entity instances that may be returned in response to processing a declarative query can be at least 100 entity instances, at least 500 entity instances, at least 1000 entity instances, at least 5,000 entity instances, at least 10,000 entity instances, at least 25,000 entity instances, at least 50,000 entity instances, at least 100,000 entity instances, at least 250,000 entity instances, between 1,000 and 10,000 entity instances, between 5,000 and 1 million entity instances, or any other suitable range within these ranges. Generating and outputting such a large number of data entity instances, along with the values of their attributes, imposes a significant computational burden on a data processing system—a significant amount of data must be retrieved from one or more data stores, processed, and then output.

[0243] The inventors have recognized that it may be useful to limit the processing performed by a data processing system when executing a declarative query, because in some circumstances, without any such limit, the number of data entity instances returned by the data processing system may be large, thereby creating a computational load on the data processing system that is a significant burden on the processor, memory, and / or network resources of the data processing system. Such a feature may be particularly important when a computer program or user enters a declarative query for processing without realizing that so much data will be returned and / or without actually needing to use the entirety of the results.

[0244] Thus, the Entity API provides several mechanisms for limiting the amount of processing performed by a data processing system when processing a declarative query. One such mechanism is that a declarative query may explicitly specify a numerical limit on the total number of results to be returned. This limit may be set globally or may be set locally for a particular portion of the declarative query. An example of setting a global limit on the total number of data entity instances to be returned is shown in Figure 4C, where, within line 414, the total number of data instances to be returned is limited to 5,000.

[0245] As another example, in some embodiments, the results may be a hierarchy with multiple levels, and the declarative query may specify that the number of data entity instances returned at each level of the hierarchy should be limited to a threshold number of results. For example, the declarative query shown in Table 7 below indicates that no more than two data entity instances should be returned at each level of the query.

[0246] [Table 7]

[0247] Table 8 shows the results of processing the declarative query in Table 7. Note that the results include only two data entity instances, and also show that more results are available (because the "exceededMaxRows" indicator is set to "true").

[0248] [Table 8]

[0249] In some embodiments, if the declarative query is hierarchical, different numeric limits may be specified at different levels of the hierarchy in the declarative query, for example, as shown below in Table 9. The results of processing such a declarative query may indicate whether a data entity instance limit has been reached at a core level (e.g., an attribute level) of the hierarchy.

[0250] [Table 9]

[0251] The inventors have recognized that by selecting a collection attribute (e.g., a collection of data element entities) and specifying a value for the "maxRows" option at that (child) level, processing a declarative query such as that shown in Table 9 above may produce unbalanced results, as each parent instance (starting from the first) is fully filled with child instances before the next parent instance is processed. Once the value of the "maxRows" option for a child level is filled, no further child instances are included in the response. For example, consider three dataset instances that can be identified by processing the declarative query in Table 9, where the first dataset instance contains three data elements, the second dataset instance contains one data element, and the third dataset instance also contains one data element. In this case, the result would include three data element instances in the first dataset instance, exhausting the number of rows allowed at the data element level (set by the "maxRows=3" option). As a result, no data element instances are returned in the second and third dataset instances.

[0252] To distribute child instances more evenly across parent instances, the Entity API allows you to specify the number of child rows per parent instance. As shown in the example in Table 10, the "collectionMaxRows" option can be used to limit the number of child rows per parent instance, while the overall row limit still applies. The results produced by processing the declarative query shown in Table 10 will include one Data Element entity instance for each DataSet Instance entity.

[0253] [Table 10]

[0254] Another mechanism for limiting the amount of processing performed by a data processing system when processing a declarative query is that, in some embodiments, the declarative query may be processed in a “streaming” or “staged” manner rather than a “batch” manner. In a batch manner, all data entity instances identified by processing the identifying portion of the declarative query may be enriched with attribute values and returned, which, as discussed above, may impose a significant computational burden on the data processing system if the number of identified data entity instances is large. Conversely, in some embodiments, the declarative query may be processed using a streaming manner, whereby the identifying portion of the declarative query may be used to identify data entity instances of interest (e.g., 5,000 data entity instances), but these instances are enriched and output in stages (e.g., in groups of 500) to (1) reduce the data processing system processing requirements at each stage, and (2) provide an opportunity, after completion of one or more stages, to cancel the execution of one or more subsequent stages if results from those stages are not required or needed. In some embodiments, after returning one set of results in a streaming manner, the next set of results is not determined and / or returned until a requestor (e.g., a user or computer software submitting a declarative query and receiving results) provides input indicating that additional results are required.

[0255] As one non-limiting example, consider that processing a declarative query results in the identification of a large number of data entity instances. Processing such a declarative query may be performed as follows: (1) generating an executable (e.g., SQL) query using the identifying portion of the declarative query and executing it to identify data entity instances of interest (e.g., 5,000 data entity instances); (2) for a first subset of the identified data entity instances (e.g., the first 500 data entity instances), generating one or more executable queries to obtain attribute values for the instances in the subset and executing the generated queries to obtain the attribute values; and (3) combining the results of the queries and outputting them (e.g., a query result) optionally with an indication that additional results are available. (4) receive an indication (e.g., from a user or from computer software submitting a declarative query) that additional results are desired; (5) generate, for a second subset of the identified data entity instances (e.g., data entity instances 501-1000), one or more executable queries to obtain attribute values for instances in the second subset; execute the generated queries to obtain the attribute values; and (3) combine and output the results of the queries (e.g., output the results for data entity instances 501-1000), optionally with an indication that additional results are available, until additional results are no longer desired or no additional results are available.

[0256] In some embodiments, the number of results returned at each stage may be set by a configurable parameter either globally for the Entity API as a whole or locally for (e.g., within) a particular declarative query.

[0257] Such a streaming architecture, which retrieves and returns results in stages, is made possible in part by the structure of the declarative queries that the Entity API processes. In particular, by having the identification portion of a declarative query processed separately from the enrichment portion of a declarative query, data entity instances can be identified by first generating and executing a set of one or more "identification" queries, and then incrementally retrieving those values (by generating and executing one or more "enrichment queries") based on whether additional results are desired and / or needed.

[0258] Searching for data using one or more search indexes In some embodiments, the processing of declarative queries described herein may be facilitated through the use of one or more search indexes. In particular, in some embodiments, a configured data processing system may be configured to maintain one or more search indexes such that users of the data processing system and / or computer programs interfacing with the data processing system may enter text search strings to locate data of interest (e.g., data entity instances, data entity definitions, etc.).

[0259] In some embodiments, such one or more search indexes may be maintained by a data processing system (e.g., data processing system 105) external to the database system (e.g., an RDBMS such as ORACLE, SQL SERVER, DB2, MONGODB, POSTGRESS, etc.) that stores the data entity instances. In this manner, once one or more search indexes are built by querying the database system, the data processing system may use the one or more search indexes to search for desired data entity instances (and their attribute values) without processing the same query again, leading to significant computational savings and faster result retrieval. Thus, the one or more search indexes described herein are distinct from any search index internal to the database system that stores the data entity instances. Furthermore, unlike an internal search index, the one or more search indexes described herein are text search indexes that assist users in entering text search strings to find data of interest.

[0260] As an example, a user may provide the search string "birthday" as input, and the data processing system may be configured to use the search index to return data such as: (1) instances of the data entity "BizTerm" for which the string "birthday" is at least part of one of its attribute values (e.g., the "name" or "description" attribute); (2) instances of the data entity "data element" for which the string "birthday" is at least part of one of its attribute values; and (3) instances of the data entity "dataset" that reference the birth date (e.g., in the value of one of its attributes).

[0261] In some embodiments, such a search index may be used to aid and expedite the processing of a declarative query. For example, in some embodiments, the identification portion of a declarative query may specify a search string (e.g., "birthday") to identify any data entity instances that reference "birthday," and the search index may be used to identify any such data entity instances instead of generating an executable (e.g., SQL) query and executing it against one or more databases. The data entity instances retrieved using the search index may then be enriched (e.g., by generating one or more executable queries for enrichment) using the techniques described herein. In this manner, using a search index instead of executing a database query may significantly accelerate the processing of the identification portion of the declarative query, shortening overall processing time and reducing the amount of computational resources (e.g., processor, memory, network resources) utilized.

[0262] An example of a declarative query with an identification portion that can be processed by using a search index is shown in Table 11. The declarative query in this example specifies the search string "birthday" using the "entitySearch" option, and the query as a whole is to identify any data entity instances that contain the search string "birthday" and enrich the results with values of the "name" and "description" attributes. Note that this query is global in that it does not search through instances of only one type of data entity, but rather searches instances of various types of data entities (e.g., "BizTerm," "Dataset," "DataElement," etc.). However, if only one type of data entity (e.g., "BizTerm") is of interest, the query can be made more specific in this regard with the option "entityClass": "BizTerm," as in other examples described herein.

[0263] [Table 11]

[0264] The declarative query example shown in Table 11 may be processed by (1) identifying one or more data entity instances of interest using one or more search indexes, (2) storing the results in a temporary table, and (3) enriching the results by generating one or more executable queries to obtain values for the attributes "name" and "description." In some embodiments, the first step may include taking into account the requested view type and / or user permissions before returning the data. The third step may be performed in any manner described herein for obtaining values for dynamic attributes.

[0265] Of course, a search index used to identify data entity instances needs to be created before it can be used to process such a declarative query. The search index may be created in any suitable manner. For example, in some embodiments, a data processing system may be configured to use any suitable search engine software library, such as APACHE LUCENE. The data processing system may be configured to generate a search index using this library. In some embodiments, the data processing system may be configured to update the search index (e.g., periodically, according to a predefined schedule, etc.). In some embodiments, a data processing system may be configured to generate one or more search engine indexes and may use one or more indexes to process declarative queries, and aspects of the technology described herein are not limited by the number of search indexes utilized.

[0266] In some embodiments, creating one or more search indexes using search software (e.g., APACHE LUCENE) requires that the search software be configured with information indicating which data entities and attributes should be indexed for searching. This allows further optimization of processing declarative queries by using the search index to retrieve these indexed values instead of executing a database query (e.g., an SQL query) to retrieve the attribute values. For example, if a data entity has three attributes (e.g., "Name," "Description," and "Business Line"), two of which (e.g., "Name" and "Description") are indexed for searching, then when processing a declarative query for this data entity, the search index may be used to retrieve the values of these two attributes instead of generating and executing an SQL or other type of executable query against a database, thereby resulting in significant acceleration. The value of the third attribute can still be retrieved, for example, by querying a relational database using an SQL query.

[0267] As noted with respect to the example declarative query in Table 11, a declarative query searches all different types of data entities within a data processing system. Thus, the results may include instances of different types of data entities (e.g., instances of different entity classes), such as an instance of a "BizTerm" data entity and an instance of a "data element" data entity. The different data entities may have some attributes in common (e.g., "name"), but different data entities generally have different attributes. To enable enrichment in this situation, a declarative query can explicitly assert the entity class of the data entities in the response (using the "entityClassSelect" notation shown in Table 12 below), which can then be enriched with attributes specific to that entity class. An example of this is shown in the declarative query in Table 12 below, where the Identify portion indicates that all data entity instances that reference "Birthday" should be searched for, but after these data entity instances are identified, (1) the values of the attributes "Name", "Description", and "Business Term Group" of every identified instance of the data entity "BizTerm", (2) the values of the attributes "Name", "Description", and "Dataset" of every identified instance of the data entity "Data Element", and (3) the values of the attributes "Name", "Description", and "Technology Group" should be obtained.

[0268] [Table 12]

[0269] Generating Executable Queries Preferred aspects of generating executable queries from declarative queries are further described in this section, and preferred aspects of generating executable queries from declarative queries can be combined with each other and with those described above. As described herein, the identification portion of a declarative query may be used to generate an executable query (e.g., an executable SQL query) for identifying data entity instances of interest. Similarly, the enrichment portion of a declarative query may be used to generate one or more executable queries (e.g., one or more executable SQL queries) for obtaining values of attributes of interest.

[0270] Executable identification and enrichment queries may be generated according to process 700 shown in Figure 7A. Process 700 may be used to generate executable queries for identifying data entity instances of interest using information specified in the identification portion of a declarative query. For example, process 700 may be used to generate executable SQL query 510 using information specified in the identification portion 504 of declarative query 502.

[0271] Process 700 may be used to generate an executable query to obtain values for attributes of interest using information specified in the enrichment portion of a declarative query. Because the enrichment portion may be used to generate multiple executable queries (e.g., multiple executable queries for different attributes or groups of different attributes, where groups may be obtained using any of the grouping techniques described herein), process 700 may be applied multiple times using different portions of the enrichment portion to generate multiple executable (e.g., SQL) queries. For example, process 700 may be used to generate executable queries 512, 514, 516, 518, and / or 530 using information specified in the enrichment portion 506 of declarative query 502.

[0272] Process 700 may be performed by any suitable computing device. Process 700 may be performed by any suitable software running on a data processing system, such as data processing system 105. For example, with reference to data processing system 105 shown in FIG. 1D, process 700 may be performed by using input transformation module 130 (e.g., to perform operations 702, 704, 706, and 708, described below).

[0273] In some embodiments, process 700 may be invoked multiple times by a process for generating multiple executable queries from a declarative query. For example, as described above with reference to Figure 3, in some embodiments, process 300 may invoke process 700 one or more times. For example, process 300 may invoke process 700 to generate one executable query from the identification portion of the declarative query and multiple executable queries from the enrichment portion of the declarative query.

[0274] Process 700 may begin at operation 702 by obtaining a portion of a declarative query. This may be any suitable portion of the declarative query. For example, if process 700 is being used to generate an executable identification query (e.g., query 510), the portion may include at least some or all of the identification portion of the declarative query (e.g., identification portion 504). As another example, if process 700 is being used to generate an executable enrichment query (e.g., queries 512, 514, 516, 518, or 530), the portion may include at least some or all of the enrichment portion of the declarative query (e.g., enrichment portion 506).

[0275] As yet another example, the declarative query may include an inline view (e.g., in an enrichment portion of the declarative query), and the portion obtained in operation 702 may include some or all of the information specified in the inline view. For example, if the inline view specifies a declarative table query, in operation 702, at least some (e.g., all) of the portion of the inline view that specifies the table query may be obtained.

[0276] Process 700 then proceeds to operation 704, where an intermediate query representation is generated for the portion of the declarative query obtained at operation 702. Generating the intermediate query representation may be performed by parsing the portion of the declarative query obtained at operation 702 and generating an intermediate representation using the parsed information. In some embodiments, the parsing may be performed by using an SQL parser, such as an ANSI SQL parser. In some embodiments, the intermediate query representation may include an Abstract Syntax Tree (AST). However, any other suitable type of intermediate representation may be used in some embodiments, and aspects of the technology described herein are not limited in this respect.

[0277] Process 700 then proceeds to operation 706, where the intermediate query representation is progressively transformed through a series of one or more transformations, each transformation augmenting the intermediate query representation (e.g., the AST) with auxiliary information that will be used to generate a database-specific executable query (e.g., a database-specific executable query) at operation 708. In the example embodiment of FIG. 7A, there are four transformations, which are described below with reference to operations 706a, 706b, 706c, and 706d and the examples of FIGS. 7B-7F. However, it should be understood that in some embodiments, one or more of the transformations described in FIG. 7A may be omitted (e.g., if there are no dynamic attributes referenced in the enrichment portion of the declarative query, dynamic reference attribute augmentation operation 706b may be omitted). Furthermore, in some embodiments, one or more other transformations may be used instead of or in addition to the transformations described herein, and aspects of the technology described herein are not limited in this respect.

[0278] 7A, an initial transformation of the intermediate query representation obtained in operation 704 is performed in operation 706a, where the intermediate query representation is augmented with unique identifier information associated with any data entity instances being searched. In this manner, the executable query generated in operation 708 obtains the unique identifiers and returns the unique identifiers along with the results of the query. Having access to the unique identifiers for each data entity instance may facilitate further access to the data entity instances, for example, to edit or delete one or more of the data entity instances.

[0279] For example, consider the identification portion 710 of the declarative query shown in Figure 7B. The intermediate query representation in identification portion 710 may be augmented with information that causes the executable query generated in act 708 to obtain a unique identifier for each "BizTerm" data entity instance that satisfies the "WHERE" clause of the query. Figure 7C shows an SQL view 720 of the augmented version of the intermediate query representation after the augmentation of act 706a. Note that the "SELECT" statement also indicates a "business term ID" in addition to a "name."

[0280] It should be understood that Figures 7C and 7D-7F, for clarity, depict SQL query views of the augmented intermediate query representation (instead of the intermediate representation itself, e.g., an augmented abstract syntax tree). Thus, in some embodiments, the intermediate SQL query views shown in Figures 7C-7F are not generated during performance of process 700. Rather, the intermediate query representation may undergo a series of one or more transformations, and after all transformations have been performed, an executable (e.g., SQL) query is generated from the intermediate query representation in operation 708.

[0281] A second transformation of the intermediate query expression is performed in operation 706b, where the intermediate query expression is further augmented with dynamic attribute reference information. The dynamic attribute reference information includes information specifying how the referenced dynamic attribute is stored in the underlying physical implementation. For example, this information may include information identifying one or more tables in which the value of the referenced dynamic attribute is stored in a database. When the value of the dynamic attribute is inherited from an attribute of another data entity instance, the information indicating where the inherited value came from may be used to identify the underlying table from which the inherited value is accessed. The dynamic attribute reference information may then be used in an executable query to access the appropriate physical table or other structure to obtain the value of the referenced dynamic attribute.

[0282] For example, the "x calculation rule" attribute in identification portion 710 is a dynamic attribute. However, "x calculation rule" is not the name of the physical table that stores the value of this attribute. In this example, the actual attribute value is stored in an "extended attribute value" table with ID "1004." As shown in SQL view 730 in FIG. 7D (which represents the augmented intermediate query representation obtained after performing operation 706b), the SQL query is now configured to select the dynamic attribute value from the extended attribute table with ID 1004.

[0283] A third transformation of the intermediate query expression is performed at operation 706c, where the intermediate query expression is augmented with view type information. In some embodiments, the view type information may be derived from the content of the declarative query itself and / or other configuration information associated with the data processing system executing process 700.

[0284] As described herein, in some embodiments, data entity instances managed by a data processing system may have associated therewith each workflow state. Additionally or alternatively, data entity instances managed by a data processing system may be versioned. In some embodiments, a declarative query may explicitly indicate that a particular "view type" is requested. The view type may indicate that data having a particular version and / or associated workflow state should be returned. For example, the view type may indicate that only data entity instances associated with a publication state should be returned. As another example, the view type may indicate that data entity instances updated within a particular time range should be returned. Other examples are described herein.

[0285] In some embodiments, if the declarative query does not specify a desired view type, the view type may be set to a default view type or may be set to a view type configured by the data processing system executing process 700. In some embodiments, even if the declarative query specifies a desired view type, the data processing system may override the requested view type with an override value.

[0286] Regardless of how the requested view type is specified, different data is returned depending on the requested view type. Because different data may be stored in different tables, in operation 706c, the intermediate query expression is augmented with view type information that indicates where the requested data is stored in physical storage (e.g., in which table). For example, the value of the "BizTerm" instance associated with the "Publish" workflow state and the value of the "BizTerm" instance associated with the "Pending Approval" workflow state may be stored in different tables.

[0287] 7B, the identification portion 710 does not specify a desired view type, and in this example, the data processing system is configured to default the view type, such that only data associated with a particular version of the data entity instance (e.g., the current version, a previous version, an even previous version, etc.) is returned. In this example, the data for the particular version of the entity instance is stored in physical tables with names beginning with the prefix "Vb." As shown in SQL view 740 in FIG. 7E (which represents the augmented intermediate query representation obtained after performance of act 706c), the SQL query is now configured to select dynamic attribute values from tables with the prefix "Vb."

[0288] A fourth transformation of the intermediate query representation is performed at operation 706d, where the intermediate query representation is augmented with permission information. In some embodiments, the data processing system is configured to allow only users or user groups authorized to access data to access those data. Thus, at operation 706d, the intermediate query representation may be augmented with permission information indicating one or more user groups to which the user submitting the declarative query (e.g., the logged-in user) belongs. This information may also be used to determine whether the user is authorized to access the requested data, so that when the executable query generated at operation 708 is executed, only data to which the user has permission is returned.

[0289] The intermediate query representation initially generated in operation 704 is augmented in operation 706 using a series of one or more transformations, and then the augmented intermediate query representation is converted into an executable query in operation 708.

[0290] In some embodiments, the executable query is for accessing data from a particular type of database system, and the augmented intermediate query representation can be converted into an executable query customized for that type of database system. For example, the database system can be a database system (e.g., a relational database system) configured to process SQL queries, and the executable query generated in operation 708 can be an executable SQL query. As another example, the database system can be a NoSQL system, and the executable query generated in operation 708 can be an executable query that the NoSQL system is configured to process.

[0291] In some embodiments, the executable query generated in operation 708 may be customized in a vendor-specific manner. For example, different vendors (e.g., MICROSOFT, ORACLE, IBM, POSTGRESQL, etc.) may implement different dialects of SQL and / or provide extensions to the SQL standard. In such situations, the executable query for the target database system (e.g., ORACLE) may be generated using syntax and / or commands implemented by the target database system (e.g., using any special syntax and / or commands implemented by ORACLE). Additionally or alternatively, the executable query generated in operation 708 may include optimizations to queries that can be supported using the target database system.

[0292] Thus, in some embodiments, an executable query generated in act 708 for one type of database (e.g., an executable SQL query for a MICROSOFT SQL SERVER database) may differ from an executable query generated in act 708 for another type of database (e.g., an executable SQL query for an IBM DB2 database), even if both queries are generated from the same underlying intermediate representation obtained by performing act 706 of process 700.

[0293] After the executable query is generated in operation 708, process 700 is complete.

[0294] Searching hierarchical data Declarative queries using recursion As described herein, data entities may be related to one another, and in some embodiments, a single declarative query may be used to retrieve information about multiple related data entities. For example, as described with reference to Figures 4D-4F, a single declarative query may specify a hierarchy of attributes (some of the attributes are attributes of different data entities), and values of the attributes in the hierarchy may be retrieved by processing the single declarative query. Furthermore, the retrieved attribute values may be output in a hierarchical format that corresponds to the hierarchical structure of the hierarchy of attributes specified in the declarative query. Indeed, in that example, the hierarchical format has the same nesting structure as the nesting structure of the attributes in the hierarchy of attributes specified in the declarative query.

[0295] In the example described with reference to Figures 4D-4F, the hierarchy of attributes whose values were to be searched for was explicitly specified as part of the declarative query. Specifically, the enrichment portion of the declarative query in that example had attributes organized in a nested manner as shown in Figure 4D. Other examples of specifying hierarchical queries using nesting are shown in Figures 9C, 9F-1, and 9F-2.

[0296] However, in some embodiments, recursion provides an alternative to using explicit nesting of attributes in enrichment sections to indicate that a hierarchical value of the attribute is desired. In particular, some data entities supported by a data processing system may be recursive. A data entity may be recursive if it has a parent of the same entity class (e.g., the same class parent). For example, a data entity is said to be recursive if it has an attribute that references a parent data entity, and the parent data entity has the same entity class. For example, the entity class "BizTerm" has the same class parent through the property "ParentBizTerm." As another example, the entity class "DataElement" has the entity class parent "DataElement." Such a recursive structure provides an alternative to using nesting to retrieve a hierarchical value of a data entity, and in some embodiments, recursion can be used to retrieve information about multiple related data entities.

[0297] An example of using recursion in a single declarative query to retrieve information about multiple related data entities is shown in Figures 8A-8C. Figure 8A shows an example data entity instance 800 of a "Data Element" data entity with the same class parent. Data entity instance 800 is named "Contact Information," has the hierarchical structure shown in Figure 8A, and may be used to store information about a user's contact information. The data entity instance "Contact Information" has three attributes, "Address," "Email," and "Phone Number," which reference respective "Data Element" instances that store address, email, and phone number information, respectively. Furthermore, the "Address" entity instance itself has four attributes, called "Address Line," "City," "State," and "Zip Code," which reference respective "Data Element" instances that store street, city, state, and zip code information, respectively.

[0298] 8B illustrates a single exemplary declarative query 805 for searching hierarchical data using recursion, according to some embodiments of the techniques described herein. As shown in FIG. 8B, the hierarchy of data elements that are children of the data element "Contact Information" may be searched using a Recursion On field 806 with a value "Children," indicating that the recursion should be downward. The enrichment section indicates, at 808, that for each attribute searched recursively, the name of the attribute should be obtained.

[0299] Figures 8C-1, 8C-2, and 8C-3 show exemplary results obtained by a data processing system by processing the example declarative query 805 shown in Figure 8B. As is apparent from Figures 8C-1, 8C-2, and 8C-3, the structure of the results is hierarchical, with a hierarchical structure that matches the nesting structure of the data elements shown in Figure 8A. At the same time, that nesting structure does not need to be explicitly stated in the declarative query 805, thereby simplifying writing queries to search hierarchical data in instances where recursion may be used.

[0300] Although the example declarative query in Figure 8B indicates that the recursion should be downward, a declarative query may use upward recursion to retrieve values. An example is shown in Figure 8D, which shows a single declarative query 810 to recurse through attributes of every parent entity of a "phone number" data entity instance. In this example, the recursion on field is set to the value "parent data element," indicating that the recursion should be upward. Exemplary results 820 obtained by a data processing system by processing the example declarative query 810 are shown in Figure 8E.

[0301] In some embodiments, a declarative query may specify a numerical limit on the number of levels of recursion to be performed by a data processing system when processing the declarative query, which may be useful in keeping result sizes manageable.

[0302] Adding a declarative query example to search hierarchical data Additional examples of declarative queries for searching hierarchical data are now described with reference to Figures 9A-9G.

[0303] FIG. 9A illustrates a portion of the entity-relationship diagram shown in FIG. 2G to help illustrate the examples of FIGS. 9B-9G. In the example of FIG. 9A, the "BizTerm" data entity 902 has attributes "Name," "Description," "PII Classification ID," "Legacy URL," "Legacy Modification Date," and "Design Document," which can take on various values. One or more of these attributes may be static. One or more of these attributes may be dynamic. The "BizTerm" data entity also has reference attributes 905, "BizTerm Group," "PII Classification," and "Data Element," which reference lists of other data entities—BizTerm Group data entity 904, PII Classification data entity 906, and one or more Data Element data entities, respectively. Additionally, BizTerm Group data entity 904 has attributes "Name," "Description," "Dictionary Type," and a reference attribute "Business Term." PII Classification data entity 906 has a name attribute.

[0304] 9B shows a pseudo-code specification of a declarative query for searching hierarchical data, according to some embodiments of the technology described herein. In particular, the specification indicates that the results should include information about all instances of the "BizTerm" data entity whose PII classification attribute is set to reference some specific instance of the PII classification data entity. For each such identified "BizTerm" instance, the results should include the values of the "Name" and "Legacy URL" attributes. In addition, for each identified specific "BizTerm" instance, the results should include the values of the "Name" and "Dictionary Type" attributes of the "BizTerm Group" data entity instance to which the BizTerm Group attribute (of the specific "BizTerm" instance) refers.

[0305] 9C shows a declarative query 910 that is a JSON implementation of the pseudo-code declarative query specification shown in FIG. 9B in accordance with some embodiments of the techniques described herein. Query 910 includes an identification portion 912 that indicates that results should include public instances of the "BizTerm" data entity whose PII classification attribute is not null and is set to reference some instance of the PII classification data entity. Query 912 includes an enrichment portion 914 that indicates that results should include attribute values for "Name," "Legacy URL," and, for the referenced BizTerm group data entity instance, values for its "Name" and "Dictionary Type" attributes.

[0306] 9D shows an example diagram of results that may be obtained by processing the JSON declarative query of FIG. 9C in accordance with some embodiments of the technology described herein. In this example, the results are shown using nested tables, but this is not a limitation of embodiments of the technology described herein. In other embodiments, the results may have a hierarchical format specified by nesting or in any other suitable manner.

[0307] Note that the enrichment section 914 in Figure 9C uses nesting to specify the desired hierarchy of attributes, and the results are presented in a hierarchical format whose structure matches the nesting structure of the enrichment section 914. Also, note that, like many of the example declarative queries described herein, there is no distinction between declarative queries in which the attributes are static and those in which the attributes are dynamic. As a result, the query author does not need to know how the attributes are stored in memory. The declarative query 910 is not only independent of whether the attributes are static or dynamic, but also independent of the type of database that actually stores the values - database-specific details are hidden.

[0308] 9E shows a pseudo-code specification of another, more complex declarative query for searching hierarchical data, in accordance with some embodiments of the techniques described herein. As shown in FIG. 9E, the specification indicates that the results should include information about all instances of the "BizTerm" data entity that reference a "BizTerm Group" entity instance with its "Dictionary Type" attribute set to "Company." For each such entity instance, the results should include the name of the entity instance, the name of the referenced BizTerm group, the name of the referenced data element instance, the name of the data set referenced by the referenced data element instance, a count of all associated data elements, a count of associated data elements that are not part of an application, and a count of associated data elements grouped by application.

[0309] 9F-1 and 9F-2 show a declarative query 920, which is a JSON implementation of the pseudo-code declarative query specification shown in FIG. 9B, in accordance with some embodiments of the technology described herein. The declarative query includes an identification portion indicating that the results should include public instances of BizTerm data entities that reference BizTerm group instances with their "dictionary type" attribute set to "company." The declarative query also includes an enrichment portion that uses nesting to pull attribute values from two different hierarchies: the BizTerm group hierarchy and the data element hierarchy.

[0310] Additionally, declarative queries include nested SQL queries to enrich the response with nested results. The nested SQL query in inline view 922 is a declarative table query. The nested SQL query in inline view 924 is a declarative table query to enrich the response with nested table results.

[0311] 9G shows an example diagram of results that may be obtained by processing the JSON declarative queries of FIGS. 9F-1 and 9F-2, in accordance with some embodiments of the technology described herein. In this example, the results are shown using nested tables, but this is not a limitation of embodiments of the technology described herein. In other embodiments, the results may have a hierarchical format specified by nesting or in any other suitable manner.

[0312] 9A-9D, the declarative query 920 in Figures 9F-1 and 9F-2 uses nesting to specify the desired hierarchy of attributes, and the results are presented in a hierarchical format shown in Figure 9G whose structure matches the nesting structure of the declarative query. The declarative query 920 is agnostic to whether the attributes are static or dynamic, and to the details of which database stores these attribute values and how this is done.

[0313] Additional Aspects of the Entity API As described herein, a data processing system (e.g., data processing system 105) may be configured to enable different types of access to data entities managed by the data processing system through entity APIs. For example, in addition to enabling querying of data entity instances, a data processing system may be configured to enable creation, updating, and deletion of data entity instances. As another example, as described herein, a data processing system may be configured to manage workflow operations using a workflow management system, and the entity API may be used as an interface to enable execution of workflow operations (e.g., submitting changes to one or more data entity instances for approval, approving or rejecting changes submitted to a data entity instance, etc.).

[0314] Thus, in some embodiments, declarative statements (e.g., specified using JSON or any other suitable format) may be used to specify a desired create, update, delete, or workflow operation (similar to how a declarative query may be used to specify a desired query), and the declarative statements may be converted into one or more executable statements (e.g., SQL statements), which may be executed by a data processing system to perform the desired operation.

[0315] Examples of declarative statements for performing create, update, or delete operations are described below with reference to Tables 13-16. In these examples, such statements may be indicated in JSON notation using "operation":="ENTITY_SAVE" (whereas declarative queries may be indicated using "ENTITY_QUERY" or "TABULAR_QUERY", as described below). In some embodiments, the "ENTITY_SAVE" statement further includes a parameter indicating the type of edit being made (e.g., using an "editType" command). For example, the value of the parameter may indicate whether a data entity instance is being created, updated, deleted, added to, or removed from a reference collection.

[0316] An illustrative example of a declarative statement for creating a data entity instance is shown in Table 13 below, where the declarative statement is for creating an instance of the "BizTerm Group" data entity with the "Name" and "Description" attributes set to "New Business Term Group" and "My First Dictionary", respectively.

[0317] Table 14 shows an example result returned in response to executing the declarative statement in Table 13 to create a new instance of the data entity "BizTermGroup." In this example, the result includes an identifier ("Composite ID" in this example) of the newly created data entity instance. This identifier may be used to identify the data entity instance for subsequent operations (e.g., update or delete). The result also indicates whether the newly created data entity instance can be updated and / or deleted by the user.

[0318] [Table 13]

[0319] [Table 14]

[0320] An illustrative example of a declarative statement for updating a data entity instance is shown below in Table 15a, where the declarative statement is for updating an instance of the "BizTerm Group" data entity to change the value of the "description" attribute to "My Really First Dictionary." In this example, the editType parameter is set to "update." Table 15b shows exemplary results returned in response to executing the declarative statement in Table 15a for updating an instance of the data entity "BizTerm Group."

[0321] [Table 15]

[0322] [Table 16]

[0323] The example declarative statement in Table 15a includes a unique identifier for the data entity instance to be updated. In this example, the unique identifier can be obtained from the results shown in FIG. 14 or by executing a declarative query to obtain the identifier. In some embodiments, a declarative query can be executed before performing an update or delete operation to obtain the identifier for the declarative entity instance to update or delete. An example of such a declarative query is shown below in Table 16a, and the results are shown in Table 16b.

[0324] The declarative query example in Table 16a specifies the option "retrieveForEdit," which causes additional information related to editing a data entity instance to be returned. As shown in Table 16b, the additional information may indicate whether a particular attribute can be updated (e.g., the value of the "description" attribute can be updated because the additional information includes the string "canUpdate":true). The additional information may also include an editContext, which indicates the version of the item to be edited. In some embodiments, this editContext may be provided in the declarative statement to allow the edit to ensure that the latest version of the attribute is being edited. This avoids conflicts if someone else makes a change after a query for the current attribute value has been made but before the new value is submitted. As shown in Table 15a, the declarative statement includes an edit context, which is returned in the results shown in Table 16b.

[0325] [Table 17]

[0326] [Table 18]

[0327] As described herein, the Entity API may be used to perform workflow operations on one or more data entity instances managed by a data processing system. To this end, in some embodiments, a data entity instance may have an attribute (e.g., referred to as a "Workflow Task") having a value indicating a workflow task (e.g., approve, reject, etc.) that may be performed on the data entity instance. This attribute value may be used to identify a data entity instance on which the workflow task may be performed, and a declarative statement (e.g., with a "WORKFLOW_ACTION" action) may be used to cause the data processing system to perform the workflow operation.

[0328] In some embodiments, declarative statements for creating, updating, deleting, or performing workflow operations on one or more data entity instances may have a hierarchical structure. For example, a declarative statement may include a hierarchy of attributes (e.g., specified using nesting or recursively), similar to how a declarative query may be used to obtain values for a hierarchy of attributes.

[0329] As described herein, values of attributes of a data entity instance may be stored by a data processing system in different tables. For example, values of static attributes and dynamic attributes may be stored in different tables. As another example, values of different dynamic attributes (e.g., different types of dynamic attributes) may be stored in different tables. Because a single declarative query can be used to access values of multiple different attributes of a data entity instance, a single declarative query can be used to access values stored across multiple different tables stored in a data processing system. Similarly, because a single declarative statement can be used to create, update, or delete values of multiple different attributes of a data entity instance, the statement can be used to create values, update, or delete values stored in multiple tables.

[0330] In some embodiments, declarative statements for creating, updating, or deleting one or more data entity instances may be processed according to process 700 described with reference to Figure 7. For example, in some embodiments, the declarative statements may first be converted into an intermediate representation (e.g., an abstract syntax tree), for example, by using an SQL parser.

[0331] The intermediate representation is then transformed using a series of one or more transformations (e.g., dynamic attribute reference augmentation 706b, view type augmentation 706c, and permission augmentation 706d described in connection with FIG. 7 and / or any other suitable transformations) to obtain a transformed intermediate representation. Because declarative statements for updating or deleting data entity instances already include one or more data entity instance identifiers, in some embodiments, the unique identifier augmentation step may not be necessary when converting declarative statements into executable statements. On the other hand, dynamic attribute reference augmentation and view type augmentation may be performed to identify tables in which attribute values to be updated or deleted are stored, and permission augmentation may be performed to ensure that the logged-in user is authorized to make the requested changes.

[0332] After the intermediate representation (e.g., an abstract syntax tree) is transformed through a series of one or more transformations, executable statements (e.g., executable SQL statements) are generated from the transformed intermediate representation. The executable statements may, in some embodiments, be customized to a particular database system. The executable statements may then be executed to process the declarative statements.

[0333] Declarative Table Queries As described herein, a declarative query may be a declarative entity query having an identification portion and an enrichment portion, and the results of processing the declarative entity query include data entity instances enriched with attribute values. Each data entity instance portion of the result may be associated with a respective entity instance ID that uniquely identifies the data entity instance. In the examples described herein, such a query may be indicated in JSON notation using "operation":="ENTITY_QUERY"

[0334] However, the inventors have recognized that SQL queries can be used directly (instead of declarative entity queries) to join data entity instances such that a single row in the result set belongs to a single data entity instance, and one or more (e.g., all) of the columns may be associated with different entity instances.

[0335] Thus, in some embodiments, a data processing system implementing the Entity API may also be configured to process declarative table queries whose results may mix data from different data entity instances, with the results organized into a matrix where one or more of the columns may optionally be tied to each data entity instance ID. In the examples described herein, a declarative table query may be indicated in JSON notation using "operation":="TABULAR_QUERY"

[0336] Figure 10A shows an illustrative example of a table query that combines information obtained from multiple data entity instances. In this example, the table query is to find the names of all "BizTerm" data entity instances that have a parent "BizTerm." As shown in Figure 10B, the result of processing this table query contains two columns: the name of the BizTerm and the name of the parent "BizTerm." Note that aliases are used to rename the columns.

[0337] In some embodiments, table queries may be used to perform aggregation functions, including, but not limited to, identifying the minimum or maximum of a set of numbers (e.g., a column), identifying the mean, standard deviation, or other statistic of a set of values (e.g., a column), counting the number of values in a set of values (e.g., a column), counting the number of values in a set of values (e.g., a column) that satisfy one or more conditions, etc.

[0338] FIG. 10C shows an illustrative example of a table query for performing an aggregation function (in this example, a count operation). The results of processing the declarative table query of FIG. 10C are shown in FIG. 10D. Note that in this example, the declarative table query requires that the result include a structural definition of the result by adding a "retrieveDefinition" global part. The structural definition is returned in the first part of the result shown in FIG. 10D. Another example of a declarative table query for performing an aggregation function is shown in inline view 924 of FIG. 9F-2, with example results shown in FIG. 9G.

[0339] As can be seen from the above example, a declarative table may include SQL statements to select desired rows and columns. Optionally, a declarative table query may have an enrichment section to enrich individual columns if entity instance IDs are available. For example, in the case of reference attributes, the entity instance values in a column may be enriched by sub-selecting additional attributes of the top-level entity, including nested attributes. This produces enriched column values that are references at the top level, like reference values in an entity query, but with additional attribute values from the sub-select in the attribute section.

[0340] For example, consider the table query shown in Figure 10A to find the names of all "BizTerm" data entity instances that have a parent "BizTerm." Now, we want to enrich each row with a reference to the BizTerm group of child business terms (alias "b") and the values of the "name" and "description" attributes of each BizTerm group instance. This can be achieved using the query shown in Table 17 below. The "fromAlias" keyword is used to identify which of the business terms should be used for enrichment. The result is shown in Table 17 and includes the "BizTerm group" reference.

[0341] [Table 19]

[0342] [Table 20]

[0343] In some embodiments, a declarative table query may be processed using process 700 of Figure 7. In some such embodiments, the declarative table query is first converted into an intermediate query representation (e.g., an abstract syntax tree), for example, by using an SQL parser. The intermediate query representation is then transformed using a series of one or more transformations (e.g., the transformations described herein in connection with Figure 7 and / or any other suitable transformations) to obtain a transformed intermediate query representation. The intermediate query representation is then converted into an executable query (e.g., an executable SQL query). The executable query, in some embodiments, may be customized to a particular database system.

[0344] Inline View As described herein, in some embodiments, the enrichment portion of a declarative query may include an inline view that includes another declarative query. Inline views enable the nesting of declarative queries. In this manner, a declarative table or entity query can be nested within another declarative query. For example, as already seen in connection with Figures 9F-1 and 9F-2, the enrichment portion of declarative query 920 includes two inline views: inline view 922 that includes an entity query and inline view 924 that includes a table query.

[0345] In some embodiments, an inline view may include four components: (1) information specifying the type of declarative query to be performed (e.g., information specifying whether the query is a declarative entity query or a declarative table query); (2) an SQL query to create the inline view, which selects inner join keys; (3) a list of column names to be used as outer join keys for joining from the outside (denoted "outerJoinKeys" in the examples of Figures 9F-1, 9F-2, 10E, and 10F); and (4) a list of column names to be used as inner join keys for joining from the inside (denoted "innerJoinKeys" in the examples of Figures 9F-1, 9F-2, 10E, and 10F).

[0346] 10E and 10F show additional examples of declarative queries that include inline views, according to some embodiments of the techniques described herein.

[0347] In particular, FIG. 10E illustrates a declarative entity query that includes an inline view with another declarative entity query within it. In this example, the overall declarative query is to select all instances of the "BizTerm" data entity that have at least one data element. Furthermore, for each such data entity instance, the goal is to obtain the collection of "applications" associated with the instance by traversing the entity model path "DataElement.DataSet.Application" according to the entity-relationship diagram shown in FIG. 2G. Without an inline view, the result would need to include all intermediate elements (i.e., data elements connected to datasets connected to applications). However, by using an inline view, there is no need to return these intermediate results, as shown in FIG. 10E.

[0348] In this example, the inline query starts with "Application" and joins down to the entity with the join attribute—in this case, the data element—because the inner and outer join attributes are set to "Business Term ID." Figures 10G-1 and 10G-2 show example results from executing the declarative entity query shown in Figure 10E.

[0349] Figure 10F shows that a declarative table query can be nested within an enrichment table query using an inline view. In this example, the inline view helps to find the count of datasets per technology group for each instance of the "Tech System" data entity. The inner and outer join attributes are set to "Tech System ID." Figures 10H-1, 10H-2, and 10H-3 show example results of executing the declarative entity query shown in Figure 10F.

[0350] Additional Implementation Details 11 illustrates an example of a suitable computing environment 1100 on which the technology described herein may be implemented. The computing environment 1100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology described herein. Neither the computing environment 1100 nor the computing environment 1100 should be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 1100.

[0351] The technology described herein is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0352] A computing environment may execute computer-executable instructions such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The techniques described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0353] 11, an exemplary system for implementing the techniques described herein includes a general-purpose computing device in the form of a computer 1100. Components of the computer 1100 may include, but are not limited to, a processing unit 1120, a system memory 1130, and a system bus 1121 that couples various system components including the system memory to the processing unit 1120. The system bus 1121 may be any of several types of buses, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the MicroChannel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus, also known as the Mezzanine bus.

[0354] Computer 1110 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computer 1110, including both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computer 1100. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0355] The system memory 1130 includes computer storage media in the form of volatile and / or nonvolatile memory such as read-only memory (ROM) 1131 and random access memory (RAM) 1132. A basic input / output system 1133 (BIOS), containing the basic routines that help to transfer information between elements within the computer 1110, such as during start-up, is typically stored in ROM 1131. RAM 1132 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by the processing unit 1120. By way of example, and not limitation, FIG. 11 illustrates operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0356] Computer 1110 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, FIG. 11 illustrates a hard disk drive 1141 that reads from or writes to non-removable, non-volatile magnetic media, a flash drive 1151 that reads from or writes to removable, non-volatile memory 1152 such as flash memory, and an optical disk drive 1155 that reads from or writes to a removable, non-volatile optical disk 1156 such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid-state RAM, solid-state ROM, etc. The hard disk drive 1141 is typically connected to the system bus 1121 through a non-removable memory interface, such as interface 1140, and the magnetic disk drive 1151 and optical disk drive 1155 are typically connected to the system bus 1121 by a removable memory interface, such as interface 1150.

[0357] The drives and associated computer storage media discussed above and illustrated in FIG. 11 provide storage of computer-readable instructions, data structures, program modules, and other data for the computer 1110. In FIG. 11, for example, hard disk drive 1141 is illustrated as storing operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components can either be the same as or different from operating system 1134, application programs 1145, other program modules 1136, and program data 1147. Operating system 1144, application programs 1145, other program modules 1146, and program data 1147 are given different numbers here to illustrate that, at a minimum, they are different copies. Actors may enter commands and information into the computer 1110 through input devices such as a keyboard 1162 and pointing device 1161, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, etc. These and other input devices are often connected to the processing unit 1120 through a user input interface 1160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, a game port, or a universal serial bus (USB). A monitor 1191 or other type of display device is also connected to the system bus 1121 via an interface, such as a video interface 1190. In addition to a monitor, computers may also include other peripheral output devices such as speakers 1197 and printer 1196, which may be connected through an output peripheral interface 1195.

[0358] The computer 1110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 1180. The remote computer 1180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 1110, although only a memory storage device 1181 is illustrated in FIG. 11. The logical connections depicted in FIG. 11 include a local area network (LAN) 1171 and a wide area network (WAN) 1173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0359] When used in a LAN networking environment, the computer 1110 is connected to the LAN 1171 through a network interface or adapter 1170. When used in a WAN networking environment, the computer 1110 typically includes a modem 1172 or other means for establishing communications over the WAN 1173, such as the Internet. The modem 1172, which may be internal or external, may be connected to the system bus 1121 via the actor input interface 1160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 1110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 11 illustrates remote application programs 1185 as residing on memory device 1181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0360] Having described several aspects of at least one embodiment of the technology described herein, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0361] Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of this disclosure. Additionally, while advantages of the technology described herein have been set forth, it should be understood that not every embodiment of the technology described herein includes every described advantage. Some embodiments may not implement any feature described herein as an advantage, and in some cases, one or more of the described features may be implemented to achieve further embodiments. Accordingly, the foregoing description and drawings are by way of example only.

[0362] The above embodiments of the technology described herein can be implemented in any of many ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided on one computer or distributed across multiple computers. Such processors may be implemented as integrated circuits, where one or more processors are within integrated circuit components, including commercially available integrated circuit components known in the art under names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, a processor may be implemented in a custom circuit such as an ASIC or semi-custom circuit resulting from constructing a programmable logic device. As a further alternative, a processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of the cores can constitute a processor. However, a processor may be implemented using circuitry in any suitable format.

[0363] Further, it should be understood that a computer may be implemented in any of many forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. Furthermore, a computer may be implemented in devices not generally considered to be computers but having suitable processing capabilities, including a personal digital assistant (PDA), a smartphone, or any other suitable portable or fixed electronic device.

[0364] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visually presenting output and a speaker or other sound-generating device for audibly presenting output. Examples of input devices that may be used in a user interface include a keyboard and pointing devices such as a mouse, touchpad, and digitizing tablet. As another example, a computer may receive information entered through speech recognition or in other audible formats.

[0365] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or as a wide area network such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0366] Also, the various methods or processes outlined herein may be coded as software executable on one or more processors utilizing any one of a variety of operating systems or platforms. Furthermore, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0367] In this regard, aspects of the technology described herein may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memory, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments described above. As is evident from the above examples, a computer-readable storage medium may retain information for a sufficient period of time to provide computer-executable instructions in a non-transitory form. Such one or more computer-readable storage media may be transportable such that the stored program(s) can be loaded into one or more different computers or other processors to implement various aspects of the technology as described above. As used herein, the term “computer-readable storage medium” encompasses only non-transitory computer-readable media that can be considered to be a manufacture (i.e., article of manufacture) or machine. Alternatively or additionally, the techniques described herein may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0368] The terms "program" or "software" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement various aspects of the technology described above. Furthermore, according to one aspect of this embodiment, it should be understood that one or more computer programs that, when executed, perform the methods of the technology described herein need not reside on one computer or processor, but may be distributed in a modular manner across several different computers or processors to implement various aspects of the technology described herein.

[0369] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0370] Additionally, data structures may be stored in computer-readable media in any suitable form. For ease of illustration, data structures may be shown with fields related through their locations in the data structure. Such relationships may similarly be achieved by assigning storage of fields to locations in the computer-readable media that convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in fields of a data structure, including the use of pointers, tags, or other mechanisms for establishing relationships between data elements.

[0371] Various aspects of the technology described herein may be used alone, in combination, or in a variety of configurations not specifically discussed in the embodiments described above, and therefore, its application is not limited to the details and arrangements of components set forth in the above description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0372] Additionally, the techniques described herein may be implemented as methods, examples of which are provided herein, including with reference to Figures 3 and 7. The acts performed as part of any of the methods may be in any suitable order. Thus, although shown as sequential acts in the exemplary embodiments, embodiments may be constructed in which acts are performed in a different order than shown, which may include performing some acts simultaneously.

[0373] Additionally, some actions are described as being performed by an "actor" or "user." It should be understood that an "actor" or "user" need not be a single individual, and that in some embodiments, actions attributed to an "actor" or "user" may be performed by an individual in combination with a team of individuals and / or computer-assisted tools or other mechanisms.

[0374] The use of ordinal numbers such as "first," "second," "third," etc. to modify claim elements in the claims does not, in itself, imply any priority, precedence, or ordering of one claim element relative to another, or the chronological order in which method actions are performed, but is merely used as a label to distinguish one claim element having a particular name from another element having the same name (other than the use of the ordinal number).

[0375] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use herein of "comprises," "includes," or "have," "contain," "involve," and variations thereof means the inclusion of the items listed thereafter and equivalents thereof, as well as additional items.

Claims

1. 1. A method, performed by a data processing system, for computationally efficient retrieval of a first attribute value of a first instance of a data entity in response to a query, comprising: (A) obtaining a query for accessing data through an application programming interface (API) of the data processing system, the query comprising: a first portion including information for identifying at least the first instance of the data entity stored in at least one data store; and obtaining, the second portion indicating at least one attribute of the data entity from which a value is to be obtained; (B) generating a plurality of executable queries, including a first set of one or more executable queries and a second set of one or more executable queries; using the first portion of the query to generate the first set of executable queries for identifying at least the first instances of the data entities stored in the at least one data store; and generating, using the second portion of the query, a second set of executable queries for retrieving attribute values of at least the first instances of the data entities identified using the first set of executable queries, the attribute values including the first attribute value. generating a (C) executing the plurality of executable queries to obtain results of the queries provided via the API, the results including at least the first attribute value of the first instance of the data entity; and (D) outputting at least some of the results including at least the first attribute value of the first instance of the data entity; and A method comprising:

2. The method of claim 1 , wherein the first and second sets of executable queries comprise executable Structured Query Language (SQL) queries.

3. the first portion includes information for identifying one or more instances of each of a plurality of data entities, the plurality of data entities including the data entity; 2. The method of claim 1 , wherein generating the first set of executable queries comprises, for each particular data entity of the plurality of data entities, generating a respective executable query to identify instances of the data entity stored in the at least one data store.

4. the at least one attribute includes a plurality of attributes of the data entity; 2. The method of claim 1 , wherein generating the second set of executable queries includes, for each particular attribute of at least some of the plurality of attributes, generating a respective executable query to obtain a value of the particular attribute of the instances of the data entities identified using the first set of executable queries.

5. the plurality of attributes includes a plurality of dynamic attributes; Generating the second set of executable queries includes: grouping the plurality of dynamic attributes into dynamic attribute groups, each of the groups having at least one of the plurality of dynamic attributes; generating, for each of the plurality of groups of dynamic attributes, a respective executable query for obtaining values of attributes in the group for the instance of the data entity; The method of claim 4, comprising:

6. The method of claim 5 , wherein grouping the plurality of dynamic attributes is performed based on whether the dynamic attributes are stored in the same table in the at least one data store.

7. Executing the plurality of executable queries includes: executing the first set of executable queries to obtain information identifying at least the first instance of the data entity; After executing the first set of executable queries, executing the second set of executable queries using the information identifying at least the first instance of the data entity; and The method of claim 1 , comprising:

8. The method of claim 7 , wherein generating the second set of executable queries is performed after executing the first set of executable queries.

9. The method of claim 7 , wherein generating the second set of executable queries is performed before executing the first set of executable queries.

10. the at least one attribute comprises a group of multiple dynamic attributes of the data entity; the second set of executable queries includes a plurality of executable queries respectively corresponding to groups of the plurality of dynamic attributes; executing the second set of executable queries includes executing each of the plurality of executable queries to obtain a respective plurality of dynamic attribute value results; The method of claim 7 , further comprising, after executing the second set of executable queries, combining the dynamic attribute value results to obtain the results for the queries.

11. the second portion of the query includes an inline view; generating the second set of executable queries includes generating executable queries using information specified in the inline view; The method of claim 1 , wherein the inline view includes a table query.

12. The method of claim 1 , wherein the second portion of the query indicates a hierarchy of attributes that includes the at least one attribute using recursion or nesting.

13. Outputting at least some of the results includes: formatting said at least some of said results in a hierarchical format corresponding to the hierarchy of said attributes; outputting said at least some of said results in said hierarchical format; and 13. The method of claim 12, comprising:

14. the first portion of the query is transformed through a series of transformations that augment the information in the first portion with information managed by the data processing system to generate the first set of executable queries configured to be executed against the at least one data store; 2. The method of claim 1 , wherein the second portion of the query is transformed through a series of transformations that augment the information in the second portion with information managed by the data processing system to generate the second set of executable queries configured to be executed against the at least one data store.

15. Generating the first set of executable queries may include at least in part generating the first executable queries from: generating an intermediate query representation using the first portion of the query; augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information, and / or permission information to obtain an augmented intermediate query representation; generating the first executable query using the augmented intermediate query representation; 2. The method of claim 1, comprising generating the hydroxybenzoate by

16. Generating the second set of executable queries may include generating the second set of executable queries at least in part by: generating an intermediate query representation using the second portion of the query; augmenting the intermediate query representation with unique identifier information, dynamic attribute reference information, view type information, and / or permission information to obtain an augmented intermediate query representation; generating the second executable query using the augmented intermediate query representation; and 2. The method of claim 1, comprising generating the hydroxybenzoate by

17. executing the plurality of executable queries includes executing the first set of executable queries to obtain information identifying a plurality of data entity instances; The method of claim 1 , wherein generating the second set of one or more executable queries is performed using the second portion of the query and the information identifying the plurality of data entity instances.

18. The method of claim 1 , wherein the query is a declarative query.

19. 1. A data processing system comprising: at least one computer hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions; the processor-executable instructions, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for obtaining a first attribute value of a first instance of a data entity in response to a query, the method comprising: (A) obtaining a query for accessing data through an application programming interface (API) of the data processing system, the query comprising: a first portion including information for identifying at least the first instance of the data entity stored in at least one data store; and a second part indicating at least one attribute of the data entity from which a value is to be obtained; and (B) generating a plurality of executable queries, including a first set of one or more executable queries and a second set of one or more executable queries; using the first portion of the query to generate the first set of executable queries for identifying at least the first instances of the data entities stored in the at least one data store; and generating, using the second portion of the query, a second set of executable queries for retrieving attribute values of at least the first instances of the data entities identified using the first set of executable queries, the attribute values including the first attribute value. generating a (C) executing the plurality of executable queries to obtain results of the queries provided via the API, the results including at least the first attribute value of the first instance of the data entity; and (D) outputting at least some of the results including at least the first attribute value of the first instance of the data entity; and a data processing system including:

20. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for obtaining a first attribute value of a first instance of a data entity in response to a query, the method comprising: (A) obtaining a query for accessing data through an application programming interface (API) of a data processing system, the query comprising: a first portion including information for identifying at least the first instance of the data entity stored in at least one data store; and a second part indicating at least one attribute of the data entity from which a value is to be obtained; and (B) generating a plurality of executable queries, including a first set of one or more executable queries and a second set of one or more executable queries; using the first portion of the query to generate the first set of executable queries for identifying at least the first instances of the data entities stored in the at least one data store; and generating, using the second portion of the query, a second set of executable queries for retrieving attribute values of at least the first instances of the data entities identified using the first set of executable queries, the attribute values including the first attribute value. generating a (C) executing the plurality of executable queries to obtain results of the queries provided via the API, the results including at least the first attribute value of the first instance of the data entity; and (D) outputting at least some of the results including at least the first attribute value of the first instance of the data entity; and At least one non-transitory computer-readable storage medium, including:

Images