JavaScript Object Notation (JSON) query to join across cloud resources

Abstract

A cloud resource join query is parsed for a join operation across cloud resources to extract join rules and a query for each cloud resource in the cloud resource join query. Results from individual cloud queries are dynamically indexed based on cloud resource pairs indicated in the join rules. A search engine uses the dynamic index to apply first-order predicates in the join rules to generate pairwise join results corresponding to the query. Results of the cloud resource join query include the pairwise join results after merging.

Description

Background Technology

[0001] This disclosure generally relates to data processing and database and file access.

[0002] An inner join (hereinafter referred to as "join") is an operation in relational algebra that combines results across indexes in a relational database based on the common values ​​of specific columns or objects. The result is a join based on the fact that each column value or specific entry of an object in the relational database specified by a join query against the database has the same value. Using the AND logical operator, joins can be combined across more than two indexes by merging the join results from each of the pairwise join operations. Attached Figure Description

[0003] The embodiments of this disclosure can be better understood by referring to the accompanying drawings.

[0004] Figure 1 This is a schematic diagram of an example system for connecting JSON query rules across cloud resources.

[0005] Figure 2 This is a schematic diagram of an example system used to parse cloud resource queries by leveraging cloud resource optimization join filters.

[0006] Figure 3 It is a graph showing the computation time and cross product size for legacy joins and optimized joins in cloud resource join queries.

[0007] Figure 4 This is a flowchart of a sample operation used to evaluate cloud resource join queries.

[0008] Figure 5 This is a flowchart of an example operation used to evaluate the results of a cloud resource JSON query based on first-order predicates to generate a pairwise join set.

[0009] Figure 6 An example computer system with a cloud resource join query resolver is described.

[0010] describe

[0011] The following description includes example systems, methods, techniques, and program flows that embody embodiments of the present disclosure. However, it is to be understood that the present disclosure can be practiced without these specific details. For example, in an illustrative example, the present disclosure relates to the results of a JavaScript Object Notation (JSON) query concatenating across cloud resources. Aspects of the present disclosure may instead be applied to the results of queries concatenating across cloud resources against other file formats. In other instances, well-known examples of instructions, protocols, structures, and techniques have not been shown in detail so as not to obscure the description.

[0012] Overview

[0013] The optimized joins for cloud resources disclosed in this paper evaluate cloud resource join queries by dynamically indexing query results from each cloud resource and evaluating and merging pairwise joins on the dynamic index of query results. A join query parser parses the cloud resource join query to isolate join rules corresponding to first-order predicates about cloud resource pairs from JavaScript object notation (“JSON”) rules corresponding to JSON files about individual cloud resources. An information retrieval system parses the JSON rules to generate a query for a database of fingerprints corresponding to cloud resources and retrieves the corresponding cloud resource fingerprint results. The optimized join filter includes a dynamic indexer that generates an index corresponding to each first-order predicate for each cloud resource pair, and a search engine uses the index to find pairwise cloud resource join results for each first-order predicate. The optimized join filter merges pairwise cloud resource join results by identifying results that include fingerprints from each cloud resource that simultaneously satisfy each first-order predicate, to generate cloud resource join query results.

[0014] Example illustration

[0015] Figure 1This is a schematic diagram of an example system for connecting JSON query rules across cloud resources. Computing device 101 transmits a cloud resource join query 100 to a join query parser 103. The join query parser 103 parses the cloud resource join query 100 and divides it into join rules 140 and JSON query rules 142 for querying individual cloud resources. Cloud resource information retrieval system 105 receives the JSON query rules 142 and generates a cloud database query 106 for one or more cloud resources. Cloud resource information retrieval system 105 transmits the cloud database query 106 to a cloud fingerprint database 107. Cloud fingerprint database 107 returns cloud resource query results, including cloud resource X query result 110 and cloud resource Y query result 108. Cloud resource information retrieval system 105 transmits the cloud resource X query result 110 and cloud resource Y query result 108 to a join filter 109. The connection filter 109 additionally receives the connection rule 140 and applies the connection rule 140 to the cloud resource X query result 110 and the cloud resource Y query result 108 to generate the connection query result 112, and the connection filter 109 transmits the connection query result 112 to the computing device 101.

[0016] Example cloud resource connection query 120 includes the following syntax:

[0017] config where api.name='cloudA'as X

[0018] config where api.name='cloudB'as Y

[0019] filter$X.pathA contains itemA and_Fn($Y.pathB)<40and$X.joinValueequals$Y.joinValue

[0020] Example cloud resource join query 120 is a combination of JSON rules defined on objects in a JSON file on a cloud resource and joins on filtered cloud resources after the JSON rules have been applied. The first line defines variable X as a cloud resource with api.name "CloudA", and the second line defines variable Y as a cloud resource with api.name "CloudB". The third line specifies the join rules (using the "filter..." syntax) for all JSON object pairs in cloud resources CloudA and CloudB filtered according to the first two lines, such that the path in X contains item A, the custom function _Fn(x) applied to "x = path in Y" is less than 40, and the value of X.joinValue is equal to Y.joinValue. Although described for two cloud resources, example cloud resource join query 120 can include filters (i.e., join rules) across any number of cloud resources. The results from the JSON rules for each cloud resource can be combined according to the join rules, using combinations of joins and merging the resulting join sets, as described in various ways below.

[0021] The join query parser 103 parses the cloud resource join query 100 into a formatted join query 102 containing join rule 140 and JSON query rule 142. The join query parser 103 can extract JSON rules from the "filter..." section of the cloud resource join query 100 and insert them into the corresponding JSON rule sections of the cloud resource join query 100. For example, sample join rule 122 after extracting the JSON rules includes the following:

[0022] filter$X.joinValue equals$Y.joinValue

[0023] Example JSON query rules 124 and 126 include the following:

[0024] config where api.name='cloudB'and json.rule=_Fn(pathB)<40as Y

[0025] config where api.name='cloudA'and json.rule=pathA contains'itemA'asX

[0026] In this example, join query parser 103 generates example JSON query rules 124 and 126 by detecting the "$..." syntax in the filter segment of example cloud resource join query 120. Subsequently, join query parser 103 identifies segments corresponding to cloud resources X and Y in the corresponding JSON rules of example cloud resource join query 120, adds the corresponding cloud resource segments to the "config..." segment, and translates the syntax using predefined syntax rules that can depend on the implemented query format. The JSON rule segments in the "filter..." segment of example cloud resource join query 120 are replaced by the corresponding cloud resource identifiers (X and Y) in the resulting example join rule 140.

[0027] The cloud resource information retrieval system 105 receives JSON query rules 142 and uses them to generate cloud database queries 106. The cloud resource information retrieval system 105 may include a JSON query parser that generates queries for cloud resources based on the logical rules in the JSON query rules 142. The JSON query parser 103 may be a compiler-compiler that can handle first-order logical rules defined by context-free syntax. The cloud resource information retrieval system 105 can use the generated logical rules to identify a set of one or more cloud resources associated with those logical rules. For example, the cloud resource information retrieval system 105 can use profiles and / or fingerprints maintained for each cloud resource to detect key / value pairs, fingerprints, and / or other metadata identifying cloud resources present in the logical rules. In the given example, the cloud resource information retrieval system 105 may identify cloud resources “cloudA” and “cloudB” based on the API names indicated in the example JSON query rule 142. Subsequently, the cloud resource information retrieval system 105 uses the generated logical rules to generate a query for the identified cloud resource. This query is performed in the cloud fingerprint database 107 using a query format specific to the identified cloud resource. The cloud resource information retrieval system 105 then transmits a cloud database query 106, including the generated query, to the cloud fingerprint database 107. The cloud fingerprint database 107 returns the query result 110 for cloud resource X and the query result 108 for cloud resource Y to the cloud resource information retrieval system 105.

[0028] The cloud fingerprint database 107 may include fingerprints of objects stored in JSON files on known cloud resources. Fingerprints may include identifiers for objects that are frequently queried for each corresponding cloud resource. The cloud fingerprint database 107 may maintain a profile for each cloud resource as it receives queries. The profile may indicate the frequency of the queried object and / or other characteristics, statistics, identifiers, etc., of each object. JSON objects with low query frequency within a time period (e.g., after a schedule-based check) may be removed from the cloud fingerprint database 107. In some embodiments, when the cloud fingerprint database 107 receives the cloud database query 106, some of the query results may return null values. However, cloud resource query results 108 and 110 may include objects that contain key / value pairs from queries that returned null values. Storing cloud resources as fingerprints reduces storage costs while maintaining the join functionality of cloud queries.

[0029] Join filter 109 receives join rule 140, cloud resource X query result 110, and cloud resource Y query result 108. Join filter 109 uses a dynamic indexer, a search engine, and a join result merger to generate join query result 112, which is then transmitted to computing device 101. Figure 2 The operation for joining cloud resource query results using join rules is described in more detail. Computing device 101 may include a user interface that allows a user to generate cloud resource join queries and adjust parameters of the algorithm used to join cloud resource queries.

[0030] Figure 2This is a schematic diagram of an example system for parsing cloud resource queries using a cloud resource optimized join filter. A cloud resource fingerprint generator 201 receives a cloud resource query result 200 and uses it to generate an updated fingerprint 202. The cloud resource fingerprint generator 201 transmits the generated fingerprint 202 to a cloud fingerprint database 203 for subsequent storage. In a separate but parallel pipeline, a computing device 231 transmits a cloud resource join query 240 to a cloud resource information retrieval system 205. The cloud resource information retrieval system 205 extracts a cloud resource query 206 from the cloud resource join query 240, and the cloud fingerprint database 203 receives the cloud resource query 206. The cloud fingerprint database 203 returns a cloud resource fingerprint result 204. The cloud resource information retrieval system 205 transmits the cloud resource fingerprint result 204 extracted from the cloud resource join query 240 and a cloud resource join rule 208 to a cloud resource optimized join filter 241, which includes a dynamic indexer 207. Dynamic indexer 207 generates an indexed cloud resource fingerprint 210, which is received by search engine 209. Search engine 209 generates a pairwise cloud resource join set 212 from the indexed cloud resource fingerprint 210. Pairwise join merger 211 receives the pairwise cloud resource join set 212 and generates an N-wise cloud resource join result 214, which is received by computing device 231.

[0031] The cloud resource fingerprint generator 201 can parse the cloud resource query result 200 to identify the cloud resource and JSON object corresponding to the query result 200. The cloud resource fingerprint generator 201 can query the cloud fingerprint database 203 for existing cloud fingerprints corresponding to the query result 200 and update these fingerprints accordingly. For example, the example updated fingerprint 220 includes the following fields:

[0032] path = 'X.pathA'

[0033] frequency = 15

[0034] path = 'X.pathB'

[0035] frequency = 0.03

[0036] The cloud resource fingerprint generator 201 can receive previous frequency values ​​for X.pathA and X.pathB from cloud resource queries and update these previous frequency values ​​based on queries for X.pathA and X.pathB in the cloud resource query results 200. In some embodiments, the cloud fingerprint includes other statistical information and / or characteristics, such as the known importance of the JSON object in the fingerprint, the fingerprint identifier, etc. The cloud fingerprint database 203 can be indexed by the cloud identifier and path location of the JSON object corresponding to the stored fingerprint.

[0037] Cloud resource join query 240 includes a combination of JSON rules and join rules, wherein the JSON rules include rules for querying JSON objects on cloud resources, and the join rules include rules for combining JSON object results across cloud resources. Cloud resource information retrieval system 205 may include a cloud resource join query parser, which can extract the JSON rules and join rules from cloud resource join query 240, and can further extract cloud resource query 206 using the JSON rules and a JSON query parser generated by a compiler using context-free syntax. Cloud resource query 206 may include a list of cloud resources and the corresponding paths in each cloud resource for which a fingerprint is to be retrieved. Cloud resource information retrieval system 205 transmits the cloud resource fingerprint results 204 returned by cloud fingerprint database 203 and the cloud resource join rules 208 generated from cloud resource join query 240 to dynamic indexer 207.

[0038] Cloud resource join rule 208 includes first-order predicates that define join rules between fingerprints retrieved for cloud resource pairs. The full join rule expressed by cloud resource join rule 208 includes the intersection of each of the pairwise join rules expressed by the first-order predicates. Dynamic indexer 207 generates an index for each first-order predicate, search engine 209 evaluates each of the first-order predicates for the corresponding fingerprint in the cloud resource fingerprint results 204, and pairwise join merger 211 performs intersection and deduplication in the pairwise cloud resource join set 212. Example first-order predicates in cloud resource join rule 208 include the following:

[0039] FOPCContains($.X.path1,$.Y.path2)and FOPCNotEquals($.Y.path1,$.Z.path2)

[0040] The first-order predicate specifies the join rule between cloud resource X and cloud resource Y, namely, the JSON object at X.path1 contains the JSON object at Y.path2. The second first-order predicate specifies the join rule between cloud resource Y and cloud resource Z, namely, the JSON object at Y.path1 is not equal to the JSON object at Z.path2, and the combined join rule specifies the triples of JSON objects across cloud resources X, Y, and Z that satisfy these two first-order predicates.

[0041] Dynamic indexer 207 generates an index for each first-order predicate between cloud resource pairs in cloud resource fingerprint results 204. Dynamic indexer 207 can use any indexing pattern that facilitates efficient file format lookup. For example, dynamic indexer 207 can represent an inverted index or use a B+ tree to generate each index. The type of index can depend on the statistics of cloud resource fingerprint results 204, such as the number of distinct fingerprints for each cloud resource, the frequency of JSON object queries corresponding to the fingerprints, or similarity of other statistics. B+ trees can be implemented for cloud resources with more distinct fingerprints in cloud resource fingerprint results 204, while balanced trees can be used for cloud resources with fewer distinct fingerprints. The index can be generated based on the first-order predicates of the corresponding cloud resources. For example, if the first-order predicates specify equality between X.path1 and Y.path2 for cloud resources X and Y respectively, the index can be generated based on object paths X.path1 and Y.path2. The index can be dynamically updated and maintained as queries in cloud resource query 206 are processed by cloud fingerprint database 203, and the index can be linked to... Figure 2 Other operations described in the text are maintained in parallel.

[0042] Search engine 209 receives indexed cloud resource fingerprints 210 generated by dynamic indexer 207 and uses the generated index to apply pairwise join rules corresponding to the first-order predicates in cloud resource join rules 208. For example, if the first-order predicate specifies that X.path1 in cloud resource X equals Y.path2 in cloud resource Y, then search engine 209 can search for fingerprints in the indexed cloud resource fingerprints 210 where the fingerprint along X.path1 in cloud resource X matches the fingerprint along Y.path2 in cloud resource Y (the index can be constructed for exactly such a lookup operation). Search engine 209 transmits a pairwise cloud resource join set 212 to pairwise join merger 211, which includes pairs of JSON objects satisfying pairwise join rules expressed as first-order predicates.

[0043] Pairwise join merger 211 merges pairwise cloud resource join sets 212 to generate N-wise cloud resource join result 214. Pairwise join merger 211 searches for tuples of fingerprints in each database represented by cloud resource join rule 208, such that a pair of fingerprints in the tuple satisfies every corresponding first-order predicate (i.e., the pair of fingerprints exists in the tuple for each pair of cloud resource join set 212). Pairwise join merger 211 can fix a pair of fingerprints in one of the pairwise cloud resource join sets 212, can scan a second resource join set in the pairwise cloud resource join set 212 for another pair of fingerprints, such that all three resulting fingerprints simultaneously satisfy all join rules for the corresponding cloud resources, and so on. This operation can be optimized using standard libraries / algorithms for multiple join sets. After merging, cloud resource optimized join filter 241 can query cloud resources for objects stored on the cloud resources corresponding to the fingerprints in the N-wise cloud resource join result 214. The pairwise join merger 211 transmits the resulting N-wise cloud resource join result 214 (where N is the number of cloud resources present in the resulting join set) to the computing device 213 as a return of the cloud resource join query 240. The pairwise join merger 211 may additionally or alternatively transmit query results corresponding to the query for cloud resources (which is directed to the objects indicated in the N-wise cloud resource join result 214) to the computing device 213.

[0044] Figure 3 This is a graph showing the computation time versus cross-product size for legacy and optimized joins for cloud resource join queries. The cross-product size represents the product of the number of item / value pairs across the JSON file for each cloud resource in the cloud resource join query. For example, the cross-product size for a cloud resource join query across cloud resources X, Y, and Z with sizes 20, 100, and 50 is 20 x 100 x 50 = 100,000. The cross-product represents the search space using the legacy join. As shown in Table 300, the optimized join presented in this paper, which uses dynamic indexes after first applying the JSON rules to the cloud resources, outperforms the legacy join after a cross-product size of 25,000. Conversely, the legacy join, which performs a brute-force search across the entire cross-product space without first applying the JSON rules, has lower overhead due to the dynamic indexes and therefore outperforms the dynamically indexed join at lower cross-product sizes, but suffers as the cross-product size increases. The threshold cross product size of 25,000 is arbitrary and depends on the implementation of legacy and optimized joins, the file format type in cloud resource join queries, and should be determined based on the empirical performance of each join.

[0045] To maintain consistency with earlier graphs(s), example operations are described with reference to join query parsers, information retrieval systems, and optimized join filters. The names chosen for the program code are not a limitation of the claims. The structure and organization of the program may vary due to platform, programmer / architect preferences, programming language, etc. Furthermore, the names of code units (programs, modules, methods, functions, etc.) may vary for the same reasons and can be arbitrary.

[0046] Figure 4 This is a flowchart of a sample operation used to evaluate a cloud resource join query. At box 401, the join query parser separates the JSON rules from the join rules in the cloud resource join query. The join rules consist of first-order predicates about objects in the cloud resource pair. The JSON rules consist of rules used to query cloud resources that include JSON files. Although described as JSON rules, these rules can be rules for any file format residing on the cloud resource.

[0047] At box 402, the optimized join filter determines whether the cross product size of a join rule is higher than an optimization threshold cross product size. The cross product size is the product of the number of item-value pairs in the file on each cloud resource, multiplying across cloud resources. This corresponds to the order of magnitude of the search space for brute-force join algorithms. The optimization threshold is chosen as a cross product size at which the optimized join filtering algorithm depicted in boxes 403, 405, 407, 409, 411, 413, and 415 outperforms the old (i.e., brute-force) join filtering algorithm. This optimization threshold can be determined empirically and may depend on the cloud resources being queried and the implementation of each algorithm. For example, the optimization threshold can be determined by running optimized and old joins on the same cloud resource join queries with varying cross product sizes and across varying cloud resources. Therefore, the optimization threshold is a cross product size at which the optimized join outperforms the old join in terms of average computation time across cloud resource join queries for that cross product size. If the cross product size is higher than the optimization threshold, the operation proceeds to box 403. Otherwise, the operation skips to box 417.

[0048] At box 403, the information retrieval system begins iterating through the JSON rules separated in the cloud resource join query. Boxes 405 and 407 depict the operations at each iteration.

[0049] At box 405, the information retrieval system uses a JSON query parser with a context-free first grammar and first-order logic to evaluate the current JSON rules to generate a cloud resource fingerprint query. The JSON query parser can be generated by a compiler—which uses a context-free grammar as input. The information retrieval system can then transform the JSON query obtained from the JSON query parser into a cloud fingerprint database-specific format (e.g., an indexing schema based on the cloud fingerprint database).

[0050] At box 407, the information retrieval system queries the cloud fingerprint database to retrieve cloud resource fingerprint results corresponding to the current JSON query. This query indicates characteristics of the objects indicated in the JSON query, such as the path corresponding to the objects in the JSON query. The query can be formatted based on the type of index used in the cloud fingerprint database. The cloud fingerprint database can be indexed, allowing it to efficiently retrieve cloud resource fingerprint results based on queries (e.g., by using the JSON file path used for cloud fingerprints).

[0051] At box 409, the information retrieval system determines whether an additional JSON rule exists in the cloud resource query. If an additional JSON rule exists, the operation returns to box 403. Otherwise, the operation proceeds to box 411.

[0052] At box 411, the optimized join filter parses the join rules from the cloud resource join query to generate first-order predicates for the cloud resource pairs. For example, the join rule could have the following form:

[0053] filter X.path1 equal Y.path2

[0054] Furthermore, the resulting first-order predicate can have the following form:

[0055] FOPCEquals(X.path1,Y.path2)

[0056] In these two instances, the join operation corresponds to the fingerprints (such as “FOPCEquals”, i.e., the first-order predicate equals syntax) of the join having equal paths 1 (in cloud resource X) and 2 (in cloud resource Y).

[0057] At box 413, the optimized join filter evaluates the results of the cloud resource JSON query based on the first-order predicate to generate paired join sets. Figure 5 The operation at box 413 is described in more detail.

[0058] At box 415, the optimized join filter merges paired join sets. The optimized join filter determines tuples of fingerprints in the cloud resource fingerprint results such that each fingerprint pair in the tuple corresponding to a cloud resource pair in the paired join set exists in that set. Standard search techniques for merging join sets (e.g., merging Structured Language Query (SQL) inner join operations) can be implemented.

[0059] If it is determined that the cross product size of the join rule is not higher than the optimized threshold cross product size, then at box 417, the optimized join filter applies legacy joins to the cloud resource join query. A legacy join can be a brute-force search of the cross products of cloud resources present in the cloud resource join query against the join rules in the cloud resource join query. Alternatively, it can use the previously natively implemented join operation.

[0060] Figure 5 This is a flowchart of example operations used to evaluate the results of a cloud resource JSON query based on a first-order predicate to generate a set of paired joins. At box 503, the optimized join filter begins iterating through the first-order predicate to be applied to the cloud resources. Operations at each iteration occur at boxes 505 and 507.

[0061] At box 505, the optimized join filter generates a dynamic index for cloud resource pairs corresponding to the current first-order predicate. The dynamic index can be generated based on the current first-order predicate (e.g., by the file path indicated in the current first-order predicate). For example, the dynamic index can be an inverted index that includes a mapping between key / value pairs in the cloud resources and their corresponding document identifiers. For two cloud resources X and Y with paths X.vpcId and Y.resourceId respectively, where X.vpcId takes values ​​from the set {a, b}, the dynamic index of X can map a and b to the set of document identifiers as follows:

[0062] a->{1,20,111,...}

[0063] b->.{15,18,19}

[0064] Key / value pairs in Y are mapped in a similar manner, such that any key / value pair in Y with the value "a" will have a document identifier in the list {1, 20, 111, ...}, and any key / value pair in Y with the value "b" will have a document identifier in the list {15, 18, 19}. Dynamic indexes can be maintained and updated when computing join sets and when updating fingerprints of cloud resources.

[0065] At box 507, the optimized join filter uses a dynamic index to search for cloud resource fingerprint results for the corresponding cloud resources to generate pairwise cloud resource join results. The optimized join filter can parse the current first-order predicate to identify fingerprints in the corresponding results of the JSON query for the corresponding cloud resource, thereby determining fingerprint pairs that satisfy the first-order predicate to add to the pairwise join results for the current first-order predicate. In the example above, the calculation of join filter $X.vpcId equals $Y.resourceId includes: searching for Y against document identifiers in the list {1,20,111,...} and determining whether the corresponding JSON file path in Y makes Y.resourceID = 'a'. The calculation of join filter additionally includes: searching for Y against document identifiers in the list {15,18,19} and determining whether the corresponding JSON file path in Y makes Y.resourceId = 'b'. The result from the join filter is the corresponding file path in X and Y for JSON file paths with matching key / value pairs in the set {a,b}.

[0066] At box 509, optimize the join filter to determine if there are any additional JSON rules in the cloud resource query. If additional JSON rules exist, the operation returns to box 503. Otherwise, Figure 5 The operation is complete.

[0067] The flowcharts are provided to aid in understanding the illustrations and not to limit the scope of the claims. The flowcharts depict example operations that can vary within the scope of the claims. Additional operations may be performed; fewer operations may be performed; operations may be performed in parallel; and operations may be performed in a different order. For example, the operations depicted in blocks 403, 405, 407, and 409 may be performed in parallel or simultaneously. Regarding... Figure 4 Instead, it is not necessary to apply legacy join operations to cloud resource join queries. It will be understood that each box in the flowchart diagram and / or block diagram, as well as combinations of boxes in the flowchart diagram and / or block diagram, can be implemented via program code. The program code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable machine or device.

[0068] As will be understood, aspects of this disclosure may be embodied as a system, method, or program code / instructions stored in one or more machine-readable media. Therefore, aspects may take the form of hardware, software (including firmware, resident software, microcode, etc.), or a combination of both, which may be collectively referred to herein as a “circuit,” “module,” or “system.” The functionality presented as individual modules / units in the example illustrations may be organized differently based on any of the following: platform (operating system and / or hardware), application ecosystem, interface, programmer preferences, programming language, administrator preferences, etc.

[0069] Any combination of one or more machine-readable media may be used. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable storage medium can be, for example, but not limited to, a system, apparatus, or device that uses any or a combination of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor technologies to store program code. More specific examples (a non-exhaustive list) of machine-readable storage media will include: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROP or flash memory), portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a machine-readable storage medium can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable storage medium is not a machine-readable signal medium.

[0070] Machine-readable signal media may include propagated data signals embodying machine-readable program code, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A machine-readable signal medium may be any machine-readable medium that is not a computer-readable storage medium and can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0071] The program code embodied on a machine-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination of the foregoing.

[0072] Program code / instructions may also be stored in a machine-readable medium that can instruct a machine to operate in a particular manner, such that the instructions stored in the machine-readable medium produce an article of art, which includes instructions that implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0073] Figure 6An example computer system with a cloud resource join query parser is depicted. The computer system includes a processor 601 (which may include multiple processors, multiple cores, multiple nodes, and / or implement multithreading, etc.). The computer system includes memory 607. Memory 607 may be system memory or any one or more of the possible implementations of a machine-readable medium described above. The computer system also includes a bus 603 and a network interface 605. The system also includes a cloud resource join query parser 611. The cloud resource join query parser 611 can parse cloud resource join queries to isolate join rules and JSON rules, parse JSON rules to generate queries against a cloud fingerprint database, retrieve corresponding query results from the cloud fingerprint database, and use dynamic indexing / search to join query results from the cloud fingerprint database. Any of the previously described functionalities can be implemented partially (or entirely) in hardware and / or on processor 601. For example, the functionality can be implemented using an application-specific integrated circuit, logic implemented in processor 601, a coprocessor on a peripheral device or card, etc. Furthermore, the implementation may include fewer components or Figure 6 Additional components not shown (e.g., video card, audio card, additional network interface, peripheral devices, etc.). Processor 601 and network interface 605 are coupled to bus 603. Although shown as coupled to bus 603, memory 607 may be coupled to processor 601.

[0074] While various implementations and exploits have been described in relation to aspects of this disclosure, it will be understood that these aspects are illustrative and the scope of the claims is not limited thereto. In general, the techniques described herein for parsing cloud resource queries to isolate join rules and JSON rules, and then subsequently applying the JSON rules to the cloud resources before the join rules to optimize join operations, can be implemented using facilities consistent with any one or more hardware systems. Many variations, modifications, additions, and improvements are possible.

[0075] Multiple instances may be provided for a component, operation, or structure described herein as a single instance. Finally, the boundaries between various components, operations, and data stores are somewhat arbitrary and specific operations are described within the context of the particular illustrative configuration. Other allocations of functionality are conceivable and may fall within the scope of this disclosure. In general, structures and functionalities presented as separate components in the example configuration can be implemented as composite structures or components. Similarly, structures and functionalities presented as single components can be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of this disclosure.

[0076] the term

[0077] For efficiency and ease of explanation, this description uses abbreviated terms related to cloud technology. When referring to "cloud resources," this description means resources of a cloud service provider. For example, cloud resources can encompass a cloud service provider's servers, virtual machines, and storage devices. More generally, cloud service provider resources accessible to customers are resources owned / managed by the cloud service provider entity and accessible via a network connection. Typically, access is made through application programming interfaces (APIs) or software development kits provided by the cloud service provider.

[0078] As used in this article, "fingerprint" refers to a representation of one or more key / value pairs stored on a cloud resource. A fingerprint can be, for example, an object stored in a JSON file on a cloud resource. The scope of a fingerprint varies with its size / range and the choice of representation. Fingerprints are used as a proxy for storage that is more efficient than individual key / value pairs on a cloud resource and allow for more efficient join operations within a smaller search space.

[0079] The phrase "at least one of..." preceding a list containing the conjunction "and" should not be considered an exclusive list and should not be interpreted as a list of categories with one item from each category, unless otherwise specifically stated. A clause containing "at least one of A, B, and C" may be infringe upon only one of the listed items, multiple of the listed items, one or more of the items in the list, and another unlisted item.

[0080] Example Implementation

[0081] Example implementations include the following.

[0082] Example 1: A method comprising: parsing a cloud resource join query to extract a first plurality of join rules and a first plurality of cloud resource queries, wherein the first plurality of join rules include a first plurality of first-order predicates for cloud resource pairs in the first plurality of cloud resources; querying a cloud fingerprint database for a first plurality of fingerprints of the first plurality of cloud resources associated with the first plurality of cloud resource queries; joining fingerprints in the first plurality of fingerprints according to the first plurality of first-order predicates in the first plurality of join rules to generate a first plurality of pairwise join results; and merging the first plurality of pairwise join results.

[0083] Example 2: According to the method described in Example 1, the fingerprints in the first plurality of fingerprints are joined according to the first predicates, which includes: generating a first plurality of dynamic indexes for each cloud resource pair corresponding to the first plurality of first predicates, and using the plurality of dynamic indexes to search for each cloud resource pair corresponding to the first plurality of first predicates for the result of satisfying at least one of the first plurality of first predicates.

[0084] Example 3: According to the method of Example 1, the first plurality of fingerprints includes the characteristics of querying one or more objects stored on the first plurality of cloud resources.

[0085] Example 4: According to the method of Example 3, the characteristics of querying one or more objects stored on a first plurality of cloud resources include the frequency of querying the one or more objects.

[0086] Example 5: The method described in Example 3 further includes updating the first plurality of fingerprints based on the most recent query to the first plurality of cloud resources.

[0087] Example 6: The method according to Example 1 further includes querying a first plurality of cloud resources for objects indicated in the merged first plurality of pairwise join results.

[0088] Example 7: The method described in Example 6 further includes transmitting at least one of the merged first plurality of paired connection results and the result of querying the first plurality of cloud resources to a computing device communicatively coupled to a user interface.

[0089] Example 8: According to the method described in Example 1, the parsing of the cloud resource join query is at least in part based on the determination that the cross product size of the cloud resource join query is higher than a threshold cross product size.

[0090] Example 9: One or more non-transitory machine-readable media, including program code for: parsing cloud resource join queries to extract a first plurality of join rules and a first plurality of cloud resource queries, wherein the first plurality of join rules include a first plurality of first-order predicates for cloud resource pairs in the first plurality of cloud resources; querying a cloud fingerprint database for a first plurality of fingerprints of the first plurality of cloud resources associated with the first plurality of cloud resource queries; joining fingerprints in the first plurality of fingerprints according to the first plurality of first-order predicates in the first plurality of join rules to generate a first plurality of pairwise join results; and merging the first plurality of pairwise join results.

[0091] Example 10: The non-transitory machine-readable medium according to Example 9, wherein the program code for concatenating fingerprints in a first plurality of fingerprints according to the first-order predicate includes program code for: generating a first plurality of dynamic indexes for each cloud resource pair corresponding to the first plurality of first-order predicates, and using the plurality of dynamic indexes to search for each cloud resource pair corresponding to the first plurality of first-order predicates for a result satisfying at least one of the first plurality of first-order predicates.

[0092] Example 11: The non-transitory machine-readable medium according to Example 9, wherein the first plurality of fingerprints includes the characteristic of querying one or more objects stored on the first plurality of cloud resources.

[0093] Example 12: The non-transitory machine-readable medium according to Example 11, wherein the characteristics of querying one or more objects stored on a first plurality of cloud resources include the frequency of querying the one or more objects.

[0094] Example 13: The non-transitory machine-readable medium according to Example 11 further includes program code for updating the first plurality of fingerprints based on the most recent query to the first plurality of cloud resources.

[0095] Example 14: The non-transitory machine-readable medium according to Example 9 further includes program code for querying a first plurality of cloud resources for objects indicated in the merged first plurality of pairwise join results.

[0096] Example 15: The non-transitory machine-readable medium according to Example 14 further includes program code for transmitting at least one of the merged first plurality of paired join results and the result of querying the first plurality of cloud resources to a computing device communicatively coupled to a user interface.

[0097] Example 16: According to the non-transitory machine-readable medium of Example 9, the program code for parsing the cloud resource join query is at least in part based on the determination that the cross product size of the cloud resource join query is higher than a threshold cross product size.

[0098] Example 17: An apparatus comprising a processor and a machine-readable medium having program code executable by the processor such that the apparatus: parses a cloud resource join query to extract a first plurality of join rules and a first plurality of cloud resource queries, wherein the first plurality of join rules include a first plurality of first-order predicates for cloud resource pairs in the first plurality of cloud resources; queries a cloud fingerprint database for a first plurality of fingerprints of the first plurality of cloud resources associated with the first plurality of cloud resource queries; joins the fingerprints in the first plurality of fingerprints according to the first plurality of first-order predicates in the first plurality of join rules to generate a first plurality of pairwise join results; and merges the first plurality of pairwise join results.

[0099] Example 18: The apparatus according to Example 17, wherein the program code executable by the processor to cause the apparatus to join fingerprints in a first plurality of fingerprints according to the first-order predicate includes program code that is executable by the processor to cause the apparatus to: generate a first plurality of dynamic indexes for each cloud resource pair corresponding to the first plurality of first-order predicates, and use the plurality of dynamic indexes to search for each cloud resource pair corresponding to the first plurality of first-order predicates for a result that satisfies at least one of the first plurality of first-order predicates.

[0100] Example 19: The apparatus according to Example 17, wherein the first plurality of fingerprints includes the characteristic of querying one or more objects stored on the first plurality of cloud resources.

[0101] Example 20: The apparatus according to Example 19, wherein the characteristics of querying one or more objects stored on a first plurality of cloud resources include the frequency of querying the one or more objects.

Claims

1. A method for querying cross-cloud resources, comprising: The cloud resource join query is parsed to extract multiple join rules and multiple cloud resource query rules, and the multiple join rules are separated from the multiple cloud resource query rules, wherein the multiple join rules include multiple first-order predicates for cloud resource pairs in multiple cloud resources; Multiple cloud resource queries are generated based on the aforementioned multiple cloud resource query rules; The cloud fingerprint database is queried using the multiple cloud resource queries to obtain multiple fingerprints, wherein each of the multiple fingerprints corresponds to a cloud resource among the multiple cloud resources; The fingerprints in the multiple fingerprints are joined according to the multiple first-order predicates in the multiple join rules to generate multiple pairwise join results; as well as Merge the multiple pairwise join results, The fingerprints connected among the multiple fingerprints according to the first-order predicate include: For each cloud resource pair corresponding to the plurality of first-order predicates, generate multiple dynamic indexes; as well as For a result that satisfies at least one of the plurality of first-order predicates, the plurality of dynamic indexes are used to search for each cloud resource pair corresponding to the plurality of first-order predicates.

2. The method of claim 1, wherein the plurality of fingerprints includes characteristics for querying one or more objects stored on the plurality of cloud resources.

3. The method of claim 2, wherein the characteristics of querying one or more objects stored on the plurality of cloud resources include the frequency of querying the one or more objects.

4. The method of claim 2, further comprising updating the plurality of fingerprints based on the most recent query to the plurality of cloud resources.

5. The method of claim 1, further comprising querying the plurality of cloud resources for objects indicated in the merged plurality of pairwise join results.

6. The method of claim 5, further comprising transmitting at least one of the merged plurality of pairwise connection results and the result of querying the plurality of cloud resources to a computing device communicatively coupled to a user interface.

7. The method according to claim 1, wherein parsing the cloud resource join query is based on determining that the cross product size of the cloud resource join query is higher than a threshold cross product size.

8. A non-transitory machine-readable medium comprising program code for performing the method according to any one of claims 1-7.

9. An apparatus comprising a processor and a machine-readable medium having program code executable by the processor to cause the apparatus to perform the method according to any one of claims 1-7.

Images