A processing method for performing a screening operation oriented to a heterogeneous memory system

Abstract

The application discloses a processing method for performing screening operation for a heterogeneous storage system, and relates to the technical field of heterogeneous storage system screening, and comprises the following steps: decomposing the conditions of a screening method into a plurality of sub-screenings, removing the conditions different from an initiator query source to be queried, screening the data in the query source according to the screening conditions after the heterogeneous sub-screenings are removed, screening other query sources, obtaining a heterogeneous result set, screening according to the relationship between an original result set and the heterogeneous result set, removing the data not meeting the relationship condition between the original result set and the heterogeneous result set from the original result set, obtaining a result set after relationship reference screening, checking the result set after relationship reference screening, screening, and obtaining the data as a final result.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous storage system screening technology, and more specifically to a processing method for performing screening operations for heterogeneous storage systems. Background Technology

[0002] Object-Relational Management (ORM) is a crucial task in object-oriented software engineering. These systems support business systems, and the most common application scenario involves storing the attributes and relationships between objects in a relational database or reconstructing the attributes and relationships from a relational database. To meet the expanding needs of business systems, it is often necessary to extend the types of storage service providers to satisfy different business requirements such as log analysis, object caching, and multi-dimensional computation. Therefore, in extended object-relational management, different storage service provider nodes are added to address the diverse storage service provision needs.

[0003] In practical applications, we often need to deal with situations where data needs to be retrieved from a storage system. The operation of retrieving data is usually called a query. For a query, the data type to be retrieved can be called the source type. If a query operation not only involves the source type, but also involves the association of the source type (including self-associations, i.e., associations between itself), such a query operation is called an association operation. Typical association operations include the following: 1. WHERE operation involving association in filtering conditions; 2. SELECT operation involving projection onto associated references; 3. Grouping operation involving association in grouping keys; 4. Grouping operation involving association in group element functions; 5. Grouping (aggregation) operation involving association in aggregation projection functions; 6. Include operation.

[0004] Association operations encompass various operations, among which the most commonly used is a filtering operation, which combines several logical operations to ultimately determine whether a target meets certain conditions. In a filtering operation, each condition can be independently computed, and conditions of the same type can be combined. These results can be considered as sub-operation results. Clearly, the results of these sub-operations, processed according to their logical rules, are the same as the result of the filtering operation. Therefore, we decompose the conditions of the filtering operation into a tree of depth 2. Its root node is the OR operation, and each child node is a Boolean expression that is not a logical operation or a logical expression with consecutive "AND" operations. The result of the operation of this tree is the same as the original filtering operation. We call these child nodes OR factors. If any one of these OR factors has a storage difference with the others, making it impossible to query both in a single query request, this set of OR factors is called a heterogeneous OR factor. The filtering operation that decomposes to obtain heterogeneous OR factors is called heterogeneous filtering. Otherwise, it is called homogeneous OR factors. The filtering operation with heterogeneous OR factors is called heterogeneous filtering operation. At the same time, these homogeneous OR factors can be combined for operation.

[0005] For heterogeneous filtering, the traditional approach is to query the decomposed or factored data sequentially, step by step, to arrive at the final result. The drawback of this approach is that it cannot merge or eliminate similar queries. For example, if we want to query data of type A, and the condition is a certain attribute of A and a certain attribute of B associated with A, the traditional approach is to first query the A data that meets the condition, and then query the B data that meets the condition from the B associations of these A data. However, if there are multiple conditions for the query, and each condition requires filtering A and B multiple times, it will lead to low efficiency, require a lot of computing resources, and cause waste. Summary of the Invention

[0006] This invention provides a processing method for performing filtering operations on heterogeneous storage systems. It can merge and eliminate similar conditions and schedule filtering operations in a single request, thereby improving query efficiency and reducing the use of computing resources.

[0007] A processing method for performing filtering operations on heterogeneous storage systems includes the following steps:

[0008] Step 1: Decompose the conditions of the filtering method into several sub-filters, remove the conditions that are different from the query source of the initiator to be queried, and the removed sub-filters are called heterogeneous sub-filters. The removed heterogeneous sub-filters are temporarily stored.

[0009] Step 2: Filter the data in the query source according to the filtering conditions after removing isomorphs to obtain the original result set;

[0010] Step 3: Based on the relationships between objects in the first query source and other different storage service providers, and the conditions in the heterogeneous sub-filtering that was removed in Step 1, filter other query sources to obtain a heterogeneous result set.

[0011] Step 4: Filter the results based on the relationship between the original result set and the heterogeneous result set. Remove data that does not meet the relationship conditions between the original result set and the heterogeneous result set from the original result set. After removing all data that does not meet the conditions, the result set after relationship reference filtering is obtained.

[0012] Step 5: Verify the result set after relational reference filtering. Use the heterogeneous filtering temporarily stored in Step 1 to filter the result set after relational reference filtering. The data obtained is the final result.

[0013] Furthermore, the original result set obtained in step 2 contains 0 or more data entries.

[0014] Furthermore, in step 1, decomposing the conditions of the screening method into several sub-screens specifically includes: decomposing the conditions of the screening method into a tree of depth 2, where the root node of the tree is an OR operation, and each child node is a Boolean expression without logical operation or a logical expression with continuous AND operation; using the child nodes as OR factors, the several sub-screens are sub-operations containing OR factors.

[0015] Furthermore, the removal of conditions different from the query source in step 1 specifically means: if there are OR factors that cannot be queried simultaneously with the query source in a single query request due to storage differences, they are identified as heterogeneous OR factors and removed. Those not removed are identified as isomorphic OR factors, and the isomorphic OR factors constitute the screening conditions in step 2.

[0016] Furthermore, the process of obtaining the heterogeneous result set in step 3 includes: creating a view and defining a set of non-intuitive attributes and a set of intuitive attributes for the view; wherein, the calculation method of the non-intuitive attributes is bound to the conditions in the heterogeneous sub-filters that were removed in step 1; the intuitive attributes use the filter key attributes of the query source in step 1 as the source.

[0017] Furthermore, non-intuitive properties return boolean values.

[0018] Furthermore, the process of obtaining the heterogeneous result set in step 3 also includes: creating a projection operation with the view as the target, and obtaining a filtering operation of the filtering attributes of the query source type; concatenating the two into a validation query, executing the validation query, and obtaining the result instance set of the view as the heterogeneous result set.

[0019] Furthermore, in step 2, after obtaining the original result set, the identifiers and results in the original result set are also constructed into a result dictionary.

[0020] Furthermore, the filtering and verification process in steps 4 and 5 specifically includes: for elements in the original result set, finding the corresponding instance from the result instance set of the view based on the identifier; performing continuous AND operations on all non-intuitive attributes of the view whose results are Boolean values; if the operation result is false, removing the element from the original result set; adding elements whose verification results are all true to the result dictionary; and finally returning the value of the result dictionary after execution as the final result.

[0021] Furthermore, the application of the method is premised on the following condition: the screening method is a screening operation that does not involve logical operations or only involves logical AND operations.

[0022] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0023] The processing method for performing filtering operations on heterogeneous storage systems provided by this invention can handle heterogeneous filtering queries that cannot be handled by general heterogeneous queries. Before this method was proposed, there was no similar method for handling heterogeneous filtering operations. This method uses a step-by-step processing approach when handling heterogeneous filtering operations, converting all filtering operations on a certain node in the heterogeneous query into OR factor operations. This allows the query results to be obtained at this node at once and data that does not need to participate in the next node's operation to be excluded in advance, which greatly improves the efficiency of the query and saves computing resources.

[0024] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0025] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0027] Figure 1 This is a flowchart illustrating the processing method for performing screening operations on heterogeneous storage systems disclosed in an embodiment of the present invention.

[0028] Figure 2 This is the flattening filtering operation disclosed in the embodiments of the present invention.

[0029] Figure 3 This invention discloses the generation of isomorphic filtering operations and heterogeneous projection operations and verification views.

[0030] Figure 4 The present invention discloses a method for obtaining result sets based on isomorphic filtering operations and heterogeneous projection operations and views.

[0031] Figure 5 This invention discloses a method for merging and obtaining the final result set in an embodiment of the invention. Detailed Implementation

[0032] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0033] definition:

[0034] Query source: For a filtering operation, the storage provider from which the data is filtered can be called the query source for this condition.

[0035] View: A type view is a partial observation of a type and the object system centered on it. Formally, a type view is a temporary type composed of elements of the type and elements of its associated (direct or indirect) types. This type is called the source of the view, and the object system centered on the source type is called the source extension. It can be represented by an association tree, with the root node representing the source type.

[0036] Intuitive and non-intuitive properties of a view: A view has several view properties. If a view property can be directly obtained from its source value without any calculation, it is called an intuitive property; otherwise, it is called a non-intuitive property.

[0037] like Figure 1 As shown, this embodiment of the invention provides a processing method for performing filtering operations on heterogeneous storage systems, including the following steps:

[0038] Step 1: Decompose the conditions of the filtering method into several sub-filters, remove the conditions that are different from the query source of the initiator to be queried, and the removed sub-filters are called heterogeneous sub-filters. The removed heterogeneous sub-filters are temporarily stored.

[0039] Step 2: Filter the data in the query source according to the filtering conditions after removing heterogeneous components to obtain the original result set, which contains more than or equal to 0 data entries.

[0040] Step 3: Based on the relationships between objects in the first query source and other different storage service providers, and the conditions in the heterogeneous sub-filtering that was removed in Step 1, filter other query sources to obtain a heterogeneous result set.

[0041] Step 4: Filter the results based on the relationship between the original result set and the heterogeneous result set. Remove data that does not meet the relationship conditions between the original result set and the heterogeneous result set from the original result set. After removing all data that does not meet the conditions, the result set after relationship reference filtering is obtained.

[0042] Step 5: Verify the result set after relational reference filtering. Use the heterogeneous filtering temporarily stored in Step 1 to filter the result set after relational reference filtering. The data obtained is the final result.

[0043] To enable those skilled in the art to more clearly understand the technical solution of the present invention, the higher-level concepts involved in the foregoing steps and the specific computer processing logic are explained below:

[0044] The relationship between the first query source and objects in other different storage service providers described in this invention is, in the specific computer underlying implementation, manifested as the association and matching attributes between data on different storage nodes. In the specific embodiments described below, this relationship is specifically implemented through the mapping between the filter key attribute of the query source and the object identifier of the data.

[0045] The method of filtering other query sources to obtain a heterogeneous result set, as described in this invention, is instantiated in a specific embodiment as follows: a temporary view is constructed based on the eliminated heterogeneous sub-filters, and a set of verification queries is constructed in combination with the filter key attributes. The result set of the view obtained after executing the verification query is the heterogeneous result set.

[0046] The present invention describes the process of removing data that does not meet the conditions from the original result set to obtain a result set after relational reference filtering and performing verification operations on it. In the specific algorithm logic, this is manifested as a process of identifier matching and Boolean operation: that is, traversing the initially filtered original result set, using object identifiers to find the corresponding verification items from the above view result instance set, and performing logical judgments on view attributes bound to heterogeneous conditions, such as continuous AND operations; if the judgment result is false, it means that the data does not meet the heterogeneous conditions, and the operation of removing (removing) it from the result set is performed. The data that is finally retained and merged is the final result.

[0047] It should be noted that the processing method for performing screening operations for heterogeneous storage systems provided by this invention is only applicable to screening operations where the screening conditions do not involve logical operations or only involve logical AND operations. This is because the multiple OR factors obtained during condition decomposition are ultimately connected using OR operations, and it cannot handle conditions that do not involve any logical operations at all, as these operations cannot be converted to OR operations; nor can it handle conditions where all logical operations are connected using AND operations, as such operations cannot be converted to OR operations.

[0048] like Figures 2-5 As shown, the specific implementation is as follows:

[0049] S1: Let the query source type be T. The nodes where the sub-conditions of this query are stored may be different, i.e., T1, T2...TN. After these conditions are flattened and filtered, several or factor terms are obtained. Heterogeneous terms are removed from them to obtain a filtering operation composed of isomorphic or factors.

[0050] S1a, Specifically: Obtain a heterogeneous filtering operation, and flatten and decompose the conditions of the heterogeneous filtering operation according to the query source of each sub-condition in the operation to obtain a set of isomorphic or factors Or1 and a set of heterogeneous or factors Or2. Remove the heterogeneous or factors Or2 from the set, retain the isomorphic or factors Or1, and temporarily store the removed heterogeneous or factors Or2.

[0051] S2: Perform the filtering operation on the isomorphic or factor composition obtained in S1 to obtain the result set Os, where the number of Os is greater than 0; if the result set obtained at this time is not greater than 0, return the result directly.

[0052] S2a, Specifically: Extract the elements from the isomorphic or factor Or1, construct the or factor into a new filtering operation where, perform this filtering operation where on the data in the query source T to obtain the result set Os, and construct the result dictionary Dict from the identifiers and results in the original result set.

[0053] S3: Create a view and define a set of non-intuitive properties for this view. These properties return boolean values, and the specific calculation method is bound to the heterogeneous items removed in S1.

[0054] It should be noted that a view is created and a set of properties are defined for it. These properties are all of type bool and are bound to the heterogeneous or factor Or2 that is removed in S1a. That is, these properties are calculated from the removed heterogeneous or factor Or2. These properties are non-direct pipe properties.

[0055] S4: Define a set of intuitive properties for the view created in S3, which are sourced from the filter key properties of the query source T in S1.

[0056] Specifically, in S3a~S4a, elements from the heterogeneous or factor Or2 temporarily stored in S1a are extracted, and all or factor items and filter key attributes of the elements are constructed as view attributes viewAttrs. Then, the view view is constructed based on these view attributes viewAttrs.

[0057] In this embodiment, a view attribute represented by a factor item and a filter key attribute can be constructed simultaneously for a heterogeneous or factor item.

[0058] S5: Create a projection operation targeting the view created in S3, create a filtering operation that filters the result set Os in S2 based on the filtering key of the query source T, and concatenate the two into a new query operation.

[0059] S5a, Specifically: Get the view from S3a~S4a, construct the projection operation select to project onto this view, get the query source type T from S1a, construct the filtering operation where for the query source filtering attribute, and concatenate the two into a validation query check.

[0060] S6: Execute the query operation of S5 to obtain the result instance set Vs of the view, where the number of elements V is greater than 0;

[0061] S6a, specifically: execute the check query in S5a to obtain the result set Vs.

[0062] S7: For element O in the result set Os of S2a, find the corresponding V from Vs according to the object identifier, and perform continuous AND operations on all attributes with Boolean values ​​in the view of S3. If the result is false, remove O from the result set Os. After all results have been processed, the result set Os is the final result.

[0063] S7a (Filtering and Removal): Specifically, loop through the result set Os in S2, and validate each element according to viewAttrs in S3a~S4a. If all validation results are true, add them to the dictionary Dict in S2a.

[0064] S7b (Result Merging): Specifically, S2a~S6a are executed in a loop, and the values ​​of the dictionary Dict after execution are returned as the result set Os.

[0065] It should be noted that in this implementation, since the union of the sub-filters is the same as the result of the original filter operation, S2a~S6a are actually performed on the sub-filters. In each loop, the results of the dictionary Dict are merged, and finally the union of all sub-filters is obtained to get the result.

[0066] This method can handle heterogeneous filtering queries that general heterogeneous queries cannot handle. Before this method was proposed, there was no similar method for handling heterogeneous filtering operations. When handling heterogeneous filtering operations, this method uses a step-by-step processing approach, converting all filtering operations on a certain node in the heterogeneous query into OR factor operations. This allows the query results to be obtained at this node all at once, and data that does not need to participate in the next node's operation is excluded in advance, which greatly improves the efficiency of the query and reduces the use of computing resources.

[0067] Using this method to handle heterogeneous filtering operations can efficiently implement filtering operations that do not involve logical operations or only involve logical AND operations. For users, they only need to handle the query methods of the actual nodes, without having to handle the overall logic of heterogeneous filtering operations. The method is fully encapsulated and efficient, and is user-friendly for specific implementation.

[0068] When processing heterogeneous filtering operations, this method no longer requires the previous common method to summarize and calculate the results of all involved nodes, reducing internal communication costs and summary calculation costs, and can effectively improve the stability and overall throughput of the system using this method.

[0069] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0070] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0071] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0072] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0073] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0074] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. A processing method for performing filtering operations on heterogeneous storage systems, characterized in that, Includes the following steps: Step 1: Decompose the conditions of the filtering method into several sub-filters, remove the conditions that are different from the query source of the initiator to be queried, and the removed sub-filters are called heterogeneous sub-filters. The removed heterogeneous sub-filters are temporarily stored. Step 2: Filter the data in the query source according to the filtering conditions after removing isomorphs to obtain the original result set; Step 3: Based on the relationships between objects in the first query source and other different storage service providers, and the conditions in the heterogeneous sub-filtering that was removed in Step 1, filter other query sources to obtain a heterogeneous result set. Step 4: Filter the results based on the relationship between the original result set and the heterogeneous result set. Remove data that does not meet the relationship conditions between the original result set and the heterogeneous result set from the original result set. After removing all data that does not meet the conditions, the result set after relationship reference filtering is obtained. Step 5: Verify the result set after relational reference filtering. Use the heterogeneous filtering temporarily stored in Step 1 to filter the result set after relational reference filtering. The data obtained is the final result.

2. The method as described in claim 1, characterized in that, The original result set obtained in step 2 contains 0 or more data entries.

3. The method as described in claim 1, characterized in that, In step 1, decomposing the conditions of the screening method into several sub-screens specifically includes: decomposing the conditions of the screening method into a tree of depth 2, with the root node of the tree being an OR operation, and each child node being a Boolean expression without logical operation or a logical expression with continuous AND operation; using the child nodes as OR factors, the several sub-screens are sub-operations containing OR factors.

4. The method as described in claim 3, characterized in that, In step 1, the condition that differs from the query source is removed. Specifically, if there is an OR factor that cannot be queried simultaneously with the query source in a single query request due to storage differences, it is identified as a heterogeneous OR factor and removed. Those that are not removed are identified as isomorphic OR factors, and the isomorphic OR factors constitute the screening conditions in step 2.

5. The method as described in claim 4, characterized in that, The process of obtaining the heterogeneous result set in step 3 includes: creating a view and defining a set of non-intuitive attributes and a set of intuitive attributes for the view; wherein, the calculation method of the non-intuitive attributes is bound to the conditions in the heterogeneous sub-filters that were removed in step 1; the intuitive attributes use the filter key attributes of the query source in step 1 as the source.

6. The method as described in claim 5, characterized in that, Non-intuitive properties return a boolean value.

7. The method as described in claim 5, characterized in that, Step 3, the process of obtaining the heterogeneous result set, also includes: creating a projection operation with the view as the target, and obtaining a filtering operation with the filtering attributes of the query source type; concatenating the two into a validation query, executing the validation query, and obtaining a set of result instances of the view as the heterogeneous result set.

8. The method as described in claim 1, characterized in that, In step 2, after obtaining the original result set, the identifiers and results in the original result set are also used to construct a result dictionary.

9. The method according to any one of claims 6 to 8, characterized in that, The filtering and verification process in steps 4 and 5 specifically includes: for each element in the original result set, finding the corresponding instance from the result instance set of the view based on the identifier; performing continuous AND operations on all non-intuitive attributes of the view whose results are Boolean values; if the result is false, removing the element from the original result set; adding elements whose verification results are all true to the result dictionary; and finally returning the value of the result dictionary after the execution as the final result.

10. The method as described in claim 1, characterized in that, The prerequisite for applying this method is that the filtering method does not involve logical operations or only involves logical AND operations.

Images