Routing method and device for multiple data sources

Abstract

The invention discloses a routing method and device for multiple data sources. According to the method, when data access is carried out on multiple data sources, the data sources pointed by the to-be-accessed data are firstly identified, and then corresponding file system initialization work is carried out according to the determined types of the data sources, so that file systems of different storage types can be correctly configured and accessed; the accuracy and the high efficiency of data access are ensured; furthermore, according to the method, the data path information of the to-be-accessed data is dynamically adjusted to be the path conforming to the data source storage format, it can be ensured that each data processing logic method can accurately position the data, the flexibility of data processing is improved, and the compatibility and maintainability of the system are enhanced; and according to the modified adaptive data path, reading the data stored in the specified path from the initialized file system, thereby achieving the purpose of data routing.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data management, and more particularly to a method and apparatus for routing multiple data sources. Background Art

[0002] With the development of big data technology, data types and storage formats are increasing. A single data source often cannot meet all business needs. Therefore, multiple different databases or storage systems are needed to meet the storage and processing requirements of different data types. When dealing with multiple data sources, especially in complex data processing, analysis, and storage environments, which may involve multiple different types of databases or data storage systems (such as relational databases, NoSQL databases, Hadoop, Kafka, etc.), how to select, route, and manage the flow of data to different data sources requires management through routing mechanisms.

[0003] Currently, common multi-data source routing solutions include static configuration routing, load balancing-based routing, and dynamic routing protocols based on network topology changes. However, with the complexity of business logic, the diversification of data types, and the increase in data volume, current multi-data source routing solutions have poor flexibility and are unable to dynamically adjust routing strategies based on real-time conditions. On the other hand, they cannot meet the system's rapid response requirements under multiple data types, resulting in poor system performance. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a routing method and apparatus for multiple data sources, which ensures the accuracy and efficiency of data access and improves the flexibility of data processing.

[0005] In a first aspect, an embodiment of the present application provides a routing method for multiple data sources, including:

[0006] Identify the data source of the data to be accessed;

[0007] Execute corresponding file system initialization according to the type of the data source;

[0008] Modifying the access path of the data to be accessed into a storage format corresponding to the data source to obtain an adapted data path;

[0009] According to the adapted data path, corresponding data is read from the file system.

[0010] In one embodiment, identifying the data source of the data to be accessed includes:

[0011] Determine the access path to the data to be accessed;

[0012] A data source is identified based on the access path.

[0013] In one embodiment, identifying the data source according to the access path includes:

[0014] Reading a data cache; wherein the data cache stores path information of data in a data source;

[0015] Performing matching verification on the access path according to the path information in the data cache;

[0016] The data source is determined based on the result of the matching verification.

[0017] In one embodiment, the data cache includes a multi-level cache; performing matching verification on the access path according to the path information in the data cache includes:

[0018] Verify each level of cache in turn according to the level of data cache;

[0019] If the verification is successful, the corresponding verification result is returned;

[0020] If the verification fails, continue to verify the lower-level cache.

[0021] In one embodiment, the data source includes a conventional data source and a core data source, and identifying the data source according to the access path includes:

[0022] Initiating a core path check on the access path;

[0023] If the core path verification succeeds, it is determined to be a core data source; otherwise, it is determined to be a regular data source.

[0024] In one embodiment, initiating a core path check on the access path includes:

[0025] Determine whether a corresponding data path exists in the first-level cache; wherein the first-level cache stores a table path of a non-partitioned table or a specified transmission protocol path;

[0026] If it exists in the first-level cache, it is determined to be a core path;

[0027] If the data path does not exist in the first-level cache, determining whether the corresponding data path exists in the second-level cache; wherein the second-level cache stores the partition path of the partition table;

[0028] If it exists in the secondary cache, it is determined to be a core path;

[0029] If the data path does not exist in the second-level cache, determine whether the corresponding data path exists in the third-level cache; wherein the third-level cache stores the table path of the partition table;

[0030] If the path does not exist in the third-level cache, determine the non-core path;

[0031] If the data path exists in the third-level cache, the intermediate key and the partition key are used to perform regular matching on the data path;

[0032] If the regular expression matches, it is determined to be a core path; if the regular expression matches, it is determined to be a non-core path.

[0033] In one embodiment, between if the path exists in the secondary cache and determining that the path is a core path, the method further includes:

[0034] Determine whether the corresponding access path exists in the partition blacklist;

[0035] If the path is in the partition blacklist, determine it as a non-core path;

[0036] If the path is not in the partition blacklist, it is determined to be a core path.

[0037] In one embodiment, before performing regular matching on the data path using the intermediate key and the partition key, the method further includes:

[0038] Determine whether a corresponding data path exists in the fourth-level cache; wherein the fourth-level cache stores the table path of the partition table and the completion status of disaster recovery;

[0039] If the path exists in the four-level cache, the use of the intermediate key and the partition key to perform regular matching on the data path is specifically as follows: using the intermediate key and the partition key to perform level-by-level regular matching on the data path; if the level-by-level regular matching passes, it is determined to be a core path; if the level-by-level regular matching fails, it is determined to be a non-core path;

[0040] If it does not exist in the fourth-level cache, the use of the intermediate key and partition key to perform regular matching on the data path is specifically: using the intermediate key and partition key to perform full regular matching on the data path; if the full regular matching passes, it is determined to be a core path; if the full regular matching fails, it is determined to be a non-core path.

[0041] In one embodiment, initiating a core path check on the access path includes:

[0042] The core path is used as the key and the description information corresponding to the core path is used as the value to form a core path key-value pair;

[0043] Performing hierarchical parsing on the access path to generate a list containing all hierarchical paths;

[0044] Using each path in the list as a key, traverse the core path key-value pairs to determine whether a corresponding value exists;

[0045] If it exists, the core path verification is determined to be successful; if it does not exist, the core path verification is determined to be failed.

[0046] In a second aspect, an embodiment of the present application provides a routing device for multiple data sources, including:

[0047] A data source identification unit, used to identify the data source of the data to be accessed;

[0048] A system initialization unit, configured to perform corresponding file system initialization according to the type of the data source;

[0049] A path adaptation unit, configured to modify the access path of the data to be accessed into a storage format corresponding to the data source, thereby obtaining an adapted data path;

[0050] A data reading unit is used to read corresponding data from the file system according to the adapted data path.

[0051] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0053] Figure 1 A schematic diagram of a process flow of a routing method for multiple data sources provided in an embodiment of the present application is shown;

[0054] Figure 2 A schematic diagram of a core verification process provided by an embodiment of the present application is shown;

[0055] Figure 3 An exemplary structural block diagram of a routing device for multiple data sources provided in an embodiment of the present application is shown;

[0056] Figure 4 A schematic diagram of the system workflow of a routing device for multiple data sources provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only the portions relevant to the invention are shown in the accompanying drawings.

[0058] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Although the embodiments of the present application provide the method operation instruction steps shown in the following embodiments or drawings, more or fewer operation instruction steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present application. In the actual processing process or when the device is executed, the method can be executed in the order of the methods shown in the embodiments or drawings or in parallel.

[0059] This application proposes a routing method for multiple data sources, please refer to Figure 1 , Figure 1 FIG. 1 shows a flow chart of a routing method for multiple data sources provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0060] S101, identifying the data source of the data to be accessed;

[0061] Before initializing the file system, the data source is first identified, and information such as the data to be accessed or the data path is analyzed and compared. According to preset rules or logic, the data source to which the data to be accessed points is determined.

[0062] S102: Execute corresponding file system initialization according to the type of data source;

[0063] Different data sources (such as local file systems, HDFS, S3, Ceph, and databases) have different storage methods, access protocols, and operation modes. This method configures the corresponding access methods, protocols, and permissions based on the determined data source type (for example, object storage or distributed file systems), and performs the corresponding file system initialization. For example, initializing a local file system may require setting the file path, while initializing a distributed file system (such as HDFS) may require configuring the NameNode address, port, and connection pool. This ensures that file systems of different storage types can be correctly configured and accessed, thereby ensuring accurate and efficient data access.

[0064] The file system refers to the data storage system of the data source, that is, the underlying storage system used to store the data to be accessed. If the data is stored in a BOS data source, the corresponding file system of the BOS is initialized; if the data is stored in an HDSF data source, the corresponding file system of the HDSF is initialized. It should be noted that the file system initialization method corresponding to different data source types can refer to the implementation of related technologies and is not limited here.

[0065] S103, modifying the access path of the data to be accessed to the storage format corresponding to the data source to obtain an adapted data path;

[0066] Different storage systems (such as object storage, distributed file systems, etc.) usually have their own file path formats and access methods, requiring the path to conform to specific rules or protocols (for example, prefix, protocol type, etc.). For example, the path in HDFS may have a different format than the path in S3, and different data sources may support different data storage formats (such as CSV, JSON, Parquet, ORC, etc.). After determining the storage source of the data, this method further dynamically adjusts the data path based on this information. Specifically, if the data is stored in BOS, the data path is modified to a path that conforms to the BOS storage format; accordingly, if the data is stored in HDFS, the data path is modified to a storage format unique to HDFS.

[0067] This dynamic adjustment process helps the system map access requests to formats supported by the storage system, ensuring that each data processing logic method can accurately locate the data and perform subsequent processing operations in the correct format. This not only improves data processing flexibility but also enhances system compatibility and maintainability.

[0068] S104: Read corresponding data from the file system according to the adapted data path.

[0069] According to the modified adapted data path, the data stored in the specified path is read from the initialized file system, thereby completing the data access request.

[0070] It should be noted that although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order. For example, S102 and S103 can be performed simultaneously, or all the shown operations must be performed to achieve the desired results.

[0071] Based on the above introduction, the method provided in this embodiment, when accessing data from multiple data sources, first identifies the data source pointed to by the data to be accessed, and then performs corresponding file system initialization work according to the type of the determined data source. This ensures that file systems for different storage types can be correctly configured and accessed, thereby ensuring the accuracy and efficiency of data access. Furthermore, the method dynamically adjusts the data path information of the data to be accessed to a path that conforms to the storage format of the data source, thereby ensuring that each data processing logic method can accurately locate the data, which not only improves the flexibility of data processing, but also enhances the compatibility and maintainability of the system. Then, according to the modified adaptive data path, the data stored in the specified path is read from the initialized file system, thereby achieving the purpose of data routing.

[0072] The above embodiments do not limit the method for identifying the data source of the data to be accessed. In actual applications, the data source can be identified based on data tags, metadata, data content, etc. For example, a specific JSON structure may indicate that the data comes from a certain API or database table. This embodiment proposes that the data source can be identified by determining the access path of the data to be accessed.

[0073] The access path contains information such as the path prefix or structure that can be used to infer the type of data source and its protocol. It can clearly identify the type of data source without relying on external configuration or complex management logic, which can reduce the complexity of external configuration. It is applicable to a large number of data source types and can achieve fast and accurate identification.

[0074] In the process of identifying data sources based on access paths, it is necessary to store and manage the paths and related information of each data source. In the case of a large number of data sources or a large amount of data path information under a data source, in order to reduce the number of database accesses and improve data retrieval speed, in one embodiment, a cache can be used to achieve the above goals, such as using the open source local cache library Caffeine (an open source, high-performance Java cache library) to store and manage the paths and related information. Specifically, the path information of the data in each data source is cached. When identifying the data source, the data cache is read and the access path is matched and verified based on the path information in the data cache. The data source is then determined based on the matching and verification results.

[0075] By caching table paths and creation time information, repeated database queries are avoided, reducing the number of database accesses. When accessing table paths or creation time information is needed, the system can read directly from the cache instead of retrieving it from the database, thereby improving data retrieval speed. Furthermore, by reducing database access and increasing data retrieval speed, the first-level cache can significantly improve overall system performance.

[0076] You can add path information from all data sources to the cache, setting up a first-level cache. However, to further improve performance and response speed while reducing the load on the underlying data sources, you can set up a multi-level cache for level-by-level verification. For example, when accessing certain data, the first level cache is checked. If a hit is found, the corresponding verification result is returned. If a miss is found, the second level cache is checked, and so on. This avoids performing large-scale cache queries for large amounts of data each time, reducing latency.

[0077] Multi-level cache combines the characteristics of different levels of cache to optimize cache hit rate. Typically, high-level cache can store recently used or most frequently accessed data, while low-level cache can store less frequently accessed but important data. For example, in a multi-level cache system, if a data item is frequently accessed, it can be stored in the first-level cache. If there is no hit in the first-level cache, the second-level cache is then checked. This level-by-level search can improve cache utilization efficiency and reduce unnecessary low-level storage access. Through level-by-level verification, the system can improve the overall hit rate by prioritizing access to fast, small-capacity caches.

[0078] In a major business scenario, there are two data sources: core data sources and regular data sources. The core data source is stored in BOS (Baidu Object Storage), and the core data is stored in the HDFS data source. Of course, other storage systems can also be configured for the two data sources, which is not limited here.

[0079] Core data sources are typically the most critical and important data sources in a system. These data sources contain core information for the system or business and have extremely high data consistency and reliability requirements. Conventional data sources refer to common data sources that are non-critical to the business or system. These data typically support auxiliary business operations but do not contain core, sensitive business information. Generally speaking, the amount of data in core data sources is far less than that in conventional data sources. To speed up the access path-based data source identification process, it is possible to set identification rules only for core paths, without setting path identification rules for conventional data sources. Only core path validation is performed on the access path. If the core path validation succeeds, it is identified as a core data source; otherwise, it is identified as a conventional data source. This reduces the difficulty and complexity of setting validation rules while also improving identification speed. A core path represents a data path that requires special processing. If a path is a core path, then the path itself and its subpaths are also considered core paths; its parent path and its sibling paths are not core paths.

[0080] To speed up the access path-based data source identification process while further reducing database access, a multi-level core path verification mechanism can be configured. In one embodiment, a first-level cache for storing table paths for non-partitioned tables or specified transport protocol paths, a second-level cache for storing partition paths for partitioned tables, and a third-level cache for storing table paths for partitioned tables can be pre-configured.

[0081] Among them, the first-level cache saves the table path of the non-partitioned table or the specified transmission protocol path, as well as path-related information, including creation time, etc. The second-level cache saves the partition path of the partitioned table, as well as partition-related information, including creation time, etc. The partition path of the partitioned table refers to the physical storage location of each partition data in the table when using the partitioning strategy to store data. Each partition corresponds to an independent storage directory or file path. The third-level cache saves the table path of the partitioned table, as well as partition table-related information, including creation time, etc. The table path of the partitioned table usually refers to the storage path of the partitioned table as a whole, that is, the overall location of the table data, not just the path of a single partition.

[0082] The core path verification process for the above cache configuration specifically includes the following four steps:

[0083] 1. Determine whether the corresponding data path exists in the first-level cache. The first-level cache stores table paths for non-partitioned tables or specified transmission protocol paths (referring to the specified transmission protocol used in the business scenario, which is not limited here). If the path exists in the first-level cache, it is determined to be the core path. If not, proceed to step 2.

[0084] 2. Determine whether the corresponding data path exists in the L2 cache; wherein the L2 cache stores the partition path of the partition table; if it exists in the L2 cache, it is determined to be the core path; if it does not exist in the L2 cache, execute step 3;

[0085] 3. Determine whether the corresponding data path exists in the third-level cache; wherein the third-level cache stores the table path of the partition table; if it does not exist in the third-level cache, determine the non-core path; if it exists in the third-level cache, execute step 4;

[0086] 4. Use the intermediate key and partition key to perform regular expression matching on the data path. If the regular expression matches, it is determined to be a core path. If the regular expression fails, it is determined to be a non-core path.

[0087] The above verification process settings configure the non-partitioned table path, partitioned table path, and table path of the partitioned table from low to high levels. The complexity of the path verification increases step by step, and in the current application scenario, this setting is consistent with the frequency of data access. In actual applications, a large number of effective core path verifications can be achieved through low-level cache matching and comparison, which can improve the cache hit rate. Finally, the intermediate key (midKey, the path between the partition table and the partition, a custom term) and the partition key are used to perform regular matching on the path, which can more accurately determine whether the path belongs to the core path. Different levels of cache store different levels of path information, so that more accurate regular matching can be performed for specific path types (such as partitioned paths and non-partitioned paths), reducing unnecessary matching operations.

[0088] In step 2 above, if the path is determined to exist in the L2 cache, a partition blacklist check can be performed before determining it as a core path. Specifically, if the path exists in the L2 cache, a check is performed to see if the corresponding access path exists in the partition blacklist. If the path exists in the partition blacklist, the path is determined to be a non-core path. If the path does not exist in the partition blacklist, the path is determined to be a core path.

[0089] In a partition table, some partitions may become unavailable due to business adjustments, data errors, temporary maintenance, etc. By using a partition blacklist, you can ensure that these illegal or unavailable partition paths are promptly excluded to prevent them from affecting the normal operation of the system.

[0090] In one embodiment, based on the above cache configuration, in order to further improve performance and reduce unnecessary computation, the following step 5 may be further performed between steps 3 and 4:

[0091] 5. Determine whether the corresponding data path exists in the fourth-level cache; the fourth-level cache stores the table path of the partition table and the disaster recovery completion status;

[0092] If the path exists in the L4 cache, the intermediate key and partition key are used to perform regular expression matching on the data path. Specifically, the intermediate key and partition key are used to perform regular expression matching on the data path level by level. If the regular expression matching succeeds, the path is determined to be a core path. If the regular expression matching fails, the path is determined to be a non-core path.

[0093] If the key does not exist in the fourth-level cache, the data path is regularly matched using the intermediate key and partition key. Specifically, the data path is regularly matched using the intermediate key and partition key. If the regular match succeeds, the path is determined to be a core path. If the regular match fails, the path is determined to be a non-core path.

[0094] With the support of the above steps, full regular expression matching is avoided every time, especially in the case of high load or high-frequency requests. By first checking the fourth-level cache, if the cache already has relevant information about the data path and disaster recovery status, that is, if the fourth-level cache match is successful, it means that the data of certain tables has been disaster-recovered. The path can be quickly confirmed through a lighter-weight step-by-step regular expression matching. This is more efficient than full matching and avoids the overhead of regular expression matching on the entire path every time. The efficiency improvement will be more significant, especially when the path is long and the structure is complex.

[0095] To deepen understanding, based on the above embodiment, this embodiment proposes a verification step in a specific application scenario. Take the core verification process under the following cache configuration as an example. Figure 2 shown.

[0096] The first-level cache stores the table path or specified transmission protocol path of non-partitioned tables; the second-level cache stores the partition path of partitioned tables; the third-level cache stores the table path of partitioned tables; the fourth-level cache stores the table path of partitioned tables, as well as the disaster recovery completion status.

[0097] Level 1 cache check:

[0098] When a data path is received, it is first looked up in the L1 cache.

[0099] If the first-level cache hits (ie, the corresponding data path is found), the path is directly determined to be the core path.

[0100] If the first-level cache misses, the verification process continues to the second-level cache.

[0101] Second level cache check:

[0102] Lookup is performed again in the second-level cache.

[0103] If the L2 cache hits, it further checks whether the path is in the partition blacklist.

[0104] If the path is in the partition blacklist, the path is determined to be a non-core path.

[0105] If the path is not in the partition blacklist, the path is determined to be a core path.

[0106] If the L2 cache misses, the check process continues with the L3 cache.

[0107] Level 3 cache check:

[0108] Lookup is performed in the L3 cache.

[0109] If the L3 cache misses (ie, the corresponding data path is not found in the L3 cache), the path is directly determined to be a non-core path.

[0110] If the third-level cache hits, the fourth-level cache verification process continues.

[0111] Level 4 cache check:

[0112] Find the corresponding data path in the fourth-level cache.

[0113] If the fourth-level cache hits, the intermediate key and partition key are used for level-by-level regular matching.

[0114] If the regular expression matching is passed at each level, the path is determined to be a core path.

[0115] If the regular expression matching fails at each level, the path is determined to be a non-core path.

[0116] If the L4 cache misses, a full regular expression match of the intermediate key and partition key is performed directly.

[0117] In the full regular expression matching, if the match succeeds, the path is determined to be a core path; if the match fails, the path is determined to be a non-core path.

[0118] Through the above process, it is possible to accurately and efficiently determine whether a data path is a core path, and make corresponding processing based on different cache hit situations and blacklist rules.

[0119] In the path verification method provided in the above embodiment, there is no limitation on the judgment logic for whether the access path hits the cache. This embodiment proposes a specific implementation of the judgment logic. The implementation process under other logics can refer to the introduction of this embodiment and the above embodiments, and will not be repeated here.

[0120] The specific judgment logic for initiating core path verification on the access path is as follows: the core path is used as the key and the description information corresponding to the core path is used as the value to form a core path key-value pair;

[0121] When the system needs to determine whether a given path is a core path, it follows these steps:

[0122] Path Hierarchy Parsing: Performs hierarchical parsing on the access path and generates a list of all hierarchical paths. Performs hierarchical parsing on the given path, from the first level path to the last level path, and generates a list of all hierarchical paths.

[0123] Cache query: Use each path in the list as a key, traverse the core path key-value pairs, and determine whether there is a corresponding value; if so, the core path verification is considered successful; if not, the core path verification is considered failed.

[0124] Traverse this hierarchical path list. For each path in the list, use it as a key. If a path key is found in the cache during the traversal process, then this path is considered a core path. At this point, the remaining paths are no longer traversed, and the given path is directly determined to be a core path. If no corresponding value is found in the cache after traversing the entire hierarchical path list, then the given path is determined to be a non-core path. In this way, the system can efficiently determine whether a given path is a core path.

[0125] Further references Figure 3 , which shows an exemplary structural block diagram of a routing device for multiple data sources according to an embodiment of the present application, which mainly includes:

[0126] A data source identification unit 101 is used to identify the data source of the data to be accessed;

[0127] The system initialization unit 102 is used to perform corresponding file system initialization according to the type of data source;

[0128] A path adaptation unit 103 is configured to modify the access path of the data to be accessed into a storage format corresponding to the data source, thereby obtaining an adapted data path;

[0129] The data reading unit 104 is configured to read corresponding data from the file system according to the adapted data path.

[0130] It should be understood that the units described in the above devices are the same as those in the reference Figure 1 The steps in the method described above correspond to each other. Therefore, the operations and features described above for the method are also applicable to the device and the units included therein, and will not be repeated here. The device can be pre-implemented in the server's browser or other security application, or loaded into the server's browser or its security application by downloading or other means.

[0131] Figure 4 The figure shows a system workflow diagram for a routing device with multiple data sources. In the figure, the data source identification unit identifies the data source through a four-level cache. The system initialization unit and path adaptation unit are integrated into the big data engine or client to read and write corresponding data from the big data storage. The system workflow for other unit configurations can be referenced in the description of this embodiment and will not be repeated here.

[0132] The division of the several units mentioned in the detailed description above is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0133] It should be noted that for details not disclosed in the routing device for multiple data sources in the embodiments of the present application, please refer to the details disclosed in the above embodiments of the present application, and no further details will be given here.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operating instructions of the methods and devices according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operating instruction, or can be implemented by a combination of dedicated hardware and computer instructions.

[0135] The units or modules involved in the embodiments described in this application may be implemented in software or hardware. The units or modules described may also be provided in a processor. The names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves.

[0136] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the aforementioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A routing method for multiple data sources, characterized in that: include: Identify the data source of the data to be accessed; Execute corresponding file system initialization according to the type of the data source; Modifying the access path of the data to be accessed into a storage format corresponding to the data source to obtain an adapted data path; According to the adapted data path, corresponding data is read from the file system.

2. The method according to claim 1, wherein The step of identifying a data source of the data to be accessed includes: Determine the access path to the data to be accessed; A data source is identified based on the access path.

3. The method according to claim 2, wherein The identifying the data source according to the access path includes: Reading a data cache; wherein the data cache stores path information of data in a data source; Performing matching verification on the access path according to the path information in the data cache; The data source is determined based on the result of the matching verification.

4. The method according to claim 3, wherein The data cache includes a multi-level cache; performing matching verification on the access path according to the path information in the data cache includes: Verify each level of cache in turn according to the level of data cache; If the verification is successful, the corresponding verification result is returned; If the verification fails, continue to verify the lower-level cache.

5. The method according to claim 2, wherein The data source includes a conventional data source and a core data source, and identifying the data source according to the access path includes: Initiating a core path check on the access path; If the core path verification succeeds, it is determined to be a core data source; otherwise, it is determined to be a regular data source.

6. The method according to claim 5, wherein Initiate a core path check on the access path, including: Determine whether a corresponding data path exists in the first-level cache; wherein the first-level cache stores a table path of a non-partitioned table or a specified transmission protocol path; If it exists in the first-level cache, it is determined to be a core path; If the data path does not exist in the first-level cache, determining whether the corresponding data path exists in the second-level cache; wherein the second-level cache stores the partition path of the partition table; If it exists in the secondary cache, it is determined to be a core path; If the data path does not exist in the second-level cache, determine whether the corresponding data path exists in the third-level cache; wherein the third-level cache stores the table path of the partition table; If the path does not exist in the third-level cache, determine the non-core path; If the data path exists in the third-level cache, the intermediate key and the partition key are used to perform regular matching on the data path; If the regular expression matches, it is determined to be a core path; if the regular expression matches, it is determined to be a non-core path.

7. The method according to claim 6, wherein If the path exists in the secondary cache, and the path is determined to be a core path, the method further includes: Determine whether the corresponding access path exists in the partition blacklist; If the path is in the partition blacklist, determine it as a non-core path; If the path is not in the partition blacklist, it is determined to be a core path.

8. The method according to claim 6, wherein Before performing regular matching on the data path using the intermediate key and the partition key, the method further includes: Determine whether a corresponding data path exists in the fourth-level cache; wherein the fourth-level cache stores the table path of the partition table and the completion status of disaster recovery; If the path exists in the four-level cache, the use of the intermediate key and the partition key to perform regular matching on the data path is specifically as follows: using the intermediate key and the partition key to perform level-by-level regular matching on the data path; if the level-by-level regular matching passes, it is determined to be a core path; if the level-by-level regular matching fails, it is determined to be a non-core path; If it does not exist in the fourth-level cache, the use of the intermediate key and partition key to perform regular matching on the data path is specifically: using the intermediate key and partition key to perform full regular matching on the data path; if the full regular matching passes, it is determined to be a core path; if the full regular matching fails, it is determined to be a non-core path.

9. The method according to claim 5, wherein Initiate a core path check on the access path, including: The core path is used as the key and the description information corresponding to the core path is used as the value to form a core path key-value pair; Performing hierarchical parsing on the access path to generate a list containing all hierarchical paths; Using each path in the list as a key, traverse the core path key-value pairs to determine whether a corresponding value exists; If it exists, the core path verification is determined to be successful; if it does not exist, the core path verification is determined to be failed.

10. A routing device for multiple data sources, characterized in that: include: A data source identification unit, used to identify the data source of the data to be accessed; A system initialization unit, configured to perform corresponding file system initialization according to the type of the data source; A path adaptation unit, configured to modify the access path of the data to be accessed into a storage format corresponding to the data source, thereby obtaining an adapted data path; A data reading unit is used to read corresponding data from the file system according to the adapted data path.

Images