A Real-Time Tiered Storage Method for Multi-Source Heterogeneous Data in IDC

Abstract

This invention discloses a real-time hierarchical storage method for multi-source heterogeneous data in data centers (IDCs), relating to the field of data storage technology. The method involves constructing an IDC platform and establishing resource pools within it. These resource pools include high-performance and low-performance resource pools, with allocation conditions set for each. A resource configuration assistant is configured within the IDC platform corresponding to each resource pool. A connection is established between the IDC platform and multi-source data endpoints, allowing the acquisition and storage of corresponding data in the main resource module and resource pools. This invention enables hierarchical storage of data based on access frequency. This hierarchical storage includes storage in the high-performance resource pool and storage in the low-performance resource pool, allowing for the allocation of corresponding storage devices based on data access frequency, thus conserving the use of high-performance storage.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and more specifically to a real-time hierarchical storage method for multi-source heterogeneous data in data centers (IDC). Background Technology

[0002] With societal development and the widespread use of computers, orders for computer hardware are increasing. These hardware can be purchased through e-commerce, offline channels, and other means. However, e-commerce and finance sectors require real-time processing of order, risk control, and supply chain data. Much data needs to be processed in real time, such as order-related data, which may require multiple accesses from both the client and merchant sides. This necessitates low-latency data storage and processing methods to support dynamic adjustments to business strategies. However, large amounts of historical order data do not require low-latency data storage and processing methods. Traditional storage methods struggle to allocate resources based on real-time data information and cannot meet the requirements for real-time data processing. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time hierarchical storage method for multi-source heterogeneous data in data centers (IDCs) to address the shortcomings in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a real-time hierarchical storage method for multi-source heterogeneous data in IDC, comprising the following steps:

[0005] An IDC platform is constructed, and a resource pool is established in the IDC platform. The resource pool includes a high-performance resource pool and a low-performance resource pool. Allocation conditions are set for the high-performance resource pool and the low-performance resource pool respectively.

[0006] In the IDC platform, a resource configuration assistant is set up for the resource pool, wherein the resource configuration assistant includes a main resource module, a central point, and multiple traction points;

[0007] Establish a connection between the IDC platform and the multi-source data terminal, collect corresponding data through the multi-source data terminal and store it in the main resource module and the resource pool;

[0008] The authorized port accesses the data in the resource pool through the central point, and dynamically allocates the data collected by the multi-source data terminal in the resource pool according to the real-time access frequency based on the allocation conditions.

[0009] In a preferred embodiment, the step of constructing an IDC platform and establishing a resource pool within the IDC platform includes:

[0010] In the IDC platform, high-performance resource pools and low-performance resource pools are built. Each of the high-performance resource pools and low-performance resource pools contains multiple unit information points.

[0011] Access ports are set for both the high-performance resource pool and the low-performance resource pool.

[0012] In a preferred embodiment, the step of constructing a high-performance resource pool and a low-performance resource pool in the IDC platform includes:

[0013] Multiple storage nodes are set up in both the high-performance resource pool and the low-performance resource pool;

[0014] Each of the multiple unit information points within the high-performance resource pool and the low-performance resource pool is connected one-to-one with the storage endpoint.

[0015] In a preferred embodiment, the step of setting up a resource configuration assistant for the resource pool in the IDC platform includes:

[0016] Set up a main resource module in the IDC platform, and set up multiple traction points in the main resource module. The number of traction points is the same as the number of unit information points in the resource pool.

[0017] A central point is set between the main resource section and the resource pool. The central point includes an access area and multiple guide points in the access area. Multiple guide points are combined according to a preset number to obtain a guide combination.

[0018] The unit information points inside the high-performance resource pool and the low-performance resource pool are combined according to a preset number to obtain multiple unit information combinations.

[0019] Multiple traction points in the main resource sector are combined according to a preset number to obtain multiple traction combinations;

[0020] The unit information combination, traction combination, and guide combination are matched one-to-one. The unit information points in the corresponding unit information combination are connected one-to-one with the guide points in the corresponding guide combination through the first channel. The traction points in the traction combination are connected one-to-one with the guide points in the corresponding guide combination through the second channel. The first channels between the corresponding unit information combination and the guide combination are interconnected. The second channels between the corresponding traction combination and the guide combination are interconnected. The first channels between the corresponding unit information combination and the guide combination and the second channels between the traction combination and the guide combination are set with the same connection rules. The connection rules are recorded in the corresponding guide combination. The connection rules include the connection relationships between multiple different guide combinations and the traction combination and the unit information combination, respectively.

[0021] The main resource sector, central point, and multiple traction points serve as resource allocation assistants for the resource pool.

[0022] In a preferred embodiment, the step of establishing the connection between the IDC platform and the multi-source data terminal, and collecting corresponding data through the multi-source data terminal and storing it in the main resource module and the resource pool, includes:

[0023] Multiple data endpoints are identified as multi-source data endpoints, and the IDC platform is connected to these multi-source data endpoints.

[0024] Data collected from multiple data sources will be used as raw data and stored in the main resource module and the resource pool respectively.

[0025] In a preferred embodiment, the step of using data collected from multiple data sources as raw data and storing it in the main resource module and the resource pool respectively includes:

[0026] The raw data collected from multiple data sources is classified into different types to obtain multiple data types.

[0027] Multiple data types are divided into multiple fragments according to a preset number;

[0028] Multiple fragments of data in a single data type are stored one-to-one in multiple traction points of a single traction combination in the main resource module, and the corresponding traction points are sorted according to the division order of multiple fragments of data in a single data type.

[0029] Multiple fragments of data in the type data are stored in the unit information points of the corresponding traction points. Based on the generation time of the type data, the unit information points corresponding to the type data in the preset recent time period are distributed in the high-performance resource pool, and the unit information points corresponding to the remaining type data are distributed in the low-performance resource pool.

[0030] The central point guides the combination of data by recording the order of the corresponding traction points, and the corresponding guide combination marks the corresponding type data keywords.

[0031] In a preferred embodiment, the authorized port accesses data in the resource pool through the central point, and dynamically allocates the data collected by the multi-source data terminal in the resource pool according to the real-time access frequency based on allocation conditions, including:

[0032] Mark the authorized port, set the connection rules for the corresponding authorized port, and obtain the corresponding boot combination in the access area of ​​the central point through the type data keyword of the authorized port. When the connection rule matches the connection rule corresponding to the boot combination, the sorting of the boot points in the boot combination is obtained.

[0033] The authorized port obtains the sorting of the corresponding unit information points according to the sorting of the guide points in the guide combination, and then obtains the corresponding fragment data in the unit information points in the resource pool, and combines the fragment data to obtain the original data;

[0034] Unauthorized ports are accessing the resource pool via its access ports;

[0035] The frequency of access to the statistical guidance combination is used to migrate the unit information points corresponding to frequencies within the high frequency range to the high-performance resource pool, and the unit information points corresponding to frequencies within the low frequency range to the low-performance resource pool.

[0036] In a preferred embodiment, the step of migrating cell information points corresponding to frequencies within a high frequency range to a high-performance resource pool and cell information points corresponding to frequencies within a low frequency range to a low-performance resource pool, based on the frequency of access to the statistically guided combination, includes:

[0037] Based on the frequency of access to the guiding combination, the unit information points corresponding to the frequencies that meet the high frequency range are disconnected from their storage locations, and the unit information points are transmitted to the high-performance resource pool. A storage location in the high-performance resource pool is randomly selected for connection, and the unit information points in the same unit information combination are distributed and stored in the high-performance resource pool.

[0038] The unit information points corresponding to frequencies within the low frequency range are disconnected from their respective storage locations. The unit information points are then transferred to a low-performance resource pool, and a storage location within the low-performance resource pool is randomly selected for connection. Unit information points within the same unit information combination are then distributed and stored in the low-performance resource pool.

[0039] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0040] 1. This invention can classify and store data according to the frequency of data access. The classification includes storage in a high-performance resource pool and storage in a low-performance resource pool. It can allocate corresponding storage according to the frequency of data access, which can save the use of high-performance storage and realize real-time perception, intelligent decision-making and dynamic adaptation.

[0041] 2. The central point of this invention includes an access area and multiple guide points within the access area, which can connect the traction point and the unit information points in the resource pool. Authorized ports access data in the resource pool through the central point. The central point stores information from the unit information points in the resource pool corresponding to the traction point. The unit information points in the resource pool are disordered and scattered, and cannot be accessed effectively by unauthorized ports, thus playing a role in data protection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0043] Figure 1 This is a flowchart of the method of the present invention.

[0044] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1, please refer to Figure 1-2 As shown in this embodiment, a real-time hierarchical storage method for multi-source heterogeneous data in IDC includes the following steps:

[0047] S1. Construct an IDC platform and establish a resource pool within the IDC platform. The resource pool includes a high-performance resource pool and a low-performance resource pool. Allocation conditions are set for the high-performance resource pool and the low-performance resource pool respectively. (There are multiple unit information points inside. Access ports are set for both the high-performance resource pool and the low-performance resource pool. These access ports are fake access ports. It is difficult to access the real data through these access ports. The real data can only be accessed through the central point.)

[0048] S2. In the IDC platform, a resource configuration assistant is set up for the resource pool. The resource configuration assistant includes a main resource module, a central point, and multiple traction points (the central point includes an access area and multiple guide points in the access area. The guide point is a virtual machine that can connect to the traction point and the unit information point in the resource pool. The authorized port accesses the data in the resource pool through the central point. The central point stores the information in the unit information point in the resource pool corresponding to the traction point. The unit information points in the resource pool are disordered and scattered, which can play a role in data protection).

[0049] S3. Establish the connection between the IDC platform and the multi-source data terminal, collect the corresponding data through the multi-source data terminal and store it in the main resource module and the resource pool;

[0050] S4. The authorized port accesses the data in the resource pool through the central point, and dynamically allocates the data collected by the multi-source data terminal in the resource pool according to the real-time access frequency based on the allocation conditions.

[0051] As described in steps S1-S4 above, data can be stored in a hierarchical manner based on the frequency of data access. This hierarchical storage includes storage in a high-performance resource pool and storage in a low-performance resource pool. The corresponding storage can be allocated according to the frequency of data access, which can save the use of high-performance storage and realize real-time perception, intelligent decision-making and dynamic adaptation.

[0052] In one embodiment, step S1 of constructing an IDC platform and establishing a resource pool within the IDC platform includes:

[0053] S11. Construct high-performance resource pools and low-performance resource pools in the IDC platform. Each of the high-performance resource pools and low-performance resource pools contains multiple unit information points.

[0054] S12. Access ports are set for both the high-performance resource pool and the low-performance resource pool.

[0055] In one embodiment, step S11, which involves constructing a high-performance resource pool and a low-performance resource pool in the IDC platform, includes:

[0056] S111. Multiple storage points are set up in both the high-performance resource pool and the low-performance resource pool.

[0057] S112. Connect the multiple unit information points inside the high-performance resource pool and the low-performance resource pool to the storage resident point one-to-one.

[0058] As described in steps S11 and S12 above, with the development of e-commerce and finance, various sectors require real-time processing of order, risk control, and supply chain data, and much data needs to be processed in real time. This necessitates low-latency data storage and processing methods to support dynamic adjustments to business strategies. Since a large amount of historical data does not have a high demand for low-latency data storage, low-performance data storage can be used, but traditional storage methods are difficult to meet this real-time requirement. IDC (Internet Data Center) is the infrastructure that provides data storage, processing, transmission, and management services for enterprises. In the process of e-commerce order management, it is necessary to obtain corresponding order data from multiple sources. These include material information about the ordered products (such as various computer hardware orders), the source of the order, and information about the order's customers. This multi-source heterogeneous data is stored through an IDC platform. Preprocessing of this heterogeneous data is required during storage, including cleaning and format conversion. Before processing the heterogeneous data, the IDC platform needs to be built. First, a high-performance resource pool and a low-performance resource pool are set up within the IDC platform. The high-performance resource pool is for high-frequency data storage, prioritizing high-quality resources to reduce access latency and meet real-time business needs. The low-performance resource pool is for low-frequency data storage. Multiple unit information is set up within both the high-performance and low-performance resource pools. The unit information points here are all virtual machines. Multiple virtual machines are stored in high-performance resource pools and low-performance resource pools. Access ports are then set for each high-performance and low-performance resource pool. These access ports do not provide authorized access but rather unauthorized access. To ensure that virtual machines are stably stored in the high-performance and low-performance resource pools, multiple storage endpoints must first be set up in the high-performance and low-performance resource pools. These storage endpoints are, for example, IP addresses, UUIDs, or logical addresses, used to fix the storage location of data. The storage endpoints can fix the location of the unit information points. Subsequently, the unit information points can move between the high-performance and low-performance resource pools. The high-performance and low-performance resource pools are interconnected, enabling data storage and better data correspondence, ensuring the stability of data storage location.

[0059] In one embodiment, step S2, which involves setting up a resource configuration assistant for the resource pool in the IDC platform, includes:

[0060] S21. Set up a main resource module in the IDC platform (the main resource module is a data storage device used to store subsequent multi-source heterogeneous data). Set up multiple traction points in the main resource module. The number of traction points is the same as the number of unit information points in the resource pool (the traction points here are virtual machines that can be stored in the main resource module to store multi-source heterogeneous data).

[0061] S22. Set up a central point between the main resource module and the resource pool. The central point includes an access area and multiple guide points in the access area. Combine the multiple guide points according to a preset number to obtain a guide combination (the guide point is used as a connection node as an access port and can determine the multi-source heterogeneous data in the resource pool that needs to be accessed).

[0062] S23. Combine the unit information points inside the high-performance resource pool and the low-performance resource pool according to a preset number to obtain multiple unit information combinations.

[0063] S24. Combine multiple traction points in the main resource module according to a preset number to obtain multiple traction combinations;

[0064] S25. Assign a one-to-one correspondence between the unit information combination, the traction combination, and the guide combination. Connect the unit information points in the corresponding unit information combination to the guide points in the corresponding guide combination through a first channel. Connect the traction points in the traction combination to the guide points in the corresponding guide combination through a second channel. Connect the first channels between the corresponding unit information combination and the guide combination to each other. Connect the second channels between the corresponding traction combination and the guide combination to each other. Set the same connection rule for the first channel between the corresponding unit information combination and the guide combination and the second channel between the traction combination and the guide combination. Record the connection rule in the corresponding guide combination. The connection rule includes the connection relationship between multiple different guide combinations and the traction combination and the unit information combination, respectively.

[0065] S26. Use the main resource sector, central point, and multiple traction points as resource allocation assistants for the resource pool;

[0066] As described in steps S21-S26 above, the IDC platform is configured with resource pools and a main resource module for storing subsequent multi-source heterogeneous data. The main resource module is not used as an accessible database, thus ensuring data stability and security. The high-performance and low-performance resource pools within the resource pool are also used to store subsequent multi-source heterogeneous data. The resource pool itself is a database accessible via external ports. The unit information points within the resource pool are used to store a portion of the multi-source heterogeneous data. A single unit information combination stores related multi-source heterogeneous data, such as information related to a specific order. This order-related information is split to obtain multiple fragments, which are stored separately in multiple unit information points within the same unit information combination. These fragments... The amount of data is the same as the number of unit information points in a unit information combination. Multiple unit information points and multiple traction points within multiple guiding points, high-performance resource pools, and low-performance resource pools are all combined after being divided according to the same preset number. Multiple unit information points within high-performance resource pools and low-performance resource pools are combined independently within each pool, resulting in multiple guiding combinations, multiple unit information combinations, and multiple traction combinations. One of each of these three combinations is selected and matched. A one-to-one connection is established between the unit information points in the corresponding unit information combination and the guiding points in the corresponding guiding combination, and a one-to-one connection is established between the traction points in the traction combination and the guiding points in the corresponding guiding combination. In a single unit information combination, the storage locations of multiple unit information points are not adjacent and continuous, but rather disordered. Multiple guiding points within the central hub enable the main resource module to guide these disordered unit information points in the resource pool. This ensures that data accessed by subsequent authorized external ports is ordered, while ports accessing data outside the central hub obtain disordered data, making accurate data retrieval difficult. This guarantees the security of data storage and access, preventing unauthorized network access to data. The first channel between the corresponding unit information combination and the guiding combination is interconnected, as are the second channels between the corresponding traction combination and the guiding combination. The first channel between the two and the second channel between the traction combination and the guide combination are set with the same connection rules. The communication opening and closing between multiple first channels and second channels are determined according to the connection rules. For example, in a single guide combination there are three guide points (guide points within the central point): y1, y2 and y3. The traction combination corresponding to this single guide combination has three traction points: d1, d2 and d3. Finally, the unit information combination corresponding to this single guide combination has three unit information points: x1, x2 and x3. There are multiple different connection methods between the guide combination and the traction combination, as well as multiple different connection methods between the unit information combination and the traction combination.The connection methods of the guiding combination and the traction combination, as well as the connection methods of the unit information combination and the traction combination, are mutually complementary, thus enabling multi-round connection methods as connection rules. For example, the first round, or the initial connection method, is: d1-y1-x1, d2-y2-x2, and d3-y3-x3. If the fragment data sequence of x1, x2, and x3 connected in the current order of y1, y2, and y3 is used as the original data, then to ensure access security, different connection rules need to be changed. Subsequent connections could be d1-y2-x1, d2-y3-x2, and d3-y1-x3. In this case, the fragment data sequence of x1, x2, and x3 connected in the current order of y2, y3, and y1 is used as the original data. This allows for continuous adjustment of the connection relationship between the guiding point in the guiding combination and the traction point in the corresponding traction combination. In the second round of connection in the above example, when switching from the first round to the second round, the communication connection between d1 and y1 is disconnected, and a new connection is established. The connection between d1 and y2, and all others, follow the same operation. The guide point in the central guide combination records the sorting of the corresponding traction points, and the corresponding guide combination marks the corresponding type data keywords. After the guide point in the guide combination is adjusted, the sorting of the corresponding traction points is re-recorded using the adjusted guide point. However, the sorting of the traction points is displayed through the corresponding guide point. Thus, the guide points in the guide combination are constantly changing. When an authorized port accesses the system, the connection rules can be changed based on the number of accesses. The authorized ports also have the same connection rules and change based on accesses. Therefore, the connection rules of the authorized ports and the connection rules of the central point change synchronously, ensuring that the connection rules can match. Only after matching can the authorized port obtain the connection rules and then retrieve fragmented data from the unit information points according to the connection rules. This provides good data protection, ensures the security of data storage, and prevents unauthorized ports from accessing the system.

[0067] In one embodiment, step S3, which involves establishing a connection between the IDC platform and the multi-source data endpoints, and collecting corresponding data through the multi-source data endpoints and storing it in the main resource module and the resource pool, includes:

[0068] S31. Identify multiple data endpoints as multi-source data endpoints and connect the IDC platform to the multi-source data endpoints;

[0069] S32. The data collected through multiple data sources is used as raw data and stored in the main resource module and the resource pool respectively.

[0070] In one embodiment, step S32, which involves using data collected from multiple data sources as raw data and storing it in the main resource module and the resource pool respectively, includes:

[0071] S321. Classify the raw data collected from multiple data sources to obtain multiple data types;

[0072] S322. Divide multiple types of data into multiple fragments according to a preset quantity;

[0073] S323. Store multiple fragment data in a single type of data one-to-one in multiple traction points of a single traction combination in the main resource module, and sort the corresponding traction points according to the division order of multiple fragment data in a single type of data.

[0074] S324. Store multiple fragments of data in the type data in the unit information points of the corresponding traction points respectively. According to the generation time of the type data, distribute the unit information points corresponding to the type data in the preset recent time period in the high-performance resource pool, and distribute the unit information points corresponding to the remaining type data in the low-performance resource pool.

[0075] S325. The sorting of the corresponding traction points is recorded by the guiding points of the guiding combination in the central point, and the corresponding type data keywords are marked by the guiding combination.

[0076] As described in steps S31-S32 above, the data terminal serves as a port for collecting various types of data, such as server log collection port, user behavior collection port, sensor data collection port, and business database collection port. It marks the time when the collected data was generated to obtain raw data, such as the time when user behavior occurred and the corresponding user behavior. The acquisition of raw data requires different ports, which are referred to here as multi-source data terminals. The data collected by the multi-source data terminals can then be transmitted to the IDC platform and stored in the main resource module and resource pool respectively. The traction points in the main resource module are set continuously, and the traction combinations are also consecutive combinations of adjacent traction points, which are ordered and not chaotic. Here, it is only necessary to divide a single type of data into multiple fragments according to the preset data. The number of fragments is the same as the number of traction points in the traction combination. The preset data is the same as the preset quantity used above. The multiple fragments of the type of data are stored one-to-one in the traction points in the main resource module. The multiple fragments are stored in the traction points in sequence. The storage order is provided to the central point. The corresponding guide point in the central point records the storage order in the traction points. Here, the record is represented by the order of the traction points. The initial storage of the data collected by the multi-source data terminals is based on the generation time of the type of data. The unit information points corresponding to the type of data within the preset time period are distributed in the high-performance resource pool, and the unit information points corresponding to the other type of data are distributed in the low-performance resource pool. Initially, data within the most recent preset time period is considered high-frequency access data, while data outside this preset time period is considered low-frequency access data. This ensures different data is stored on storage devices with varying latency. The current connection structure is: Main Resource Block - Central Point - High-Performance Resource Pool - Low-Performance Resource Pool. The Main Resource Block is not accessible externally; it only connects to the guiding point within the Central Point. This connection establishes an association and is not used for data transmission within the Main Resource Block. The Central Point, High-Performance Resource Pool, and Low-Performance Resource Pool are accessible via external ports. The Central Point is accessible via authorized ports; the High-Performance Resource Pool and Low-Performance Resource Pool are accessible via unauthorized external ports.

[0077] In one embodiment, step S4, where the authorized port accesses data in the resource pool through the central point and dynamically allocates data collected by the multi-source data terminal in the resource pool according to real-time access frequency based on allocation conditions, includes:

[0078] S41. Mark the authorized port, set the connection rules for the corresponding authorized port, and obtain the corresponding boot combination in the access area of ​​the central point through the type data keyword of the authorized port. When the connection rule matches the connection rule corresponding to the boot combination, the sorting of the boot points in the boot combination is obtained.

[0079] S42. The authorized port obtains the sorting of the corresponding unit information points according to the sorting of the boot points in the boot combination, and then obtains the corresponding fragment data in the unit information points in the resource pool, and combines the fragment data to obtain the original data.

[0080] S43. Unauthorized ports access resources through the resource pool's access ports;

[0081] S44. Statistically guide the access frequency of the combined data, migrate the unit information points corresponding to frequencies within the high frequency range to the high-performance resource pool, and migrate the unit information points corresponding to frequencies within the low frequency range to the low-performance resource pool.

[0082] In one embodiment, step S44, which involves migrating cell information points corresponding to frequencies within a high frequency range to a high-performance resource pool and migrating cell information points corresponding to frequencies within a low frequency range to a low-performance resource pool, includes:

[0083] S441. Based on the frequency of the guided combination being accessed, disconnect the unit information point corresponding to the frequency in the high frequency range from the storage point where it is located, transmit the unit information point to the high performance resource pool, randomly select the storage point in the high performance resource pool for connection, and distribute the unit information points in the same unit information combination in the high performance resource pool.

[0084] S442. Disconnect the unit information points corresponding to frequencies that meet the low frequency range from their respective storage points, transmit the unit information points to the low performance resource pool, randomly select storage points in the low performance resource pool for connection, and distribute the unit information points in the same unit information combination in the low performance resource pool.

[0085] As described in steps S41-S44 above, the authorized ports access data through the central point. Other unauthorized ports are not authorized to access the central point. The access area of ​​the central point contains multiple guide combinations, all of which are subject to a single connection rule. Regardless of which guide combination in the access area is accessed, when an access count occurs, the connection rule rolls over to the next guide combination's connection relationship with the traction combination and the unit information combination. Simultaneously, the connection rules set on the authorized ports also automatically roll over to the next guide combination's connection relationship with the traction combination and the unit information combination. Here, connection rules are set specifically for the authorized ports; the authorized ports themselves cannot view these rules. Yes, it's only used for subsequent access. The central point is used for matching connection rules. Authorized ports obtain the corresponding guidance combination in the access area of ​​the central point through type data keywords. For example, if you need to obtain the log information of a certain order, the log is a data type. The data corresponding to a single data type for a certain order will be stored in a unit information combination. When the connection rule matches the connection rule corresponding to the guidance combination, the order of the guidance points in the guidance combination is obtained. Only when there is a match is the order of the guidance points in the guidance combination given. Authorized ports obtain the order of the corresponding unit information points according to the order of the guidance points in the guidance combination, and then obtain the corresponding fragment data in the unit information points in the resource pool. The fragment data is combined to obtain the original data; unauthorized ports obtain the original data through the resource pool. Access is made through the access port, or an unauthorized port accidentally accesses the central point, but without a successful match of the connection rules, no useful information can be obtained. It provides good data protection for the original data, protecting both the original data in the main resource module and allowing the setting up of a resource pool for port access. The resource pool also provides data security protection. Furthermore, based on the frequency of access to the guiding combination, the unit information points corresponding to frequencies within the high-frequency range are disconnected from their respective storage locations, and the unit information points are transmitted to the high-performance resource pool. A storage location within the high-performance resource pool is randomly selected for connection. Unit information points within the same unit information combination are distributed and stored within the high-performance resource pool; the unit information points corresponding to frequencies within the low-frequency range are... The unit information point is disconnected from its storage endpoint and transmitted to a low-performance resource pool. A storage endpoint in the low-performance resource pool is randomly selected for connection. Unit information points in the same unit information combination are distributed and stored in the low-performance resource pool. Data can be hierarchically stored according to the frequency of data access. This hierarchical storage includes storage in the high-performance resource pool and storage in the low-performance resource pool. The corresponding storage can be allocated according to the frequency of data access, which can save the use of high-performance storage, reduce the cost and difficulty of use, and realize real-time perception (access frequency) - intelligent decision-making (determining the range of access frequency) - dynamic adaptation (dynamic migration between the high-performance resource pool and the low-performance resource pool).

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An IDC-oriented multi-source heterogeneous data real-time hierarchical storage method, characterized in that, Includes the following steps: An IDC platform is constructed, and a resource pool is established in the IDC platform. The resource pool includes a high-performance resource pool and a low-performance resource pool. Allocation conditions are set for the high-performance resource pool and the low-performance resource pool respectively. In the IDC platform, a resource configuration assistant is set up for the resource pool, wherein the resource configuration assistant includes a main resource module, a central point, and multiple traction points; The step of setting up a resource configuration assistant for the resource pool in the IDC platform includes: Set up a main resource module in the IDC platform, and set up multiple traction points in the main resource module. The number of traction points is the same as the number of unit information points in the resource pool. A central point is set between the main resource section and the resource pool. The central point includes an access area and multiple guide points in the access area. Multiple guide points are combined according to a preset number to obtain a guide combination. The unit information points inside the high-performance resource pool and the low-performance resource pool are combined according to a preset number to obtain multiple unit information combinations. Multiple traction points in the main resource sector are combined according to a preset number to obtain multiple traction combinations; The unit information combination, traction combination, and guide combination are matched one-to-one. The unit information points in the corresponding unit information combination are connected one-to-one with the guide points in the corresponding guide combination through the first channel. The traction points in the traction combination are connected one-to-one with the guide points in the corresponding guide combination through the second channel. The first channels between the corresponding unit information combination and the guide combination are interconnected. The second channels between the corresponding traction combination and the guide combination are interconnected. The first channels between the corresponding unit information combination and the guide combination and the second channels between the traction combination and the guide combination are set with the same connection rules. The connection rules are recorded in the corresponding guide combination. The connection rules include the connection relationships between multiple different guide combinations and the traction combination and the unit information combination, respectively. The main resource sector, central point, and multiple traction points serve as resource allocation assistants for the resource pool; Establish a connection between the IDC platform and the multi-source data terminal, collect corresponding data through the multi-source data terminal and store it in the main resource module and the resource pool; The authorized port accesses the data in the resource pool through the central point, and dynamically allocates the data collected by the multi-source data terminal in the resource pool according to the real-time access frequency based on the allocation conditions.

2. The IDC-oriented multi-source heterogeneous data real-time hierarchical storage method according to claim 1, characterized in that, The steps of constructing an IDC platform and establishing a resource pool within the IDC platform include: In the IDC platform, high-performance resource pools and low-performance resource pools are built. Each of the high-performance resource pools and low-performance resource pools contains multiple unit information points. Access ports are set for both the high-performance resource pool and the low-performance resource pool.

3. The IDC-oriented multi-source heterogeneous data real-time hierarchical storage method according to claim 2, characterized in that, The steps for building high-performance resource pools and low-performance resource pools in the IDC platform include: Multiple storage nodes are set up in both the high-performance resource pool and the low-performance resource pool; Each of the multiple unit information points within the high-performance resource pool and the low-performance resource pool is connected one-to-one with the storage endpoint.

4. The IDC-oriented multi-source heterogeneous data real-time hierarchical storage method according to claim 1, characterized in that, The steps of establishing the connection between the IDC platform and the multi-source data terminals, and collecting corresponding data through the multi-source data terminals and storing it in the main resource module and the resource pool, include: Multiple data endpoints are identified as multi-source data endpoints, and the IDC platform is connected to these multi-source data endpoints. Data collected from multiple data sources will be used as raw data and stored in the main resource module and the resource pool respectively.

5. A method for real-time hierarchical storage of multi-source heterogeneous data for IDC as described in claim 4, characterized in that, The step of using data collected from multiple data sources as raw data and storing it in the main resource module and the resource pool respectively includes: The raw data collected from multiple data sources is classified into different types to obtain multiple data types. Multiple data types are divided into multiple fragments according to a preset number; Multiple fragments of data in a single data type are stored one-to-one in multiple traction points of a single traction combination in the main resource module, and the corresponding traction points are sorted according to the division order of multiple fragments of data in a single data type. Multiple fragments of data in the type data are stored in the unit information points of the corresponding traction points. Based on the generation time of the type data, the unit information points corresponding to the type data in the preset recent time period are distributed in the high-performance resource pool, and the unit information points corresponding to the remaining type data are distributed in the low-performance resource pool. The central point guides the combination of data by recording the order of the corresponding traction points, and the corresponding guide combination marks the corresponding type data keywords.

6. A method for real-time hierarchical storage of multi-source heterogeneous data for IDC as described in claim 1, characterized in that, The authorized port accesses data in the resource pool through the central point, and dynamically allocates data collected by the multi-source data terminal in the resource pool according to the real-time access frequency based on allocation conditions, including: Mark the authorized port, set the connection rules for the corresponding authorized port, and obtain the corresponding boot combination in the access area of ​​the central point through the type data keyword of the authorized port. When the connection rule matches the connection rule corresponding to the boot combination, the sorting of the boot points in the boot combination is obtained. The authorized port obtains the sorting of the corresponding unit information points according to the sorting of the guide points in the guide combination, and then obtains the corresponding fragment data in the unit information points in the resource pool, and combines the fragment data to obtain the original data; Unauthorized ports are accessing the resource pool via its access ports; The frequency of access to the statistical guidance combination is used to migrate the unit information points corresponding to frequencies within the high frequency range to the high-performance resource pool, and the unit information points corresponding to frequencies within the low frequency range to the low-performance resource pool.

7. A method for real-time hierarchical storage of multi-source heterogeneous data for IDC as described in claim 6, characterized in that, The steps of determining the frequency of access to the statistically guided combination, migrating unit information points corresponding to frequencies within a high frequency range to a high-performance resource pool, and migrating unit information points corresponding to frequencies within a low frequency range to a low-performance resource pool, include: Based on the frequency of access to the guiding combination, the unit information points corresponding to the frequencies that meet the high frequency range are disconnected from their storage locations, and the unit information points are transmitted to the high-performance resource pool. A storage location in the high-performance resource pool is randomly selected for connection, and the unit information points in the same unit information combination are distributed and stored in the high-performance resource pool. The unit information points corresponding to frequencies within the low frequency range are disconnected from their respective storage locations. The unit information points are then transferred to a low-performance resource pool, and a storage location within the low-performance resource pool is randomly selected for connection. Unit information points within the same unit information combination are then distributed and stored in the low-performance resource pool.

Images