Distributed multi-center oriented multi-level cache automatic distribution method and device

Abstract

The invention discloses a distributed multi-center oriented multi-level cache automatic distribution method and device, and belongs to the technical field of data services. The method comprises the steps that a distributed multi-center data service system is constructed, the distributed multi-center data service system comprises a data service main center and a plurality of data service branch centers, and each data service branch center is connected with the data service main center; the data service main center obtains and analyzes the data distribution request, determines to-be-distributed data, classifies the to-be-distributed data, and divides each type of data into a plurality of segments; for each data segment, selecting a candidate node with the highest link transmission efficiency as a data sender of the data segment; and the data sender corresponding to each data segment sends the corresponding data segment to the data requester. According to the invention, the resource overall utilization efficiency of the distributed multi-center data service system is maximized, so that the overall efficiency of the distributed multi-center data service system for external distribution service is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data services, and in particular relates to a distributed multi-center oriented multi-level cache automatic distribution method and device. Background Art

[0002] With the development of network technology, large-scale data service systems are increasingly built using distributed multi-center architectures. This type of architecture has initially achieved efficient external data distribution services, but there are often problems such as backlog of distribution tasks, low efficiency in network resource utilization, and difficulty in resource coordination and scheduling.

[0003] The prior art CN104506632A is a construction method for a distributed multi-center data service system, but it only realizes functions such as data processing at the production node, that is, when providing external services, it only uses the data service main center to provide services, and only provides information retrieval capabilities, and does not realize the ability to actively push and distribute data. It is a "one-to-one" data service method. In terms of data distribution and link selection, the prior art CN115038125A realizes that when there are multiple links between the data sending node and the data receiving node, the data is divided into sub-packets of unequal sizes according to the link load conditions, and sent from multiple links respectively. However, it does not involve the situation where there may be multiple data sending starting points, and is not suitable for distributed multi-center "many-to-many" data distribution application scenarios. Therefore, it is urgently necessary to study a distributed multi-center multi-level cache automatic distribution method and device, which can efficiently distribute the "many-to-many" data of the distributed multi-center service system to maximize the data service efficiency of this type of data service system. Summary of the invention

[0004] The present invention provides a distributed multi-center oriented multi-level cache automatic distribution method and device, which are used to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, a distributed multi-center multi-level cache automatic distribution method is disclosed, the method comprising:

[0006] Step S1: constructing a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, and determines by calculation a number of data service sub-centers that receive data to be stored, and determines by calculation a data service sub-center that distributes data to be distributed;

[0007] Step S2: The data service main center obtains and analyzes the data distribution request, determines the data to be distributed, classifies the data to be distributed, and then divides each type of data into several segments. The data to be distributed is recorded as L = {l1, l2..., l m}, 1≤i≤m, l i Data segments divided into data to be distributed;

[0008] For each data segment, the following operations are performed:

[0009] Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node;

[0010] For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester;

[0011] Select the candidate node with the highest link transmission efficiency as the data sender of the data segment;

[0012] Step S3: The data senders corresponding to the data segments send the corresponding data segments to the data requesters respectively.

[0013] Preferably, determining, for each candidate node, a link transmission efficiency for the candidate node to send the data segment to the data requester comprises:

[0014] Determine all transmission paths from the candidate node to the data requester, the number of the transmission paths is denoted as k, and the set of transmission paths is denoted as R = {r1, r2..., r k Each transmission path r j Denoted as r j = {r j1 ,r j2 ,…,r jh}, 1≤j≤k, 1≤g≤h, h is the component transmission path r j The number of transmission links;

[0015] Determine the maximum transmission delay corresponding to each transmission path

[0016]

[0017] Among them, t jg The transmission path r j The transmission path r jg The transmission delay, t j The transmission path r j transmission delay;

[0018] Determine the maximum bandwidth corresponding to each transmission path

[0019]

[0020] Among them, w jg The transmission path r j The transmission path r jg The remaining bandwidth, w j The transmission path r j The remaining bandwidth;

[0021] Determine the maximum amount of data to be transmitted corresponding to each transmission path

[0022] Among them, S j The transmission path r j The total amount of data in all corresponding data segments;

[0023] The link transmission efficiency of the candidate node sending the data segment to the data requester is:

[0024]

[0025] Among them, m j The transmission path r j Packet loss rate;

[0026] The link transmission efficiency of the candidate node sending the data segment to the data requester is max{V k}.

[0027] Preferably, the data distribution request includes a user identifier, requested data type and data attributes.

[0028] Preferably, the data service main center determines the popularity of the data according to the data distribution situation and the weight coefficient of the data, and caches and maintains the data based on the popularity of the data.

[0029] Preferably, the data heat H is expressed as follows:

[0030] H={N7,N 14 ,N 30 ,size,dataValue}

[0031] N7 is the external service index of the data in the last seven days. 14 N is the external service index in the last 14 days. 30 It is the external service index in the last 30 days. size is the data size. dataValue is the data value, which is the normalized value set when the data is produced.

[0032] In a second aspect, a multi-level cache automatic distribution device for distributed multi-centers is disclosed, the device comprising:

[0033] Service system construction module: configured to construct a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center, and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, determine a number of data service sub-centers that receive data to be stored through calculation, and determine a data service sub-center that distributes data to be distributed through calculation;

[0034] Computing module: configured to obtain and analyze data distribution requests for the data service main center, determine the data to be distributed, classify the data to be distributed, and then divide each type of data into several segments, the data to be distributed is recorded as L = {l1, l2, ..., l m}, 1≤i≤m, l i Data segments divided into data to be distributed;

[0035] For each data segment, the following operations are performed:

[0036] Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node;

[0037] For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester;

[0038] Select the candidate node with the highest link transmission efficiency as the data sender of the data segment;

[0039] Data distribution module: configured so that the data senders corresponding to the respective data segments send the corresponding data segments to the data requesters respectively.

[0040] In a third aspect, an electronic device is disclosed, the electronic device comprising:

[0041] at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0044] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is disclosed, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0045] The present invention has the following technical effects:

[0046] The present invention realizes efficient data distribution services through global perception of data storage, analysis and calculation of optimal transmission paths, automatic data caching and automatic maintenance, and maximizes the overall resource utilization efficiency of the distributed multi-center data service system, thereby improving the overall performance of the system architecture in distributing services to the outside.

[0047] The present invention has more service nodes and more distribution service sources. Different from the existing distribution service system, where one service center serves multiple demand parties, the present invention proposes to build a distributed service architecture of "one main center + multiple sub-nodes" to solve the problem of distribution service interruption caused by network failure of a service center or other resource downtime.

[0048] The transmission link of the present invention is optimal and the service timeliness is faster. Compared with the original one-to-one transmission link, the innovative many-to-many data transmission link efficiency evaluation method proposed by the present invention monitors the link parameters in real time, comprehensively considers multiple factors such as link transmission delay, current available bandwidth, average packet loss rate, and transmission tasks waiting to be executed, and automatically finds the fastest and optimal path to improve data transmission efficiency;

[0049] The present invention performs multi-domain cache management, and the service provision is more efficient. The multi-domain cache management method innovatively proposed by the present invention, based on global information (data usage frequency, data type, data cache status of all nodes, etc.) and individual information of the cache domain (cache domain size), does not need to temporarily call data from the production node, and determines the necessity of cache retention of each data through calculation, so as to ensure the best transmission efficiency as much as possible under limited cache space. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a multi-level cache automatic distribution method for distributed multi-centers;

[0051] Figure 2 This is a schematic diagram of the distributed multi-center data service system architecture;

[0052] Figure 3 This is a diagram of the deployment of distributed multi-center data service software;

[0053] Figure 4 It is another flow chart of the automatic distribution method of multi-level cache for distributed multi-centers;

[0054] Figure 5 This is a schematic diagram of the structure of a multi-level cache automatic distribution device for distributed multi-centers. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, the present invention provides a distributed multi-center multi-level cache automatic distribution method, the method comprising:

[0057] Step S1: constructing a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, and determines by calculation a number of data service sub-centers that receive data to be stored, and determines by calculation a data service sub-center that distributes data to be distributed;

[0058] Step S2: The data service main center obtains and analyzes the data distribution request, determines the data to be distributed, classifies the data to be distributed, and then divides each type of data into several segments. The data to be distributed is recorded as L = {l1, l2..., l m}, 1≤i≤m, l i Data segments divided into data to be distributed;

[0059] For each data segment, the following operations are performed:

[0060] Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node;

[0061] For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester;

[0062] Select the candidate node with the highest link transmission efficiency as the data sender of the data segment;

[0063] Step S3: The data senders corresponding to the data segments send the corresponding data segments to the data requesters respectively.

[0064] like Figure 2-Figure 3 As shown, the architecture of the distributed multi-center data service system and the software deployment mode and functions of the present invention, wherein the data service main center is denoted as C, and the data service sub-nodes are denoted as P = {p1, p2..., p n}.

[0065] Furthermore, the data distribution request includes a user identifier, a requested data type and data attributes.

[0066] The step of determining, for each candidate node, a link transmission efficiency of the candidate node sending the data segment to the data requesting party includes:

[0067] Determine all transmission paths from the candidate node to the data requester, the number of the transmission paths is denoted as k, and the set of transmission paths is denoted as R = {r1, r1..., r k Each transmission path r j Denoted as r j = {r j1 ,r j2 ,…,r jh}, 1≤j≤k, 1≤g≤h, h is the component transmission path r j The number of transmission links;

[0068] Determine the maximum transmission delay corresponding to each transmission path

[0069]

[0070] Among them, t jg The transmission path r j The transmission path r jg The transmission delay, t j The transmission path r j transmission delay;

[0071] Determine the maximum bandwidth corresponding to each transmission path

[0072]

[0073] Among them, w jg The transmission path r j The transmission path r jg The remaining bandwidth, w j The transmission path r j The remaining bandwidth;

[0074] Determine the maximum amount of data to be transmitted corresponding to each transmission path

[0075] Among them, S j The transmission path r j The total amount of data in all corresponding data segments;

[0076] The link transmission efficiency of the candidate node sending the data segment to the data requester is:

[0077]

[0078] Among them, m j The transmission path r j Packet loss rate;

[0079] The link transmission efficiency of the candidate node sending the data segment to the data requester is max{V k}.

[0080] Furthermore, the data service main center determines the popularity of the data according to the data distribution situation and the weight coefficient of the data, and caches and maintains the data based on the popularity of the data.

[0081] The data heat H is expressed as follows

[0082] H={N7,N 14 ,N 30 ,size,dataValue}

[0083] N7 is the external service index of the data in the last seven days. 14 is the external service index in the last 14 days, N 30 It is the external service index in the last 30 days. size is the data size. dataValue is the data value, which is the normalized value set when the data is produced.

[0084] N7、N 14 、N 30 They are all normalized values ​​obtained by dividing the number of times the data is distributed externally in a specified time period by the number of times the data is distributed externally the most times in the specified time period.

[0085] Each node specifies a scoring table corresponding to the file size according to its own cache size, and the file size can be converted into a normalized score value according to the scoring table.

[0086] Then, by comparing the modulus lengths of H, when an alarm is triggered when the buffer space is insufficient, the data with the smallest modulus length of H is deleted.

[0087] The present invention is used to provide "many-to-many" data distribution services to multiple user data demand terminals, wherein the distributed multi-center service system is composed of a main center and multiple sub-nodes. In view of the low efficiency of the existing "one-to-one" and "one-to-many" data distribution between the supply side and the application side, the optimal transmission link to the user side is automatically analyzed and calculated by globally sensing multiple data service nodes and transmission paths, so as to provide "many-to-many" efficient data services for multiple users. The method also generates a normalized data heat score based on global information such as data usage frequency, data caching status of all nodes, and individual information of the cache domain by adopting a multi-level data cache management method, and manages and maintains the cache based on the heat and cache usage. It can ensure the optimal transmission efficiency as much as possible within a limited cache space, and realize efficient data distribution services.

[0088] The present invention comprises the following steps:

[0089] S1: The user proposes a data demand through the software deployed on his terminal (denoted as T). The demand contains elements including the demand ID, the data category of the demand, and the data attribute requirements. The demand proposed by the user is denoted as Q{id, DataType, Properties};

[0090] S11: The terminal T software stores the user's needs and updates the demand status according to the data received;

[0091] S12: Send the demand Q to the data service main center C;

[0092] S2: After receiving the demand Q, the data service main center C stores and records the demand Q, parses Q, matches the demand element data category DataType and data attribute Properties with the existing data directory, and obtains the list of data to be distributed L = {l1, l2..., l m};

[0093] S3: An innovative method for evaluating the efficiency of data transmission links is proposed to select the optimal transmission path. The data service center C globally searches for the storage status of each data in the list of data to be distributed L, which is recorded as That is, the i-th data is stored in o (o≥1) copies. The following describes it separately according to the different situations of o.

[0094] S31: If o=1, then there is only one copy of the data stored, the starting point and the end point of the transmission have been determined, and it is only necessary to find the best transmission path from the starting point to the end point. In this case, the data transmission link efficiency evaluation algorithm is as follows:

[0095] Find all the paths between the starting point and the end point, denoted as R = {r1, r2..., r k}, there are k in total.

[0096] For each channel, its composition is r j = {r j1 , r j2 ..., r jh}(j={1,2,...,k}), that is, the j-th path consists of jh segments in total.

[0097] The transmission delay of each segment is t jg | g＝1，2，...，h , then the total transmission delay The maximum transmission delay of all links can be calculated

[0098]

[0099] The current remaining bandwidth of each path is denoted as w jg | g＝1，2，...，h , then the bandwidth of a link is estimated by the minimum value of the segment residual bandwidth Maximum bandwidth of all links

[0100]

[0101] The size of the file waiting to be transmitted on each link is S j , the maximum size of files waiting to be transferred on all links

[0102]

[0103] The transmission efficiency calculation formula for each link is:

[0104]

[0105] Where m j is the transmission packet loss rate of the link.

[0106] S32: If o>1, there are multiple possible data transmission starting points. For each transmission starting point, call the method of S31 to calculate the efficiency of all feasible links of each starting point respectively, select the one with the best efficiency, and compare it with the link with the best efficiency starting from other nodes, so as to obtain the link with the global best efficiency.

[0107] S4: According to the transmission efficiency evaluation value calculated in S3, the most efficient link is selected for data transmission, and each node on the transmission path performs data caching.

[0108] S5: An innovative method for managing and maintaining the cache is proposed, which is used for daily management and maintenance of the data cache of all nodes. The data service main center calculates the data heat according to the data distribution and importance, and maintains the data cache of each node based on the data heat, node cache status, etc.

[0109] like Figure 5 As shown, the present invention provides a distributed multi-center multi-level cache automatic distribution device, the device comprising:

[0110] Service system construction module: configured to construct a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center, and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, and determines by calculation a number of data service sub-centers that receive data to be stored, and determines by calculation a data service sub-center that distributes data to be distributed;

[0111] Computing module: configured to obtain and analyze data distribution requests for the data service main center, determine the data to be distributed, classify the data to be distributed, and then divide each type of data into several segments, the data to be distributed is recorded as L = {l1, l2..., l m}, 1≤i≤m, l i Data segments divided into data to be distributed;

[0112] For each data segment, the following operations are performed:

[0113] Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node;

[0114] For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester;

[0115] Select the candidate node with the highest link transmission efficiency as the data sender of the data segment;

[0116] Data distribution module: configured so that the data senders corresponding to the respective data segments send the corresponding data segments to the data requesters respectively.

[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-level cache automatic distribution method for distributed multi-centers, characterized in that: The method comprises: Step S1: constructing a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, and determines by calculation a number of data service sub-centers that receive data to be stored, and determines by calculation a data service sub-center that distributes data to be distributed; Step S2: The data service main center obtains and analyzes the data distribution request, determines the data to be distributed, classifies the data to be distributed, and then divides each type of data into several segments. The data to be distributed is recorded as L = {l1, l2..., l m }, 1≤i≤m, l i Data segments divided into data to be distributed; For each data segment, the following operations are performed: Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node; For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester; Select the candidate node with the highest link transmission efficiency as the data sender of the data segment; Step S3: The data senders corresponding to the data segments send the corresponding data segments to the data requesters respectively.

2. The method according to claim 1, characterized in that The step of determining, for each candidate node, a link transmission efficiency of the candidate node sending the data segment to the data requesting party includes: Determine all transmission paths from the candidate node to the data requester, the number of the transmission paths is denoted as k, and the set of transmission paths is denoted as R = {r1, r2..., r k Each transmission path r j Denoted as r j = {r j1 ,r j2 ,…,r jh }, 1≤j≤k, 1≤g≤h, h is the component transmission path r j The number of transmission links; Determine the maximum transmission delay corresponding to each transmission path Among them, t jg The transmission path r j The transmission section r jg The transmission delay, t j The transmission path r j transmission delay; Determine the maximum bandwidth corresponding to each transmission path Among them, w jg The transmission path r j The transmission section r jg The remaining bandwidth, w j The transmission path r j The remaining bandwidth; Determine the maximum amount of data to be transmitted corresponding to each transmission path Among them, S j The transmission path r j The total amount of data in all corresponding data segments; The link transmission efficiency of the candidate node sending the data segment to the data requester is: Among them, m j The transmission path r j Packet loss rate; The link transmission efficiency of the candidate node sending the data segment to the data requester is max{V k }.

3. The method according to claim 1, characterized in that The data distribution request includes a user identifier, requested data type and data attributes.

4. The method according to claim 1, characterized in that The data service main center determines the popularity of the data according to the data distribution situation and the weight coefficient of the data, and caches and maintains the data based on the popularity of the data.

5. The method according to claim 4, characterized in that The data heat H is expressed as follows: H={N7,N 14 ,N 30 ,size,dataValue} N7 is the external service index of the data in the last seven days. 14 N is the external service index in the last 14 days. 30 It is the external service index in the last 30 days. size is the data size. dataValue is the data value, which is the normalized value set when the data is produced.

6. A multi-level cache automatic distribution device for distributed multi-centers, characterized in that: The device comprises: Service system construction module: configured to construct a distributed multi-center data service system, the distributed multi-center data service system includes a data service main center, and multiple data service sub-centers, each of which is connected to the data service main center; the data service sub-center is used to store data, the data service main center is used to store data, manage the data service sub-centers and perform calculations for data storage and data distribution, determine a number of data service sub-centers that receive data to be stored through calculation, and determine a data service sub-center that distributes data to be distributed through calculation; Computing module: configured to obtain and analyze data distribution requests for the data service main center, determine the data to be distributed, classify the data to be distributed, and then divide each type of data into several segments, the data to be distributed is recorded as L = {l1, l2, ..., l m }, 1≤i≤m, l i Data segments divided into data to be distributed; For each data segment, the following operations are performed: Determine a data service sub-center and / or a data service main center storing the data segment; and use each of the data service sub-center and / or the data service main center storing the data segment as a candidate node; For each candidate node, determining a link transmission efficiency for the candidate node to send the data segment to a data requester; Select the candidate node with the highest link transmission efficiency as the data sender of the data segment; Data distribution module: configured so that the data senders corresponding to the respective data segments send the corresponding data segments to the data requesters respectively.

7. An electronic device, characterized in that: The device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images