Large-scale scene-oriented variable sharing method and device, equipment and storage medium

Abstract

The invention discloses a variable sharing method, device and equipment for a large-scale scene and a storage medium, and belongs to the field of distribution.The variable sharing method comprises the steps that node data of a shared network and target delay parameters of the shared network are obtained; generating a plurality of network topology structures of the shared network according to the transmission performance parameter and the target delay parameter; according to the transmission performance parameter and the target delay parameter, determining a traffic evaluation value of the network topology structure, and determining the network topology structure corresponding to the minimum traffic evaluation value as a target network topology structure; and adjusting the connection relationship between the nodes in the shared network according to the target network topology structure. According to the network topology structure generated by the scheme of the invention, data transmission between the writing node and the reading node is only transmitted to the parent level from the child level in the writing node forest, and is only transmitted to the child level from the parent level in the reading node forest, so that the problem that a shared variable mechanism can only be applied to small and medium-scale server clusters in the prior art is solved.

Description

Technical Field

[0001] The present application belongs to the field of distributed computing, and specifically relates to a shared variable method, device, equipment and storage medium for large-scale scenarios. Background Art

[0002] In modern distributed computing environments, multiple computing nodes need to share data to achieve collaborative work. Shared variables are special variables that allow multiple nodes to access and modify them simultaneously to ensure data consistency, reliability, and concurrency. Shared variables play an important role in distributed computing, distributed storage, and distributed databases, supporting data sharing and collaboration between multiple active nodes. However, as the number of nodes increases, maintaining the consistency and timeliness of shared variables becomes more complex, especially in large-scale cluster environments.

[0003] At present, the commonly used shared variable mechanism in distributed systems mainly relies on strong consistency algorithms. These algorithms ensure strong consistency of data in the system through complex mechanisms. In order to implement shared variables in large-scale scenarios, CRDT (Conflict-free Replicated Data Type) was introduced. CRDT ensures strong final consistency of data through the characteristics of merging and commutativity, avoiding conflicts in concurrent modifications and network transmission.

[0004] However, in large-scale clusters, frequent network communication and data synchronization are required to adapt to large-scale scenarios, but this leads to a significant increase in resource consumption and affects the overall performance of the system. Summary of the invention

[0005] The present application aims to provide a shared variable method, device, electronic device and computer-readable storage medium for large-scale scenarios, at least to solve the problem that distributed shared variables can only be applied to small and medium-sized server clusters.

[0006] In a first aspect, the present application embodiment discloses a shared variable method for large-scale scenarios, including:

[0007] Obtain node data of a shared network and a target delay parameter of the shared network; the node data includes transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameter is used to constrain the longest time interval from when a data variable is written into any one of the write nodes of the shared network to when the data variable is output to any one of the read nodes of the shared network;

[0008] According to the transmission performance parameter and the target delay parameter, a plurality of network topology structures of the shared network are generated; the network topology structures include a write node forest composed of write nodes in the network, and a read node forest composed of read nodes in the network; the write node forest includes at least one write tree, and the read node forest includes at least one read tree; the root write node of each write tree is connected to the root read node of each read tree one by one;

[0009] Determine the traffic evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum traffic evaluation value as the target network topology structure;

[0010] The connection relationship between the nodes in the shared network is adjusted according to the target network topology structure to obtain a target shared network.

[0011] In a second aspect, the present application also discloses a shared variable device for large-scale scenarios, including:

[0012] A data acquisition module, used to acquire node data of a shared network and a target delay parameter of the shared network; the node data includes transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameter is used to constrain the longest time interval from when a data variable is written into any one of the write nodes of the shared network to when the data variable is output to any one of the read nodes of the shared network;

[0013] A topology generation module, configured to generate a plurality of network topology structures of the shared network according to the transmission performance parameter and the target delay parameter; the network topology structure comprises a write node forest composed of write nodes in the network, and a read node forest composed of read nodes in the network; the write node forest comprises at least one write tree, and the read node forest comprises at least one read tree; the root write node of each write tree is connected one by one with the root read node of each read tree;

[0014] A topology selection module, configured to determine a flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum flow evaluation value as the target network topology structure;

[0015] The network adjustment module is used to adjust the connection relationship between the nodes in the shared network according to the target network topology structure to obtain the target shared network.

[0016] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In summary, in the embodiment of the present application, by obtaining the transmission performance parameters and target delay parameters of each node in the shared network, the data transmission between any write node and any read node in the network topology structure generated by the solution of the present application is utilized to ensure that the data transmission from the child-level node to the parent-level node in the write node forest, and the unidirectional transmission from the parent-level node to the child-level node in the read node forest, so that the generated network topology structure can meet the connection performance and delay requirements of each node in the network, and then evaluate the traffic of different network topologies to further select the network structure with the best utilization of network resources, and further reduce the network communication overhead. So that the distributed shared variables can be adapted to a larger number of server clusters. Therefore, based on the method of the embodiment of the present application, an efficient shared variable mechanism for large-scale scenarios is realized, which solves the problem that the distributed shared variable solution in the prior art can only be applied to small and medium-sized server clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the attached picture:

[0020] Figure 1 It is a flowchart of a shared variable method for large-scale scenarios provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a network topology structure provided in an embodiment of the present application;

[0022] Figure 3 It is a flowchart of another shared variable method for large-scale scenarios provided by an embodiment of the present application;

[0023] Figure 4 It is a principle schematic diagram of the data expansion rate of a node in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a method for adding nodes in an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a method for exiting a node in an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of the life cycle evolution process of a shared variable under an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of a data structure in which the method of the present application is applied to a digital object architecture;

[0028] Fig. 9 is a flowchart of a node receiving and processing digital object information provided by an embodiment of the present application;

[0029] Fig.10 It is a flowchart of the steps of periodically sending protocol information by a write node provided in an embodiment of the present application;

[0030] Fig.11 It is a flowchart of the steps of a read node periodically sending protocol information provided by an embodiment of the present application;

[0031] Fig.12 This is an operation flow chart of a CRDT variable network simulation experiment program obtained based on the method provided in the embodiment of the present application;

[0032] Fig.13 It is a block diagram of a shared variable device for large-scale scenarios provided by an embodiment of the present application;

[0033] Fig.14 is a block diagram of an electronic device according to an embodiment of the present application;

[0034] Fig.15 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] Figure 1 This is a shared variable method for large-scale scenarios provided by this embodiment.

[0038] The method may include the following steps:

[0039] Step 101: Obtain node data of a shared network and a target delay parameter of the shared network.

[0040] Among them, the node data includes the transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameters are used to constrain the maximum time interval from the writing of data variables to any write node of the shared network to the output of data variables to any read node of the shared network.

[0041] In some embodiments of the present application, the node data and target delay parameters of the shared network are obtained. The node data includes the transmission performance parameters of each node in the shared network, which are used to characterize the data transmission performance when the node is connected to other nodes. The target delay parameter is used to constrain the longest time interval from the writing of the data variable to the writing node of any shared network to the output of the data variable to the reading node of any shared network. By obtaining these data, basic information can be provided for the subsequent generation of network topology and optimization of network performance.

[0042] For example, in a large-scale distributed system, suppose there is a shared network with 100 write nodes and 200 read nodes. First, obtain the transmission performance parameters of each node, such as bandwidth, delay, and maximum number of connections. Assume that the transmission performance parameters of node A are: bandwidth 10Gbps, delay 5ms, and maximum number of connections 3. Then, set the target delay parameter to 100ms, that is, the longest time interval from the data variable being written to any write node to the data variable being output to any read node is 100ms. By collecting this information, it can provide the necessary data support for generating and optimizing the network topology in the subsequent steps.

[0043] Step 102: Generate multiple network topology structures of the shared network according to the transmission performance parameter and the target delay parameter.

[0044] Among them, Figure 2As shown in the figure, the circular nodes represent read nodes, the triangular nodes represent write nodes, the arrows between the nodes represent the data transmission direction, and the network topology structure includes a write node forest {W} composed of write nodes in the network, and a read node forest {R} composed of read nodes in the network; the write node forest {W} contains at least one write tree W, and the read node forest {R} contains at least one read tree R; the root write node w1 of each write tree is connected one by one with the root read node r1 of each read tree.

[0045] In some embodiments of the present application, multiple network topologies of a shared network are generated based on transmission performance parameters and target delay parameters. The network topology includes a write node forest consisting of write nodes in the network and a read node forest consisting of read nodes in the network. The write node forest contains at least one write tree, and the read node forest contains at least one read tree. The root write node of each write tree is connected one-to-one with the root read node of each read tree. By generating multiple network topologies, the performance of different topologies in terms of transmission performance and delay can be evaluated, thereby selecting the optimal network structure.

[0046] For example, in a large-scale distributed system, assume that there are 100 write nodes and 200 read nodes. Multiple network topologies will be generated based on the transmission performance parameters and target latency parameters obtained in the previous step. For example, a topology is generated in which the write node forest contains 10 write trees, each with 10 write nodes; the read node forest contains 20 read trees, each with 10 read nodes. The root write node of each write tree is connected to the root read node of each read tree one by one. By generating such a network topology, its performance in data transmission performance and latency can be further evaluated to select the optimal network structure.

[0047] Step 103: Determine the flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum flow evaluation value as the target network topology structure.

[0048] In some embodiments of the present application, the flow evaluation value of each network topology is determined based on the transmission performance parameter and the target delay parameter. The flow evaluation value is used to measure the efficiency and overhead of the network topology during data transmission. By calculating the flow evaluation value of each network topology, the performance of different topologies can be compared, and the network topology with the smallest flow evaluation value can be selected as the target network topology. This ensures that the utilization of network resources is optimized and the network communication overhead is reduced while meeting the target delay parameter.

[0049] For example, in a large-scale distributed system, assume that three different network topologies are generated. Based on the transmission performance parameters and target delay parameters obtained in the previous step, the traffic evaluation value of each topology can be calculated. For example, the traffic evaluation value of topology A is 500GB, the traffic evaluation value of topology B is 450GB, and the traffic evaluation value of topology C is 480GB. By comparing these three traffic evaluation values, it is determined that the traffic evaluation value of topology B is the smallest, so topology B is determined as the target network topology. In this way, the utilization of network resources can be optimized to the maximum extent while ensuring data transmission performance and delay requirements.

[0050] Step 104: Adjust the connection relationship between nodes in the shared network according to the target network topology structure to obtain the target shared network.

[0051] In some embodiments of the present application, the connection relationship between nodes in the shared network is adjusted according to the determined target network topology to obtain the target shared network. By adjusting the connection relationship between nodes, it is possible to ensure that the network topology meets the requirements of the transmission performance parameters and the target delay parameters, thereby optimizing the utilization of network resources, reducing communication overhead, and improving the efficiency and reliability of data transmission.

[0052] For example, in a large-scale distributed system, assume that the target network topology has been determined to contain 10 write trees and 20 read trees. According to the target network topology, adjust the connection relationship between the nodes in the shared network. For example, write nodes A, B, C, D, and E are used as the root nodes of five write trees, and other write nodes are connected to these root nodes according to a hierarchical relationship. At the same time, read nodes F, G, H, I, and J are used as the root nodes of five read trees, and other read nodes are connected to these root nodes according to a hierarchical relationship. Finally, ensure that the root write node of each write tree is connected to the root read node of each read tree one by one. Through such adjustments, a shared network that conforms to the target network topology is obtained, thereby optimizing data transmission performance and network resource utilization.

[0053] In summary, in the embodiment of the present application, by obtaining the transmission performance parameters and target delay parameters of each node in the shared network, the data transmission between any write node and any read node in the network topology structure generated by the solution of the present application is utilized to ensure that the data transmission from the child-level node to the parent-level node in the write node forest, and the unidirectional transmission from the parent-level node to the child-level node in the read node forest, so that the generated network topology structure can meet the connection performance and delay requirements of each node in the network, and then evaluate the traffic of different network topologies to further select the network structure with the best utilization of network resources, and further reduce the network communication overhead. So that the distributed shared variables can be adapted to a larger number of server clusters. Therefore, based on the method of the embodiment of the present application, an efficient shared variable mechanism for large-scale scenarios is realized, which solves the problem that the distributed shared variable solution in the prior art can only be applied to small and medium-sized server clusters.

[0054] Figure 3 Another shared variable method for large-scale scenarios provided in the embodiment of the present application may include the following steps:

[0055] Step 201: Acquire node data of a shared network and a target delay parameter of the shared network.

[0056] Among them, the node data includes the transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameters are used to constrain the maximum time interval from the writing of data variables to any write node of the shared network to the output of data variables to any read node of the shared network.

[0057] The method shown in this step has been explained in step 101 and will not be repeated here.

[0058] Step 202: Generate multiple network topology structures of the shared network according to the transmission performance parameter and the target delay parameter.

[0059] Among them, the network topology structure includes a write node forest composed of write nodes in the network, and a read node forest composed of read nodes in the network; the write node forest contains at least one write tree, and the read node forest contains at least one read tree; the root write node of each write tree is connected one by one with the root read node of each read tree.

[0060] The method shown in this step has been explained in step 102 and will not be repeated here.

[0061] Optionally, the transmission performance parameter includes a maximum number of connections of the node, and step 202 includes the following sub-steps:

[0062] Sub-step 2021, using the node connectivity of the write tree as the first traversal variable parameter, and the maximum number of connections as the maximum value of the first traversal parameter, to generate multiple write node forests.

[0063] Among them, the first traversal variable parameter is used to represent the node degree of the write tree in the write node forest.

[0064] In some embodiments of the present application, the node connectivity of the write tree is used as the first traversal variable parameter, and the maximum number of connections is used as the maximum value of the first traversal parameter, thereby generating multiple write node forests. Node connectivity refers to the number of other nodes that a node can directly connect to. By traversing different node connectivity, multiple write node forests with different structures can be generated to evaluate their performance and latency in subsequent steps.

[0065] For example, in a large-scale distributed system, suppose there are 100 write nodes and the maximum number of connections of the nodes is 10. Then the node connectivity of the write tree is first used as the first traversal variable parameter, and its range is set from 1 to 10. Then, multiple write node forests are generated. For example, when the node connectivity is 3, the generated write node forest may contain several write trees, and the node degree of each tree is 3. For another example, when the node connectivity is 5, the generated write node forest may contain several write trees, and the node degree of each tree is 5. By traversing different node connectivity, multiple write node forests are generated, providing a basis for evaluating their performance and latency in subsequent steps.

[0066] Sub-step 2022, using the node connectivity of the read tree as the second traversal variable parameter, and the maximum number of connections as the maximum value of the second traversal parameter, to generate multiple read node forests.

[0067] The second traversal variable parameter is used to represent the node degree of the tree read in the read node forest.

[0068] In some embodiments of the present application, the node connectivity of the read tree is used as the second traversal variable parameter, and the maximum number of connections is used as the maximum value of the second traversal parameter, thereby generating multiple read node forests. Node connectivity refers to the number of other nodes that a node can directly connect to. By traversing different node connectivity, multiple read node forests with different structures can be generated to evaluate their performance and latency in subsequent steps.

[0069] For example, in a large-scale distributed system, suppose there are 200 read nodes and the maximum number of connections of the nodes is 8. First, the node connectivity of the read tree is used as the second traversal variable parameter, and its range is set from 1 to 8. Then, multiple read node forests are generated. For example, when the node connectivity is 4, the generated read node forest may contain several read trees, and the node degree of each tree is 4. For another example, when the node connectivity is 6, the generated read node forest may contain several read trees, and the node degree of each tree is 6. By traversing different node connectivity, multiple read node forests are generated, providing a basis for evaluating their performance and latency in subsequent steps.

[0070] Sub-step 2023 , connect each write node forest to each read node forest respectively to obtain multiple candidate topologies, and test the maximum transmission delay of each candidate topology.

[0071] In some embodiments of the present application, each write node forest is connected to each read node forest to obtain multiple alternative topologies, and the maximum transmission delay of each alternative topology is tested. By combining different write node forests with read node forests, multiple alternative network topologies can be generated. Then, the maximum transmission delay of each alternative topology is tested to evaluate its performance during data transmission. The maximum transmission delay refers to the longest time required from the data variable being written to any write node to the data variable being output to any read node.

[0072] For example, in a large-scale distributed system, assume that three write node forests A, B, and C, and two read node forests X and Y have been generated. First, each write node forest is connected to each read node forest to generate six alternative topologies. For example, write node forest A is connected to read node forest X to generate alternative topology AX; write node forest A is connected to read node forest Y to generate alternative topology AY, and so on. Next, test the maximum transmission delay of each alternative topology. For example, the maximum transmission delay of alternative topology AX is 90ms, and the maximum transmission delay of alternative topology AY is 110ms. By testing the maximum transmission delay of these alternative topologies, their performance in the data transmission process can be evaluated, providing a basis for selecting the optimal network topology in subsequent steps.

[0073] In sub-step 2024, the candidate topologies corresponding to the maximum transmission delays greater than the target delay parameter in the candidate topologies are deleted, and the remaining candidate topologies are used as the network topology.

[0074] In some embodiments of the present application, the alternative topology whose maximum transmission delay is greater than the target delay parameter will be deleted, and the remaining alternative topology will be used as the network topology. The target delay parameter refers to the longest time interval from the writing of a data variable to any write node to the output of the data variable to any read node. By deleting the alternative topology that does not meet the target delay parameter, it can be ensured that the network topology finally selected meets the requirements in terms of data transmission performance and delay, thereby optimizing the utilization of network resources and communication efficiency.

[0075] For example, in a large-scale distributed system, assume that the target delay parameter is 100ms. Six alternative topologies have been generated through the previous steps, and the maximum transmission delay of each alternative topology has been tested. For example, the maximum transmission delay of alternative topology AX is 90ms, AY is 110ms, BX is 85ms, BY is 105ms, CX is 120ms, and CY is 100ms. According to the target delay parameter, the alternative topologies AY, BY, and CX with a maximum transmission delay greater than 100ms are deleted. The remaining alternative topologies AX, BX, and CY are used as network topologies that meet the target delay parameters. In this way, it is ensured that the final selected network topology meets the requirements in terms of data transmission performance and delay, thereby optimizing the utilization of network resources and communication efficiency.

[0076] Step 203: Determine the flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum flow evaluation value as the target network topology structure.

[0077] The method shown in this step has been explained in step 103 and will not be repeated here.

[0078] Optionally, in order to determine the flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, step 203 includes the following sub-steps:

[0079] Sub-step 2031, according to the transmission performance parameter and the target delay parameter, respectively determine the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value of the network topology structure.

[0080] In some embodiments of the present application, the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value of the network topology are determined respectively according to the transmission performance parameter and the target delay parameter. The write traffic evaluation value refers to the total amount of data transmission in the write node forest, the read traffic evaluation value refers to the total amount of data transmission in the read node forest, and the broadcast traffic evaluation value refers to the total amount of data transmission between the write node forest and the read node forest. By calculating these traffic evaluation values ​​respectively, the performance and overhead of the network topology during data transmission can be comprehensively evaluated.

[0081] For example, in a large-scale distributed system, suppose there is a network topology AX, which contains 100 write nodes and 200 read nodes. According to the transmission performance parameters and the target delay parameters, the write traffic evaluation value is first calculated. For example, suppose that the amount of data generated by each write node per unit time is 10MB, and the total data transmission volume of the write node forest is 1000MB. Next, the read traffic evaluation value is calculated. For example, suppose that the amount of data received by each read node per unit time is 5MB, and the total data transmission volume of the read node forest is 1000MB. Finally, the broadcast traffic evaluation value is calculated. For example, suppose that the data transmission volume between the write node forest and the read node forest is 500MB. By calculating these traffic evaluation values ​​separately, the performance and overhead of the network topology during data transmission can be comprehensively evaluated, providing a basis for selecting the optimal network topology in subsequent steps.

[0082] Sub-step 2032, taking the sum of the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value as the traffic evaluation value of the network topology structure.

[0083] In some embodiments of the present application, the sum of the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value is used as the traffic evaluation value of the network topology. By adding these three traffic evaluation values, a comprehensive traffic evaluation value can be obtained to measure the total overhead of the entire network topology during data transmission. This comprehensive traffic evaluation value can help compare different network topologies and select the optimal structure to optimize the utilization of network resources and communication efficiency.

[0084] For example, in a large-scale distributed system, assume that the write traffic assessment value, read traffic assessment value, and broadcast traffic assessment value of a certain network topology AX have been calculated separately. For example, the write traffic assessment value is 1000MB, the read traffic assessment value is 1200MB, and the broadcast traffic assessment value is 500MB. Adding these three assessment values ​​together, the comprehensive traffic assessment value of the network topology is 2700MB. In this way, the total overhead of the network topology during data transmission can be comprehensively measured, providing a basis for selecting the optimal network topology in subsequent steps.

[0085] As shown in the figure, in the embodiment of the present application, since the data transmission between nodes only occurs between the parent node and the child node in the corresponding forest, and only occurs between the root nodes between forests, when the flow evaluation value is actually calculated, the nonlinear increment existing when the data from different child nodes is merged after being transferred to the parent node is measured by defining the data expansion rate. The expansion rate of the variable is used to describe the change in the size of the two data before and after the merger: the degree to which the total memory occupied by the variable changes during the transmission and merging process of multiple nodes. :

[0086] like Figure 4 As shown, it is an example of a data expansion rate of 1: when the w4 node transmits data to the w3 node, since the data expansion rate is 1, it means that in the process of upward transmission, the parent-level node obtains one unit of data to merge the data of each child node. Therefore, when the amount of data sent by the w4 node is one unit (td), the amount of data actually obtained by the w3 node is 3*1+1=4(td), the amount of data obtained by the w2 node is 3*4+1=13(td), and the amount of data obtained by the w1 node is 3*13+1=40td.

[0087] Preferably, based on the different data expansion rates of the nodes, the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value have different calculation methods:

[0088] For unified expression, in the following formulas:

[0089] D w Indicates the write traffic evaluation value, D r Indicates the read traffic evaluation value, D rw Indicates the broadcast traffic evaluation value;

[0090] W represents the total number of nodes in the corresponding write node forest, w represents the total number of trees in the corresponding write node forest, R represents the total number of nodes in the corresponding read node forest, r represents the total number of trees in the corresponding read node forest, n represents the target number of transmissions of the node's transmission cycle T, and t represents the synchronization time interval of the node;

[0091] d represents the size of each data transmission, A represents the size of a data variable with all the data in the value range; d' represents the size of n data variables of size d after merging: d'=A(1-λ n ), λ represents the approximate parameter:

[0092] K w represents the degree of the corresponding write tree;

[0093] H w Indicates the corresponding tree height of the write tree:

[0094] N w Indicates the write node forest parameters:

[0095] Optionally, when the data expansion rate of the shared network is 0 (i.e., it remains unchanged after merging, typically a register structure), the write traffic evaluation value is calculated according to the following formula:

[0096] D w=(Ww)nd;

[0097] The estimated read traffic value is calculated according to the following formula:

[0098] D r =(Rr)nd;

[0099] The broadcast traffic assessment value is calculated according to the following formula:

[0100] D rw =rwnd.

[0101] Optionally, when the data expansion rate of the shared network is greater than 0 and less than 1 (some data between replicas can be merged, while some data cannot be merged, typically a set structure with repeated values), the write traffic assessment value is calculated according to the following formula:

[0102]

[0103] The estimated read traffic value is calculated according to the following formula:

[0104] D r =(Rr)nA(1-λ Wt );

[0105] The broadcast traffic assessment value is calculated according to the following formula:

[0106]

[0107] Optionally, when the data expansion rate of the shared network is 1 (the size after merging n replicas is the sum of the sizes of n replicas, typically in a list structure), the write traffic assessment value is calculated according to the following formula:

[0108]

[0109] The estimated read traffic value is calculated according to the following formula:

[0110] Dr = n(Rr)Wtd = (Rr)WTd;

[0111] The broadcast traffic assessment value is calculated according to the following formula:

[0112] D rw =wrnN w td=wrN w Td.

[0113] Step 204: Adjust the connection relationship between nodes in the shared network according to the target network topology structure to obtain the target shared network.

[0114] The method shown in this step has been described in step 104 and will not be repeated here.

[0115] Optionally, step 204 includes the following sub-steps:

[0116] Sub-step 2041, adjusting the connection relationship of the write nodes in the shared network to a first target topology structure to generate a target write node forest, and adjusting the connection relationship of the read nodes in the shared network to a second target topology structure to generate a target read node forest.

[0117] The first target topology structure is a subgraph of the write node forest of the target network topology structure, and the second target topology structure is a subgraph of the read node forest of the target network topology structure.

[0118] In some embodiments of the present application, the connection relationship of the write nodes in the shared network is adjusted to a first target topology to generate a target write node forest, and the connection relationship of the read nodes in the shared network is adjusted to a second target topology to generate a target read node forest. The first target topology is a subgraph of the write node forest of the target network topology, and the second target topology is a subgraph of the read node forest of the target network topology. By adjusting the connection relationship between the write nodes and the read nodes, it can be ensured that the network topology meets the requirements of the transmission performance parameters and the target delay parameters, thereby optimizing the utilization rate of network resources and communication efficiency.

[0119] For example, in a large-scale distributed system, it is assumed that the target network topology has been determined, in which the write node forest contains 10 write trees and the read node forest contains 20 read trees. First, the connection relationship of the write nodes in the shared network is adjusted to the first target topology. For example, write nodes A, B, C, D, and E are respectively used as the root nodes of five write trees, and other write nodes are connected to these root nodes according to a hierarchical relationship to form a target write node forest. Next, the connection relationship of the read nodes in the shared network is adjusted to the second target topology. For example, read nodes F, G, H, I, and J are respectively used as the root nodes of five read trees, and other read nodes are connected to these root nodes according to a hierarchical relationship to form a target read node forest. In this way, a write node forest and a read node forest that conform to the target network topology are generated, thereby optimizing data transmission performance and network resource utilization.

[0120] Sub-step 2042: Connect all root write nodes of the target write node forest to all root read nodes of the target read node forest one by one to obtain a target shared network.

[0121] In some embodiments of the present application, all root write nodes of the target write node forest are connected to all root read nodes of the target read node forest one by one to obtain a target shared network. By connecting the root nodes of the write node forest and the read node forest, it is possible to ensure that data is efficiently transmitted between the write nodes and the read nodes, thereby achieving an optimized configuration of the entire network. This step is to complete the final network topology so that it meets the requirements of the transmission performance parameters and the target delay parameters.

[0122] For example, in a large-scale distributed system, suppose the target write node forest contains 5 write trees, with root write nodes A1, A2, A3, A4, and A5; the target read node forest contains 4 read trees, with root read nodes B1, B2, B3, and B4. Then each root write node will be connected to each root read node one by one. For example, A1 is connected to B1 to B4, A2 is connected to B1 to B4, and so on, until A5 is connected to B1 to B4. In this way, a target shared network is formed to ensure that data can be efficiently transmitted between write nodes and read nodes, thereby optimizing the overall performance and resource utilization of the network.

[0123] Step 205: In response to the joining command of the dynamic node, determine the first node type of the dynamic node, and obtain the transmission performance parameter of the dynamic node.

[0124] The first node type is used to characterize the type of the node forest to which the dynamic node joins.

[0125] In some embodiments of the present application, when a join command of a dynamic node is received, the first node type of the dynamic node is determined and its transmission performance parameters are obtained. The first node type is used to characterize the type of the node forest that the dynamic node will join, that is, whether the dynamic node joins the write node forest or the read node forest. Transmission performance parameters include bandwidth, delay, and data transmission rate, etc., which are used to evaluate the transmission performance of the dynamic node in the network. By determining the node type and obtaining the transmission performance parameters, necessary information support can be provided for the joining of the dynamic node to ensure the optimization and stability of the network topology.

[0126] For example, in a large-scale distributed system, suppose there is a new dynamic node X that needs to join the shared network. After receiving the joining command of node X, its first node type will be determined. For example, node X is determined to join the write node forest. Next, the transmission performance parameters of node X are obtained, such as bandwidth of 5Gbps, latency of 10ms, and data transmission rate of 500MB / s. By determining the type of node X and obtaining its transmission performance parameters, necessary information support can be provided for the joining of node X, ensuring that its transmission performance in the write node forest meets the requirements of the network, thereby optimizing the performance and stability of the overall network.

[0127] Step 206: according to the transmission performance parameter of the dynamic node, add the dynamic node to the node forest corresponding to the corresponding first node type to update the target shared network.

[0128] In some embodiments of the present application, the dynamic node will be added to the corresponding node forest according to the transmission performance parameters of the dynamic node to update the target shared network. The transmission performance parameters of the dynamic node include bandwidth, delay, and data transmission rate, etc. These parameters are used to evaluate the transmission capability of the dynamic node in the network. According to the first node type of the dynamic node (i.e., write node or read node), it is added to the corresponding write node forest or read node forest. In this way, it can be ensured that the addition of the dynamic node will not affect the overall performance and stability of the network, and the target shared network is updated to reflect the new node connection relationship.

[0129] For example, Figure 5 As shown in the figure, in a large-scale distributed system, suppose a new dynamic node needs to join the shared network. Then first obtain the node's transmission performance parameters, such as bandwidth of 8Gbps, latency of 7ms, and data transmission rate of 800MB / s. According to the first node type determined in the previous step, suppose node r x is determined to join the read node forest. Next, node r x Add it to the read node forest, for example, it can be used as a leaf node of a read tree, and adjust the corresponding connection relationship to ensure the efficiency and stability of data transmission. In this way, the target shared network is updated to include new dynamic nodes and ensure that the overall performance and stability of the network are optimized. Similarly, the node w x Join the write node forest.

[0130] Optionally, step 206 includes the following sub-steps:

[0131] Sub-step 2061, taking the tree with the least number of nodes in the node forest corresponding to the first node type corresponding to the dynamic node as the target tree, and searching for the target insertion port in the insertion ports of the target tree according to the transmission performance parameters of the dynamic node.

[0132] In some embodiments of the present application, when a dynamic node is added, it will be inserted into the tree with the least number of nodes. First, according to the first node type of the dynamic node (i.e., write node or read node), the tree with the least number of nodes is found in the corresponding node forest as the target tree. Then, according to the transmission performance parameters of the dynamic node, a suitable target insertion port is found in the insertion port of the target tree. The insertion port refers to the position in the target tree that can accommodate the new node. By selecting a suitable insertion port, it can be ensured that the addition of the dynamic node will not affect the overall performance and stability of the network.

[0133] For example, in a large-scale distributed system, suppose there is a new dynamic node M that needs to join the shared network. First, determine that the first node type of node M is a write node. Then, find the tree with the least number of nodes in the write node forest. Assume that the tree is write tree A, which currently contains 8 nodes. Next, according to the transmission performance parameters of node M (for example, the bandwidth is 6Gbps and the delay is 8ms), find a suitable target insertion port in write tree A. Assume that the transmission performance parameters of a node N of write tree A match those of node M, and node N has an idle insertion port. In this way, the target insertion port of node M is determined to prepare for its joining write tree A.

[0134] Optionally, each insertion port has a corresponding transmission performance threshold. In order to find a target insertion port in the insertion ports of the target tree according to the transmission performance parameter of the dynamic node, sub-step 2061 includes the following sub-steps:

[0135] Sub-step 20611, determining the second node type of each node in the target tree, and sorting the nodes in the target tree according to the second node type.

[0136] The second node type is used to represent the level of the node in the target tree in the corresponding node forest.

[0137] In some embodiments of the present application, the second node type of each node in the target tree is determined, and the nodes in the target tree are sorted according to the second node type. The second node type is used to characterize the hierarchical position of the node in the target tree in the corresponding node forest. By determining the hierarchical position of each node, the nodes in the target tree can be sorted, so as to more effectively find a suitable insertion port in subsequent steps.

[0138] For example, in a large-scale distributed system, assume that the target tree is a write tree C, which contains nodes C1, C2, C3, C4, and C5. First, determine the second node type of each node. For example, node C1 is the root node, and its second node type is 1; nodes C2 and C3 are second-layer nodes, and their second node type is 2; nodes C4 and C5 are third-layer nodes, and their second node type is 3. Next, sort the nodes in the target tree according to the second node type, from large to small, that is, the order is C1, C2, C3, C4, C5. In this way, the hierarchical position of each node in the target tree can be clarified, providing a basis for finding a suitable insertion port in subsequent steps.

[0139] Sub-step 20612, determining the matching of the transmission performance threshold corresponding to the insertion port and the transmission performance parameter of the dynamic node in descending order of the corresponding second node type, and determining the first insertion port characterized as matching as the target insertion port.

[0140] In some embodiments of the present application, the matching of the transmission performance threshold corresponding to the insertion port and the transmission performance parameter of the dynamic node is determined in descending order of the second node type, and the first insertion port characterized as matching is determined as the target insertion port. The transmission performance threshold refers to the maximum transmission performance parameter that each insertion port can support, such as bandwidth and delay. By comparing the transmission performance parameters of the dynamic node with the transmission performance threshold of the insertion port, the most suitable insertion port can be found to ensure that the addition of the dynamic node will not affect the overall performance and stability of the network.

[0141] For example, in a large-scale distributed system, assume that the target tree is a write tree D, which contains nodes D1, D2, D3, D4, and D5, which have been arranged in descending order according to the second node type. Assume that node D1 is the root node, and the transmission performance threshold of its insertion port is bandwidth 10Gbps and delay 5ms; nodes D2 and D3 are second-layer nodes, and the transmission performance threshold of their insertion ports is bandwidth 8Gbps and delay 7ms; nodes D4 and D5 are third-layer nodes, and the transmission performance threshold of their insertion ports is bandwidth 6Gbps and delay 10ms. Assume that there is a dynamic node E, whose transmission performance parameters are bandwidth 7Gbps and delay 8ms. According to the order of the second node type from large to small, the insertion port of node D5 is first checked, and it is found that its transmission performance threshold does not match. Then the insertion ports of nodes D2 and D3 are checked, and it is found that the transmission performance threshold of the insertion port of node D2 matches the transmission performance parameter of node E. Therefore, the insertion port of node D2 is determined as the target insertion port, and preparations are made for the addition of dynamic node E.

[0142] Sub-step 2062, when the target tree contains a target insertion port, insert the dynamic node into the target insertion port so that the dynamic node obtains the data variable from the parent node of the dynamic node.

[0143] In some embodiments of the present application, when the target tree includes a target insertion port, the dynamic node is inserted into the target insertion port so that the dynamic node can obtain data variables from its parent node. The target insertion port refers to a position in the target tree that can accommodate a new node. By inserting the dynamic node into the target insertion port, it can be ensured that the dynamic node can normally participate in data transmission and processing, thereby maintaining the overall performance and stability of the network.

[0144] For example, in a large-scale distributed system, suppose there is a new dynamic node M that needs to join the shared network, and the target insertion port has been determined in the previous step. Assume that the target insertion port is located on node N of the write tree A. Insert node M to the target insertion port of node N, making node M a child node of node N. In this way, node M can obtain data variables from its parent node N and participate in the data transmission and processing of the write tree A, thereby ensuring the overall performance and stability of the network.

[0145] Sub-step 2063 , when the target tree does not contain the target insertion port, update the node data of the shared network with the dynamic node and the corresponding transmission performance parameter, and return to step 201 .

[0146] In some embodiments of the present application, when the target tree does not contain a target insertion port, the node data of the shared network is updated with the dynamic node and the corresponding transmission performance parameters, and the process returns to step 201, i.e., a new target shared network is reestablished. The target insertion port refers to a position in the target tree that can accommodate a new node. If there is no suitable insertion port in the target tree, the node data of the shared network needs to be updated to reflect the joining requirements of the dynamic node, and the new network topology is obtained from step 201 again. This ensures that the network topology can dynamically adapt to the joining requirements of new nodes and maintain the overall performance and stability of the network.

[0147] For example, in a large-scale distributed system, it is assumed that a new dynamic node P needs to join the shared network, and the target tree has been determined to be the write tree B in sub-step 2061. However, there is no suitable target insertion port in the write tree B to accommodate node P. In this case, node P and its transmission performance parameters (e.g., bandwidth of 4 Gbps and delay of 12 ms) are updated to the node data of the shared network. Then, the process returns to step 201: the node data and target delay parameters of the shared network are reacquired to generate a new network topology to ensure that node P can be effectively added to the shared network, thereby maintaining the overall performance and stability of the network.

[0148] In the relevant embodiments of the present application, when a node joins, in addition to the need to reconnect to the network, the read node also needs to perform data recovery processing to restore the data to a global complete state. This process can rely on local storage, or combine data from other read nodes to restore data. That is, relying on the characteristics of CRDT, during data recovery, new incremental data generated by the network can still be directly merged with local data until the stock data is restored, and the data copy of the node can reach the latest state.

[0149] Step 207 : In response to the exit command of the target dynamic node in the target shared network, determine the third node type of the target dynamic node.

[0150] The third node type is used to represent the level of the target dynamic node in the corresponding node forest.

[0151] In some embodiments of the present application, when receiving the exit command of the target dynamic node, the third node type of the target dynamic node is determined. The third node type is used to characterize the level of the target dynamic node in the corresponding node forest, that is, the specific location of the target dynamic node in the write node forest or the read node forest. By determining the third node type, the location of the target dynamic node in the network topology can be clarified, thereby providing necessary information support for subsequent node exit operations, ensuring that the adjustment and update of the network topology can proceed smoothly.

[0152] For example, in a large-scale distributed system, suppose there is a dynamic node Z that needs to exit from the shared network. Then after receiving the exit command of node Z, its third node type will be determined first. For example, node Z is determined to be an intermediate node in the read node forest. By querying the network topology, the hierarchical position of node Z in the read node forest is confirmed, for example, it is located in the second layer of a certain read tree. By determining the third node type of node Z, necessary information support can be provided for the exit of node Z, ensuring that after node Z exits, the network topology can be adjusted and updated in time to maintain the overall performance and stability of the network.

[0153] Step 208: according to the third node type, withdraw the target dynamic node from the target shared network to update the target shared network.

[0154] In some embodiments of the present application, the target dynamic node will be withdrawn from the target shared network according to the third node type to update the target shared network. The third node type characterizes the hierarchical position of the target dynamic node in the corresponding node forest. By determining the hierarchical position of the target dynamic node, the network topology can be adjusted in a targeted manner to ensure that the overall performance and stability of the network are not affected after the node is withdrawn. The process of updating the target shared network includes reconfiguring the connection relationship between the nodes to adapt to the new topology after the node is withdrawn.

[0155] For example, in a large-scale distributed system, suppose that there is a dynamic node W that needs to exit from the shared network. First, based on the third node type determined in the previous step, assume that node W is determined to be an intermediate node in the write node forest. By querying the network topology, confirm the hierarchical position of the node in the write node forest, for example, it is located at the second to last layer of a write tree. Next, remove node W from the write node forest and reconfigure the connection relationship between its parent node and child nodes to ensure the continuity of data transmission and the stability of the network. In this way, the target shared network is updated so that it can adapt to the new topology after the node exits, ensuring that the overall performance and stability of the network are optimized.

[0156] Optionally, step 208 includes the following sub-steps:

[0157] Sub-step 2081 , when the third node type is characterized as a leaf node, withdraw the target dynamic node from the target shared network.

[0158] In some embodiments of the present application, when the third node type is characterized as a leaf node, the target dynamic node is withdrawn from the target shared network. A leaf node refers to a node that has no child nodes in a tree structure. Since a leaf node has no child nodes, its withdrawal will not affect the data transmission and network structure of other nodes. Therefore, the target dynamic node can be directly removed from the target shared network to update the network topology.

[0159] For example, Figure 5 As shown, in a large-scale distributed system, it is assumed that a dynamic node w3 needs to exit from the shared network. Then, first, according to the third node type determined in the previous step (assuming that the node has been determined as a leaf node in the read node forest). At this time, since node w3 is a leaf node and has no child nodes, its exit will not affect the data transmission and network structure of other nodes. Therefore, node w3 can be directly removed from the read node forest. In this way, the target shared network is updated so that it can adapt to the new topology after node w3 exits, ensuring the overall performance and stability of the network.

[0160] Sub-step 2082, when the third node type is characterized as not being a leaf node, determining a target child node from the child nodes of the target dynamic node to serve as a replacement node for the target dynamic node, and withdrawing the target dynamic node from the target shared network.

[0161] In some embodiments of the present application, when the third node type is characterized as not being a leaf node, a target child node is determined from the child nodes of the target dynamic node as a replacement node for the target dynamic node, and the target dynamic node is withdrawn from the target shared network. Non-leaf nodes refer to nodes with child nodes in a tree structure. In order to ensure the integrity of the network topology and the continuity of data transmission, it is necessary to select a target child node to replace the position of the target dynamic node. In this way, the overall performance and stability of the network can be maintained after the target dynamic node withdraws.

[0162] For example, Figure 6 As shown in Figure 1, in a large-scale distributed system, suppose there is a dynamic node w1 that needs to exit from the shared network. First, according to the third node type determined in the previous step (assuming that node w1 has been determined as a non-leaf node in the write node forest), and has two child nodes w 21 and w 22 In order to ensure the integrity of the network topology, one of the two child nodes will be selected as the target child node. Assume that the child node w is selected 22 Then, remove node w1 from the write node forest and replace its child node w 22 Promoted to the position of its parent node, that is, let w 22 The child nodes of w 21 As w 22 , and let w 22 As the child node of the original parent node w1, it takes over the connection relationship and data transmission tasks of node w1. In this way, the target shared network is updated to adapt to the new topology after the exit of node w1, ensuring the overall performance and stability of the network.

[0163] In some embodiments of the present application, node exit may require reorganization of the network topology to avoid affecting the subtree network. The specific method can refer to the process from step 101 to step 104, which will not be repeated here. At this time, in order to adapt to the changes in the network, the neighboring nodes need to make adjustments in time to maintain the normal operation of the system. In addition, data can be persistently backed up to speed up data recovery progress when restoring next time.

[0164] like Figure 7 As shown, the method proposed in the embodiment of the present application can be used by the program in the form of a third-party library, or the shared variable mechanism can be integrated into the syntax level of the programming language by modifying the lexical and grammatical structure of the programming language to provide native support at the programming language level. After integration into the programming language, the shared variable life cycle can be divided into the following stages:

[0165] S1: Initialization: Shared variables are passed parameters (variable type, variable ID, node ID) through code to create objects;

[0166] S2: Obtain metadata: Metadata can be obtained through local configuration files or other network configuration systems, including node metadata and variable metadata;

[0167] S3: Parameter setting: After obtaining the metadata, further parameter setting is performed on the initialization object;

[0168] S4: Local data recovery: If there is local data saved when you exited last time, you can recover the local data;

[0169] S5: Session setting: create a session about this variable and join the session management;

[0170] S6: Networking: Networking with neighboring nodes to confirm that the network connection is normal;

[0171] S7: Data synchronization: When a node is shut down, if the rest of the network is still operating, it will request other nodes to synchronize data until the data is updated to the latest global state.

[0172] S8: In operation: the stage of normal service provision;

[0173] S9: Exit preparation: When the node is ready to exit, a series of preparations are performed, including notification of neighbor nodes, link disconnection, data archiving and backup, etc.

[0174] S10: Destruction: After the preparation work is completed, it can be officially destroyed.

[0175] refer to Figure 8 This is a specific embodiment of the method of the present application applied to the Digital Object Architecture (DOA), in which the node management is implemented:

[0176] like Figure 8 As shown, by storing digital objects (DO) as shared variables, it is possible to store metadata in the identification resolution system of Shu Rui in the Digital Internet Open Source Warehouse (BDWare), so that the metadata of the variables and the metadata of the nodes are integrated, stored in a lightweight data exchange format (JavaScript Object Notation, json) structure, stored together in the identification resolution system, and assigned a unique identifier.

[0177] When the node is started, each variable is read and a shared variable (SharableVar) object is generated for each node. Considering that a physical node may logically assume both the roles of a read node and a write node, as mentioned above, there are differences between the variable copies on the read and write nodes. The read node has all global states, while the write node is not guaranteed to have complete global states. Therefore, the SharableVar object contains two CRDT member fields, writerVar and readerVar, which are actually two states of a shared variable on different logical nodes. WriterVar is used when a node writes to a variable. The feedback (effect) generated by it will be stored in the queue write variable change queue (writerVarDeltaQueue) and wait to be sent to its parent node in a centralized manner when the next data is transmitted. ReaderVar is used when a node reads a variable. When a variable receives an effect, it will be merged into readerVar and stored in the read variable change queue (readerVarDeltaQueue) to wait to be sent to its leaf node in a centralized manner when the next data is transmitted.

[0178] As Figure 8 A further embodiment of Fig. 9 As shown, it is a flowchart of a node receiving and processing digital object information provided by an embodiment of the present application:

[0179] Based on the BDWare data system, the communication protocol implemented in this article uses the Diagnostic over Internet Protocol (DOIP) protocol based on automotive Ethernet, and adds an extended operation to the original DOIP request type: 0.DOIP / Op.SyncVar is used for data synchronization related to shared variables.

[0180] The properties of the protocol information (DoipMessage) of this operation are as follows:

[0181] Variable ID (varId); the type of network to which the transmission message belongs (type); the message transmitted in the read forest network (r2r); the message transmitted in the write forest network (w2w); the message sender is the message transmitted by the read-write network broadcast layer (w2r); the transmission message payload (content), serialize the effect into a json string.

[0182] When the variable receives a message of type "r2r" or "w2r", it performs the following operations:

[0183] M11: The effect in the payload needs to be merged into readerVar;

[0184] M12: Add to the readerVarDeltaQueue queue;

[0185] When the variable receives a message of type "w2w", it performs the following operations:

[0186] M21: Merge the effect in the payload into writerVar;

[0187] M22 is added to the writerVarDeltaQueue queue.

[0188] As Figure 8 A further embodiment of Fig.10 As shown, it is a flowchart of the steps of the write node periodically sending DoipMessage provided by the embodiment of the present application:

[0189] After encapsulating DoipMessage, determine whether the node itself is the root node of the forest and perform the following operations:

[0190] N1: If the node itself is not the root node of the forest, it sends a message to the parent node;

[0191] N2: When the own node is the root node of the forest, broadcast the message to the reading forest nodes.

[0192] As Figure 8 A further embodiment of Fig.11 As shown, it is a flowchart of the steps of the reading node periodically sending DoipMessage provided by the embodiment of the present application:

[0193] L1: Broadcast messages to child nodes.

[0194] like Fig.12 As shown, based on the method of the embodiment of the present application, the present application also implements a CRDT variable network simulation experiment program suitable for large-scale scenarios:

[0195] The simulation node and its corresponding components implement the TickComponent interface method. The main program drives each component to change its state by continuously adjusting the clock. The description of each component is as follows:

[0196] Node simulation component (Node): contains a unique identifier, a network interface card (NetworkInterfaceCard), and a simulated data processor (DataProcessor):

[0197] DataProcessor is used to generate data and process received data, and contains a buffer queue for temporarily storing data to be sent;

[0198] NetworkInterfaceCard, used to simulate network management and record neighbor node information;

[0199] In addition, the above structure also includes simulated bandwidth (Bandwidth), which is used for data upload and download, and contains a buffer queue to temporarily store data, and a simulated transmission process component (Transmission), which is used to simulate a transmission process, including sender node information and receiver node information, as well as data payload, and simulated data packet (DataPacket), which stores CRDT variables and can perform merge operations.

[0200] Specifically, the simulation process is as follows:

[0201] K1: cluster node initialization;

[0202] K2: Network topology networking;

[0203] K3: Analog data transmission;

[0204] K4: Data analysis report output;

[0205] The key process of the program is to simulate data transmission. The main process maintains a global clock, and each component needs to implement several interfaces contained in TickComponent to complete the work of starting, during, and ending each clock cycle. The life cycle of a data packet is as follows:

[0206] K31: DataProcessor generates data at fixed intervals and adds it to the buffer queue;

[0207] K32: DataProcessor packages and merges the data in the buffer queue according to a certain rule at regular intervals and sends it to NetworkInterfaceCard;

[0208] K33: NetworkInterfaceCard packages the data into a Transmission object, assembles the target node information, and adds it to the Bandwidth buffer queue;

[0209] K34: Bandwidth allocates a fixed traffic limit in each cycle, transmits tasks in the buffer queue to the target node, and completes a certain amount of data transmission in each clock cycle;

[0210] K35: After the data transmission is completed, the target node unpacks the data packet and adds it to the buffer queue of the DataProcessor, waiting for the next round of transmission;

[0211] K36: The entire life cycle is completed until the data packet is no longer propagated by the node.

[0212] In order to support the simulation of large-scale clusters, relying solely on the memory of a single machine cannot support the simulation of all nodes. Therefore, in the specific code implementation, all of the above components are persisted using the embedded database RocksDB, and GuavaCache is used to implement the memory buffer, which is responsible for swapping data in and out between memory and hard disk.

[0213] In summary, in the embodiment of the present application, by obtaining the transmission performance parameters and target delay parameters of each node in the shared network, the data transmission between any write node and any read node in the network topology structure generated by the solution of the present application is utilized to ensure that the data transmission from the child-level node to the parent-level node in the write node forest, and the unidirectional transmission from the parent-level node to the child-level node in the read node forest, so that the generated network topology structure can meet the connection performance and delay requirements of each node in the network, and then evaluate the traffic of different network topologies to further select the network structure with the best utilization of network resources, and further reduce the network communication overhead. So that the distributed shared variables can be adapted to a larger number of server clusters. Therefore, based on the method of the embodiment of the present application, an efficient shared variable mechanism for large-scale scenarios is realized, which solves the problem that the distributed shared variable solution in the prior art can only be applied to small and medium-sized server clusters.

[0214] refer to Fig.13 , which shows a shared variable device 30 for large-scale scenarios provided by an embodiment of the present application, including:

[0215] The data acquisition module 301 is used to acquire the node data of the shared network and the target delay parameter of the shared network; the node data includes the transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameter is used to constrain the longest time interval from when the data variable is written into any write node of the shared network to when the data variable is output to any read node of the shared network;

[0216] A topology generation module 302 is used to generate multiple network topology structures of the shared network according to the transmission performance parameter and the target delay parameter; the network topology structure includes a write node forest composed of write nodes in the network, and a read node forest composed of read nodes in the network; the write node forest includes at least one write tree, and the read node forest includes at least one read tree; the root write node of each write tree is connected to the root read node of each read tree one by one;

[0217] A topology selection module 303 is used to determine the flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum flow evaluation value as the target network topology structure;

[0218] The network adjustment module 304 is used to adjust the connection relationship between the nodes in the shared network according to the target network topology structure to obtain the target shared network.

[0219] Optionally, the transmission performance parameter includes a maximum number of connections of a node, and the topology generation module 302 includes:

[0220] A write forest generation submodule is used to generate multiple write node forests by taking the node connectivity of the write tree as the first traversal variable parameter and the maximum number of connections as the maximum value of the first traversal parameter; the first traversal variable parameter is used to characterize the node degree of the write tree in the write node forest;

[0221] The read forest generation submodule is used to use the node connectivity of the read tree as the second traversal variable parameter and the maximum number of connections as the maximum value of the second traversal parameter to generate multiple read node forests; the second traversal variable parameter is used to characterize the node degree of the read tree in the read node forest;

[0222] A combining submodule, used for connecting each write node forest with each read node forest respectively to obtain multiple candidate topology structures, and testing the maximum transmission delay of each candidate topology structure;

[0223] The exclusion submodule is used to delete the alternative topology structures corresponding to the maximum transmission delay greater than the target delay parameter among the alternative topology structures, and use the remaining alternative topology structures as the network topology structures.

[0224] Optionally, the topology selection module 303 includes:

[0225] A partial evaluation submodule, for determining a write traffic evaluation value, a read traffic evaluation value, and a broadcast traffic evaluation value of a network topology structure according to a transmission performance parameter and a target delay parameter;

[0226] The comprehensive evaluation submodule is used to use the sum of the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value as the traffic evaluation value of the network topology structure.

[0227] Optionally, the network adjustment module 304 includes:

[0228] A forest adjustment submodule, configured to adjust the connection relationship of the write nodes in the shared network to a first target topology structure to generate a target write node forest, and to adjust the connection relationship of the read nodes in the shared network to a second target topology structure to generate a target read node forest; the first target topology structure is a subgraph of the write node forest of the target network topology structure, and the second target topology structure is a subgraph of the read node forest of the target network topology structure;

[0229] The root connection submodule is used to connect all the root write nodes of the target write node forest with all the root read nodes of the target read node forest one by one to obtain a target shared network.

[0230] Optionally, the shared variable device 30 for large-scale scenarios further includes:

[0231] A node acquisition module, configured to determine a first node type of the dynamic node in response to a join command of the dynamic node, and to acquire a transmission performance parameter of the dynamic node; the first node type is used to characterize a type of the node forest to which the dynamic node joins;

[0232] The node adding module is used to add the dynamic node to the node forest corresponding to the corresponding first node type according to the transmission performance parameters of the dynamic node, so as to update the target shared network.

[0233] Optionally, the node joining module includes:

[0234] A port search submodule, used to take a tree with the least number of nodes in a node forest corresponding to a first node type corresponding to a dynamic node as a target tree, and to search for a target insertion port in the insertion ports of the target tree according to a transmission performance parameter of the dynamic node;

[0235] A node insertion submodule, used for inserting a dynamic node into a target insertion port when the target tree contains the target insertion port, so that the dynamic node obtains a data variable from a parent node of the dynamic node;

[0236] The network reconstruction submodule is used to update the node data of the shared network with the dynamic nodes and the corresponding transmission performance parameters when the target tree does not contain the target insertion port, and return to the step of obtaining the node data of the shared network and the target delay parameters of the shared network.

[0237] Optionally, each insertion port has a corresponding transmission performance threshold, and the port search submodule includes:

[0238] A port lookup unit, used to determine a second node type of each node in the target tree, and sort the nodes in the target tree according to the second node type; the second node type is used to characterize the level of the node in the target tree in the corresponding node forest;

[0239] The port selection unit is used to determine the matching of the transmission performance threshold corresponding to the insertion port and the transmission performance parameter of the dynamic node in descending order of the corresponding second node type, and determine the first insertion port characterized as matching as the target insertion port.

[0240] Optionally, the shared variable device 30 for large-scale scenarios further includes:

[0241] A node selection module, configured to determine a third node type of the target dynamic node in response to an exit command of the target dynamic node in the target shared network; the third node type is used to characterize the level of the target dynamic node in the corresponding node forest;

[0242] The node exit module is used to exit the target dynamic node from the target shared network according to the third node type, so as to update the target shared network.

[0243] Optionally, the node exit module includes:

[0244] A leaf node exit submodule, used for exiting the target dynamic node from the target shared network when the third node type is characterized as a leaf node;

[0245] The other node exit submodule is used to determine a target child node from the child nodes of the target dynamic node as a replacement node for the target dynamic node and exit the target dynamic node from the target shared network when the third node type is characterized as not being a leaf node.

[0246] In summary, in the embodiment of the present application, by obtaining the transmission performance parameters and target delay parameters of each node in the shared network, the data transmission between any write node and any read node in the network topology structure generated by the solution of the present application is utilized to ensure that the data transmission from the child-level node to the parent-level node in the write node forest, and the unidirectional transmission from the parent-level node to the child-level node in the read node forest, so that the generated network topology structure can meet the connection performance and delay requirements of each node in the network, and then evaluate the traffic of different network topologies to further select the network structure with the best utilization of network resources, and further reduce the network communication overhead. So that the distributed shared variables can be adapted to a larger number of server clusters. Therefore, based on the method of the embodiment of the present application, an efficient shared variable mechanism for large-scale scenarios is realized, which solves the problem that the distributed shared variable solution in the prior art can only be applied to small and medium-sized server clusters.

[0247] Reference Fig.14, the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0248] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0249] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0250] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0251] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0252] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0253] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0254] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0255] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0256] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0257] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0258] Fig.15 600 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. Fig.15 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0259] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0260] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0261] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A shared variable method for large-scale scenarios, characterized in that: include: Acquire node data of a shared network and a target delay parameter of the shared network; the node data includes transmission performance parameters of each node in the shared network; The transmission performance parameter is used to characterize the data transmission performance when the corresponding node is connected to other nodes; the target delay parameter is used to constrain the longest time interval from when a data variable is written into any write node of the shared network to when the data variable is output to any read node of the shared network; Generating a plurality of network topology structures of the shared network according to the transmission performance parameter and the target delay parameter; the network topology structures comprising a write node forest formed by write nodes in the network and a read node forest formed by read nodes in the network; The write node forest includes at least one write tree, and the read node forest includes at least one read tree; the root write node of each write tree is connected one-to-one with the root read node of each read tree; Determine the traffic evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum traffic evaluation value as the target network topology structure; The connection relationship between the nodes in the shared network is adjusted according to the target network topology structure to obtain a target shared network.

2. The method according to claim 1, characterized in that The transmission performance parameter includes the maximum number of connections of the node, and the generating of multiple network topology structures of the shared network according to the transmission performance parameter and the target delay parameter includes: The node connectivity of the write tree is used as a first traversal variable parameter, and the maximum number of connections is used as a maximum value of the first traversal parameter to generate multiple write node forests; the first traversal variable parameter is used to characterize the node degree of the write tree in the write node forest; The node connectivity of the read tree is used as a second traversal variable parameter, and the maximum number of connections is used as the maximum value of the second traversal parameter to generate multiple read node forests; the second traversal variable parameter is used to characterize the node degree of the read tree in the read node forest; Connecting each of the write node forests to each of the read node forests to obtain a plurality of candidate topologies, and testing a maximum transmission delay of each of the candidate topologies; The alternative topology structures corresponding to the maximum transmission delays in the alternative topology structures that are greater than the target delay parameter are deleted, and the remaining alternative topology structures are used as the network topology structures.

3. The method according to claim 1, characterized in that The determining, according to the transmission performance parameter and the target delay parameter, a flow evaluation value of the network topology structure comprises: Determine, according to the transmission performance parameter and the target delay parameter, a write traffic evaluation value, a read traffic evaluation value, and a broadcast traffic evaluation value of the network topology structure; The sum of the write traffic evaluation value, the read traffic evaluation value, and the broadcast traffic evaluation value is used as the traffic evaluation value of the network topology structure.

4. The method according to claim 1, characterized in that The method further comprises: In response to a join command of a dynamic node, determining a first node type of the dynamic node and acquiring a transmission performance parameter of the dynamic node; the first node type is used to characterize a type of a node forest to which the dynamic node joins; According to the transmission performance parameter of the dynamic node, the dynamic node is added to the node forest corresponding to the corresponding first node type to update the target shared network.

5. The method according to claim 4, characterized in that Each of the insertion ports has a corresponding transmission performance threshold, and searching for a target insertion port in the insertion ports of the target tree according to the transmission performance parameter of the dynamic node includes: Determine a second node type of each of the nodes in the target tree, and sort the nodes in the target tree according to the second node type; the second node type is used to characterize the level of the node in the target tree in the corresponding node forest; According to the order of the corresponding second node types from large to small, the matching of the transmission performance threshold corresponding to the insertion port and the transmission performance parameter of the dynamic node is determined, and the first insertion port characterized as matching is determined as the target insertion port.

6. The method according to claim 1, characterized in that The method further comprises: In response to an exit command for a target dynamic node in the target shared network, determining a third node type of the target dynamic node; the third node type is used to characterize a level of the target dynamic node in a corresponding node forest; According to the third node type, the target dynamic node is withdrawn from the target shared network to update the target shared network.

7. The method according to claim 6, characterized in that The step of withdrawing the target dynamic node from the target shared network according to the third node type to update the target shared network includes: In a case where the third node type is characterized as a leaf node, withdrawing the target dynamic node from the target shared network; In the case that the third node type is characterized as not being a leaf node, a target child node is determined from the child nodes of the target dynamic node to serve as a replacement node for the target dynamic node, and the target dynamic node is withdrawn from the target shared network.

8. A shared variable device for large-scale scenarios, characterized in that: include: A data acquisition module, used to acquire node data of a shared network and a target delay parameter of the shared network; The node data includes transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance of the corresponding node when connected to other nodes; the target delay parameter is used to constrain the longest time interval from when a data variable is written into any write node of the shared network to when the data variable is output to any read node of the shared network; A topology generation module, configured to generate a plurality of network topology structures of the shared network according to the transmission performance parameter and the target delay parameter; the network topology structures comprising a write node forest composed of write nodes in the network, and a read node forest composed of read nodes in the network; The write node forest includes at least one write tree, and the read node forest includes at least one read tree; the root write node of each write tree is connected one-to-one with the root read node of each read tree; A topology selection module, configured to determine a flow evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determine the network topology structure corresponding to the minimum flow evaluation value as the target network topology structure; The network adjustment module is used to adjust the connection relationship between the nodes in the shared network according to the target network topology structure to obtain the target shared network.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images