Shared variable method, device and equipment for large-scale scenarios and storage medium

Abstract

The application discloses a shared variable method and device for large-scale scenarios, equipment and a storage medium, and belongs to the distributed field, and comprises the following steps: acquiring node data of a shared network and a target delay parameter of the shared network; generating a plurality of network topology structures of the shared network according to a transmission performance parameter and the target delay parameter; determining a traffic evaluation value of the network topology structure according to the transmission performance parameter and the target delay parameter, and determining a network topology structure corresponding to the smallest traffic evaluation value as a target network topology structure; and adjusting the connection relationship between nodes in the shared network according to the target network topology structure. Through the network topology structure generated by the scheme, data transmission between write nodes and read nodes will only be transmitted from a child level to a parent level in a write node forest, and will only be transmitted from the parent level to the child level in a read node forest, thereby solving the problem that the shared variable mechanism can only be applied to a small and medium-sized server cluster in the prior art.

Description

Technical Field

[0001] This application belongs to the field of distributed systems, specifically relating to a method, apparatus, device, and storage medium for sharing variables in large-scale scenarios. Background Technology

[0002] In modern distributed computing environments, multiple computing nodes need to share data to collaborate. Shared variables are special variables that allow multiple nodes to access and modify them simultaneously, ensuring data consistency, reliability, and concurrency. Shared variables play a crucial role in distributed computing, distributed storage, and distributed databases, supporting data sharing and collaboration among multiple active nodes. However, as the number of nodes increases, maintaining the consistency and timeliness of shared variables becomes more complex, especially in large-scale cluster environments.

[0003] Currently, commonly used shared variable mechanisms in distributed systems primarily rely on strong consistency algorithms. These algorithms ensure strong data consistency within the system through complex mechanisms. To achieve shared variables in large-scale scenarios, CRDT (Conflict-free Replicated Data Type) has been introduced. CRDT guarantees strong eventual consistency of data through its merge and exchange properties, avoiding conflicts during concurrent modifications and network propagation.

[0004] However, in large-scale clusters, frequent network communication and data synchronization are required to adapt to large-scale scenarios, which leads to a significant increase in resource consumption and affects the overall performance of the system. Summary of the Invention

[0005] This application aims to provide a method, apparatus, electronic device, and computer-readable storage medium for sharing variables in large-scale scenarios, at least to solve the problem that distributed shared variables can only be applied to small and medium-sized server clusters.

[0006] In a first aspect, embodiments of this application disclose a shared variable method for large-scale scenarios, including:

[0007] Obtain node data of the shared network and target latency parameters of the shared network; the node data includes transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes; the target latency parameters are used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network.

[0008] Based on the transmission performance parameters and the target latency parameters, multiple network topologies of the shared network are generated; the network topologies include a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network; the write node forest contains at least one write tree and the read node forest contains at least one read tree; the root write node of each write tree is connected to the root read node of each read tree one by one.

[0009] Based on the transmission performance parameters and the target delay parameters, the traffic evaluation value of the network topology is determined, and the network topology corresponding to the minimum traffic evaluation value is determined as the target network topology.

[0010] Adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

[0011] Secondly, embodiments of this application also disclose a shared variable device for large-scale scenarios, comprising:

[0012] The data acquisition module is used to acquire node data of the shared network and the target latency parameter of the shared network; the node data includes the transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes; the target latency parameter is used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network.

[0013] A topology generation module is used to generate multiple network topologies of the shared network based on the transmission performance parameters and the target delay parameters. The network topologies include a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network. The write node forest contains at least one write tree, and the read node forest contains at least one read tree. The root write node of each write tree is connected to the root read node of each read tree.

[0014] The topology selection module is used to determine the traffic evaluation value of the network topology based on the transmission performance parameters and the target delay parameters, and to determine the network topology corresponding to the minimum traffic evaluation value as the target network topology.

[0015] The network adjustment module is used to adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

[0016] Thirdly, embodiments of this application also disclose an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0017] Fourthly, embodiments of this application also disclose a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0018] In summary, in this embodiment, by obtaining the transmission performance parameters and target latency parameters of each node in the shared network, and utilizing the unidirectional transmission property of the network topology generated by the scheme of this application—that data transmission between any write node and any read node is guaranteed to be from child-level nodes to parent-level nodes in the write node forest and from parent-level nodes to child-level nodes in the read node forest—the generated network topology can meet the connection performance and latency requirements of each node in the network. Then, by evaluating the traffic of different network topologies, the network structure with the optimal utilization of network resources is further selected, further reducing network communication overhead. This allows distributed shared variables to be adapted to larger-scale server clusters. Therefore, the method based on this embodiment achieves an efficient shared variable mechanism for large-scale scenarios, solving the problem that existing distributed shared variable schemes can only be applied to small- to medium-sized server clusters. Attached Figure Description

[0019] In the attached diagram:

[0020] Figure 1 This is a flowchart illustrating the steps of a shared variable method for large-scale scenarios provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the network topology provided in the embodiments of this application;

[0022] Figure 3 This is a flowchart illustrating the steps of another shared variable method for large-scale scenarios provided in this application embodiment;

[0023] Figure 4 This is a schematic diagram illustrating the principle of the data expansion rate of nodes in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram illustrating the method of adding nodes in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the method for exiting a node in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram illustrating the lifecycle evolution process of shared variables in the embodiments of this application;

[0027] Figure 8 This is a data structure diagram illustrating the application of the method in this application to a digital object architecture;

[0028] Figure 9 This is a flowchart illustrating how a node receives and processes digital object information, as provided in an embodiment of this application.

[0029] Figure 10 This is a flowchart illustrating the steps of a write node periodically sending protocol information, as provided in an embodiment of this application.

[0030] Figure 11 This is a flowchart illustrating the steps of a read node periodically sending protocol information, as provided in an embodiment of this application.

[0031] Figure 12 This is a flowchart of the CRDT variable network simulation experiment program obtained based on the method provided in the embodiments of this application;

[0032] Figure 13 This is a block diagram of a shared variable device for large-scale scenarios provided in an embodiment of this application;

[0033] Figure 14 This is a block diagram of an electronic device provided in one embodiment of this application;

[0034] Figure 15 This is a block diagram of an electronic device according to another embodiment of the present application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] Figure 1 This embodiment provides a shared variable method for large-scale scenarios.

[0038] The method may include the following steps:

[0039] Step 101: Obtain the node data of the shared network and the target latency parameters of the shared network.

[0040] The node data includes transmission performance parameters of each node in the shared network. The transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes. The target delay parameter is used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network.

[0041] In some embodiments of this application, node data and target latency parameters of the shared network will be acquired. Node data includes transmission performance parameters of each node in the shared network, which characterize the data transmission performance when a node connects to other nodes. The target latency parameter constrains the longest time interval from when a data variable is written to any write node in the shared network to when the data variable is output to any read node in the shared network. Acquiring this data provides fundamental information for subsequently generating the network topology and optimizing network performance.

[0042] For example, in a large-scale distributed system, suppose there is a shared network containing 100 write nodes and 200 read nodes. First, obtain the transmission performance parameters for each node, such as bandwidth, latency, and maximum number of connections. Assume node A has the following transmission performance parameters: bandwidth 10Gbps, latency 5ms, and maximum number of connections 3. Then, set the target latency parameter to 100ms, meaning the longest time interval between writing data to any write node and outputting data to any read node is 100ms. By collecting this information, necessary data support can be provided for generating and optimizing the network topology in subsequent steps.

[0043] Step 102: Generate multiple network topologies for the shared network based on transmission performance parameters and target delay parameters.

[0044] Among them, such as Figure 2As shown in the diagram, circular nodes represent read nodes, triangular nodes represent write nodes, and arrows between nodes indicate the direction of data transmission. The network topology includes a forest of write nodes {W} consisting of write nodes in the network, and a forest of read nodes {R} consisting of read nodes in the network. The forest of write nodes {W} contains at least one write tree W, and the forest of read nodes {R} contains at least one read tree R. The root write node w1 of each write tree is connected one-to-one with the root read node r1 of each read tree.

[0045] In some embodiments of this application, multiple network topologies of the shared network are generated based on transmission performance parameters and target latency parameters. The network topologies include a write node forest composed of write nodes and a read node forest composed of read nodes. Each write node forest contains at least one write tree, and each read node forest contains at least one read tree. The root write node of each write tree is connected one-to-one with the root read node of each read tree. By generating multiple network topologies, the performance of different topologies in terms of transmission performance and latency can be evaluated, thereby selecting the optimal network structure.

[0046] For example, in a large-scale distributed system, suppose there are 100 write nodes and 200 read nodes. Based on the transmission performance parameters and target latency parameters obtained in previous steps, multiple network topologies will be generated. For instance, a topology might be generated where the write node forest contains 10 write trees, each with 10 write nodes; and the read node forest contains 20 read trees, each with 10 read nodes. The root write node of each write tree is connected one-to-one with the root read node of each read tree. By generating such network topologies, their performance in terms of data transmission performance and latency can be further evaluated to select the optimal network structure.

[0047] Step 103: Based on the transmission performance parameters and the target delay parameters, determine the traffic evaluation value of the network topology, and determine the network topology corresponding to the minimum traffic evaluation value as the target network topology.

[0048] In some embodiments of this application, a traffic evaluation value for each network topology is determined based on transmission performance parameters and target delay parameters. The traffic evaluation value measures the efficiency and overhead of the network topology during data transmission. By calculating the traffic evaluation value for each network topology, the performance of different topologies can be compared, and the network topology with the lowest traffic evaluation value can be selected as the target network topology. This ensures that network resource utilization is optimized and network communication overhead is reduced while meeting the target delay parameters.

[0049] For example, in a large-scale distributed system, suppose three different network topologies are generated. Based on the transmission performance parameters and target latency parameters obtained in previous steps, the traffic assessment value for each topology can be calculated. For example, the traffic assessment value for topology A is 500GB, for topology B it is 450GB, and for topology C it is 480GB. By comparing these three traffic assessment values, topology B is determined to have the lowest traffic assessment value, and therefore, topology B is selected as the target network topology. This maximizes the utilization of network resources while ensuring data transmission performance and latency requirements are met.

[0050] Step 104: Adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

[0051] In some embodiments of this application, the connections between nodes in the shared network are adjusted according to a determined target network topology to obtain the target shared network. By adjusting the connections between nodes, it can be ensured that the network topology meets the requirements of transmission performance parameters and target delay parameters, thereby optimizing the utilization of network resources, reducing communication overhead, and improving the efficiency and reliability of data transmission.

[0052] For example, in a large-scale distributed system, suppose the target network topology is determined to contain 10 write trees and 20 read trees. Based on this target network topology, the connections between nodes in the shared network are adjusted. For instance, write nodes A, B, C, D, and E are designated as the root nodes of five write trees, and other write nodes are connected to these root nodes according to their hierarchical relationships. Similarly, read nodes F, G, H, I, and J are designated as the root nodes of five read trees, and other read nodes are connected to these root nodes according to their hierarchical relationships. Finally, ensure that the root write node of each write tree is connected to the root read node of each read tree. Through this adjustment, a shared network conforming to the target network topology is obtained, thereby optimizing data transmission performance and network resource utilization.

[0053] In summary, in this embodiment, by obtaining the transmission performance parameters and target latency parameters of each node in the shared network, and utilizing the unidirectional transmission property of the network topology generated by the scheme of this application—that data transmission between any write node and any read node is guaranteed to be from child-level nodes to parent-level nodes in the write node forest and from parent-level nodes to child-level nodes in the read node forest—the generated network topology can meet the connection performance and latency requirements of each node in the network. Then, by evaluating the traffic of different network topologies, the network structure with the optimal utilization of network resources is further selected, further reducing network communication overhead. This allows distributed shared variables to be adapted to larger-scale server clusters. Therefore, the method based on this embodiment achieves an efficient shared variable mechanism for large-scale scenarios, solving the problem that existing distributed shared variable schemes can only be applied to small- to medium-sized server clusters.

[0054] Figure 3 Another shared variable method for large-scale scenarios provided in this application embodiment may include the following steps:

[0055] Step 201: Obtain the node data of the shared network and the target latency parameters of the shared network.

[0056] The node data includes transmission performance parameters of each node in the shared network. The transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes. The target delay parameter is used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network.

[0057] The method shown in this step has been explained in step 101 and will not be repeated here.

[0058] Step 202: Generate multiple network topologies for the shared network based on transmission performance parameters and target delay parameters.

[0059] The network topology includes a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network; the write node forest contains at least one write tree and the read node forest contains at least one read tree; the root write node of each write tree is connected to the root read node of each read tree one by one.

[0060] The method shown in this step has been explained in step 102 and will not be repeated here.

[0061] Optionally, the transmission performance parameters include the maximum number of connections for the node, and step 202 includes the following sub-steps:

[0062] In sub-step 2021, the node connectivity of the written tree is used as the first traversal variable parameter, and the maximum number of connections is used as the maximum value of the first traversal parameter to generate multiple write node forests.

[0063] The first traversal variable parameter is used to characterize the degree of the node in the write node forest that is written to the tree.

[0064] In some embodiments of this application, the node connectivity of the write tree is used as a variable parameter for the first traversal, and the maximum number of connections is used as the maximum value of the first traversal parameter, thereby generating multiple write node forests. Node connectivity refers to the number of other nodes that a node can directly connect to. By traversing different node connectivity, multiple write node forests with different structures can be generated so that their performance and latency can be evaluated in subsequent steps.

[0065] For example, in a large-scale distributed system, suppose there are 100 write nodes and the maximum number of connections per node is 10. First, the node connectivity of the write trees is used as a variable parameter for the first traversal, ranging from 1 to 10. Then, multiple write node forests are generated. For example, when the node connectivity is 3, the generated write node forest might contain several write trees, each with a node degree of 3. Similarly, when the node connectivity is 5, the generated write node forest might contain several write trees, each with a node degree of 5. By traversing different node connectivity levels, multiple write node forests are generated, providing a basis for evaluating their performance and latency in subsequent steps.

[0066] In sub-step 2022, the node connectivity of the read tree is used as the variable parameter for the second traversal, and the maximum number of connections is used as the maximum value of the second traversal parameter to generate multiple read node forests.

[0067] The second traversal variable parameter is used to characterize the degree of the nodes in the read node forest.

[0068] In some embodiments of this application, the node connectivity of the read tree is used as a variable parameter for the second traversal, and the maximum number of connections is used as the maximum value of the second traversal parameter, thereby generating multiple read node forests. Node connectivity refers to the number of other nodes that a node can directly connect to. By traversing different node connectivity, multiple read node forests with different structures can be generated so that their performance and latency can be evaluated in subsequent steps.

[0069] For example, in a large-scale distributed system, suppose there are 200 read nodes, and the maximum number of connections per node is 8. First, the node connectivity of the read trees is used as a variable parameter in the second traversal, ranging from 1 to 8. Then, multiple read node forests are generated. For example, when the node connectivity is 4, the generated read node forest may contain several read trees, each with a node degree of 4. Similarly, when the node connectivity is 6, the generated read node forest may contain several read trees, each with a node degree of 6. By traversing different node connectivity levels, multiple read node forests are generated, providing a basis for evaluating their performance and latency in subsequent steps.

[0070] Sub-step 2023 connects each write node forest to each read node forest to obtain multiple alternative topologies and tests the maximum transmission latency of each alternative topology.

[0071] In some embodiments of this application, each write node forest is connected to each read node forest to obtain multiple alternative topologies, and the maximum transmission latency of each alternative topology is tested. By combining different write node forests and read node forests, multiple alternative network topologies can be generated. Then, the maximum transmission latency of each alternative topology is tested to evaluate its performance during data transmission. The maximum transmission latency refers to the longest time required from when a data variable is written to any write node to when the data variable is output to any read node.

[0072] For example, in a large-scale distributed system, suppose three write node forests A, B, and C, and two read node forests X and Y have been generated. First, each write node forest is connected to each read node forest, generating six candidate topologies. For example, writing node forest A is connected to read node forest X, generating candidate topology AX; writing node forest A is connected to read node forest Y, generating candidate topology AY, and so on. Next, the maximum transmission latency of each candidate topology is tested. For example, the maximum transmission latency of candidate topology AX is 90ms, and the maximum transmission latency of candidate topology AY is 110ms. By testing the maximum transmission latency of these candidate topologies, their performance during data transmission can be evaluated, providing a basis for selecting the optimal network topology in subsequent steps.

[0073] Sub-step 2024: Delete the candidate topology structure corresponding to the maximum transmission delay that is greater than the target delay parameter, and use the remaining candidate topology structure as the network topology structure.

[0074] In some embodiments of this application, candidate topologies with a maximum transmission delay greater than the target delay parameter are deleted, and the remaining candidate topologies are used as the network topology. The target delay parameter refers to the longest time interval from when a data variable is written to any write node to when the data variable is output to any read node. By deleting candidate topologies that do not meet the target delay parameter, it can be ensured that the final selected network topology meets the requirements in terms of data transmission performance and latency, thereby optimizing the utilization of network resources and communication efficiency.

[0075] For example, in a large-scale distributed system, suppose the target latency parameter is 100ms. Six candidate topologies have been generated through previous steps, and the maximum transmission latency of each candidate topology has been tested. For example, the maximum transmission latency of candidate topologies AX is 90ms, AY is 110ms, BX is 85ms, BY is 105ms, CX is 120ms, and CY is 100ms. Based on the target latency parameter, candidate topologies AY, BY, and CX with a maximum transmission latency greater than 100ms are removed. The remaining candidate topologies AX, BX, and CY are selected as the network topologies that meet the target latency parameter. In this way, it is ensured that the final selected network topology meets the requirements in terms of data transmission performance and latency, thereby optimizing network resource utilization and communication efficiency.

[0076] Step 203: Based on the transmission performance parameters and the target delay parameters, determine the traffic evaluation value of the network topology, and determine the network topology corresponding to the minimum traffic evaluation value as the target network topology.

[0077] The method shown in this step has been explained in step 103 and will not be repeated here.

[0078] Optionally, in order to determine the traffic assessment value of the network topology based on the transmission performance parameters and the target delay parameters, step 203 includes the following sub-steps:

[0079] Sub-step 2031: Based on the transmission performance parameters and the target delay parameters, determine the write traffic evaluation value, read traffic evaluation value, and broadcast traffic evaluation value of the network topology, respectively.

[0080] In some embodiments of this application, write traffic assessment values, read traffic assessment values, and broadcast traffic assessment values ​​for the network topology are determined based on transmission performance parameters and target delay parameters, respectively. The write traffic assessment value refers to the total amount of data transmitted in the write node forest, the read traffic assessment value refers to the total amount of data transmitted in the read node forest, and the broadcast traffic assessment value refers to the total amount of data transmitted between the write node forest and the read node forest. By calculating these traffic assessment values ​​separately, the performance and overhead of the network topology during data transmission can be comprehensively evaluated.

[0081] For example, in a large-scale distributed system, suppose there is a network topology AX containing 100 write nodes and 200 read nodes. Based on transmission performance parameters and target latency parameters, the write traffic assessment is first calculated. For example, assuming each write node generates 10MB of data per unit time, the total data transmission volume of the write node forest is 1000MB. Next, the read traffic assessment is calculated. For example, assuming each read node receives 5MB of data per unit time, the total data transmission volume of the read node forest is 1000MB. Finally, the broadcast traffic assessment is calculated. For example, assuming the data transmission volume between the write node forest and the read node forest is 500MB. By calculating these traffic assessments separately, the performance and overhead of the network topology during data transmission can be comprehensively evaluated, providing a basis for selecting the optimal network topology in subsequent steps.

[0082] Sub-step 2032 uses the sum of the write traffic assessment value, the read traffic assessment value, and the broadcast traffic assessment value as the traffic assessment value for the network topology.

[0083] In some embodiments of this application, the sum of write traffic assessment, read traffic assessment, and broadcast traffic assessment is used as the traffic assessment value for the network topology. By adding these three traffic assessment values, a comprehensive traffic assessment value can be obtained, which measures the total overhead of the entire network topology during data transmission. This comprehensive traffic assessment value can help compare different network topologies and select the optimal structure to optimize network resource utilization and communication efficiency.

[0084] For example, in a large-scale distributed system, suppose the estimated write traffic, estimated read traffic, and estimated broadcast traffic for a certain network topology AX have been calculated separately. For instance, the estimated write traffic is 1000MB, the estimated read traffic is 1200MB, and the estimated broadcast traffic is 500MB. Adding these three estimates together yields a comprehensive estimated traffic value of 2700MB for this network topology. This method comprehensively measures the total overhead of the network topology during data transmission, providing a basis for selecting the optimal network topology in subsequent steps.

[0085] As shown in the figure, in the embodiments of this application, due to data transmission between nodes, it only occurs between parent and child nodes in the corresponding forest, and only between root nodes in the forest. Therefore, when calculating the traffic evaluation value, the data expansion rate is defined to measure the non-linear increment that exists when data from different child nodes is transmitted to the parent node and then merged. The expansion rate of the variable is used to describe the change in size of the two data sets before and after merging: the degree to which the total memory occupied by the variable changes during the transmission and merging process across multiple nodes.

[0086] like Figure 4 As shown, here is an example of a data inflation rate of 1: When node w4 transmits data to node w3, since the data inflation rate is 1, it means that during the upward transmission process, the parent node receives one unit of data to merge the data of each child node. Therefore, when node w4 sends one unit of data (td), node w3 actually receives 3*1+1=4(td), while node w2 receives 3*4+1=13(td), and node w1 receives 3*13+1=40td.

[0087] Preferably, based on the different data expansion rates of the nodes, the write traffic assessment value, read traffic assessment value, and broadcast traffic assessment value are calculated in different ways:

[0088] For the sake of consistency, in the following formulas:

[0089] D w This indicates writing the traffic evaluation value, D r D represents the read traffic assessment value. rw This represents the broadcast traffic assessment value;

[0090] W represents the total number of nodes in the corresponding write node forest, w represents the total number of trees in the corresponding write node forest, R represents the total number of nodes in the corresponding read node forest, r represents the total number of trees in the corresponding read node forest, n represents the target number of transmissions in the node's transmission period T, and t represents the node's synchronization time interval.

[0091] d represents the size of each data transmission; A represents the size of a data variable containing all data in its value range; d' represents the size of n data variables of size d combined: d' = A(1-λ) / 2 n ), where λ represents the approximate parameter:

[0092] K w Indicates the degree of the corresponding write tree;

[0093] H w Indicates the height of the corresponding write tree:

[0094] N w Indicates the forest parameters for writing nodes:

[0095] Optionally, if the data bloat rate of the shared network is 0 (i.e., it remains unchanged after merging, typically in a register structure), the write traffic estimate is calculated using the following formula:

[0096] D w=(Ww)nd;

[0097] The read traffic assessment value is calculated using the following formula:

[0098] D r =(Rr)nd;

[0099] Broadcast traffic assessment values ​​are calculated using the following formula:

[0100] D rw =rwnd.

[0101] Optionally, when the data bloat rate of the shared network is greater than 0 and less than 1 (some data between replicas can be merged, while some data cannot be merged, typically a set structure with duplicate values), the write traffic assessment value is calculated according to the following formula:

[0102]

[0103] The read traffic assessment value is calculated using the following formula:

[0104] D r =(Rr)nA(1-λ) Wt );

[0105] Broadcast traffic assessment values ​​are calculated using the following formula:

[0106]

[0107] Optionally, when the data bloat rate of the shared network is 1 (the size after merging n replicas is the sum of the sizes of the n replicas, typically a list structure), the write traffic assessment value is calculated according to the following formula:

[0108]

[0109] The read traffic assessment value is calculated using the following formula:

[0110] Dr=n(Rr)Wtd=(Rr)WTd;

[0111] Broadcast traffic assessment values ​​are calculated using the following formula:

[0112] D rw =wrnN w td=wrN w Td.

[0113] Step 204: Adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

[0114] The method shown in this step has been explained in step 104 and will not be repeated here.

[0115] Optionally, step 204 includes the following sub-steps:

[0116] Sub-step 2041: Adjust the connection relationship of the write nodes in the shared network to the first target topology to generate the target write node forest, and adjust the connection relationship of the read nodes in the shared network to the second target topology to generate the target read node forest.

[0117] The first target topology is a subgraph of the write node forest of the target network topology, and the second target topology is a subgraph of the read node forest of the target network topology.

[0118] In some embodiments of this application, the connection relationships of write nodes in the shared network are adjusted to a first target topology to generate a target write node forest, and the connection relationships of read nodes in the shared network are adjusted to a second target topology to generate a target read node forest. The first target topology is a subgraph of the write node forest of the target network topology, and the second target topology is a subgraph of the read node forest of the target network topology. By adjusting the connection relationships of write and read nodes, it can be ensured that the network topology meets the requirements of transmission performance parameters and target latency parameters, thereby optimizing the utilization of network resources and communication efficiency.

[0119] For example, in a large-scale distributed system, assuming the target network topology is already determined, where the write node forest contains 10 write trees and the read node forest contains 20 read trees, the connection relationships of the write nodes in the shared network are first adjusted to the first target topology. For example, write nodes A, B, C, D, and E are designated as the root nodes of five write trees, and other write nodes are connected to these root nodes according to their hierarchical relationships, forming the target write node forest. Next, the connection relationships of the read nodes in the shared network are adjusted to the second target topology. For example, read nodes F, G, H, I, and J are designated as the root nodes of five read trees, and other read nodes are connected to these root nodes according to their hierarchical relationships, forming the target read node forest. In this way, write node forests and read node forests conforming to the target network topology are generated, thereby optimizing data transmission performance and network resource utilization.

[0120] Sub-step 2042 connects all root write nodes of the target write node forest to all root read nodes of the target read node forest to obtain the target shared network.

[0121] In some embodiments of this application, all root write nodes of the target write node forest are connected one by one to all root read nodes of the target read node forest to obtain a target shared network. By connecting the root nodes of the write node forest and the read node forest, efficient data transmission between write nodes and read nodes can be ensured, thereby achieving optimized configuration of the entire network. This step is to complete the final network topology, making it meet the requirements of transmission performance parameters and target latency parameters.

[0122] For example, in a large-scale distributed system, suppose the target write node forest contains 5 write trees, with root write nodes A1, A2, A3, A4, and A5; and the target read node forest contains 4 read trees, with root read nodes B1, B2, B3, and B4. Each root write node would be connected to each root read node one by one. For example, A1 would be connected to B1 through B4, A2 to B1 through B4, and so on, until A5 is connected to B1 through B4. In this way, a target shared network is formed, ensuring efficient data transfer between write and read nodes, thereby optimizing the overall network performance and resource utilization.

[0123] Step 205: In response to the dynamic node join command, determine the first node type of the dynamic node and obtain the transmission performance parameters of the dynamic node.

[0124] The first node type is used to characterize the type of node forest that dynamic nodes are added to.

[0125] In some embodiments of this application, upon receiving a join command for a dynamic node, the first node type of the dynamic node is determined, and its transmission performance parameters are obtained. The first node type characterizes the type of node forest the dynamic node will join, i.e., whether the dynamic node is joining a write node forest or a read node forest. Transmission performance parameters include bandwidth, latency, and data transmission rate, used to evaluate the transmission performance of the dynamic node in the network. By determining the node type and obtaining the transmission performance parameters, necessary information support can be provided for the joining of dynamic nodes, ensuring the optimization and stability of the network topology.

[0126] For example, in a large-scale distributed system, suppose a new dynamic node X needs to join the shared network. Upon receiving the join command from node X, its initial node type is determined. For instance, node X is determined to join the write node forest. Next, node X's transmission performance parameters are obtained, such as a bandwidth of 5Gbps, a latency of 10ms, and a data transfer rate of 500MB / s. By determining node X's type and obtaining its transmission performance parameters, necessary information can be provided to support node X's joining, ensuring that its transmission performance in the write node forest meets network requirements, thereby optimizing the overall network performance and stability.

[0127] Step 206: Based on the transmission performance parameters of the dynamic nodes, add the dynamic nodes to the node forest corresponding to the first node type to update the target shared network.

[0128] In some embodiments of this application, dynamic nodes are added to the corresponding node forest based on their transmission performance parameters to update the target shared network. The transmission performance parameters of dynamic nodes include bandwidth, latency, and data transmission rate, which are used to evaluate the transmission capabilities of dynamic nodes in the network. Based on the first node type of the dynamic node (i.e., write node or read node), it is added to the corresponding write node forest or read node forest. This method ensures that the addition of dynamic nodes does not affect the overall performance and stability of the network, while updating the target shared network to reflect the new node connection relationships.

[0129] For example, such as Figure 5 As shown, in a large-scale distributed system, suppose a new dynamic node needs to join the shared network. First, the node's transmission performance parameters will be obtained, such as bandwidth of 8Gbps, latency of 7ms, and data transfer rate of 800MB / s. Based on the first node type determined in previous steps, assume node r... x It was determined to be added to the read node forest. Next, node r... x Adding a node to a read node forest, for example, as a leaf node in a read tree, and adjusting the corresponding connections ensures efficient and stable data transmission. In this way, the target shared network is updated to include new dynamic nodes, ensuring optimized overall network performance and stability. Similarly, node w can be added to the read node forest. x Add it to the write node forest.

[0130] Optionally, step 206 includes the following sub-steps:

[0131] Sub-step 2061: Select the tree with the fewest nodes in the node forest corresponding to the first node type of the dynamic node as the target tree, and find the target insertion port in the insertion port of the target tree according to the transmission performance parameters of the dynamic node.

[0132] In some embodiments of this application, when a dynamic node is added, it is inserted into the tree with the fewest nodes. First, based on the dynamic node's primary node type (i.e., write node or read node), the tree with the fewest nodes in the corresponding node forest is found as the target tree. Then, based on the dynamic node's transmission performance parameters, a suitable target insertion port is found among the insertion ports in the target tree. The insertion port refers to the position in the target tree that can accept a new node. By selecting a suitable insertion port, it can be ensured that the addition of a dynamic node will not affect the overall performance and stability of the network.

[0133] For example, in a large-scale distributed system, suppose a new dynamic node M needs to join the shared network. First, determine that node M's primary node type is a write node. Then, find the tree with the fewest nodes in the write node forest, let's say this tree is write tree A, currently containing 8 nodes. Next, based on node M's transmission performance parameters (e.g., bandwidth 6Gbps, latency 8ms), find a suitable target insertion port in write tree A. Suppose that a node N in write tree A has transmission performance parameters that match node M, and node N has a free insertion port. In this way, determine node M's target insertion port, preparing it for joining write tree A.

[0134] Optionally, each insertion port has a corresponding transmission performance threshold. In order to find the target insertion port among the insertion ports of the target tree based on the transmission performance parameters of the dynamic node, sub-step 2061 includes the following sub-steps:

[0135] Sub-step 20611: Determine the second node type of each node in the target tree, and sort the nodes in the target tree according to the second node type.

[0136] The second node type is used to represent the level of a node in the target tree within the corresponding node forest.

[0137] In some embodiments of this application, a second node type is determined for each node in the target tree, and the nodes in the target tree are sorted according to the second node type. The second node type is used to characterize the hierarchical position of the node in the target tree within the corresponding node forest. By determining the hierarchical position of each node, the nodes in the target tree can be sorted, thereby more effectively finding suitable insertion ports in subsequent steps.

[0138] For example, in a large-scale distributed system, suppose the target tree is a write tree C, containing nodes C1, C2, C3, C4, and C5. First, determine the second node type for each node. For example, node C1 is the root node, and its second node type is 1; nodes C2 and C3 are second-level nodes, and their second node type is 2; nodes C4 and C5 are third-level nodes, and their second node type is 3. Next, sort the nodes in the target tree according to their second node type, from largest to smallest, i.e., the order is C1, C2, C3, C4, C5. In this way, the hierarchical position of each node in the target tree can be clearly defined, providing a basis for finding suitable insertion ports in subsequent steps.

[0139] Sub-step 20612: According to the order of the corresponding second node type from largest to smallest, determine the matching degree between the transmission performance threshold corresponding to the insertion port and the transmission performance parameters of the dynamic node, and determine the first insertion port with matching degree as the target insertion port.

[0140] In some embodiments of this application, the matching degree between the transmission performance threshold corresponding to the insertion port and the transmission performance parameters of the dynamic node is determined according to the second node type from largest to smallest, and the first insertion port with a matching degree is determined as the target insertion port. The transmission performance threshold refers to the maximum transmission performance parameters that each insertion port can support, such as bandwidth and latency. By comparing the transmission performance parameters of the dynamic node with the transmission performance threshold of the insertion port, the most suitable insertion port can be found, ensuring that the addition of the dynamic node does not affect the overall performance and stability of the network.

[0141] For example, in a large-scale distributed system, suppose the target tree is a write tree D, containing nodes D1, D2, D3, D4, and D5, already arranged in descending order of second-level node type. Assume node D1 is the root node, with a transmission performance threshold of 10Gbps bandwidth and 5ms latency for its insertion port; nodes D2 and D3 are second-level nodes, with a transmission performance threshold of 8Gbps bandwidth and 7ms latency for their insertion ports; and nodes D4 and D5 are third-level nodes, with a transmission performance threshold of 6Gbps bandwidth and 10ms latency for their insertion ports. Assume there is a dynamic node E with transmission performance parameters of 7Gbps bandwidth and 8ms latency. Following the descending order of second-level node type, first check the insertion port of node D5, finding that its transmission performance threshold does not match. Next check the insertion ports of nodes D2 and D3, finding that the transmission performance threshold of node D2's insertion port matches the transmission performance parameters of node E. Therefore, the insertion port of node D2 is determined as the target insertion port, preparing for the addition of the dynamic node E.

[0142] Sub-step 2062: If the target tree contains a target insertion port, insert the dynamic node into the target insertion port so that the dynamic node obtains data variables from the parent node of the dynamic node.

[0143] In some embodiments of this application, if the target tree contains a target insertion port, a dynamic node will be inserted into the target insertion port so that the dynamic node can obtain data variables from its parent node. The target insertion port refers to a location in the target tree that can accept a new node. By inserting a dynamic node into the target insertion port, it can be ensured that the dynamic node can participate normally in data transmission and processing, thereby maintaining the overall performance and stability of the network.

[0144] For example, in a large-scale distributed system, suppose a new dynamic node M needs to join the shared network, and its target insertion port has been determined in previous steps. Assume the target insertion port is located on node N, which is writing to tree A. Inserting node M into the target insertion port of node N makes node M a child node of node N. In this way, node M can obtain data variables from its parent node N and participate in the data transmission and processing in tree A, thereby ensuring the overall performance and stability of the network.

[0145] Sub-step 2063: If the target tree does not contain the target insertion port, update the node data of the shared network with the dynamic node and the corresponding transmission performance parameters, and return to step 201.

[0146] In some embodiments of this application, if the target tree does not contain a target insertion port, the node data of the shared network is updated with dynamic nodes and their corresponding transmission performance parameters, and the process returns to step 201, i.e., a new target shared network is re-established. The target insertion port refers to a location in the target tree that can accommodate a new node. If there is no suitable insertion port in the target tree, the node data of the shared network needs to be updated to reflect the dynamic node addition requirements, and the process restarts from step 201 to obtain a new network topology. This ensures that the network topology can dynamically adapt to the new node addition requirements, maintaining the overall performance and stability of the network.

[0147] For example, in a large-scale distributed system, suppose a new dynamic node P needs to join the shared network, and the target tree has been determined as write tree B in sub-step 2061. However, there is no suitable target insertion port in write tree B to accept node P. In this case, node P and its transmission performance parameters (e.g., bandwidth of 4Gbps, latency of 12ms) are updated in the node data of the shared network. Then, the process returns to step 201: the node data and target latency parameters of the shared network are retrieved again to generate a new network topology, ensuring that node P can be effectively joined to the shared network, thereby maintaining the overall performance and stability of the network.

[0148] In the relevant embodiments of this application, when a node is added, in addition to network reconnection, the read node also needs to perform data recovery processing to restore the data to a state consistent with the global complete state. This process can rely on local storage or combine data from other read nodes to restore the data. That is, relying on the characteristics of CRDT, during the data recovery process, new incremental data generated by the network can still be directly merged with local data until the existing data is fully restored, at which point the node's data copy will reach the latest state.

[0149] Step 207: In response to the exit command for the target dynamic node in the target shared network, determine the third node type of the target dynamic node.

[0150] The third node type is used to represent the level of the target dynamic node in the corresponding node forest.

[0151] In some embodiments of this application, upon receiving a withdrawal command from a target dynamic node, the third node type of the target dynamic node is determined. The third node type characterizes the target dynamic node's hierarchy within the corresponding node forest, i.e., its specific position within the write or read node forest. Determining the third node type clarifies the target dynamic node's position in the network topology, providing necessary information for subsequent node withdrawal operations and ensuring smooth adjustments and updates to the network topology.

[0152] For example, in a large-scale distributed system, suppose a dynamic node Z needs to leave the shared network. Upon receiving the leave command from node Z, its third node type will be determined first. For instance, node Z might be identified as an intermediate node in a read node forest. By querying the network topology, node Z's hierarchical position within the read node forest is confirmed, such as being at the second level of a certain read tree. Determining node Z's third node type provides the necessary information to support its exit, ensuring that the network topology can be adjusted and updated promptly after node Z's departure, thus maintaining the overall performance and stability of the network.

[0153] Step 208: Based on the third node type, remove the target dynamic node from the target shared network to update the target shared network.

[0154] In some embodiments of this application, a target dynamic node is removed from the target shared network based on a third node type to update the target shared network. The third node type represents the hierarchical position of the target dynamic node in the corresponding node forest. By determining the hierarchical position of the target dynamic node, the network topology can be adjusted in a targeted manner to ensure that the overall performance and stability of the network are not affected after the node leaves. The process of updating the target shared network includes reconfiguring the connection relationships between nodes to adapt to the new topology after the node leaves.

[0155] For example, in a large-scale distributed system, suppose a dynamic node W needs to leave the shared network. First, based on the third node type determined in previous steps, assume node W is identified as an intermediate node in the write node forest. By querying the network topology, the node's hierarchical position in the write node forest is confirmed, for example, it might be located at the second-to-last level of a certain write tree. Next, node W is removed from the write node forest, and the connections between its parent and child nodes are reconfigured to ensure data transmission continuity and network stability. In this way, the target shared network is updated to adapt to the new topology after the node's departure, ensuring optimized overall network performance and stability.

[0156] Optionally, step 208 includes the following sub-steps:

[0157] Sub-step 2081: If the third node type is represented as a leaf node, the target dynamic node is removed from the target shared network.

[0158] In some embodiments of this application, when the third node type is represented as a leaf node, the target dynamic node is removed from the target shared network. A leaf node is a node that has no child nodes in the tree structure. Since a leaf node has no child nodes, its removal will not affect the data transmission of other nodes or the network structure. Therefore, the target dynamic node can be directly removed from the target shared network to update the network topology.

[0159] For example, such as Figure 5 As shown, in a large-scale distributed system, suppose a dynamic node w3 needs to leave the shared network. First, based on the third node type determined in the previous steps (assuming the node has been identified as a leaf node in the read node forest), since node w3 is a leaf node with no children, its departure will not affect data transmission or network structure for other nodes. Therefore, node w3 can be directly removed from the read node forest. In this way, the target shared network is updated to adapt to the new topology after node w3's departure, ensuring the overall performance and stability of the network.

[0160] Sub-step 2082: If the third node type is not a leaf node, determine a target child node from the child nodes of the target dynamic node as a replacement node for the target dynamic node, and remove the target dynamic node from the target shared network.

[0161] In some embodiments of this application, when the third node type is characterized as not a leaf node, a target child node is determined from the child nodes of the target dynamic node as a replacement node for the target dynamic node, and the target dynamic node is removed from the target shared network. A non-leaf node is a node with child nodes in a tree structure. To ensure the integrity of the network topology and the continuity of data transmission, a target child node needs to be selected to replace the position of the target dynamic node. In this way, the overall performance and stability of the network can be maintained after the target dynamic node is removed.

[0162] For example, such as Figure 6 As shown, in a large-scale distributed system, suppose a dynamic node w1 needs to leave the shared network. First, based on the previous steps that have already determined its third node type (assuming node w1 has been identified as a non-leaf node in the write node forest), and it has two child nodes w1... 21 and w 22 To ensure the integrity of the network topology, one of these two child nodes will be selected as the target child node. Let's assume child node w is selected. 22 As the target child node. Then, remove node w1 from the write node forest, and set its child node w as the target child node. 22 Promote it to the position of its parent node, that is, let w 22 The child nodes, and w 21 As w 22 The child nodes, and let w 22 As a child node of the original parent node w1, it takes over the connection relationships and data transmission tasks of node w1. In this way, the target shared network is updated to adapt to the new topology after node w1 leaves, ensuring the overall performance and stability of the network.

[0163] In some embodiments of this application, node exit may require network topology reorganization to prevent subtree networks from being affected. Specific methods can be found in steps 101 to 104, and will not be repeated here. In this case, neighboring nodes need to make timely adjustments to adapt to network changes and maintain normal system operation. Additionally, persistent data backup can be performed to accelerate data recovery during the next recovery process.

[0164] like Figure 7 As shown, the method proposed in this application can be used by programs as a third-party library, or it can be integrated into the syntax level of a programming language by modifying its lexical and syntactic structures, providing native support at the programming language level. After integration into the programming language, the lifecycle of shared variables can be divided into the following stages:

[0165] S1: Initialization: Shared variables are used to create objects by passing parameters (variable type, variable ID, node ID) through code;

[0166] S2: Obtain metadata: Metadata can be obtained through local configuration files or other network configuration systems, including node metadata and variable metadata;

[0167] S3: Parameter settings: After obtaining the metadata, further parameter settings are performed on the initialized object;

[0168] S4: Local Data Recovery: If there was local data saved during the last exit, you can perform local data recovery.

[0169] S5: Session Settings: Create a session for this variable and add it to session management;

[0170] S6: Networking: Establish a network with neighboring nodes and confirm that the network connection is normal;

[0171] S7: Data Synchronization: During node shutdown, if the rest of the network continues to operate, request other nodes to perform data synchronization until the data is updated to the latest global state;

[0172] S8: In operation: The stage where normal service is provided;

[0173] S9: Exit Preparation: When a node is ready to exit, it performs a series of preparatory tasks, including notifying neighboring nodes, disconnecting links, archiving and backing up data, etc.

[0174] S10: Destruction: After the preparation work is completed, it can be officially destroyed.

[0175] refer to Figure 8 This application describes a specific implementation of node management within a Digital Object Architecture (DOA) framework, based on the method described in this application.

[0176] like Figure 8 As shown, by storing digital objects (DOs) as shared variables, it is possible to store metadata in the identifier resolution system of the DataNetwork open source repository (BDWare). This integrates the metadata of variables and the metadata of nodes, storing them in a lightweight data exchange format (JavaScript ObjectNotation, JSON) structure, and jointly storing them in the identifier resolution system, assigning a unique identifier.

[0177] Once a node starts, it reads each variable, generating a shared variable (SharableVar) object for each node. Considering that a physical node may logically act as both a read and write node, and as mentioned earlier, the variable replicas on read and write nodes are different—read nodes possess the entire global state, while write nodes are not guaranteed to have a complete global state—the SharableVar object contains two CRDT member fields: writerVar and readerVar. These represent two states of a shared variable on different logical nodes. writerVar is used when a node performs a write operation on the variable; the resulting effect is stored in the writerVarDeltaQueue, waiting to be sent to its parent node during the next data transmission. readerVar is used when a node performs a read operation on the variable; when a variable receives an effect, it is merged into readerVar and stored in the readerVarDeltaQueue, waiting to be sent to its leaf nodes during the next data transmission.

[0178] As Figure 8 Further embodiments, such as Figure 9 The diagram shown is a flowchart illustrating how a node receives and processes digital object information, as provided in an embodiment of this application.

[0179] Based on the BDWare data system, the communication protocol implemented in this paper uses the Diagnostic over Internet Protocol (DOIP) protocol based on vehicle Ethernet. In addition, an extended operation, 0.DOIP / Op.SyncVar, is added to the original DOIP request type for data synchronization related to shared variables.

[0180] The properties of the protocol information (DoipMessage) for this operation are as follows:

[0181] Variable ID (varId); Network type of the transmitted message (type), reading messages transmitted in the forest network (r2r); writing messages transmitted in the forest network (w2w); Message sender is reading and writing messages transmitted in the network broadcast layer (w2r); Transmit message payload (content), serializing the effect into a JSON string.

[0182] When the variable receives a message of type "r2r" or "w2r", perform the following operations:

[0183] M11: The effect in the payload needs to be merged into readerVar;

[0184] M12: Add to the readerVarDeltaQueue queue;

[0185] When the variable receives a message of type "w2w", perform the following operations:

[0186] M21: Merge the effects in the payload into writerVar;

[0187] M22 is added to the writerVarDeltaQueue queue.

[0188] As Figure 8 Further embodiments, such as Figure 10 The diagram shown is a flowchart illustrating the steps of a write node periodically sending DoipMessages, as provided in an embodiment of this application.

[0189] After encapsulating DoipMessage, determine whether the current node is the root node of the forest and perform the following operations:

[0190] N1: If the node itself is not the root node of the forest, send a message to the parent node;

[0191] N2: When the node itself is the root node of the forest, broadcast messages to the reading forest nodes.

[0192] As Figure 8 Further embodiments, such as Figure 11 The diagram shown is a flowchart illustrating the steps of a read node periodically sending DoipMessages, as provided in an embodiment of this application.

[0193] L1: Broadcast messages to child nodes.

[0194] like Figure 12 As shown, based on the method of the embodiments of this application, this application also implements a CRDT variable network simulation experimental program suitable for large-scale scenarios:

[0195] The simulated nodes and their corresponding components implement the `TickComponent` interface method. The main program drives the state transitions of each component by continuously adjusting the clock. The descriptions of each component are as follows:

[0196] Node simulation component: includes a unique identifier, a Network Interface Card, and a simulated data processor.

[0197] The DataProcessor is used to generate and process received data, and contains a buffer queue for temporarily storing data to be sent.

[0198] NetworkInterfaceCard is used to simulate network management and record neighbor node information;

[0199] In addition, the above structure also includes a simulated bandwidth for data upload and download, containing a buffer queue to temporarily store data, a simulated transmission process component (Transmission) to simulate a transmission process, including sender node information and receiver node information, as well as data payload, and a simulated data packet (DataPacket), which stores CRDT variables and can perform merging operations.

[0200] Specifically, the simulation process is as follows:

[0201] K1: Cluster node initialization;

[0202] K2: Network topology;

[0203] K3: Analog data transmission;

[0204] K4: Data analysis report output;

[0205] The key part of the program is the simulated data transmission. The main process maintains a global clock, and each component needs to implement several interfaces contained in TickComponent to complete the tasks at the beginning, middle, and end of each clock cycle. The lifecycle of a data packet is as follows:

[0206] K31: The DataProcessor generates data at fixed intervals and adds it to the buffer queue;

[0207] K32: The DataProcessor packages and merges the data in the buffer queue according to a certain rule at fixed intervals and sends it to the NetworkInterfaceCard;

[0208] K33: NetworkInterfaceCard packages data into a Transmission object, assembles the target node information, and adds it to the Bandwidth buffer queue;

[0209] K34: Bandwidth is allocated a fixed traffic limit in each cycle, and transmits tasks in the buffer queue to the target node. A certain amount of data can be transmitted each clock cycle.

[0210] K35: After the data transmission is completed, the target node unpacks the data packet and adds it to the DataProcessor's buffer queue, waiting for the next round of transmission;

[0211] K36: The entire lifecycle is completed when the data packet is no longer propagated by the node.

[0212] To support the simulation of a large-scale cluster, relying solely on the memory of a single machine is insufficient to simulate all nodes. Therefore, in the specific code implementation, all the above components are persisted using the embedded database RocksDB, and a memory buffer is implemented using GuavaCache to handle the swapping of data between memory and disk.

[0213] In summary, in this embodiment, by obtaining the transmission performance parameters and target latency parameters of each node in the shared network, and utilizing the unidirectional transmission property of the network topology generated by the scheme of this application—that data transmission between any write node and any read node is guaranteed to be from child-level nodes to parent-level nodes in the write node forest and from parent-level nodes to child-level nodes in the read node forest—the generated network topology can meet the connection performance and latency requirements of each node in the network. Then, by evaluating the traffic of different network topologies, the network structure with the optimal utilization of network resources is further selected, further reducing network communication overhead. This allows distributed shared variables to be adapted to larger-scale server clusters. Therefore, the method based on this embodiment achieves an efficient shared variable mechanism for large-scale scenarios, solving the problem that existing distributed shared variable schemes can only be applied to small- to medium-sized server clusters.

[0214] refer to Figure 13 It illustrates a shared variable device 30 for large-scale scenarios provided in an embodiment of this application, comprising:

[0215] The data acquisition module 301 is used to acquire node data of the shared network and the target delay parameter of the shared network. The node data includes the transmission performance parameters of each node in the shared network. The transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes. The target delay parameter is used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network.

[0216] The topology generation module 302 is used to generate multiple network topologies of the shared network based on transmission performance parameters and target delay parameters. The network topology includes a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network. The write node forest contains at least one write tree, and the read node forest contains at least one read tree. The root write node of each write tree is connected to the root read node of each read tree.

[0217] The topology selection module 303 is used to determine the traffic evaluation value of the network topology based on the transmission performance parameters and the target delay parameters, and to determine the network topology corresponding to the minimum traffic evaluation value as the target network topology.

[0218] The network adjustment module 304 is used to adjust the connection relationship between nodes in the shared network according to the target network topology to obtain the target shared network.

[0219] Optionally, the transmission performance parameters include the maximum number of connections per node, and the topology generation module 302 includes:

[0220] The write forest generation submodule is used to generate multiple write node forests by using the node connectivity of the written tree as the first traversal variable parameter and the maximum number of connections as the maximum value of the first traversal parameter; the first traversal variable parameter is used to characterize the node degree of the written tree in the write node forest.

[0221] The read forest generation submodule is used to generate multiple read node forests by using the node connectivity of the read tree as the second traversal variable parameter and the maximum number of connections as the maximum value of the second traversal parameter; the second traversal variable parameter is used to characterize the node degree of the read tree in the read node forest.

[0222] The combination submodule is used to connect each write node forest to each read node forest to obtain multiple alternative topologies and test the maximum transmission latency of each alternative topology.

[0223] The exclusion submodule is used to delete the candidate topology structures whose maximum transmission delay is greater than the target delay parameter, and to use the remaining candidate topology structures as the network topology structures.

[0224] Optionally, the topology selection module 303 includes:

[0225] The partial evaluation submodule is used to determine the write traffic evaluation value, read traffic evaluation value, and broadcast traffic evaluation value of the network topology based on the transmission performance parameters and target delay parameters, respectively.

[0226] The comprehensive evaluation submodule is used to sum the write traffic evaluation value, read traffic evaluation value, and broadcast traffic evaluation value as the traffic evaluation value of the network topology.

[0227] Optionally, the network adjustment module 304 includes:

[0228] The forest adjustment submodule is used to adjust the connection relationships of write nodes in the shared network to a first target topology to generate a target write node forest, and to adjust the connection relationships of read nodes in the shared network to a second target topology to generate a target read node forest; the first target topology is a subgraph of the write node forest of the target network topology, and the second target topology is a subgraph of the read node forest of the target network topology.

[0229] The root connection submodule is used to connect all root write nodes of the target write node forest to all root read nodes of the target read node forest one by one to obtain the target shared network.

[0230] Optionally, the shared variable device 30 for large-scale scenarios also includes:

[0231] The node acquisition module is used to respond to the dynamic node join command, determine the first node type of the dynamic node, and obtain the transmission performance parameters of the dynamic node; the first node type is used to characterize the type of node forest to which the dynamic node joins;

[0232] The node addition module is used to add dynamic nodes to the node forest corresponding to the first node type based on the transmission performance parameters of the dynamic nodes, so as to update the target shared network.

[0233] Optionally, the node addition module includes:

[0234] The port lookup submodule is used to select the tree with the fewest nodes in the node forest corresponding to the first node type of the dynamic node as the target tree, and to find the target insertion port in the insertion port of the target tree according to the transmission performance parameters of the dynamic node.

[0235] The node insertion submodule is used to insert a dynamic node into the target insertion port when the target tree contains the target insertion port, so that the dynamic node can obtain data variables from the parent node of the dynamic node.

[0236] The network reconstruction submodule is used to update the node data of the shared network with dynamic nodes and corresponding transmission performance parameters when the target tree does not contain the target insertion port, and then return to the steps of obtaining the node data of the shared network and the target latency parameters of the shared network.

[0237] Optionally, each inserted port has a corresponding transmission performance threshold, and the port lookup submodule includes:

[0238] The port lookup unit is used to determine the second node type of each node in the target tree and sort the nodes in the target tree according to the second node type; the second node type is used to represent the level of the node in the target tree in the corresponding node forest;

[0239] The port selection unit is used to determine the matching degree between the transmission performance threshold corresponding to the insertion port and the transmission performance parameters of the dynamic node in descending order of the corresponding second node type, and to determine the first insertion port whose matching degree is characterized as a match as the target insertion port.

[0240] Optionally, the shared variable device 30 for large-scale scenarios also includes:

[0241] The node selection module is used to determine the third node type of the target dynamic node in response to the exit command of the target dynamic node in the target shared network; the third node type is used to characterize the level of the target dynamic node in the corresponding node forest.

[0242] The node exit module is used to remove the target dynamic node from the target shared network based on the third node type, so as to update the target shared network.

[0243] Optionally, the node exit module includes:

[0244] The Leaf Node Exit Submodule is used to remove the target dynamic node from the target shared network when the third node type is represented as a leaf node.

[0245] The Other Node Exit Submodule is used to determine a target child node from the child nodes of the target dynamic node when the third node type is not a leaf node, to serve as a replacement node for the target dynamic node, and to exit the target dynamic node from the target shared network.

[0246] In summary, in this embodiment, by obtaining the transmission performance parameters and target latency parameters of each node in the shared network, and utilizing the unidirectional transmission property of the network topology generated by the scheme of this application—that data transmission between any write node and any read node is guaranteed to be from child-level nodes to parent-level nodes in the write node forest and from parent-level nodes to child-level nodes in the read node forest—the generated network topology can meet the connection performance and latency requirements of each node in the network. Then, by evaluating the traffic of different network topologies, the network structure with the optimal utilization of network resources is further selected, further reducing network communication overhead. This allows distributed shared variables to be adapted to larger-scale server clusters. Therefore, the method based on this embodiment achieves an efficient shared variable mechanism for large-scale scenarios, solving the problem that existing distributed shared variable schemes can only be applied to small- to medium-sized server clusters.

[0247] Reference Figure 14The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0248] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0249] Memory 504 is used to store various types of data to support the operation of electronic device 500. Examples of this data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0250] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0251] Multimedia component 508 includes an interface that provides an output interface between electronic device 500 and user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When electronic device 500 is in an operating mode, such as shooting mode or multimedia mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0252] Audio component 510 is used to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) used to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0253] Input / output (I / O) interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0254] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or a component of electronic device 500, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0255] Communication component 516 facilitates wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0256] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of this application.

[0257] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of an electronic device 500 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0258] Figure 15 This is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. (See also...) Figure 15 The electronic device 600 includes a processing component 622, which further includes one or more processors, and memory resources represented by memory 632 for storing instructions, such as application programs, that can be executed by the processing component 622. The application programs stored in memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 622 is configured to execute instructions to perform the methods provided in the embodiments of this application.

[0259] Electronic device 600 may also include a power supply component 626 configured to perform power management of electronic device 600, a wired or wireless network interface 650 configured to connect electronic device 600 to a network, and an input / output (I / O) interface 658. Electronic device 600 may operate on an operating system stored in memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0260] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0261] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A shared variable method for large-scale scenarios, characterized in that, Applied to distributed systems, including: Obtain node data of the shared network and target latency parameters of the shared network; the node data includes transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes; the target latency parameters are used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network. Based on the transmission performance parameters and the target latency parameters, multiple network topologies of the shared network are generated; the network topologies include a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network; the write node forest contains at least one write tree and the read node forest contains at least one read tree; the root write node of each write tree is connected to the root read node of each read tree one by one. Based on the transmission performance parameters and the target delay parameters, the traffic evaluation value of the network topology is determined, and the network topology corresponding to the minimum traffic evaluation value is determined as the target network topology. Adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

2. The method as described in claim 1, characterized in that, The transmission performance parameters include the maximum number of connections for the node. Generating multiple network topologies for the shared network based on the transmission performance parameters and the target latency parameters includes: The node connectivity of the written tree is used as the first traversal variable parameter, and the maximum number of connections is used as the maximum value of the first traversal variable parameter to generate multiple write node forests; the first traversal variable parameter is used to characterize the node degree of the written tree in the write node forest. The node connectivity of the read tree is used as the second traversal variable parameter, and the maximum number of connections is used as the maximum value of the second traversal variable parameter to generate multiple read node forests; the second traversal variable parameter is used to characterize the node degree of the read tree in the read node forest. Each of the write node forests is connected to each of the read node forests to obtain multiple alternative topologies, and the maximum transmission latency of each alternative topology is tested. The candidate topology structure corresponding to the maximum transmission delay greater than the target delay parameter is deleted, and the remaining candidate topology structure is used as the network topology structure.

3. The method as described in claim 1, characterized in that, Determining the traffic evaluation value of the network topology based on the transmission performance parameters and the target delay parameters includes: Based on the transmission performance parameters and the target delay parameters, the write traffic evaluation value, read traffic evaluation value, and broadcast traffic evaluation value of the network topology are determined respectively. The sum of the write traffic assessment value, the read traffic assessment value, and the broadcast traffic assessment value is used as the traffic assessment value of the network topology.

4. The method as described in claim 1, characterized in that, The method further includes: In response to a dynamic node joining command, the first node type of the dynamic node is determined, and the transmission performance parameters of the dynamic node are obtained; the first node type is used to characterize the type of node forest to which the dynamic node joins; Based on the transmission performance parameters of the dynamic node, the dynamic node is added to the node forest corresponding to the first node type to update the target shared network.

5. The method as described in claim 4, characterized in that, Based on the transmission performance parameters of the dynamic node, the dynamic node is added to the node forest corresponding to the corresponding first node type to update the target shared network, including: The tree with the fewest nodes in the node forest corresponding to the first node type of the dynamic node is taken as the target tree, and the target insertion port is found in the insertion port of the target tree according to the transmission performance parameters of the dynamic node. If the target tree contains the target insertion port, the dynamic node is inserted into the target insertion port so that the dynamic node obtains data variables from the parent node of the dynamic node; If the target tree does not contain the target insertion port, update the node data of the shared network with the dynamic node and the corresponding transmission performance parameters, and return to the step of obtaining the node data of the shared network and the target delay parameters of the shared network. Each insertion port has a corresponding transmission performance threshold. The step of finding the target insertion port among the insertion ports of the target tree based on the transmission performance parameters of the dynamic node includes: Determine the second node type of each node in the target tree, and sort the nodes in the target tree according to the second node type; the second node type is used to characterize the level of the node in the target tree in the corresponding node forest; According to the order of the corresponding second node types from largest to smallest, determine the matching degree between the transmission performance threshold corresponding to the insertion port and the transmission performance parameters of the dynamic node, and determine the first insertion port whose matching degree is characterized as a match as the target insertion port.

6. The method as described in claim 1, characterized in that, The method further includes: In response to an exit command for a target dynamic node in the target shared network, a third node type of the target dynamic node is determined; the third node type is used to characterize the level of the target dynamic node in the corresponding node forest. Based on the third node type, the target dynamic node is removed from the target shared network to update the target shared network.

7. The method as described in claim 6, characterized in that, The step of removing the target dynamic node from the target shared network according to the third node type to update the target shared network includes: If the third node type is characterized as a leaf node, the target dynamic node will be removed from the target shared network; If the third node type is not a leaf node, a target child node is determined from the child nodes of the target dynamic node to serve as a replacement node for the target dynamic node, and the target dynamic node is removed from the target shared network.

8. A shared variable device for large-scale scenarios, characterized in that, Applied to distributed systems, including: The data acquisition module is used to acquire node data of the shared network and the target latency parameter of the shared network; the node data includes the transmission performance parameters of each node in the shared network; the transmission performance parameters are used to characterize the data transmission performance of the corresponding node when it is connected to other nodes; the target latency parameter is used to constrain the longest time interval from when the data variable is written to any write node of the shared network to when the data variable is output to any read node of the shared network. A topology generation module is used to generate multiple network topologies of the shared network based on the transmission performance parameters and the target delay parameters. The network topologies include a write node forest composed of write nodes in the network and a read node forest composed of read nodes in the network. The write node forest contains at least one write tree, and the read node forest contains at least one read tree. The root write node of each write tree is connected to the root read node of each read tree. The topology selection module is used to determine the traffic evaluation value of the network topology based on the transmission performance parameters and the target delay parameters, and to determine the network topology corresponding to the minimum traffic evaluation value as the target network topology. The network adjustment module is used to adjust the connection relationships between nodes in the shared network according to the target network topology to obtain the target shared network.

9. An electronic device, characterized in that, include: Processor; memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 7.

Images