Information synchronization methods, systems, electronic devices and computer-readable storage media

By monitoring the CT module's updates to the southbound database tables through the local controller, the problem of untimely synchronization of session information in the load balancing module was resolved, thereby improving the information consistency of the load balancing module and enhancing system reliability.

CN116192876BActive Publication Date: 2026-03-13PURPLE MOUNTAIN LAB
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-24
Publication Date
2026-03-13

Smart Images

  • Figure CN116192876B_ABST
    Figure CN116192876B_ABST
Patent Text Reader

Abstract

This invention discloses an information synchronization method, system, electronic device, and computer-readable storage medium. The method includes: a local controller monitoring a connected tracking CT module to obtain a target monitoring result; if the target monitoring result indicates that a target event has been detected by the CT module, the local controller obtains target information based on the content of the target event; based on the target information, the local controller modifies the initial table of the southbound database to an updated table, so that load balancing modules corresponding to multiple local controllers can obtain target session information. The target session information is obtained from the updated table, which is retrieved by each of the multiple local controllers from the southbound database. This invention solves the technical problem of difficulty in timely synchronization of session information when synchronizing session information of multiple load balancing modules in related technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to an information synchronization method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] Cloud data centers, compared to traditional data centers, are larger in scale in terms of hardware infrastructure such as computing, networking, and storage. They also integrate more new technologies such as cloud computing, software-defined networking (SDN), and big data, enabling them to provide richer cloud services. To cope with the exponential growth of data and high concurrency access, large data centers need to deploy load balancing modules to handle massive workloads from both external and internal sources and improve resource utilization. Load balancing modules handle almost all external inbound traffic and half of the internal traffic, making them crucial to the performance of services hosted in the data center. Load balancing modules can evenly distribute workloads among a group of backend servers (load balancing resource pools), effectively handling client requests to the servers. On one hand, load balancing modules are a vital building block for improving server resource utilization in cloud data centers. Based on information from the network layer, transport layer, and applications, appropriate load balancing algorithms are used to distribute requests evenly to nodes in the server cluster, ensuring that each node's resources are fully utilized. On the other hand, load balancing modules also ensure service reliability by ensuring high availability for multiple users through the collaboration of multiple devices and failover mechanisms. Timely synchronization of information from multiple load balancing modules is crucial to ensuring the proper functioning of each module. However, there are technical challenges in synchronizing session information across multiple load balancing modules in related technologies.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an information synchronization method, system, electronic device, and computer-readable storage medium to at least solve the technical problem in the related art of difficulty in timely synchronizing session information when synchronizing session information of multiple load balancing modules.

[0005] According to one aspect of the present invention, an information synchronization method is provided, comprising: a local controller monitoring a connection to a tracking CT module to obtain a target monitoring result; if the target monitoring result indicates that a target event has occurred in the CT module, the local controller obtains target information based on the content of the target event, wherein the target event is a change in the initial session information in the CT module; the local controller, based on the target information, modifies the initial table of the southbound database in the southbound database to an updated table of the southbound database, so that load balancing modules corresponding to multiple local controllers respectively obtain the target session information, wherein the target session information is obtained based on the updated table of the southbound database, and the updated table of the southbound database is obtained by the multiple local controllers respectively from the southbound database.

[0006] Optionally, the step of changing the initial table of the southbound database to an updated table of the southbound database based on the target information includes: when the target information includes 5-tuple information, determining whether the 5-tuple information matches the corresponding item information in the pre-stored information in the initial table of the southbound database to obtain a first matching result, wherein the 5-tuple information includes a target layer 4 protocol type, a target IP address group, and a target port group, the target IP address group includes a source IP address and a destination IP address, and the target port group includes a source port and a destination port; when the first matching result is that the 5-tuple information matches the target information, the initial table of the southbound database is updated. If the corresponding information in the pre-stored information cannot be matched, a target row is added to the initial table of the southbound database to obtain the updated table of the southbound database. The target row includes the target information. The mismatch between the five-tuple information and the corresponding information in the pre-stored information includes: the target layer 4 protocol type does not match the corresponding information in the pre-stored information, or the target layer 4 protocol type matches the corresponding information in the pre-stored information, but the target IP address group does not match the corresponding information in the pre-stored information, and the target port group does not match the corresponding information in the pre-stored information.

[0007] Optionally, when the target information includes quintuple information, after determining whether the quintuple information matches the corresponding item information in the pre-stored information in the initial table of the southbound database to obtain a first matching result, the method further includes: when the target information includes other information, if the first matching result indicates that the quintuple information matches the corresponding item information in the pre-stored information, determining a predetermined row in the pre-stored information that matches the quintuple information, and predetermined row information in the predetermined row, wherein the other information is information in the target information other than the quintuple information; determining whether each item in the other information matches the corresponding item information in the predetermined row information to obtain a second matching result; if the second matching result indicates that each item in the other information does not match the corresponding item information in the predetermined row information, updating the predetermined row in the initial table of the southbound database to the target row to obtain the updated table of the southbound database, wherein the target row includes the target information.

[0008] Optionally, the step of changing the initial table of the southbound database to the updated table of the southbound database based on the target information includes: calling the target interface; and changing the initial table of the southbound database to the updated table of the southbound database based on the target interface and the target information.

[0009] According to one aspect of the present invention, an information synchronization method is provided, comprising: when the initial table of the southbound database in the southbound database is updated to an updated table of the southbound database, multiple local controllers respectively obtain the updated table of the southbound database, wherein the updated table of the southbound database is obtained by modifying the initial table of the southbound database according to target information, the target information being obtained based on the content of a target event when any one of the multiple local controllers detects a target event in the connection tracking CT module, the target event being a change in the initial session information in the CT module; the multiple local controllers respectively determine target session information based on the updated table of the southbound database; and the multiple local controllers respectively send the target session information to a corresponding load balancing module.

[0010] According to one aspect of the present invention, an information synchronization system is provided, comprising: a plurality of target modules, a southbound database, each of the plurality of target modules including an Open Virtual Switch (OVS), a local controller, and a Connection Tracking (CT) module, the local controller including a load balancing module, wherein the OVS in each of the plurality of target modules is used to receive packets, obtain target information based on the packets, and change the initial session information in the corresponding CT module based on the target information, the local controller in each of the plurality of target modules is used to monitor the corresponding CT module, obtain a target monitoring result, and wherein the target monitoring result is that the target CT module has been monitored. When a target event occurs in the CT module, the target information is obtained based on the content of the target event. Based on the target information, the initial table of the southbound database in the southbound database is changed to the updated table of the southbound database. The target event is a change in the initial session information in the CT module. When the initial table of the southbound database in the southbound database is updated to the updated table of the southbound database in the multiple target modules, the multiple local controllers obtain the updated table of the southbound database, determine the target session information based on the updated table of the southbound database, and send the target session information to the corresponding load balancing module respectively, so that the target session information in the multiple load balancing modules is the same.

[0011] Optionally, the system also includes a white-box switch, wherein the OVS receives the messages sent by the white-box switch, wherein the white-box switch includes the cloud-based open network software SONiC white-box switch.

[0012] Optionally, the load balancing module includes: a target-level flow table for the pipeline, wherein the pipeline includes multi-level flow tables.

[0013] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the information synchronization method described in any of the preceding claims.

[0014] According to one aspect of the present invention, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the information synchronization method described in any of the preceding claims.

[0015] In this embodiment of the invention, the local controller first listens to the connection tracking CT module to obtain the target listening result. Then, if the target listening result indicates that a target event has occurred in the CT module, the local controller obtains the target information based on the content of the target event. The target event is a change in the initial session information in the CT module. The local controller then updates the initial table in the southbound database to an updated table based on the target information, so that the load balancing modules corresponding to multiple local controllers can obtain the target session information. The target session information is obtained from the updated table in the southbound database, which is obtained by each of the multiple local controllers from the southbound database. Because the local controller can promptly determine the target event of a change in session information in the CT module by listening to the connection tracking CT module and obtain the target information based on the target event content, the local controller can promptly obtain the target information corresponding to newly arrived packets. Since the local controller changes the initial table of the southbound database to the updated table of the southbound database based on the target information, and the information in the updated table of the southbound database can be session information, and the target information is also session information, it can be ensured that the updated table of the southbound database accurately reflects the latest session information. This achieves the technical effect of timely updating the session information in the updated table of the southbound database. Because the southbound database is connected to multiple local controllers, multiple load balancing modules in multiple local controllers can obtain the target session information in the updated table of the southbound database, and the target session information in multiple load balancing modules is the same. This solves the technical problem of difficulty in timely synchronizing session information when synchronizing session information of multiple load balancing modules in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of an information synchronization method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of the second information synchronization method according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram illustrating the function of the local controller provided in an optional embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of a message transmission system provided by an optional embodiment of the present invention;

[0021] Figure 5 This is a structural block diagram of an information synchronization system according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0025] The connection tracking (CT) module (conntrack) is a module in the Linux kernel used to discover and track the state of connections. Specifically, it includes: 1) extracting tuple information from packets to identify data flows and corresponding connections; 2) maintaining a state database (conntracktable) for all connections, such as connection creation time, number of packets sent, number of bytes sent, etc.; 3) reclaiming expired connections; and 4) providing services for higher-level functions (such as Network Address Translation (NAT).

[0026] Open vSwitch (OVS): Open vSwitch (OVS) is a high-quality virtual switch that supports multi-layer data forwarding. Primarily deployed on servers, it offers superior programmability compared to traditional switches, while retaining the network isolation and data forwarding capabilities of traditional switches. It runs on each virtualized physical machine and provides remote management. OVS offers two protocols for remote management in virtualized environments: OpenFlow, which manages switch behavior through flow tables, and the OVSDB management protocol, used to expose the switch's port status. The OpenFlow protocol can be used in SDN networks to separate the network forwarding plane and control plane.

[0027] Local Controller: The OVN local controller (ovn-controller) is a background process that can connect to the OVN southbound database via the OVSDB protocol and to the virtual switch OVS via the OpenFlow protocol. The local controller in this solution also includes a load balancing module, which can implement load balancing (LB) functionality.

[0028] Southbound Database: OVN Southbound DB is the core of the system. The client consists of the upper-layer OVN core background process ovn-northd and the lower-layer local controller ovn-controller running on each compute node.

[0029] White-box switches: White-box switches adopt an open device architecture and a hardware-software decoupling approach, allowing for customization of the underlying hardware and upper-layer software according to business needs. Compared to the bundled hardware and software of traditional switches, this significantly reduces costs. Furthermore, in terms of software functionality, secondary development based on open-source software can be performed, reducing development cycles and costs. White-box switches support programmable hardware data planes and containerized software deployment. By customizing data plane forwarding logic through software definition, they also fully leverage modern cloud computing technologies for rapid upgrades and iterations of network functions. This enhances network flexibility, agility, and determinism, optimizes network performance, and enables the network to meet complex business requirements. Additionally, containerized deployment simplifies management and maintenance, reducing network operation and maintenance costs.

[0030] Session Information: In this application, session information may refer to a session, which in computers, especially in web applications, is known as "session control." A session object stores the attributes and configuration information required for a specific user session. This ensures that variables stored in the session object are not lost when a user navigates between web pages within the application, but persist throughout the user session. When a user requests a web page from the application, the web server automatically creates a session object if the user does not already have one. The server terminates the session when it expires or is abandoned.

[0031] CT Session: Session information related to the connection tracking CT module.

[0032] Example 1

[0033] According to an embodiment of the present invention, an embodiment of an information synchronization method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] First, it should be noted that the session information in this application includes the source IP address of the forward packet, the destination IP address of the forward packet, the source port of the forward packet, the destination port of the forward packet, the source IP address of the reverse packet, the destination IP address of the reverse packet, the source port of the reverse packet, the destination port of the reverse packet, the Layer 3 protocol type, the Layer 4 protocol type, the Source Network Address Translation (SNAT) address, the Destination Network Address Translation (DNAT) address, the SNAT port, and the DNAT port. The load balancing module can select an appropriate path for transmission of packets arriving at the OVS corresponding to the load balancing module based on the content of the session information.

[0035] Figure 1 This is a flowchart of an information synchronization method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step S102: The local controller listens to the connected tracking CT module and obtains the target listening results;

[0037] In the technical solution provided by step S102 of the present invention, the executing entity is a local controller. In the scenario in which the method is implemented in this application, multiple local controllers may be included, that is, multiple local controllers may each monitor the CT module of the Linux kernel to obtain the target monitoring result. In this way, the target monitoring result can be obtained quickly and timely.

[0038] Step S104: If the target monitoring result is that a target event has been detected in the CT module, the local controller obtains the target information based on the content of the target event, wherein the target event is a change in the initial session information in the CT module.

[0039] In the technical solution provided in step S104 of the present invention, the initial session information includes multiple session information entries, and the target information is a session information entry related to the CT session. When the target monitoring result indicates that a target event has occurred in the CT module, the local controller can quickly and promptly obtain the target information based on the content of the target event. In this way, it can be ensured that the local controller can promptly capture the target information corresponding to newly arrived packets and promptly grasp the target information, that is, promptly grasp the latest session information.

[0040] It should be noted that there are many situations that can cause changes in the session information in the CT module. For example, when a message arrives at the OVS, the OVS will obtain the target information corresponding to the message and change the session information in the CT module based on the target information. That is, the session information in the CT module has changed. In this case, the local controller will listen for the target event in the CT module. Here, the OVS, the CT module and the local controller are located on the same hardware.

[0041] In step S106, the local controller changes the initial table of the southbound database to the updated table of the southbound database based on the target information, so that the load balancing modules corresponding to the multiple local controllers can obtain the target session information. The target session information is obtained based on the updated table of the southbound database, which is obtained by the multiple local controllers from the southbound database respectively.

[0042] It should be noted that the information in the southbound database update table is session information, which can be CT session information. The southbound database update table includes multiple session information entries.

[0043] In the technical solution provided by step S106 of the present invention, the local controller can promptly change the initial table of the southbound database to the updated table of the southbound database based on the target information. In this way, it can be ensured that the session information in the updated table of the southbound database takes into account the newly arrived packets and is the latest and most accurate.

[0044] It should be noted that when the session information in the southbound database update table is CT session information, the southbound database update table can be a CT flow table related to the CT session information, namely the Ct_Flow table. The Ct_Flow table includes multiple entries, for example, the types of multiple entries can be: forward packet source IP address entry, forward packet destination IP address entry, forward packet source port entry, forward packet destination port entry, reverse packet source IP address entry, reverse packet destination IP address entry, reverse packet source port entry, reverse packet destination port entry, Layer 3 protocol type entry, Layer 4 protocol type entry, Source Network Address Translation (SNAT) address entry, Destination Network Address Translation (DNAT) address entry, SNAT port entry, and DNAT port entry.

[0045] It should also be noted that a message arriving at OVS is an intermediate step in the process of a message traveling from the client to the server, or vice versa. When a message is initially sent from the client to the server, the message from the client to the server is a forward message, and the message returned from the server to the client is a reverse message. Conversely, when a message is initially sent from the server to the client, the message from the server to the client is a forward message, and the message returned from the client to the server is a reverse message. The source IP address of a forward message is stored in the forward message source IP address field of the corresponding row. The destination IP address of a forward packet is stored in the forward packet destination IP address field of the corresponding row, the source port of the forward packet is stored in the forward packet source port field of the corresponding row, and the destination port of the forward packet is stored in the forward packet destination port field of the corresponding row. Similarly, the source IP address of a reverse packet is stored in the reverse packet source IP address field of the corresponding row, the destination IP address of the reverse packet is stored in the reverse packet destination IP address field of the corresponding row, and the source port of the reverse packet is stored in the reverse packet source port field of the corresponding row, and the destination port of the reverse packet is stored in the reverse packet destination port field of the corresponding row. Layer 3 protocol type fields, Layer 4 protocol type fields, SNAT address fields, DNAT address fields, SNAT port fields, and DNAT port fields can store information corresponding to both forward and reverse packets. When the row corresponding to these fields stores information about a forward packet, these fields store information about a forward packet; when the row corresponding to these fields stores information about a reverse packet, these fields store information about a reverse packet.

[0046] It should also be noted that a predetermined duration can be set for each piece of information in the southbound database update table, for example, 20 minutes. The existence duration of all information in the southbound database update table should be promptly determined. If the existence duration in the southbound database table exceeds the predetermined duration, the timeout information should be identified and deleted. In this way, the information in the southbound database update table will only have been used within the predetermined duration, and timeout information that has not been used for a period of time can be deleted, saving space occupied by the southbound database update table.

[0047] Through steps S102 to S106, the local controller first listens to the connection tracking CT module to obtain the target listening result. Then, if the target listening result indicates that a target event has occurred in the CT module, the local controller obtains the target information based on the content of the target event. The target event is a change in the initial session information in the CT module. The local controller then updates the initial table in the southbound database to an updated table based on the target information, so that the load balancing modules corresponding to multiple local controllers can obtain the target session information. The target session information is obtained from the updated table in the southbound database, which is obtained by each local controller from the southbound database. Because the local controller can promptly determine the target event of a change in session information in the CT module by listening to the connection tracking CT module and obtain the target information based on the target event content, the local controller can promptly obtain the target information corresponding to newly arrived packets. Since the local controller changes the initial table of the southbound database to the updated table of the southbound database based on the target information, and the information in the updated table of the southbound database can be session information, and the target information is also session information, it can be ensured that the updated table of the southbound database accurately reflects the latest session information. This achieves the technical effect of timely updating the session information in the updated table of the southbound database. Because the southbound database is connected to multiple local controllers, multiple load balancing modules in multiple local controllers can obtain the target session information in the updated table of the southbound database, and the target session information in multiple load balancing modules is the same. This solves the technical problem of difficulty in timely synchronizing session information when synchronizing session information of multiple load balancing modules in related technologies.

[0048] As an optional embodiment, based on the target information, the initial table of the southbound database in the southbound database is changed to an updated table of the southbound database, including: if the target information includes 5-tuple information, determining whether the 5-tuple information matches the corresponding information in the pre-stored information in the initial table of the southbound database to obtain a first matching result, wherein the 5-tuple information includes the target Layer 4 protocol type, the target IP address group, and the target port group, the target IP address group includes the source IP address and the destination IP address, and the target port group includes the source port and the destination port; if the first matching result is that the 5-tuple information does not match the corresponding information in the pre-stored information, a target row is added to the initial table of the southbound database to obtain an updated table of the southbound database, wherein the target row includes the target information, and the 5-tuple information does not match the corresponding information in the pre-stored information including: the target Layer 4 protocol type does not match the corresponding information in the pre-stored information, or the target Layer 4 protocol type matches the corresponding information in the pre-stored information but the target IP address group does not match the corresponding information in the pre-stored information and the target port group does not match the corresponding information in the pre-stored information.

[0049] In this embodiment, the pre-stored information is the existing session information in the initial table of the southbound database. The target information includes the target forward packet source IP address, target forward packet destination IP address, target forward packet source port, target forward packet destination port, target reverse packet source IP address, target reverse packet destination IP address, target reverse packet source port, target reverse packet destination port, target Layer 3 protocol type, target Layer 4 protocol type, target SNAT address, target DNAT address, target SNAT port, and target DNAT port. The target Layer 3 protocol type can be a network layer protocol type, and the target Layer 4 protocol type can be a transport layer protocol type. Using the five-tuple information in the target information, it can be quickly determined whether the target information matches the pre-stored information in the initial table of the southbound database. If they do not match, it means that the target information has not been stored in the initial table of the southbound database before, and a target row needs to be added to the initial table of the southbound database to include the target information, resulting in an updated table of the southbound database containing the target information. In this way, the target information can be quickly stored in the southbound database, ensuring that the updated table of the southbound database is up-to-date and comprehensive.

[0050] It should be noted that, when the target packet is a forward packet, the source IP address mentioned above refers to the source IP address of the target forward packet, and the destination IP address mentioned above refers to the destination IP address of the target forward packet. The target IP address group mentioned above includes the source IP address and the destination IP address of the target forward packet; the source port mentioned above refers to the source port of the target forward packet, and the destination port mentioned above refers to the destination port of the target forward packet. The target port group mentioned above includes the source port and the destination port of the target forward packet; the target information includes the source IP address, destination IP address, source port, destination port, target Layer 4 protocol type, target Layer 3 protocol type, target SNAT address, target DNAT address, target SNAT port, and target DNAT port; the 5-tuple information includes the source IP address, destination IP address, source port, destination port, and target Layer 4 protocol type. When the target packet is a reverse packet, the source IP address mentioned above is the source IP address of the target reverse packet, the destination IP address mentioned above is the destination IP address of the target reverse packet, and the target IP address group mentioned above includes the source IP address and the destination IP address of the target reverse packet; the source port mentioned above is the source port of the target reverse packet, the destination port mentioned above is the destination port of the target reverse packet, and the target port group mentioned above includes the source port and the destination port of the target reverse packet; the target information includes the source IP address, destination IP address, source port, destination port, target Layer 4 protocol type, target Layer 3 protocol type, target SNAT address, target DNAT address, target SNAT port, and target DNAT port; the 5-tuple information includes the source IP address, destination IP address, source port, destination port, and target Layer 4 protocol type.

[0051] It should also be noted that when the 5-tuple information includes the source IP address, destination IP address, source port, destination port, and Layer 4 protocol type of the target forward packet, the first matching result can be determined as a match between the 5-tuple information and the corresponding information in the pre-stored information only if the source IP address, destination IP address, and Layer 4 protocol type are the same as their corresponding information in the pre-stored information, or if the source port, destination port, and Layer 4 protocol type are the same as their corresponding information in the pre-stored information. Otherwise, it is considered a mismatch. If the five-tuple information includes the target reverse packet source IP address, target reverse packet destination IP address, target reverse packet source port, target reverse packet destination port, and target Layer 4 protocol type, the first matching result can be determined as a match between the five-tuple information and the corresponding information in the pre-stored information only if the target reverse packet source IP address, target reverse packet destination IP address, and target Layer 4 protocol type are the same as the corresponding information in the pre-stored information, or if the target reverse packet source port, target reverse packet destination port, and target Layer 4 protocol type are the same as the corresponding information in the pre-stored information. Otherwise, it is a mismatch.

[0052] As an optional embodiment, when the target information includes quintuple information, after determining whether the quintuple information matches the corresponding item information in the pre-stored information in the initial table of the southbound database and obtaining a first matching result, the method further includes: when the target information includes other information, and when the first matching result indicates that the quintuple information matches the corresponding item information in the pre-stored information, determining the predetermined row in the pre-stored information that matches the quintuple information, and the predetermined row information in the predetermined row, wherein the other information refers to the information in the target information other than the quintuple information; determining whether each item in the other information matches the corresponding item information in the predetermined row information to obtain a second matching result; and when the second matching result indicates that each item in the other information does not match the corresponding item information in the predetermined row information, updating the predetermined row in the initial table of the southbound database to the target row to obtain an updated table of the southbound database, wherein the target row includes the target information.

[0053] In this embodiment, if the first matching result shows a match between the quintuple information and the corresponding item in the pre-stored information, then it is determined whether each item in the target information (excluding the quintuple information) matches the corresponding item in the predetermined row information, thus obtaining a second matching result. If the second matching result shows a mismatch between each item in the other information and the corresponding item in the predetermined row information, that is, the initial table of the southbound database stores the target information, but the target information has changed and needs to be updated, the predetermined row including the target information needs to be updated in the initial table of the southbound database, resulting in an updated table of the southbound database including the target information. In this way, the target information can be quickly updated into the corresponding position of the updated table of the southbound database, ensuring that the updated table is accurate and avoiding the storage of invalid information that needs updating in the updated table, thus saving space occupied by the updated table.

[0054] It should be noted that, when the target packet is a forward packet, the target information includes the target forward packet source IP address, the target forward packet destination IP address, the target forward packet source port, the target forward packet destination port, the target Layer 4 protocol type, the target Layer 3 protocol type, the target SNAT address, the target DNAT address, the target SNAT port, and the target DNAT port; the 5-tuple information includes the target forward packet source IP address, the target forward packet destination IP address, the target forward packet source port, the target forward packet destination port, and the target Layer 4 protocol type; other information includes the target Layer 3 protocol type, the target SNAT address, the target DNAT address, the target SNAT port, and the target DNAT port. When the target packet is a reverse packet, the target information includes the target reverse packet source IP address, the target reverse packet destination IP address, the target reverse packet source port, the target reverse packet destination port, the target Layer 4 protocol type, the target Layer 3 protocol type, the target SNAT address, the target DNAT address, the target SNAT port, and the target DNAT port; the 5-tuple information includes the target reverse packet source IP address, the target reverse packet destination IP address, the target reverse packet source port, the target reverse packet destination port, and the target Layer 4 protocol type; other information includes the target Layer 3 protocol type, the target SNAT address, the target DNAT address, the target SNAT port, and the target DNAT port.

[0055] As an optional embodiment, changing the initial table of the southbound database to the updated table of the southbound database based on the target information includes: calling the target interface; and changing the initial table of the southbound database to the updated table of the southbound database based on the target interface and the target information.

[0056] In this embodiment, the target interface may include the conntrack API interface in the CT module, that is, an API interface that can implement the connection tracking function. The local controller can call the target interface and, based on the target interface and target information, change the initial table of the southbound database to the updated table of the southbound database. Since the API interface can be a set of predefined programs or protocols, the operation of changing the initial table of the southbound database to the updated table of the southbound database based on the target information in the optional embodiment of this application can be implemented by predefining the API interface. Therefore, by directly calling the API interface, the above steps can be implemented quickly and conveniently, thereby enabling flexible, accurate, and rapid changes to the initial table of the southbound database to the updated table of the southbound database.

[0057] Figure 2 This is a flowchart of the second information synchronization method according to an embodiment of the present invention, as follows: Figure 2 As shown, the method includes the following steps:

[0058] In step S202, when the initial table of the southbound database in the southbound database is updated to the updated table of the southbound database, multiple local controllers obtain the updated table of the southbound database respectively. The updated table of the southbound database is obtained by changing the initial table of the southbound database according to the target information. The target information is obtained according to the content of the target event when any one of the multiple local controllers listens to the connection tracking CT module and a target event occurs. The target event is that the initial session information in the CT module has changed.

[0059] In the technical solution provided by step S202 of the present invention, since multiple local controllers are connected to the southbound database, multiple local controllers can know in a timely manner that the initial table of the southbound database in the southbound database has been updated. As a result, multiple local controllers can obtain the updated table of the southbound database in a timely manner, ensuring that multiple local controllers can obtain the same session information in a timely manner.

[0060] In step S204, multiple local controllers update tables based on the southbound database to determine the target session information;

[0061] In the technical solution provided in step S204 of the present invention, the target session information is all the session information in the southbound database update table. Multiple local controllers can quickly determine the same target session information based on the southbound database update table, thereby avoiding the loss caused by different session information in different local controllers.

[0062] In step S206, multiple local controllers send target session information to their respective load balancing modules.

[0063] In the technical solution provided by step S206 of the present invention, since the target session information in multiple local controllers is the same, after multiple local controllers send the target session information to their respective load balancing modules, the target session information in multiple load balancing modules is the same, thereby achieving the effect of timely synchronization of information in multiple load balancing modules. When a packet arrives at the OVS, the corresponding load balancing module, since the target session information in the load balancing module is the same as the session information in the updated southbound database update table, and the session information in the southbound database update table is accurate and comprehensive, can accurately determine the path for transmitting the packet based on the accurate target session information, thereby transmitting the packet quickly and accurately. Since the session information in multiple load balancing modules can be synchronized in a timely manner, the consistency of multiple load balancing modules can be ensured. When one load balancing module is deleted, its corresponding OVS and local controller are also deleted synchronously, and other load balancing modules can perform the same operation as the deleted load balancing module without affecting the transmission of packets. For example, three load balancing modules in three local controllers are load balancing module a1, load balancing module b1, and load balancing module c1, corresponding to OVS a2, OVS b2, and OVS c2, respectively. The same message i will randomly arrive at any of the three OVSs. If message i has previously arrived at OVS a2 within a predetermined time, meaning that load balancing module a1 corresponding to OVS a2 has processed message i within the predetermined time, load balancing module a1 will contain session information j corresponding to message i. Based on session information j, load balancing module a1 can select an accurate complete transmission path x for message i, and OVS a2 will transmit message i according to the complete transmission path x. When load balancing module a1 is deleted, its corresponding OVS a1 is also deleted synchronously, and newly arriving message i cannot reach OVS a2. When message i arrives randomly at OVS b2, since the target session information in load balancing module b1 is the same as that in load balancing module a2 (i.e., load balancing module b1 also includes session information j corresponding to message i), load balancing module b1 can select an accurate complete transmission path x for message i based on session information j. OVS b2 then transmits message i according to the complete transmission path x. In other words, when a load balancing module is deleted, other load balancing modules can perform the same operations as the deleted load balancing module without affecting message transmission.

[0064] It's important to note that when adding a load balancing module, the local controller corresponding to the newly added module will promptly update the tables in the southbound database to determine the target session information and send this information to the new load balancing module. This ensures that the session information in the newly added load balancing module is identical to the session information in the existing load balancing modules, guaranteeing consistency.

[0065] Through steps S202 to S206, when the initial table in the southbound database is updated to the updated table, multiple local controllers obtain the updated table and determine the target session information based on it. Then, each local controller sends the target session information to its corresponding load balancing module. Because multiple local controllers promptly obtain the updated table and determine the target session information accordingly, they can send the target session information to the corresponding load balancing module in a timely manner, ensuring that the load balancing modules can obtain the latest target session information promptly. Since the target session information in multiple local controllers is determined by each local controller based on the updated tables in the southbound database, the target session information in multiple local controllers is the same. Furthermore, since each local controller sends the target session information to its corresponding load balancing module, the target session information in multiple load balancing modules is the same. This consistency among multiple load balancing modules achieves the technical effect of timely synchronization of session information in multiple load balancing modules, thereby solving the technical problem of difficulty in timely synchronization of session information when synchronizing session information of multiple load balancing modules in related technologies.

[0066] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0067] In related technologies, there is a technical problem that it is difficult to synchronize session information in a timely manner when synchronizing session information of multiple load balancing modules.

[0068] In view of this, an optional embodiment of the present invention provides an information synchronization method, which can synchronize the session information of multiple load balancing modules in a timely manner, ensuring the consistency of multiple load balancing modules, and enabling multiple load balancing modules to form a load balancing pool. Taking the arrival of a target packet at one of multiple OVSs, and the local controller corresponding to that OVS being local controller a, as an example, the entire process of the optional embodiment of this application will be described as follows:

[0069] S1, multiple local controllers listen to the connected tracking CT module respectively to obtain the target monitoring results;

[0070] S2, when the target packet sent by the white-box switch arrives at the OVS corresponding to local controller a, the target monitoring result obtained by local controller a is that the target event has occurred in the corresponding CT module. That is, the session information in the CT module has been changed by the OVS corresponding to local controller a. Local controller a obtains the target information based on the target event content. The target information includes the source IP address of the target forward packet, the destination IP address of the target forward packet, the source port of the target forward packet, the destination port of the target forward packet, the source IP address of the target reverse packet, the destination IP address of the target reverse packet, the source port of the target reverse packet, the destination port of the target reverse packet, the target Layer 3 protocol type, the target Layer 4 protocol type, the target SNAT address, the target DNAT address, the target SNAT port, and the target DNAT port.

[0071] It should be noted that when the target packet is a forward packet, the five-tuple information includes the source IP address of the target forward packet, the destination IP address of the target forward packet, the source port of the target forward packet, the destination port of the target forward packet, and the target Layer 4 protocol type. When the target packet is a reverse packet, the five-tuple information includes the source IP address of the target reverse packet, the destination IP address of the target reverse packet, the source port of the target reverse packet, the destination port of the target reverse packet, and the target Layer 4 protocol type.

[0072] S3, Local controller a determines whether the five-tuple information matches the corresponding item information in the pre-stored information in the initial table of the southbound database, and obtains the first matching result;

[0073] S4.1, if the first matching result is that the five-tuple information does not match the corresponding item information in the pre-stored information, add a target row including the target information to the initial table of the southbound database to obtain the updated table of the southbound database;

[0074] S4.2, if the first matching result is that the quintuple information matches the corresponding item information in the pre-stored information, determine the pre-defined row in the pre-stored information that matches the quintuple information, and the pre-defined row information in the pre-defined row; determine whether each item in the target information other than the quintuple information matches the corresponding item information in the pre-defined row information, and obtain the second matching result;

[0075] S4.2.1, if there is a mismatch between each item in the other information in the second matching result and the corresponding item in the predetermined row information, update the target row including the target information in the predetermined behavior in the initial table of the southbound database to obtain the updated table of the southbound database;

[0076] S4.2.2 If the second matching result shows that each item in the other information matches the corresponding item in the predetermined row information, it indicates that the target information is stored in the initial table of the southbound database. No operation needs to be performed on the initial table of the southbound database, and the initial table of the southbound database is determined to be the updated table of the southbound database.

[0077] It should be noted that if information in the southbound database update table that has been deleted for a period of time exceeding the predetermined duration is deleted, multiple local controllers will also obtain the southbound database update table separately.

[0078] S5, when the initial table of the southbound database in the southbound database is updated to the updated table of the southbound database, multiple local controllers obtain the updated table of the southbound database respectively, wherein the multiple local controllers include controller a.

[0079] S6, multiple local controllers update tables based on the southbound database to determine the target session information;

[0080] S7, multiple local controllers send target session information to the corresponding load balancing module respectively.

[0081] Through the above steps S1-S7, session information in multiple load balancing modules can be synchronized in a timely manner, ensuring that the session information in multiple load balancing modules is the same, thus making multiple load balancing modules consistent. When one of the multiple load balancing modules is deleted or a load balancing module is added, it will not affect the transmission of packets, thereby improving the transmission efficiency of packets.

[0082] It should be noted that the above process may involve two threads performing the corresponding operations. Figure 3 This is a schematic diagram illustrating the function of the local controller provided in an optional embodiment of the present invention, such as... Figure 3 As shown, the local controller has two threads, which perform the following operations:

[0083] The first thread is started. When the target message arrives at the OVS corresponding to the local controller, the conntrack API interface is called to determine the target information corresponding to the target message. The target information is stored in ct_flow_cache and the main thread is triggered to execute. The main thread is started and the conntrack API interface is called. Based on the target information in ct_flow_cache, the initial table of the southbound database in the southbound database is changed to the updated table of the southbound database.

[0084] It's important to note that the conntrack API is an interface within the CT module, and ct_flow_cache is a high-speed cache. Since the two threads each have their own corresponding operations, running them in a dual-threaded manner ensures that their operations are performed independently, preventing errors. ct_flow_cache ensures that neither thread can access the other's memory space, thus preventing memory corruption between the two threads.

[0085] Based on the above embodiments and optional embodiments, this application provides another optional implementation method. This optional implementation method provides a message transmission system, taking a system including a router, a white-box switch, three Open Virtual Switches (OVS), and four servers as an example, as described below:

[0086] Figure 4 This is a schematic diagram of a message transmission system provided by an optional embodiment of the present invention. Figure 4 In the middle, the four layers from top to bottom are: router, white-box switch, 3 OVS, and 4 servers, as follows: Figure 4 As shown, the router sends the target packet to the white-box switch. The white-box switch, based on the Policy Based Routing (PBR) table, sends the target packet to multiple OVSs via Overlay Equal Cost Multipath (ECMP). The multiple OVSs then transmit the packet according to the target session information in their respective load balancing modules. Since the multiple load balancing modules are consistent, and the target session information in the multiple load balancing modules is the same and up-to-date, the multiple OVSs can accurately transmit the packet based on the target session information in their respective load balancing modules, thus improving the efficiency and accuracy of packet transmission.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0089] Example 2

[0090] According to embodiments of the present invention, a system for implementing the above-described information synchronization method is also provided. Figure 5 This is a structural block diagram of an information synchronization system according to an embodiment of the present invention, such as... Figure 5 As shown, the system includes multiple target modules 502 and a southbound database 504, and all target modules 502 are connected to the southbound database 504. The system will be described in detail below.

[0091] Each of the multiple target modules 502 includes a virtual switch (OVS), a local controller, and a connection tracking (CT) module. The local controller includes a load balancing module.

[0092] In each of the multiple target modules 502, the OVS is used to receive messages, obtain target information based on the messages, and modify the initial session information in the corresponding CT module based on the target information.

[0093] In each of the multiple target modules 502, the local controller monitors the corresponding CT module and obtains the target monitoring result. If the target monitoring result indicates that a target event has been detected in the CT module, the target information is obtained based on the content of the target event. Based on the target information, the initial table of the southbound database in the southbound database 504 is changed to the updated table of the southbound database. The target event is a change in the initial session information in the CT module.

[0094] When the initial table in the southbound database 504 is updated to the updated table in the southbound database, the multiple local controllers in the multiple target modules 502 obtain the updated table in the southbound database, determine the target session information based on the updated table, and send the target session information to the corresponding load balancing modules respectively, so that the target session information in the multiple load balancing modules is the same.

[0095] As an optional embodiment, the system also includes a white-box switch, wherein the OVS receives messages sent by the white-box switch, wherein the white-box switch includes the cloud-based open networking software SONiC white-box switch.

[0096] In this embodiment, the white-box switch can be configured with functional parameters, resulting in better performance. The SONiC (Software for Open Networking in the Cloud) white-box switch can serve as the first-level hardware load balancer in the information synchronization system. It can perform equal cost multi-path routing (ECMP) based on the Virtual Internet Protocol (VIP) address, accurately sending packets to the OVS.

[0097] As an optional embodiment, the load balancing module includes: a target-level flow table for the pipeline, wherein the pipeline includes multi-level flow tables.

[0098] In this embodiment, the pipeline includes multi-level flow tables. The target-level flow table of the pipeline can serve as a second-level software load balancer. That is, the target-level flow table of the pipeline can implement load balancing functionality, performing source-consistent load sharing based on the IP address and port number of the packets, which can better achieve the load balancing function.

[0099] It should be noted here that the multiple target modules 502 and the southbound database 504 are used to implement the steps involved in this application in the information synchronization method. The steps of the multiple ends are implemented in the same instances and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0100] Example 3

[0101] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the information synchronization method described above.

[0102] Example 4

[0103] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the above-described information synchronization methods.

[0104] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0105] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An information synchronization method, characterized by, The method comprises the following steps: The local controller listens to a connection tracking (CT) module to obtain a target monitoring result; In a case where the target monitoring result is that the CT module has a target event, the local controller obtains target information according to the target event content, wherein the target event is a change in initial session information in the CT module; The local controller changes an initial table in a southbound database to an updated table in the southbound database according to the target information, so that a load balancing module corresponding to each of a plurality of local controllers obtains target session information, wherein the target session information is obtained according to the updated table in the southbound database, and the updated table in the southbound database is obtained by each of the plurality of local controllers from the southbound database, wherein the target session information in each of the plurality of load balancing modules is the same, and the target session information is all session information in the updated table in the southbound database.

2. The method of claim 1, wherein, The step of changing the initial table in the southbound database to the updated table in the southbound database according to the target information comprises the following steps: In a case where the target information includes five-tuple information, it is determined whether the five-tuple information matches corresponding item information in pre-stored information in the initial table in the southbound database to obtain a first matching result, wherein the five-tuple information includes a target four-layer protocol type, a target IP address group, and a target port group, the target IP address group includes a source IP address and a destination IP address, and the target port group includes a source port and a destination port; In a case where the first matching result is that the five-tuple information cannot match the corresponding item information in the pre-stored information, a target row is added in the initial table in the southbound database to obtain the updated table in the southbound database, wherein the target row includes the target information, and the case where the five-tuple information cannot match the corresponding item information in the pre-stored information includes a case where the target four-layer protocol type does not match the corresponding item information in the pre-stored information, or a case where the target four-layer protocol type matches the corresponding item information in the pre-stored information, but the target IP address group does not match the corresponding item information in the pre-stored information, and the target port group does not match the corresponding item information in the pre-stored information.

3. The method of claim 2, wherein, After the step of determining whether the five-tuple information matches the corresponding item information in the pre-stored information in the initial table in the southbound database to obtain the first matching result, the method further comprises the following steps: In a case where the target information includes other information, in a case where the first matching result is that the five-tuple information matches the corresponding item information in the pre-stored information, a predetermined row in the pre-stored information that matches the five-tuple information and predetermined row information in the predetermined row are determined, wherein the other information is information in the target information other than the five-tuple information; It is determined whether each item of information in the other information matches corresponding item information in the predetermined row information to obtain a second matching result. In a case that the second matching result is that each item of information in the other information does not match a corresponding item of information in the predetermined row information, the target row in the southbound database initial table is updated to obtain a southbound database updated table, wherein the target information is included in the target row.

4. The method of claim 1, wherein, The changing the southbound database initial table in the southbound database into the southbound database updated table according to the target information comprises: calling a target interface; changing the southbound database initial table in the southbound database into the southbound database updated table according to the target interface and the target information.

5. An information synchronization method characterized by, comprises: In a case that the southbound database initial table in the southbound database is updated to the southbound database updated table, a plurality of local controllers respectively acquire the southbound database updated table, wherein the southbound database updated table is obtained by changing the southbound database initial table according to target information, the target information is obtained according to target event content in a case that a target event occurs in a connection tracking (CT) module listened to by any one of the plurality of local controllers, and the target event is a change of initial session information in the CT module; the plurality of local controllers respectively determine target session information according to the southbound database updated table; the plurality of local controllers respectively send the target session information to corresponding load balancing modules, wherein target session information in a plurality of load balancing modules is the same, and the target session information is all session information in the southbound database updated table.

6. An information synchronization system, characterized by comprises: a plurality of target modules, a southbound database, each target module in the plurality of target modules comprises an open virtual switch (OVS), a local controller, and a connection tracking (CT) module, and the local controller comprises a load balancing module, wherein the OVS in each target module in the plurality of target modules is configured to receive a packet, acquire target information according to the packet, and change initial session information in a corresponding CT module according to the target information, the local controller in each target module in the plurality of target modules is configured to listen to a corresponding CT module to obtain a target listening result, in a case that the target listening result is that the CT module is listened to to have a target event, obtain target information according to target event content, and change a southbound database initial table in the southbound database into a southbound database updated table according to the target information, wherein the target event is a change of initial session information in the CT module, In a case where a southbound database initial table of the plurality of local controllers in the plurality of target modules in the southbound database is updated to a southbound database update table, the plurality of local controllers acquire the southbound database update table, determine target session information according to the southbound database update table, and respectively send the target session information to corresponding load balancing modules, so that the target session information in the plurality of load balancing modules is the same, wherein the target session information in the plurality of load balancing modules is the same, and the target session information is all session information in the southbound database update table.

7. The system of claim 6, wherein, The system further includes a white box switch, wherein The OVS receives the packet sent by the white box switch, and the white box switch includes a SONiC white box switch.

8. The system of claim 6, wherein, The load balancing module includes a target stage flow table of a pipeline, and the pipeline includes multiple stage flow tables.

9. An electronic device, comprising: The system further includes a processor and a memory. The processor is configured to execute the instructions to implement the information synchronization method of any one of claims 1 to 5. When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the information synchronization method of any one of claims 1 to 5. ​ 10. A computer-readable storage medium, characterized in that, ​

Citation Information

Patent Citations

  • Session information synchronization method, device and system

    CN107846433A

  • Data transmission method and device based on load balancing, equipment and storage medium

    CN113691460A