Cluster data synchronization method, system and device and storage medium

Abstract

The invention discloses a cluster data synchronization method, system and device and a storage medium. The method comprises the following steps: receiving a synchronization request of a user through a first node in a cluster; wherein the first node is any node in the cluster; responding to the synchronization request, and determining a node set appointed by the user to perform data synchronization and file data needing to be synchronized; wherein the node set comprises at least one target node needing data synchronization; establishing a connection between the first node and each of the target nodes; and reading the file data through the first node and then transmitting the file data to each target node. According to the method in the embodiment of the invention, the cluster data synchronization efficiency can be effectively improved, the processing time consumption is reduced, and the related data are synchronized through concurrent transmission, so that the error probability can be reduced, and the data synchronization accuracy can be improved. The method can be widely applied to the field of financial science and technology.

Description

Technical Field

[0001] This application relates to the field of fintech, and in particular to a method, system, device and storage medium for cluster data synchronization. Background Art

[0002] In the financial field, ensuring data consistency and real-time performance among various systems is a key requirement. With the increasing complexity and scale of financial institutions' operations, the traditional single-machine processing mode has become difficult to meet the requirements of high concurrency, low latency, and high reliability. Therefore, cluster technology has been widely used to improve the performance, availability, and scalability of systems. A cluster refers to a group of computers connected together through a network, which work together to provide services with high availability, high performance, or high scalability. Cluster technology can be applied to various scenarios, such as database management, file storage, and computing task processing. By combining multiple computers (nodes) into a cluster, it is possible to achieve task load processing capabilities, data redundancy, and fault recovery capabilities that are difficult to achieve with a single computer. In a cluster, each individual computer is usually referred to as a node or instance, and these nodes work together to complete specific tasks.

[0003] In the related art, when controlling a cluster, it is necessary to perform unified scheduling on each node. For example, when it is necessary to update the related functions of the cluster nodes, it is often necessary to synchronously adjust the configuration content of each node (such as adding configurations, deleting configurations, or modifying configurations, etc.) because the financial system usually requires high availability, high performance, and strict security guarantees. However, it has been found in current practical applications that the console applications configured on each node in the cluster are different, and some nodes do not even have a console application configured. When synchronizing data among the nodes in the cluster, it is only possible to set and adjust the relevant data one by one at the dimension of each node, resulting in very low efficiency of cluster data synchronization and prone to errors.

[0004] In summary, the problems existing in the related art need to be solved urgently. Summary of the Invention

[0005] The purpose of this application is to solve at least to some extent one of the technical problems existing in the related art.

[0006] To this end, an object of an embodiment of this application is to provide a method for cluster data synchronization, which can effectively improve the efficiency of cluster data synchronization, reduce the processing time-consuming, and the relevant data is synchronized through concurrent transmission, which can reduce the probability of errors and improve the accuracy of data synchronization.

[0007] To achieve the above technical purpose, the technical solutions adopted in the embodiments of this application include:

[0008] On the one hand, an embodiment of the present application provides a cluster data synchronization method, including:

[0009] Receiving a synchronization request from a user through a first node in the cluster; wherein, the first node is any node in the cluster;

[0010] Responding to the synchronization request, determining a set of nodes specified by the user for data synchronization and file data to be synchronized; wherein, the set of nodes includes at least one target node for data synchronization;

[0011] Establishing connections between the first node and each of the target nodes;

[0012] After reading the file data through the first node, concurrently transmitting it to each of the target nodes.

[0013] In addition, according to the cluster data synchronization method of the above embodiment of the present application, the following additional technical features may also be included:

[0014] Further, in an embodiment of the present application, the method further includes:

[0015] Installing and deploying the same console application on each node of the cluster; wherein, the console application is used to execute data synchronization tasks;

[0016] Configuring the same authentication information for each node.

[0017] Further, in an embodiment of the present application, the authentication information includes account verification information; and the responding to the synchronization request includes:

[0018] Detecting the login account name and password information of the user;

[0019] Authenticating the login account name and the password information according to the account verification information;

[0020] When it is determined that the authentication of the login account name and the password information passes, responding to the synchronization request.

[0021] Further, in an embodiment of the present application, the responding to the synchronization request, determining the set of nodes specified by the user for data synchronization and the file data to be synchronized, includes:

[0022] Responding to the synchronization request, presenting the node list and file list of the cluster to the user;

[0023] Responding to a first selection operation of the user on the node list, determining the set of nodes specified by the user for data synchronization;

[0024] In response to the user's second selection operation on the file list, determine the file data specified by the user for synchronization.

[0025] Further, in an embodiment of the present application, establishing a connection between the first node and each of the target nodes includes:

[0026] Establish TCP connections between the first node and each of the target nodes through multi-threaded concurrency;

[0027] Authenticate the TCP connections based on the authentication information.

[0028] Further, in an embodiment of the present application, responding to the synchronization request and determining the file data to be synchronized includes:

[0029] Receive, by the first node, the file data uploaded by the user along with the synchronization request;

[0030] Determine the file data as the file data to be synchronized.

[0031] Further, in an embodiment of the present application, the method further includes:

[0032] Detect the synchronization time taken for a single target node;

[0033] Detect the total execution time of the data synchronization task corresponding to the synchronization request;

[0034] Determine performance data based on the ratio of the total execution time and the synchronization time taken;

[0035] Feed back the performance data to the user.

[0036] On the other hand, an embodiment of the present application further provides a cluster data synchronization system, including:

[0037] A receiving unit, configured to receive a synchronization request from a user through a first node in the cluster; wherein, the first node is any node in the cluster;

[0038] A response unit, configured to respond to the synchronization request and determine a set of nodes specified by the user for data synchronization and the file data to be synchronized; wherein, the set of nodes includes at least one target node that needs to perform data synchronization;

[0039] An establishing unit, configured to establish connections between the first node and each of the target nodes;

[0040] A transmission unit, configured to concurrently transmit the file data to each of the target nodes after reading the file data through the first node.

[0041] On the other hand, an embodiment of the present application provides a computer device, including:

[0042] At least one processor;

[0043] At least one memory, configured to store at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the above-mentioned cluster data synchronization method.

[0045] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to implement the above-mentioned cluster data synchronization method when executed by the processor.

[0046] Advantages and beneficial effects of the present application will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the present application:

[0047] A cluster data synchronization method disclosed in an embodiment of the present application receives a synchronization request of a user through a first node in a cluster; wherein, the first node is any node in the cluster; responds to the synchronization request to determine a set of nodes specified by the user for data synchronization and file data to be synchronized; wherein, the set of nodes includes at least one target node that needs to perform data synchronization; establishes a connection between the first node and each of the target nodes; and concurrently transmits the file data to each of the target nodes after reading the file data through the first node. The method in the embodiment of the present application can effectively improve the efficiency of cluster data synchronization, reduce the processing time, and the relevant data is synchronized through concurrent transmission, which can reduce the probability of errors and improve the accuracy of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the accompanying drawings related to the technical solutions in the embodiments of the present application or the prior art. It should be understood that the accompanying drawings in the following introduction are only for conveniently and clearly presenting some embodiments of the technical solutions of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0049] Figure 1 It is a schematic diagram of an implementation environment of a cluster data synchronization method provided in an embodiment of the present application;

[0050] Figure 2 It is a schematic flowchart of a cluster data synchronization method provided in an embodiment of the present application;

[0051] Figure 3 It is a schematic flowchart of a process for responding to a synchronization request provided in an embodiment of the present application;

[0052] Figure 4 It is a schematic flowchart of a process for determining a set of nodes and file data that need to perform data synchronization provided in an embodiment of the present application;

[0053] Figure 5 It is a schematic flowchart of a process for establishing a connection between a first node and a target node provided in an embodiment of the present application;

[0054] Figure 6 It is a schematic flowchart of a process for determining performance data of a data synchronization task provided in an embodiment of the present application;

[0055] Figure 7 It is a schematic structural diagram of a cluster data synchronization system provided in an embodiment of the present application;

[0056] Figure 8 It is a schematic structural diagram of a computer device provided in an embodiment of the present application. Detailed implementation manners

[0057] The present application will be further described below in conjunction with the accompanying drawings of the specification and specific embodiments. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0058] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] First, several terms involved in the present application are parsed:

[0061] 1) Cluster: A group of computers connected together through a network that work together to provide services with high availability, high performance, or high scalability.

[0062] 2) Streaming Media, a technology for continuously transmitting audio and video content over a network, enabling end devices to start playing media files before the data is fully downloaded.

[0063] 3) Transmission Control Protocol (TCP), a connection-oriented, reliable, byte-stream-based transport layer communication protocol. It is responsible for reliably delivering data from one application end to another.

[0064] 4) Secure Shell (SSH), a network protocol used to encrypt the security of remote logins and other network services. SSH is mainly used to provide secure remote login and file transfer functions in an insecure network environment.

[0065] In the financial field, ensuring data consistency and real-time performance among various systems is a key requirement. With the increasing complexity and scale of financial institutions' operations, the traditional single-machine processing mode has become difficult to meet the requirements of high concurrency, low latency, and high reliability. Therefore, cluster technology has been widely applied to improve system performance, availability, and scalability. A cluster refers to a group of computers connected together via a network, which work together to provide services with high availability, high performance, or high scalability. Cluster technology can be applied to various scenarios, such as database management, file storage, and computing task processing. By combining multiple computers (nodes) into a cluster, it is possible to achieve task load processing capabilities, data redundancy, and fault recovery capabilities that are difficult to achieve with a single computer. In a cluster, each individual computer is usually referred to as a node or instance, and these nodes work together to complete specific tasks.

[0066] In related technologies, when controlling a cluster, it is necessary to perform unified scheduling of each node. For example, when it is necessary to update the relevant functions of the cluster nodes, it is often necessary to synchronously adjust the configuration content of each node (such as adding configuration, deleting configuration, or modifying configuration, etc.) because the financial system usually requires high availability, high performance, and strict security guarantees. However, it has been found in current practical applications that the console applications configured on each node within the cluster are different, and some nodes do not even have a console application configured. When synchronizing data among the nodes within the cluster, it is only possible to set and adjust the relevant data one by one at the dimension of each node, resulting in very low efficiency of cluster data synchronization and prone to errors.

[0067] To solve the problems existing in the related art, the embodiments of the present application provide a cluster data synchronization method, system, device, and storage medium. The first node in the cluster receives a synchronization request from a user; wherein, the first node is any node in the cluster; the synchronization request is responded to, and the node set specified by the user for data synchronization and the file data to be synchronized are determined; wherein, the node set includes at least one target node that needs to perform data synchronization; a connection is established between the first node and each of the target nodes; after the file data is read by the first node, it is concurrently transmitted to each of the target nodes. The method in the embodiments of the present application can effectively improve the efficiency of cluster data synchronization, reduce the processing time, and moreover, the relevant data is synchronized through concurrent transmission, which can reduce the probability of errors and improve the accuracy of data synchronization.

[0068] Applying the method in the embodiments of the present application to the financial field can effectively improve the speed and efficiency of transaction processing; moreover, the financial industry has extremely high requirements for data accuracy and consistency, especially when it comes to cross-regional and cross-system transactions. Through the method in the embodiments of the present application, the configuration content of the system can be updated through concurrent transmission and real-time synchronization to ensure that the data in all relevant systems is consistent, and reduce risks caused by inconsistent data, such as duplicate transactions and incorrect bookkeeping.

[0069] Of course, the cluster data synchronization method provided in the embodiments of the present application is not limited to being implemented in a financial scenario and can also be executed in some other application scenarios:

[0070] Exemplarily, in some embodiments, the cluster data synchronization method in the embodiments of the present application can be applied to the scenario of a shopping application. For example, in a shopping application, it is often necessary to ensure that the data related to product inventory is consistent among multiple cluster servers to avoid over-selling; or ensure that account information (such as address, payment information) is consistent among multiple servers to prevent order anomalies. The cluster data synchronization method in the embodiments of the present application can be applied to the above shopping application. In this way, when using the shopping application on different terminal devices, the smooth progress of transactions can be better guaranteed, which is beneficial to improving the shopping experience.

[0071] Exemplarily, in some embodiments, the method provided in the embodiments of the present application can be applied to the scenario of streaming media applications. Currently, with the continuous development of information technology, there are many streaming media applications. Streaming Media is a technology for continuously transmitting audio and video content over the network, enabling terminal devices to start playing media files before the data is fully downloaded. Streaming media scenarios typically involve real-time or near-real-time content transmission, which may include online videos, audio live broadcasts, Internet TV, distance education courses, game live broadcasts, video conferences, etc.

[0072] The data synchronization method provided in the embodiments of the present application can be applied to the scenario of streaming media. By building a cluster application, a streaming media service can be implemented. To ensure that the functions of each node in the cluster are consistent, the cluster data synchronization method in the embodiments of the present application can be used to perform unified function configuration on each node to ensure the browsing experience of the streaming media service.

[0073] Of course, it should be noted that the above application scenarios only serve as examples and do not mean to limit the actual application of the method in the embodiments of the present application. Those skilled in the art can understand that in different application scenarios, the method provided in the embodiments of the present application can be used to perform specified tasks.

[0074] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the implementation environment of the cluster data synchronization method provided in the embodiments of the present application. The main software and hardware entities of this implementation environment mainly include a user terminal 110 and a server 120, and the user terminal 110 is communicatively connected to the server 120. Among them, this cluster data synchronization method can be configured to be executed on the server 120 side and implemented based on the data interaction between the user terminal 110 and the server 120.

[0075] Exemplarily, for example, in some scenarios, the user terminal 110 can be a device used by the manager or maintainer of the cluster, and the server 120 can be a server device in the cluster. When implementing the method in the embodiments of the present application, the manager or maintainer of the cluster can send a synchronization request to the server 120 through the user terminal 110. The server 120 can act as the first node in the cluster, respond to the synchronization request, determine the set of nodes specified by the user for data synchronization and the file data to be synchronized. Then establish connections with other target nodes in the cluster, read the file data to be transmitted, and concurrently transmit it to each target node, thereby realizing the synchronization processing of the cluster data.

[0076] Specifically, the user terminal 110 in the present application may include, but is not limited to, any one or more of a smart watch, a smart phone, a computer, a personal digital assistant (PDA), a smart voice interaction device, a smart home appliance, or a vehicle-mounted terminal. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0077] A communication connection may be established between the user terminal 110 and the server 120 through a wireless network or a wired network. The wireless network or wired network uses standard communication technologies and / or protocols. The network may be set to the Internet or any other network, such as any combination including, but not limited to, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network, or a virtual private network.

[0078] Of course, it can be understood that Figure 1 the implementation environment in Figure 1 is only an optional application scenario of the cluster data synchronization method provided in the embodiments of the present application. The actual application is not fixed to the

[0079] Next, in combination with the Figure 1 shown implementation environment, the cluster data synchronization method provided in the embodiments of the present application will be described in detail.

[0080] First, please refer to Figure 2 , Figure 2 which is a schematic flowchart of the cluster data synchronization method provided in the embodiments of the present application. Figure 2 The shown cluster data synchronization method can be applied to relevant computer devices in the server 120, but is not limited to the above form. Figure 2 The method in

[0081] Step 210: Receive a synchronization request from a user through a first node in the cluster; wherein, the first node is any node in the cluster;

[0082] Step 220: Respond to the synchronization request, determine the set of nodes specified by the user for data synchronization and the file data to be synchronized; wherein, the set of nodes includes at least one target node that needs to perform data synchronization.

[0083] Step 230: Establish connections between the first node and each of the target nodes.

[0084] Step 240: Read the file data through the first node and concurrently transmit it to each of the target nodes.

[0085] In the embodiments of the present application, a method for cluster data synchronization is provided. This method can effectively improve the efficiency of cluster data synchronization, reduce the processing time consumption, and the relevant data is synchronized through concurrent transmission, which can reduce the probability of errors and improve the accuracy of data synchronization.

[0086] In the embodiments of the present application, the cluster can be a cluster used by relevant institutions in the financial field. The nodes in the cluster can be any servers responsible for processing financial services, such as application servers (such as transaction processing systems, customer relationship management systems (CRM), risk management platforms, etc.), database servers (used to store and manage financial data, including customer information, transaction records, account balances, market data, etc.), cache servers (used to cache frequently accessed data, such as the latest market quotes, popular trading pairs, etc.), and other various computer devices. The present application does not make any restrictions on this.

[0087] Specifically, when executing the method for cluster data synchronization provided in the embodiments of the present application, the first node in the cluster can receive the user's synchronization request, then respond to the synchronization request, determine the set of nodes specified by the user for data synchronization and the file data to be synchronized. Then, connections can be established between the first node and each target node that needs to perform data synchronization, and the file data is read through the first node and concurrently transmitted to each target node, thereby completing the data synchronization of the cluster.

[0088] Next, in combination with the overview of the method for cluster data synchronization provided in the embodiments of the present application, each step involved in the method for cluster data synchronization will be introduced and explained.

[0089] In step 210, when the user has a need for cluster data synchronization, a synchronization request can be sent to any node in the cluster. In the embodiments of the present application, the node to which the user sends the synchronization request is denoted as the first node. The specific situation where the user has a need for cluster data synchronization is not limited in the present application. Exemplarily, for example, in some embodiments, it may be that the cluster needs to perform configuration file synchronization, that is, to maintain the same configuration file on all nodes in the cluster to ensure consistent settings on each node. At this time, the user can initiate a synchronization request for this need. In some embodiments, it may be that the cluster needs to perform database synchronization, that is, to synchronize data between multiple database nodes to ensure data consistency and high availability. At this time, the user can initiate a synchronization request for this need. In some embodiments, it may also be that the cluster needs to perform file system synchronization or configuration management synchronization.

[0090] Of course, it can be understood that the need for cluster data synchronization can be determined according to the actual situation. When there is any type of need, the user can send a synchronization request to the first node to trigger cluster data synchronization.

[0091] In the embodiments of the present application, for the synchronization request, its data format can be customized according to specific application scenarios and requirements, and usually includes the following parts: Request header: It may include request type (GET, POST, etc.), content type (such as JSON, XML, etc.), authentication information (token, API key, etc.). Request body: Specific data carried according to business requirements, such as data objects to be synchronized, operation instructions, etc. Metadata: It can include timestamp, source IP address, request ID, etc., for purposes such as logging and tracking requests. In the embodiments of the present application, the specific data content of the synchronization request is not limited.

[0092] For each node in the cluster, they can configure relevant listeners to receive the user's synchronization request. For the relevant protocol process, it can be implemented with reference to the prior art, and the present application does not limit this.

[0093] In step 220, after the first node receives the user's synchronization request, it can respond to the synchronization request, determine the set of nodes specified by the user for data synchronization and the file data to be synchronized. In the embodiments of the present application, the nodes specified by the user for data synchronization are denoted as target nodes. It can be understood that in the set of nodes, it can include one or more target nodes for data synchronization. In the embodiments of the present application, the number of target nodes in the set of nodes is not limited and can be flexibly specified according to user needs. Exemplarily, in some embodiments, the set of nodes can include some nodes in the cluster. Of course, in other embodiments, the set of nodes can also include all nodes in the cluster.

[0094] In step 220, according to the synchronization request, it is also possible to determine the file data specified by the user that needs to be synchronized. The present application does not limit the specific type of file data. Exemplarily, in some embodiments, the file data specified by the user that needs to be synchronized may be the configuration file data of the cluster. For example, the file data may store configuration parameters of relevant application programs, such as database connection strings, API keys, environment variables, etc. In some embodiments, the file data specified by the user that needs to be synchronized may be log file data. The log file data can be used to record various information during the operation of the application program, including error logs, access logs, etc. Synchronizing the log file data to each target node can help monitor and debug problems in the cluster and provide historical data for analysis. In some embodiments, the file data specified by the user that needs to be synchronized may also be ordinary data, such as data generated or processed by the cluster, such as user data, transaction records, etc. The present application does not limit this.

[0095] In step 230, after determining each target node, a connection can be established between the first node and the target nodes. The present application does not limit the specific implementation manner of establishing the connection. Exemplarily, in some embodiments, a data connection relationship between the first node and the target nodes can be established through relevant network transmission protocols. For example, the TCP protocol, SSH protocol, etc. can be used. In some embodiments, configuration management tools (such as Ansible, Puppet, Chef) can help automate the connection between cluster nodes. In some embodiments, message queues (such as RabbitMQ, Kafka) etc. can be used to implement the data connection between the first node and the target nodes.

[0096] Specifically, taking the establishment of a data connection relationship between the first node and the target nodes using the TCP protocol as an example, its application process generally includes: Before data transmission, TCP will first establish a connection (three-way handshake). After the data transmission is completed, the connection will be disconnected (four-way handshake). For each data transmission task, large chunks of data will be split into multiple small segments (referred to as segments or packets). Each segment has a sequence number, and the receiving party (target node) can reorganize the data according to the sequence number. After receiving the data segment, the receiving party will send an acknowledgment reply (ACK), and the sending party (first node) will judge which data has been successfully delivered based on these acknowledgments. Through the sliding window mechanism, the sending party can control the sending rate to avoid the receiving party being unable to process too much data.

[0097] In step 240, after establishing the connection relationship between the first node and each target node, the first node can read the file data and then concurrently transmit it to each target node. In the implementation of this application, concurrent transfer refers to a technology that simultaneously performs multiple data transfer tasks at the same time. This technology can significantly improve the efficiency and speed of data transfer, especially when the network bandwidth is sufficient.

[0098] In the embodiment of this application, for the first node, it can start multiple threads in one process, and each thread is responsible for an independent transfer task. This can make full use of the computing power of the multi-core processor and improve the overall transfer efficiency. Through the concurrent transfer method, multiple transfer tasks can be carried out simultaneously, making full use of the network bandwidth and improving the overall transfer speed. Moreover, if a certain transfer task fails, other tasks will not be affected and can continue, which improves the stability and reliability of the system.

[0099] Furthermore, in the embodiment of this application, the first node can be any node in the cluster, that is, the data synchronization task can be triggered and executed by any node in the cluster. This decentralized implementation method can enhance the robustness and reliability of the system. Since there is no single central node, even if some nodes fail or are attacked, the entire system can still operate normally and execute the relevant cluster data synchronization tasks, which greatly improves the reliability and stability of the system. Moreover, this application eliminates the intermediate link in the traditional centralized system, reducing the transaction cost and management cost. In specific implementation, the network scale can be expanded by simply adding new nodes without complex configuration and management.

[0100] It can be understood that a cluster data synchronization method provided in the embodiment of this application includes: receiving a synchronization request from a user through a first node in the cluster, where the first node is any node in the cluster; responding to the synchronization request to determine the set of nodes specified by the user for data synchronization and the file data to be synchronized, where the set of nodes includes at least one target node that needs to perform data synchronization; establishing a connection between the first node and each of the target nodes; and after reading the file data through the first node, concurrently transmitting it to each of the target nodes. The method in the embodiment of this application can effectively improve the efficiency of cluster data synchronization, reduce the processing time, and the relevant data is synchronized through concurrent transfer, which can reduce the probability of errors and improve the accuracy of data synchronization.

[0101] Specifically, in some embodiments, the cluster data synchronization method provided in the embodiment of this application further includes:

[0102] Install and deploy the same console application on each of the nodes in the cluster; wherein, the console application is used to execute data synchronization tasks;

[0103] Configure the same authentication information for each of the nodes.

[0104] In the embodiments of the present application, in order to achieve efficient data synchronization and management, the same console application can be installed and deployed on each node, and the same authentication information can be configured for these nodes. It can be understood that in the financial field, the requirements for security and compliance are extremely high. Especially when it comes to system configuration updates or data synchronization, relevant operations must be ensured to comply with security standards and compliance requirements before they can be executed. Therefore, in the embodiments of the present application, an authentication policy can be configured on the nodes to achieve secure and reliable permission management.

[0105] Specifically, in the embodiments of the present application, through a unified console application, an administrator can conveniently monitor and manage each node in the cluster, and the operations implemented are more unified, without the need to adapt to the operation modes of each different console application. In the embodiments of the present application, the console application can facilitate graphical operations. For example, functions such as application management, data source, log monitoring, JMS messages, log configuration, system parameters, and JMS storage can be added, deleted, modified, etc. on the console application. The present application does not limit this. Specifically, for example, in some embodiments, the console application can be a Rockyas application, and the present application does not limit this.

[0106] In the embodiments of the present application, when installing and deploying the same console application on the nodes, first, a suitable data synchronization tool can be selected according to business requirements or the console application can be custom-developed, and the console application can be packaged into an executable file or a container image. Then, the operating systems and dependent environments of all nodes can be adjusted to be the same, and the installation script or command can be manually run on each node, or a configuration management tool (such as Ansible, Puppet, Chef) or a CI / CD tool (such as Jenkins) can also be used for automatic deployment. After the installation is completed, it can be verified on each node whether the console application is correctly installed and can run normally.

[0107] In the embodiments of the present application, the same authentication information can also be configured for the nodes to ensure that only authorized users or services can access and operate the console application. All nodes use the same authentication information, which simplifies management and maintenance. Specifically, in the embodiments of the present application, the authentication information can adopt account verification information in the form of a username and password, an API key, or an SSL / TLS certificate, etc. The present application does not limit this. For the authentication information, it can be written into a configuration file, and then the configuration file can be distributed to each node.

[0108] Specifically, in some embodiments, the authentication information includes account verification information; referring to Figure 3 , the response to the synchronization request includes:

[0109] Detect the user's login account name and password information;

[0110] Authenticate the login account name and the password information according to the account verification information;

[0111] When it is determined that the authentication of the login account name and the password information passes, respond to the synchronization request.

[0112] In the embodiments of the present application, in some cases, before responding to the user's synchronization request, it can be detected first. Specifically, the login account name and password information provided by the user can be detected. For example, the account name and password information required for login can be collected from the user through the login interface or other means. Then, a preliminary format check is performed on the input information, such as checking whether the account name is empty and whether the password length meets the requirements.

[0113] Then, the login account name and password information can be authenticated according to the account verification information. Exemplarily, in order to protect the security of user information, the password can be encrypted (such as using a hash algorithm), and then compared with the encrypted password stored in the database. If the two are consistent, the user identity is considered legal and data synchronization is allowed. Moreover, in the embodiments of the present application, a session can be created for the authenticated user, and a unique session ID can be assigned. This ID will be used to track the user's status during the entire access period. The user can initiate a synchronization request through this session, thereby triggering the cluster data synchronization task.

[0114] It can be understood that in the embodiments of the present application, by authenticating the user's synchronization request, the security of the system can be improved and the risk probability of being invaded can be reduced.

[0115] Specifically, in some embodiments, referring to Figure 4 , the response to the synchronization request to determine the node set specified by the user for data synchronization and the file data to be synchronized includes:

[0116] Respond to the synchronization request and display the node list and file list of the cluster to the user;

[0117] In response to the user's first selection operation on the node list, determine the node set specified by the user for data synchronization;

[0118] In response to the user's second selection operation on the file list, determine the file data specified by the user to be synchronized.

[0119] In the embodiments of the present application, when responding to a synchronization request, a node list of the current cluster and a related file list may be displayed to the user. Here, the node list may include information of all nodes in the cluster. Based on the node list, the user may perform a selection operation to specify a set of nodes for which data synchronization is required. Similarly, the file list may include relevant file information in the cluster. Based on the file list, the user may perform another selection operation to specify the file data to be synchronized. In the embodiments of the present application, the user's selection operation on the node list is denoted as the first selection operation, and the user's selection operation on the file list is denoted as the second selection operation.

[0120] Exemplarily, for example, first, all node information and file lists of the current cluster may be obtained from a backend server or database. The obtained data may be organized into a format easy to display on the front end, such as a JSON object. Then, the node list and file list may be displayed through a front-end page, and forms such as tables and lists may be used to facilitate the user's intuitive viewing and selection. On the front-end page, the user's first selection operation on the node list may be captured through an event listener (such as a click event), and the selected nodes by the user may be recorded, which may be a single node or a set of nodes, and the present application does not limit this. Also, the user's second selection operation on the file list is captured, and the selected files by the user are recorded to obtain the file data to be synchronized. In some embodiments, visual feedback may also be provided to the user, such as highlighting the selected files, or displaying the quantity and names of the selected files at the bottom of the page to confirm the user's operation.

[0121] In some embodiments, referring to Figure 5 , establishing a connection between the first node and each of the target nodes includes:

[0122] Establishing a TCP connection between the first node and each of the target nodes through multi-threaded concurrency;

[0123] Authenticating the TCP connection based on the authentication information.

[0124] In the embodiments of the present application, the process of establishing connections between the first node and each target node mainly involves two parts: establishing TCP connections concurrently with multiple threads and authenticating TCP connections based on authentication information. Among them, multi-threading means that multiple threads run simultaneously in a program, and each thread can independently execute different tasks. Multi-threaded concurrency can significantly improve the response speed and processing power of the system. In the embodiments of the present application, on the first node, multiple threads can be started, and each thread is responsible for establishing a TCP connection with a target node. In this way, connections can be established with multiple target nodes simultaneously, improving the connection efficiency. To better manage and reuse thread resources, thread pool technology can be used. The thread pool pre-creates a group of threads. When there is a new connection task, an idle thread is taken from the thread pool to execute the task. After the task is completed, the thread returns to the thread pool and waits for the next task.

[0125] In addition, in the embodiments of the present application, the TCP connection can also be authenticated based on authentication information. Specifically, the first node can send authentication information to the target node through the established TCP connection. After receiving the authentication information, the target node verifies it. The verification can be completed by comparing the pre-stored authentication information or through a third-party authentication service. The target node returns the authentication result (success or failure) to the first node. If the authentication is successful, the target node confirms that the connection is valid and can start normal communication; if the authentication fails, the target node closes the connection and refuses communication.

[0126] Based on the strategy provided in the embodiments of the present application, an efficient and secure TCP connection can be established between the first node and each target node, ensuring reliable data transmission and system security.

[0127] Specifically, in some embodiments, responding to the synchronization request and determining the file data to be synchronized includes:

[0128] Receiving, by the first node, the file data uploaded by the user along with the synchronization request;

[0129] Determining the file data as the file data to be synchronized.

[0130] In the embodiments of the present application, when synchronizing file data, the file data can be not only on the cluster nodes. In some embodiments, it can also be file data uploaded by the user. This file data can be uploaded along with the synchronization request. When the first node receives the synchronization request, if it finds that there is accompanying uploaded file data, it can directly determine this file data as the file data to be synchronized.

[0131] Specifically, in some embodiments, referring to Figure 6 , the cluster data synchronization method provided in the embodiments of the present application further includes:

[0132] Detect the synchronization time consumption corresponding to a single target node;

[0133] Detect the total execution time consumption of the data synchronization task corresponding to the synchronization request;

[0134] Determine performance data according to the ratio of the total execution time consumption and the synchronization time consumption;

[0135] Feedback the performance data to the user.

[0136] In the embodiments of the present application, during the data synchronization process, performance analysis operations can also be performed. Specifically, in the embodiments of the present application, the synchronization time consumption corresponding to a single target node can be detected, and the total execution time consumption of the data synchronization task corresponding to the synchronization request can be detected. According to the ratio of the total execution time consumption and the synchronization time consumption, performance data can be determined, and this performance data can be fed back to the user.

[0137] It can be understood that in the embodiments of the present application, in the way of collecting concurrent transmission, normally, the total execution time consumption and the synchronization time consumption corresponding to a single target node should be relatively close. Therefore, the value of the performance data should be close to 1. If the user finds that the value of this performance data exceeds 1 by a large margin, the relevant transmission path can be detected to troubleshoot the transmission fault between the first node and the target node, thereby improving the operation stability of the system.

[0138] Refer to Figure 7 , in the embodiments of the present application, a cluster data synchronization system is further proposed, including:

[0139] A receiving unit 710, configured to receive a synchronization request from a user through a first node in the cluster; wherein, the first node is any node in the cluster;

[0140] A response unit 720, configured to respond to the synchronization request, determine the node set specified by the user for data synchronization and the file data to be synchronized; wherein, the node set includes at least one target node that needs to perform data synchronization;

[0141] A establishing unit 730, configured to establish connections between the first node and each of the target nodes;

[0142] A transmission unit 740, configured to read the file data through the first node and concurrently transmit it to each of the target nodes.

[0143] Optionally, in some embodiments, the system further includes an initialization unit, and the initialization unit is specifically configured to:

[0144] Install and deploy the same console application on each of the nodes in the cluster; wherein, the console application is used to execute data synchronization tasks;

[0145] Configure the same authentication information for each of the nodes.

[0146] Optionally, in some embodiments, the authentication information includes account verification information; and the response unit is specifically configured to:

[0147] Detect the login account name and password information of the user;

[0148] Perform authentication on the login account name and the password information according to the account verification information;

[0149] When it is determined that the authentication of the login account name and the password information passes, respond to the synchronization request.

[0150] Optionally, in some embodiments, the response unit is specifically configured to:

[0151] Respond to the synchronization request and display the node list and file list of the cluster to the user;

[0152] In response to a first selection operation of the user on the node list, determine the set of nodes specified by the user for data synchronization;

[0153] In response to a second selection operation of the user on the file list, determine the file data specified by the user for synchronization.

[0154] Optionally, in some embodiments, the establishment unit is specifically configured to:

[0155] Establish TCP connections between the first node and each of the target nodes through multi-threaded concurrency;

[0156] Authenticate the TCP connections based on the authentication information.

[0157] Optionally, in some embodiments, the response unit is specifically configured to:

[0158] Receive the file data uploaded by the user along with the synchronization request through the first node;

[0159] Determine the file data as the file data to be synchronized.

[0160] Optionally, in some embodiments, the system further includes a feedback unit, and the feedback unit is specifically configured to:

[0161] Detect the synchronization time consumed for a single target node;

[0162] Detect the total execution time of the data synchronization task corresponding to the synchronization request;

[0163] Determine performance data according to the ratio of the total execution time to the synchronization time;

[0164] Feedback the performance data to the user.

[0165] It can be understood that the content in the above embodiments of the cluster data synchronization method is applicable to the embodiments of this processing system. The functions specifically implemented in the embodiments of this processing system are the same as those in the above embodiments of the cluster data synchronization method, and the beneficial effects achieved are also the same as those in the above embodiments of the cluster data synchronization method.

[0166] Referring to Figure 8 , an embodiment of the present application also discloses a computer device, including:

[0167] At least one processor 810;

[0168] At least one memory 820, configured to store at least one program;

[0169] When at least one program is executed by at least one processor 820, at least one processor 820 implements the above embodiments of the cluster data synchronization method.

[0170] It can be understood that the content in the above embodiments of the cluster data synchronization method is applicable to the embodiments of this computer device. The functions specifically implemented in the embodiments of this computer device are the same as those in the above embodiments of the cluster data synchronization method, and the beneficial effects achieved are also the same as those in the above embodiments of the cluster data synchronization method.

[0171] An embodiment of the present application also discloses a computer-readable storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to implement the above embodiments of the cluster data synchronization method when executed by the processor.

[0172] It can be understood that the content in the above embodiments of the cluster data synchronization method is applicable to the embodiments of this computer-readable storage medium. The functions specifically implemented in the embodiments of this computer-readable storage medium are the same as those in the above embodiments of the cluster data synchronization method, and the beneficial effects achieved are also the same as those in the above embodiments of the cluster data synchronization method.

[0173] A cluster data synchronization method disclosed in an embodiment of the present application receives a synchronization request from a user through a first node in the cluster; wherein, the first node is any node in the cluster; responds to the synchronization request to determine a set of nodes specified by the user for data synchronization and file data to be synchronized; wherein, the set of nodes includes at least one target node that needs to perform data synchronization; establishes connections between the first node and each of the target nodes; reads the file data through the first node and concurrently transmits it to each of the target nodes. The method in the embodiment of the present application can effectively improve the efficiency of cluster data synchronization, reduce the processing time-consuming, and the relevant data is synchronized through concurrent transmission, which can reduce the probability of errors and improve the accuracy of data synchronization.

[0174] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and the sub-operations described as part of a larger operation are performed independently.

[0175] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. Rather, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present application as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0176] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, etc., all kinds of media that can store program codes.

[0177] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0178] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts (electronic devices) having one or more wirings, portable computer disk cartridges (magnetic devices), random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fiber devices, and portable compact disc read-only memories (CDROMs). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or, if necessary, other suitable processing, and then storing it in a computer memory.

[0179] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0180] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0181] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application. The scope of the present application is defined by the claims and their equivalents.

[0182] The above has specifically described the preferred embodiments of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the present application.

[0183] In the description of this specification, the description with reference to the terms "one embodiment", "another embodiment", or "certain embodiments", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0184] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for cluster data synchronization, characterized in that, including: receiving a synchronization request from a user through a first node in a cluster; wherein, the first node is any node in the cluster; responding to the synchronization request to determine a set of nodes specified by the user for data synchronization and file data to be synchronized; wherein, the set of nodes includes at least one target node for data synchronization; establishing connections between the first node and each of the target nodes; reading the file data through the first node and concurrently transmitting it to each of the target nodes.

2. The method for synchronizing cluster data according to claim 1, wherein The method further includes: installing and deploying the same console application on each node in the cluster; wherein, the console application is used to execute data synchronization tasks; configuring the same authentication information for each node.

3. The method for synchronizing cluster data according to claim 2, wherein The authentication information includes account verification information; The responding to the synchronization request includes: detecting the user's login account name and password information; performing authentication on the login account name and the password information according to the account verification information; when it is determined that the authentication of the login account name and the password information passes, responding to the synchronization request.

4. A method for cluster data synchronization according to claim 1, characterized in that, The responding to the synchronization request to determine the set of nodes specified by the user for data synchronization and the file data to be synchronized includes: responding to the synchronization request to display a list of nodes and a list of files in the cluster to the user; responding to a first selection operation of the user on the list of nodes to determine the set of nodes specified by the user for data synchronization; responding to a second selection operation of the user on the list of files to determine the file data specified by the user to be synchronized.

5. A method for cluster data synchronization according to claim 2, characterized in that, The establishing connections between the first node and each of the target nodes includes: establishing TCP connections between the first node and each of the target nodes through multi-threaded concurrency; authenticating the TCP connections based on the authentication information.

6. A method for cluster data synchronization according to claim 1, characterized in that, The responding to the synchronization request to determine the file data to be synchronized includes: receiving, through the first node, file data uploaded by the user along with the synchronization request; determining the file data as the file data to be synchronized.

7. A method for cluster data synchronization according to any one of claims 1-6, characterized in that, The method further includes: detecting the synchronization time taken for a single target node; detecting the total execution time of the data synchronization task corresponding to the synchronization request; determining performance data according to the ratio of the total execution time to the synchronization time; feeding back the performance data to the user.

8. A cluster data synchronization system, characterized in that, including: a receiving unit, configured to receive a synchronization request from a user through a first node in a cluster; wherein, the first node is any node in the cluster; a responding unit, configured to respond to the synchronization request to determine a set of nodes specified by the user for data synchronization and file data to be synchronized; wherein, the set of nodes includes at least one target node for data synchronization; an establishing unit, configured to establish connections between the first node and each of the target nodes; a transmitting unit, configured to read the file data through the first node and concurrently transmit it to each of the target nodes.

9. A computer device, characterized in that, including: at least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the cluster data synchronization method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the cluster data synchronization method according to any one of claims 1-7 when executed by the processor.

Images