Data synchronization method and device and storage medium

Abstract

The invention provides a data synchronization method and device and a storage medium, and is used for providing a scheme for data synchronization between RabbitMQ and Kafka. The method is applied to a framework and a distributed processing engine, and comprises the following steps: acquiring starting parameter information; the starting parameter information comprises configuration parameters of a source message queue system and a target message queue system for indicating execution of data synchronization; establishing a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system, and monitoring whether the source message queue system triggers a source message; and when a source message triggered by the source message queue is monitored, sending the source message to the target message queue system.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, device and storage medium. Background Art

[0002] In modern distributed systems and microservice architectures, message queue systems are widely used to achieve decoupling and asynchronous communication between systems. RabbitMQ and Kafka are two common message queue systems, each with different characteristics and application scenarios.

[0003] However, with the improvement of enterprise informatization, the data interaction requirements between systems are becoming more and more complex, especially in cross-departmental and cross-environment data synchronization and real-time processing. Therefore, how to achieve data synchronization between RabbitMQ and Kafka, two common message queue systems, has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, device and storage medium to solve the problems existing in the related technologies. The technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method, which is applied to a framework and a distributed processing engine, including:

[0006] Obtaining startup parameter information, wherein the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization;

[0007] Based on the configuration parameters of the source message queue system and the target message queue system, establish a connection with the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message;

[0008] When a source message triggered by the source message queue is monitored, the source message is sent to the target message queue system.

[0009] In one implementation, the source message queue system and the target message queue system are respectively any one of RabbitMQ, encrypted Kafka, and unencrypted Kafka.

[0010] In one embodiment, establishing a connection with the source message queue system and the target message queue system based on configuration parameters of the source message queue system and the target message queue system includes:

[0011] Based on the host address, virtual host, port number, user name and queue name corresponding to the source message queue system, searching for the corresponding source configuration file;

[0012] Based on the source configuration file, establishing a connection with the source message queue system; and,

[0013] Based on the host address, virtual host, port number, user name and queue name corresponding to the target message queue system, searching for the corresponding target configuration file;

[0014] Based on the target configuration file, a connection with the target message queue system is established.

[0015] In one embodiment, the method further comprises:

[0016] Configuration verification is performed on the source configuration file and the target configuration file respectively.

[0017] In one embodiment, sending the source message to the target message queue system includes:

[0018] Performing encryption processing and / or persistence processing on the source message to obtain a processed source message;

[0019] The processed source message is sent to the target message queue system.

[0020] In one embodiment, the method further comprises:

[0021] After determining that monitoring result information indicating that the source message was successfully sent is received, determining whether message confirmation information is received;

[0022] When it is determined that the message confirmation information is received, it is determined that the source message has been processed by the target message queue system.

[0023] In one implementation, the startup parameter information is configured in any one of a development environment, a test environment, and a production environment.

[0024] In a second aspect, the embodiment of the present application further provides a data synchronization device, which is applied to a framework and a distributed processing engine, including:

[0025] An obtaining unit, used to obtain startup parameter information, wherein the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization;

[0026] A processing unit, configured to establish a connection with the source message queue system and the target message queue system based on configuration parameters of the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message;

[0027] The synchronization unit is used to send the source message to the target message queue system when monitoring the source message triggered by the source message queue.

[0028] In one implementation, the source message queue system and the target message queue system are respectively any one of RabbitMQ, encrypted Kafka, and unencrypted Kafka.

[0029] In one embodiment, when the processing unit is used to establish a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system, it is specifically used to:

[0030] Based on the host address, virtual host, port number, user name and queue name corresponding to the source message queue system, searching for the corresponding source configuration file;

[0031] Based on the source configuration file, establishing a connection with the source message queue system; and,

[0032] Based on the host address, virtual host, port number, user name and queue name corresponding to the target message queue system, searching for the corresponding target configuration file;

[0033] Based on the target configuration file, a connection with the target message queue system is established.

[0034] In one embodiment, the data synchronization device further includes a verification unit, and the verification unit is used to:

[0035] Configuration verification is performed on the source configuration file and the target configuration file respectively.

[0036] In one embodiment, when the synchronization unit is used to send the source message to the target message queue system, it is specifically used to:

[0037] Performing encryption processing and / or persistence processing on the source message to obtain a processed source message;

[0038] The processed source message is sent to the target message queue system.

[0039] In one implementation, the data synchronization device further includes a verification unit, and the verification unit is used to:

[0040] After determining that monitoring result information indicating that the source message was successfully sent is received, determining whether message confirmation information is received;

[0041] When it is determined that the message confirmation information is received, it is determined that the source message has been processed by the target message queue system.

[0042] In one implementation, the startup parameter information is configured in any one of a development environment, a test environment, and a production environment.

[0043] In a third aspect, an embodiment of the present application further provides a computer device, comprising: a memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned aspects, wherein the memory and the processor communicate with each other through an internal connection path.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is implemented.

[0045] The advantages or beneficial effects of the above technical solution include at least:

[0046] In the present application, the framework and the distributed processing engine can obtain startup parameter information, which includes configuration parameters for indicating the source message queue system and the target message queue system for performing data synchronization; and, based on the configuration parameters of the source message queue system and the target message queue system, establish a connection with the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message; when the source message triggered by the source message queue is monitored, send the source message to the target message queue system. Specifically, since the framework and the distributed processing engine can execute any stream data program in a data parallel and pipeline manner, the configuration parameters of the two message queue systems to be transmitted can be received based on the engine, and a connection can be established based on this to achieve real-time data transmission and processing between the two message queue systems.

[0047] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0049] Figure 1 An example diagram of an application scenario provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0051] Figure 3 A structural block diagram of a data synchronization device provided in an embodiment of the present application;

[0052] Figure 4 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0054] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0055] RabbitMQ refers to an open source message broker software that implements the Advanced Message Queuing Protocol (AMQP). It can also be called a message-oriented middleware or message queue system. It is suitable for scenarios that require complex routing logic and reliable message delivery, such as order processing, task scheduling, etc. The RabbitMQ server is written in Erlang language. The cluster of RabbitMQ servers and the handling of failover are built on the open telecommunications platform framework. All major programming languages ​​of RabbitMQ have client libraries that communicate with the proxy interface.

[0056] Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. Actions can be understood as user web browsing, searching and other user actions. Actions are a key factor in many social functions on modern networks. Therefore, analyzing and processing the action stream data corresponding to actions has become an urgent problem to be solved. Related technologies generally process action stream data based on throughput requirements and in combination with log aggregation. In addition, Kafka is a publish-subscribe model, that is, producers are the objects that publish information, and consumers are the objects that subscribe to published information. This message queue system is suitable for scenarios that process large amounts of real-time data streams, such as log collection, real-time analysis, and other scenarios.

[0057] Flink is a framework and distributed processing engine that can execute any streaming data program in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.

[0058] Kerberos is a network authentication protocol designed to provide strong authentication services for client / server applications through a key system. The implementation of this authentication process does not rely on the authentication of the host operating system, does not require trust based on the host address, does not require the physical security of all hosts on the network, and assumes that data packets transmitted on the network can be arbitrarily read, modified, and inserted.

[0059] The JAR package is an archive file format on the Java platform, which is used to package multiple Java class files, resource files, and metadata into one file.

[0060] In related technologies, message queue systems are widely used to achieve decoupling and asynchronous communication between systems. Among them, RabbitMQ and Kafka are two common message queue systems, each with different characteristics and application scenarios.

[0061] As the degree of enterprise informatization increases, the data interaction requirements between message queue systems are becoming more and more complex, especially in cross-department and cross-environment data synchronization and real-time processing, facing the following challenges:

[0062] (1) Heterogeneous system integration is difficult. Specifically, different message queue systems have different protocols and data formats, making integration complex and error-prone. Therefore, there is an urgent need for a solution that can efficiently convert data to achieve synchronous data transmission, thereby ensuring data consistency and compatibility between different systems.

[0063] (2) High demand for real-time data processing. Specifically, many business scenarios require real-time data synchronization and processing, which is difficult to meet with the traditional batch processing mode. Therefore, the data synchronization transmission solution also requires powerful real-time stream processing capabilities to ensure timely data transmission and processing.

[0064] (3) Security and configuration management are complex. Specifically, different systems and environments have different security requirements, such as encrypted transmission and identity authentication. Therefore, the data synchronization solution also requires flexible configuration management to support dynamic switching between multiple accounts and multiple environments.

[0065] However, existing message queue synchronization tools and cross-system data synchronization solutions have the following limitations when facing the above challenges:

[0066] 1. Single system support: Many tools only support RabbitMQ or Kafka, making it difficult to achieve two-way real-time synchronization.

[0067] (II) Batch processing mode is inefficient. Specifically, traditional tools mostly use batch processing mode, which has large data transmission delays and cannot meet real-time processing requirements. When processing large amounts of real-time data, performance and reliability are poor.

[0068] (III) Inflexible configuration management. Specifically, the configuration management functions of existing tools are limited and cannot meet the complex needs of multiple accounts and multiple environments. There is a lack of comprehensive support for scenarios with high security requirements (such as Kerberos encryption).

[0069] In view of this, an embodiment of the present application provides a data synchronization method, through which a framework and a distributed processing engine can obtain startup parameter information, the startup parameter information including configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization; and, based on the configuration parameters of the source message queue system and the target message queue system, establish a connection with the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message; when the source message triggered by the source message queue is monitored, send the source message to the target message queue system. Specifically, since the framework and the distributed processing engine can execute any stream data program in a data parallel and pipeline manner, the configuration parameters of the two message queue systems for data to be transmitted can be received based on the engine, and a connection can be established based on this to achieve real-time data transmission and processing between the two message queue systems.

[0070] After introducing the design ideas of the embodiments of the present application, the following is a brief introduction to the application scenarios to which the technical solutions in the embodiments of the present application are applicable. It should be noted that the application scenarios described in the embodiments of the present application are for the purpose of more clearly illustrating the technical solutions in the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0071] See also Figure 1 As shown, it is a schematic diagram of a scenario to which the embodiment of the present application can be applied. The scenario includes a message queue device 101, a computer device 102 and a message queue device 103. The data synchronization method of the embodiment of the present application can be Figure 1 The message queue device 101, the computer device 102 and the message queue device 103 can be implemented in collaboration, or can be implemented only by the computer device 102, which is not limited in the embodiments of the present application. The computer device 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 that interacts with the information collection device 101.

[0072] Optionally, the data synchronization solution provided in the embodiment of the present application may be used in the following scenarios in the future: internal enterprise system integration, cross-departmental data sharing and collaboration, data migration and synchronization in multi-cloud environments, and Internet of Things (IoT) data processing and analysis, etc. These application scenarios will achieve data synchronization through the efficient and reliable data synchronization system provided by the present application, thereby improving data processing efficiency to adapt to complex enterprise needs and changes in the technical environment.

[0073] In the embodiment of the present application, the aforementioned message queue device 101 may be a device with RabbitMQ deployed, a device with encrypted Kafka deployed, or a device with unencrypted Kafka deployed. The message queue device 101 may generate corresponding messages and may be referred to as a producer or a device corresponding to a source message queue system. The message queue device 103 may be a device with RabbitMQ deployed, a device with encrypted Kafka deployed, or a device with unencrypted Kafka deployed. The message queue device 103 may consume corresponding messages and may be referred to as a consumer or a device corresponding to a target message queue system.

[0074] It can be seen that the data synchronization object in the embodiment of the present application can be between RabbitMQ and RabbitMQ, two consumer queue systems; or between RabbitMQ and Kafka, two consumer queue systems; or between Kafka and RabbitMQ, two consumer queue systems; or between Kafka and Kafka, two consumer queue systems. In other words, in one embodiment, RabbitMQ can be both a producer of messages and a consumer of RabbitMQ, and in another embodiment, Kafka can be both a producer of messages and a consumer of Kafka.

[0075] In an embodiment of the present application, a computer device 102 with a framework and a distributed processing engine is deployed through a network 104, which can obtain startup parameter information, which includes configuration parameters of a source message queue system and a target message queue system for indicating data synchronization; and, based on the configuration parameters of the source message queue system and the target message queue system, establish a connection with the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message; when a source message triggered by the source message queue is monitored, the source message is sent to the target message queue system.

[0076] In the specific implementation process, multiple computer devices 102 may interact with multiple message queue devices 101 and message queue devices 103, or one computer device 102 may interact with multiple message queue devices 101 and message queue devices 103, or one computer device 102 may interact with one message queue device 101 and message queue device 103, which is not limited in the embodiments of the present application. Figure 1 The interaction between a computer device 102 and a message queue device 101 and a message queue device 103 is taken as an example.

[0077] In the embodiment of the present application, the message queue device 101, the computer device 102 and the message queue device 103 can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, or physical servers, or server clusters, but are not limited to this. Among them, the message queue device 101 and the message queue device 103 can be connected to the computer device 102 through one or more networks 104. The network 104 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network, or it can be a wireless fidelity (WIREless-Fidelity, WIFI) network. Of course, it can also be other possible networks, and the embodiment of the present application does not limit this.

[0078] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, such as only a computer device implementing a data synchronization method, which is not limited in the embodiments of the present application. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0079] To further illustrate the scheme of the data synchronization method provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the actual processing process or when the device is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or drawings (for example, an application environment of a parallel processor or multi-threaded processing).

[0080] The following combination Figure 2 The method flow chart shown illustrates the data synchronization method in the embodiment of the present application. The method flow can be executed by a computer device deployed with a framework and a distributed processing engine. The specific implementation flow of the method is as follows:

[0081] S201: obtaining startup parameter information, where the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization.

[0082] In the embodiment of the present application, a data synchronization system can be built based on the framework and the distributed processing engine, and the data synchronization system can be deployed on a computer device. Optionally, if the framework and the distributed processing engine are Apache Flink, the user uploads the tool JAR package to the Apache Flink Dashboard, configures the necessary startup parameters, and clicks "Submit", and the data synchronization system built based on the framework and the distributed processing engine can be implemented.

[0083] In an embodiment of the present application, the data synchronization system can queue-connect two message queue systems to ensure stable connection and efficient communication between the two message queue systems. Specifically, the data synchronization system ensures stable communication with RabbitMQ and Kafka through refined connection management. The data synchronization system establishes an initial connection with RabbitMQ and Kafka at startup and continuously monitors the connection status. Through the optimized connection initialization and maintenance mechanism, the stability and efficiency of the connection are ensured throughout the message transmission process. The design of the connection simplifies resource consumption, avoids unnecessary complexity, and is suitable for efficient data transmission in single queue and single topic scenarios.

[0084] In the embodiment of the present application, users can flexibly configure various configurations of Apache Flink, including RabbitMQ and Kafka connection information, account passwords, security authentication information, etc., to ensure the flexibility and security of the data synchronization system in different environments.

[0085] Optionally, users can configure multi-queue management. Specifically, users can set multiple sets of RabbitMQ and Kafka account passwords, addresses, authentication files, certificates and other information in Apache Flink, supporting dynamic switching in different environments. For example, different accounts and connection information can be used in development, testing and production environments to ensure the security and isolation of the data synchronization system.

[0086] Optionally, the user can dynamically configure the loading information. Specifically, the data synchronization system supports loading configuration files through startup parameters. For example, the user can specify different configuration files at startup to run in different environments.

[0087] Optionally, users can configure management information securely. Specifically, users can configure Kerberos authentication, key and certificate management, and RabbitMQ account and password authentication based on the data synchronization system. Among them, Kerberos authentication is used to manage Kafka's Kerberos encrypted authentication information, thereby ensuring the security and reliability of message transmission in an environment using Kerberos authentication. Key and certificate management is used for centralized management of keys and certificates, thereby ensuring the secure storage and use of encrypted authentication information. RabbitMQ account and password authentication is used to use account and password for verification when connecting to RabbitMQ, thereby ensuring the security and reliability of message transmission.

[0088] In an embodiment of the present application, Apache Flink can obtain startup parameter information based on data triggered by information set by the user. In other words, the startup parameter information can be configured in any one of the development environment, the test environment, and the production environment. Exemplarily, the startup parameter information can be shown in Table 1 below.

[0089] Table 1

[0090]

[0091] In an embodiment of the present application, the source message queue system and the target message queue system are respectively any one of encrypted Kafka, RabbitMQ and unencrypted Kafka.

[0092] In the embodiment of the present application, if the source message queue system and the target message queue system are both RabbitMQ, that is, the two message queue systems for data synchronization are RabbitMQ, then the startup parameter information is, for example: -Dsource=R-Dtarget=R-Ds-host=ip-Ds-virtualHost=test

[0093] -Ds-port=5672-Ds-username=guest-Ds-password=guest-Ds-queue=source_queue-Dt-host=119.91.206.46-Dt-virtualHost=test-Dt-port=5672-Dt-username=guest-Dt-password=guest-Dt-queue=target_queue.

[0094] In the embodiment of the present application, if the source message queue system is RabbitMQ and the target message queue system is encrypted Kafka, the startup parameter information is, for example: -Dsource=R-Dtarget=K2

[0095] -Ds-virtualHost=test-Ds-host=ip-Ds-port=5672-Ds-username=guest

[0096] -Ds-password=guest-Ds-queue=source_queue-Dt-servers=kafka.abc.com:9092

[0097] -Dt-topic=target_topic_sasl-Dt-kafkaId=dqysh020036.

[0098] In the embodiment of the present application, if the source message queue system is encrypted Kafka and the target message queue system is RabbitMQ, the startup parameter information is, for example: -Dsource=K2-Dtarget=R

[0099] -Ds-kafkaId=dqysh020036-Ds-servers=kafka.abc.com:9092

[0100] -Ds-topic=source_topic_sasl-Dt-host=ip-Dt-virtualHost=test-Dt-port=5672

[0101] -Dt-username=guest-Dt-password=guest-Dt-queue=target_queue.

[0102] In the embodiment of the present application, if the source message queue system and the target message queue system are both encrypted Kafka, that is, the two message queue systems for data synchronization are encrypted Kafka, then the startup parameter information is, for example: -Dsource=K2-Dtarget=K2-Ds-kafkaId=dqysh020036

[0103] -Ds-servers=kafka.abc.com:9092-Ds-topic=source_topic_sasl

[0104] -Dt-servers=kafka.abc.com:9092-Dt-topic=target_topic_sasl

[0105] -Dt-kafkaId=dqysh020036.

[0106] In the embodiment of the present application, if the source message queue system and the target message queue system are both unencrypted Kafka, that is, the two message queue systems for data synchronization are unencrypted Kafka, then the startup parameter information is, for example: -Dsource=K2-Dtarget=K2-Ds-servers=kafka.abc.com:9092

[0107] -Ds-topic=source_topic_sasl-Dt-servers=kafka.abc.com:9092

[0108] -Dt-topic=target_topic_sasl-Dt-kafkaId=dqysh020036.

[0109] In other words, the difference between the startup parameter information when both the source message queue system and the target message queue system are unencrypted Kafka and the startup parameter information when both the source message queue system and the target message queue system are encrypted Kafka is that the s-kafkaId or t-kafkaId parameter is not required.

[0110] S202: Establishing a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system, and monitoring whether the source message queue system triggers a source message.

[0111] In an embodiment of the present application, the data synchronization system can establish a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system. Specifically, the corresponding source configuration file can be searched based on the host address, virtual host, port number, user name and queue name corresponding to the source message queue system; the connection with the source message queue system can be established based on the source configuration file; and the corresponding target configuration file can be searched based on the host address, virtual host, port number, user name and queue name corresponding to the target message queue system; the connection with the target message queue system can be established based on the target configuration file.

[0112] In a possible implementation, the data synchronization system can also perform configuration verification on the source configuration file and the target configuration file respectively. That is, when the data synchronization system starts loading the configuration file, Apache Flink will verify the configuration items to ensure that all required configuration items have been correctly filled in to avoid system failures caused by configuration errors.

[0113] In the embodiment of the present application, the data synchronization system sets a dedicated message listener to monitor the message queues and topics in RabbitMQ and Kafka. When a new message arrives, the listener can immediately capture and trigger the corresponding consumption process. In this way, through an efficient monitoring mechanism, the message can be processed at the moment it is generated, ensuring the real-time and efficient consumption of the message and achieving the goal of data synchronization.

[0114] S203: When a source message triggered by a source message queue is monitored, the source message is sent to a target message queue system.

[0115] In the embodiment of the present application, the source message can be encrypted and / or persisted to obtain the processed source message, and the processed source message can be sent to the target message queue system. Optionally, the source message can also be monitored, and when it is determined that the monitoring result information indicating that the source message is successfully sent is received, it is determined whether the message confirmation information is received; when it is determined that the message confirmation information is received, it is determined that the source message has been processed by the target message queue system.

[0116] For example, if the target message queue system is RabbitMQ, the processed message can be sent to the specified RabbitMQ queue through the AMQP protocol of RabbitMQ. Specifically, during the message sending process, the message persistence mechanism can be used, so that even if the RabbitMQ server is restarted, the reliable transmission of the message can be guaranteed, thereby ensuring that the processed message will not be lost during the transmission process.

[0117] And, when the source message queue system is (that is, the producer is RabbitMQ), it can automatically pull messages from the aforementioned specified RabbitMQ queue and implement a strict message confirmation mechanism. Specifically, once the message is successfully processed, a confirmation signal, that is, the aforementioned message confirmation information, can be sent to ensure that the message will not be consumed repeatedly.

[0118] Specifically, a message confirmation mechanism (ACK) can be used. After a message is sent to a queue, RabbitMQ will wait for the target message queue system (i.e., the consumer) to confirm that the message has been successfully processed, such as sending an ACK. If the consumer fails to process the message, RabbitMQ will put the message back into the queue to ensure that the message will not be lost or consumed repeatedly.

[0119] For example, if the target message queue system is Kafka, the processed message can be sent to the specified Kafka topic using either synchronous or asynchronous sending. The synchronous sending method can ensure that the message confirmation information is returned immediately after the processed message is successfully transmitted, thereby improving the reliability of the data synchronization transmission system. Specifically, the asynchronous sending method returns immediately after placing the processed message into the Kafka sending buffer, and the background thread is responsible for sending the processed messages in batches to the Kafka cluster. In this way, the sending efficiency of the processed message can be significantly improved. Optionally, if the business scenario is a high-throughput business scenario, the asynchronous sending method can be used to improve the synchronous transmission efficiency of the processed message.

[0120] And, if the source message queue system is Kafka, you can pull messages from the specified Kafka topic and process them, while accurately managing the offset of the message. Among them, the submission of the offset of the message can ensure the orderly consumption of the message and avoid data loss or repeated processing. In addition, the breakpoint resumption function of message processing can be realized through the management of the offset.

[0121] Specifically, when using message offsets to manage message confirmations, each message can have a unique offset. When the consumer processes the message, it submits the unique offset corresponding to the message, thereby indicating that the message has been successfully processed based on the received unique offset.

[0122] In an embodiment of the present application, during the message sending process between two message queue systems, monitoring result information can be monitored and sent. If the monitoring result information indicates that the processed message fails to be sent, a retry operation can be performed and a failure log can be recorded so that subsequent investigation and processing of the failed message can be performed based on the log. Specifically, the number of retries and the interval time corresponding to the aforementioned retry operation can be personalized. For example, the user sets the number of retries to 3 times and the interval time to 1 second each time, thereby ensuring that the processed message can eventually be successfully sent.

[0123] In one embodiment, in order to improve the robustness of the system, the data synchronization system integrates disconnection detection and automatic reconnection mechanisms. When the connection between two message queue systems is disconnected due to network problems or server outages, the data synchronization system will detect it in real time and immediately attempt to reestablish the connection. In this way, the continuity of message transmission between the two message queue systems can be ensured as much as possible, and data flow can be restored as soon as possible even under adverse conditions. In addition, it should be noted that the automatic reconnection mechanism of Apache Flink in the embodiment of the present application supports custom configuration, which allows users to personalize the reconnection interval and maximum number of attempts based on actual implementation requirements to balance recovery speed and system stability.

[0124] In one embodiment, the data synchronization system is designed with a fault recovery and data protection mechanism. When the connection between two message queue systems fails, the two message queue systems will not only automatically reconnect, but also ensure that the data not transmitted during the disconnection period is not lost. Optionally, through message persistence or caching technology, the system can continue to transmit unfinished messages after the connection is restored, thereby ensuring data consistency and integrity. In this way, the fault tolerance of the data synchronization system can be improved and the reliability of synchronously transmitted data can be improved.

[0125] It should be noted that although the data synchronization method is introduced above with the source message queue system being Kafka and the target message queue system being Kafka, or the source message queue system being RabbitMQ and the target message queue system being RabbitMQ as examples, those skilled in the art will appreciate that the present application should not be limited thereto. In fact, users can flexibly set the source message queue system and the target message queue system according to their personal preferences and / or actual application scenarios, and the present application does not limit this.

[0126] In this way, according to the solution provided by the above embodiment of the present application, the data synchronization system can support Kafka's Kerberos encrypted and non-encrypted connection methods, and quickly match the configuration through the startup parameters, greatly improving the flexibility, security and ease of use of the system. In addition, it can be run in clusters in Flink session mode, support large-scale data processing and expansion, simplify the deployment process, reduce maintenance costs, and ensure the stability and scalability of the system. And because of the unique combination of Flink's stream processing capabilities, the data synchronization between RabbitMQ and Kafka is not only limited to simple message transmission, but also can achieve complex real-time data processing and analysis. It can also support dynamic switching and configuration management of multiple environments, that is, support switching configurations in different environments (development, testing, production) to meet the needs of various business scenarios. Furthermore, through a multi-level security mechanism, the security and integrity of the data transmission process are ensured.

[0127] Figure 3 FIG. 2 shows a structural block diagram of a data synchronization device according to an embodiment of the present application. Figure 3 As shown, the device may include:

[0128] An obtaining unit 301 is used to obtain startup parameter information, where the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization;

[0129] The processing unit 302 is used to establish a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message;

[0130] The synchronization unit 303 is used to send the source message to the target message queue system when monitoring the source message triggered by the source message queue.

[0131] In one implementation, the source message queue system and the target message queue system are respectively any one of RabbitMQ, encrypted Kafka, and unencrypted Kafka.

[0132] In one implementation, when the processing unit 302 is used to establish a connection with the source message queue system and the target message queue system based on the configuration parameters of the source message queue system and the target message queue system, it is specifically used to:

[0133] Based on the host address, virtual host, port number, user name and queue name corresponding to the source message queue system, search for the corresponding source configuration file;

[0134] Based on the source configuration file, establishing a connection with the source message queue system; and,

[0135] Based on the host address, virtual host, port number, user name and queue name corresponding to the target message queue system, search for the corresponding target configuration file;

[0136] Based on the target configuration file, a connection is established with the target message queue system.

[0137] In one embodiment, the data synchronization device further includes a verification unit, which is used to:

[0138] Perform configuration verification on the source configuration file and the target configuration file respectively.

[0139] In one implementation, when the synchronization unit 303 is used to send the source message to the target message queue system, it is specifically used to:

[0140] Performing encryption processing and / or persistence processing on the source message to obtain a processed source message;

[0141] Send the processed source message to the target message queue system.

[0142] In one embodiment, the data synchronization device further includes a verification unit, and the verification unit is used to:

[0143] After determining that monitoring result information indicating that the source message was successfully sent is received, determining whether message confirmation information is received;

[0144] When it is determined that the message confirmation information is received, it is determined that the source message has been processed by the target message queue system.

[0145] In one implementation, the startup parameter information is configured in any one of a development environment, a test environment, and a production environment.

[0146] The functions of each unit in the data synchronization device in the embodiment of the present application can be found in the corresponding description in the above method, and will not be repeated here.

[0147] Figure 4 FIG. 2 shows a block diagram of a computer device according to an embodiment of the present application. Figure 4 As shown, the computer device includes: a processor 401 and a memory 402. The memory 402 stores instructions, and the instructions are loaded and executed by the processor 401 to implement the data synchronization method in the above embodiment. The number of the memory 402 and the processor 401 can be one or more.

[0148] The computer device also includes:

[0149] The communication interface 403 is used to communicate with external devices and perform data exchange transmission.

[0150] If the processor 401, the memory 402 and the communication interface 403 are implemented independently, the processor 401, the memory 402 and the communication interface 403 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0151] Optionally, in a specific implementation, if the processor 401, the memory 402, and the communication interface 403 are integrated on a chip, the processor 401, the memory 402, and the communication interface 403 can communicate with each other through an internal interface.

[0152] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method provided in the embodiment of the present application is implemented.

[0153] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0154] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0155] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0156] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0158] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0159] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0161] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0162] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data synchronization method, characterized in that: Applied to frameworks and distributed processing engines, including: Obtaining startup parameter information, wherein the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization; Based on the configuration parameters of the source message queue system and the target message queue system, establish a connection with the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message; When a source message triggered by the source message queue is monitored, the source message is sent to the target message queue system.

2. The method according to claim 1, characterized in that The source message queue system and the target message queue system are respectively any one of RabbitMQ, encrypted Kafka and unencrypted Kafka.

3. The method according to claim 2, characterized in that Based on the configuration parameters of the source message queue system and the target message queue system, establishing a connection with the source message queue system and the target message queue system includes: Based on the host address, virtual host, port number, user name and queue name corresponding to the source message queue system, searching for the corresponding source configuration file; Based on the source configuration file, establishing a connection with the source message queue system; and, Based on the host address, virtual host, port number, user name and queue name corresponding to the target message queue system, searching for the corresponding target configuration file; Based on the target configuration file, a connection with the target message queue system is established.

4. The method according to claim 3, characterized in that: The method further comprises: Configuration verification is performed on the source configuration file and the target configuration file respectively.

5. The method according to any one of claims 1 to 4, characterized in that Sending the source message to the target message queue system comprises: Performing encryption processing and / or persistence processing on the source message to obtain a processed source message; The processed source message is sent to the target message queue system.

6. The method according to claim 5, characterized in that The method further comprises: After determining that monitoring result information indicating that the source message was successfully sent is received, determining whether message confirmation information is received; When it is determined that the message confirmation information is received, it is determined that the source message has been processed by the target message queue system.

7. The method according to any one of claims 1 to 4, characterized in that The startup parameter information is configured in any one of a development environment, a test environment, and a production environment.

8. A data synchronization device, characterized in that: Applied to frameworks and distributed processing engines, including: An obtaining unit, used to obtain startup parameter information, wherein the startup parameter information includes configuration parameters for indicating a source message queue system and a target message queue system for performing data synchronization; A processing unit, configured to establish a connection with the source message queue system and the target message queue system based on configuration parameters of the source message queue system and the target message queue system, and monitor whether the source message queue system triggers a source message; The synchronization unit is used to send the source message to the target message queue system when monitoring the source message triggered by the source message queue.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the method according to any one of claims 1 to 7 is implemented.

Images