Logstash-based data exchange system and method

Abstract

The invention particularly relates to a data exchange system and method based on Logstash. The data exchange system based on the Logstash comprises a relational database, a Logstash data processing center, an Elasticsearch cluster, a message queue and a website front end, the relational database is used for storing government affair data; the Logstash data processing center is used for extracting data from the relational database at regular time; the Elasticsearch cluster is used for storing the data processed by the Logstash data processing center, and the Logstash data processing center is used for storing the data processed by the Logstash data processing center; the message queue is used for receiving deletion events of the relational database; and the website front end is used for reading the data from the Elasticsearch cluster and displaying the data. According to the data exchange system and method based on the Logstash, seamless synchronization and real-time display of the data between the relational database and the Logstash are achieved, the data exchange efficiency is improved, and the real-time performance of data updating and the consistency of the data are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data exchange system and method based on Logstash. Background Art

[0002] In the field of data management, relational database management systems (RDBMS) such as MySQL and PostgreSQL are widely used due to their data consistency and transaction management capabilities. These systems can ensure the accuracy and reliability of data and are suitable for processing complex queries and transaction operations. However, with the continuous growth of data volume and the diversification of data types, relational databases have gradually exposed performance bottlenecks in processing large-scale data analysis and real-time data retrieval.

[0003] To solve this problem, a solution is needed that can efficiently process large amounts of data and provide fast retrieval capabilities. As an open source server-side data processing pipeline, Logstash can collect data from multiple sources, dynamically convert data, and send data to a specified repository. Logstash has powerful data processing capabilities and can clean, format, and convert data. It supports multiple data formats and protocols, making it an ideal tool for data synchronization and conversion.

[0004] Although Logstash has significant advantages in data processing and conversion, in practical applications, how to integrate it with existing relational database systems to achieve efficient synchronization and real-time update of data is still a technical challenge. Especially in the management field, the real-time and accuracy of data are crucial to decision support, so a reliable data exchange system and method is needed to ensure seamless synchronization of data between relational databases and Logstash.

[0005] In order to solve the problems in the prior art of low data exchange efficiency, insufficient real-time performance of data update and difficulty in ensuring data consistency, the present invention proposes a data exchange system and method based on Logstash. Summary of the invention

[0006] In order to make up for the defects of the prior art, the present invention provides a simple and efficient data exchange system and method based on Logstash.

[0007] The present invention is achieved through the following technical solutions:

[0008] A data exchange system based on Logstash, including a relational database, a Logstash data processing center, an Elasticsearch cluster, a message queue and a website front end;

[0009] The relational database is used to store government data;

[0010] The Logstash data processing center is used to extract data from the relational database at regular intervals;

[0011] The Elasticsearch cluster is used to store data processed by the Logstash data processing center;

[0012] The message queue is used to receive deletion events of the relational database;

[0013] The website front end is used to read data from the Elasticsearch cluster and display it.

[0014] The Logstash data processing center is configured with an input plug-in, a filter plug-in and an output plug-in;

[0015] The input plug-in is used to connect to the relational database and extract data regularly;

[0016] The filter plug-in is used to convert the extracted data;

[0017] The output plug-in is used to write the processed data to the Elasticsearch cluster and delete the data in the Elasticsearch cluster according to the deletion event in the message queue.

[0018] The website front end is responsible for reading data from the Elasticsearch cluster through the REST API of the Elasticsearch cluster.

[0019] A data exchange method based on Logstash includes the following steps:

[0020] Step S1: Configure the input plug-in in the Logstash data processing center, set a scheduled task, and extract data from the relational database;

[0021] Step S2: Configure the filter plug-in in the Logstash data processing center to perform necessary conversion processing on the extracted data, including data cleaning and format conversion, to remove sensitive information and convert the extracted data into a custom format;

[0022] Step S3: Configure the output plug-in in the Logstash data processing center and specify the connection information and index settings of the Elasticsearch cluster;

[0023] Use the output plug-in to write the processed data to the Elasticsearch cluster and form an index so that the website front end can read it;

[0024] Step S4: When data in the relational database is deleted, the Logstash data processing center is notified through a trigger mechanism (such as a message queue, a trigger, etc.);

[0025] After receiving the deletion notification, the Logstash data processing center calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data in the Elasticsearch cluster.

[0026] In the step S1, when configuring the input plug-in in the Logstash data processing center, the connection information and query statement of the relational database are specified;

[0027] Use the schedule function of the Logstash data processing center to customize the time interval for data extraction.

[0028] In step S3, the deletion status of the data is recorded through the filter plug-in and a deletion mark is added.

[0029] In step S4, when the data in the relational database is deleted, the corresponding data in the Elasticsearch cluster is deleted synchronously, and the steps are as follows:

[0030] Step S4.1, set a trigger in the relational database, when the data is deleted, the trigger publishes the deletion event to the message queue;

[0031] Step S4.2, the Logstash data processing center subscribes to the above message queue and receives the deletion event;

[0032] Step S4.3: The Logstash data processing center constructs a deletion request based on the received deletion event, and calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data.

[0033] A data exchange device based on Logstash, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0034] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0035] The beneficial effects of the present invention are as follows: the data exchange system and method based on Logstash realize seamless synchronization and real-time display of data between a relational database and Logstash, which not only improves the data exchange efficiency, but also ensures the real-time nature of data update and the consistency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Attached Figure 1 Schematic diagram of data flow of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0039] The Logstash-based data exchange system includes a relational database, a Logstash data processing center, an Elasticsearch cluster, a message queue, and a website front end;

[0040] The relational database is used to store government data;

[0041] The Logstash data processing center is used to extract data from the relational database at regular intervals;

[0042] The Elasticsearch cluster is used to store data processed by the Logstash data processing center;

[0043] The message queue is used to receive deletion events of the relational database;

[0044] The website front end is used to read data from the Elasticsearch cluster and display it.

[0045] The Logstash data processing center is configured with an input plug-in, a filter plug-in and an output plug-in;

[0046] The input plug-in is used to connect to the relational database and extract data regularly;

[0047] The filter plug-in is used to convert the extracted data;

[0048] The output plug-in is used to write the processed data to the Elasticsearch cluster and delete the data in the Elasticsearch cluster according to the deletion event in the message queue.

[0049] The website front end reads data from the Elasticsearch cluster through the REST API of the Elasticsearch cluster.

[0050] The data exchange method based on Logstash includes the following steps:

[0051] Step S1: Configure the input plug-in in the Logstash data processing center, set a scheduled task, and extract data from the relational database;

[0052] Step S2: Configure the filter plug-in in the Logstash data processing center to perform necessary conversion processing on the extracted data, including data cleaning and format conversion, to remove sensitive information and convert the extracted data into a custom format;

[0053] Step S3: Configure the output plug-in in the Logstash data processing center and specify the connection information and index settings of the Elasticsearch cluster;

[0054] Use the output plug-in to write the processed data to the Elasticsearch cluster and form an index so that the website front end can read it;

[0055] Step S4: When data in the relational database is deleted, the Logstash data processing center is notified through a trigger mechanism (such as a message queue, a trigger, etc.);

[0056] After receiving the deletion notification, the Logstash data processing center calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data in the Elasticsearch cluster.

[0057] In the step S1, when configuring the input plug-in in the Logstash data processing center, the connection information and query statement of the relational database are specified;

[0058] Use the schedule function of the Logstash data processing center to customize the time interval for data extraction.

[0059] In step S3, the deletion status of the data is recorded through the filter plug-in and a deletion mark is added.

[0060] In step S4, when the data in the relational database is deleted, the corresponding data in the Elasticsearch cluster is deleted synchronously, and the steps are as follows:

[0061] Step S4.1, set a trigger in the relational database, when the data is deleted, the trigger publishes the deletion event to the message queue;

[0062] Step S4.2, the Logstash data processing center subscribes to the above message queue and receives the deletion event;

[0063] Step S4.3: The Logstash data processing center constructs a deletion request based on the received deletion event, and calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data.

[0064] The Logstash-based data exchange device includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0065] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0066] Compared with the prior art, the data exchange system and method based on Logstash has the following characteristics:

[0067] (1) Improved data exchange efficiency: By leveraging the powerful data processing capabilities of the Logstash data processing center, efficient data exchange from the relational database to the Elasticsearch cluster is achieved.

[0068] (2) Real-time data update is achieved: Through scheduled tasks and trigger mechanisms, real-time data update can be ensured to meet the website front-end's demand for real-time data.

[0069] (3) Ensure data consistency: Through the data synchronization mechanism, the data in the relational database and the Elasticsearch cluster can be kept consistent, avoiding the generation of data islands.

[0070] (4) Improved data retrieval capabilities: The efficient search capabilities of the Elasticsearch engine are utilized to improve data retrieval efficiency and provide rapid data support for management decisions.

[0071] (5) Enhanced system flexibility and scalability: The system design takes flexibility and scalability into consideration, and the logic of data extraction and processing can be adjusted according to the characteristics and requirements of the data.

[0072] (6) Improved user experience: Real-time data display and efficient data retrieval improve the user experience of website users.

[0073] (7) Ensure data security and privacy: During the process of data extraction, conversion, and loading, sensitive information is cleaned and desensitized to ensure data security and user privacy.

[0074] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A data exchange system based on Logstash, characterized by: Including relational database, Logstash data processing center, Elasticsearch cluster, message queue and website front end; The relational database is used to store government data; The Logstash data processing center is used to extract data from the relational database at regular intervals; The Elasticsearch cluster is used to store data processed by the Logstash data processing center; The message queue is used to receive deletion events of the relational database; The website front end is used to read data from the Elasticsearch cluster and display it.

2. The data exchange system based on Logstash according to claim 1, characterized in that: The Logstash data processing center is configured with an input plug-in, a filter plug-in and an output plug-in; The input plug-in is used to connect to the relational database and extract data regularly; The filter plug-in is used to convert the extracted data; The output plug-in is used to write the processed data to the Elasticsearch cluster and delete the data in the Elasticsearch cluster according to the deletion event in the message queue.

3. The data exchange system based on Logstash according to claim 1, characterized in that: The website front end is responsible for reading data from the Elasticsearch cluster through the REST API of the Elasticsearch cluster.

4. A data exchange method based on Logstash, characterized in that: The following steps are involved: Step S1: Configure the input plug-in in the Logstash data processing center, set a scheduled task, and extract data from the relational database; Step S2: Configure the filter plug-in in the Logstash data processing center to convert the extracted data, including data cleaning and format conversion, to remove sensitive information and convert the extracted data into a custom format; Step S3: Configure the output plug-in in the Logstash data processing center and specify the connection information and index settings of the Elasticsearch cluster; Use the output plug-in to write the processed data to the Elasticsearch cluster and form an index so that the website front end can read it; Step S4: When the data in the relational database is deleted, the Logstash data processing center is notified through a trigger mechanism; After receiving the deletion notification, the Logstash data processing center calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data in the Elasticsearch cluster. In the step S1, when configuring the input plug-in in the Logstash data processing center, the connection information and query statement of the relational database are specified; Use the schedule function of the Logstash data processing center to customize the time interval for data extraction.

5. The data exchange method based on Logstash according to claim 4, characterized in that: In step S3, the deletion status of the data is recorded through the filter plug-in and a deletion mark is added.

6. The data exchange method based on Logstash according to claim 4, characterized in that: In step S4, when the data in the relational database is deleted, the corresponding data in the Elasticsearch cluster is deleted synchronously, and the steps are as follows: Step S4.1, set a trigger in the relational database, when the data is deleted, the trigger publishes the deletion event to the message queue; Step S4.2, the Logstash data processing center subscribes to the above message queue and receives the deletion event; Step S4.3: The Logstash data processing center constructs a deletion request based on the received deletion event, and calls the REST API of the Elasticsearch cluster through the output plug-in to delete the corresponding data.

7. A data exchange device based on Logstash, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 4 to 6 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 4 to 6 is implemented.

Images