Data synchronization method, system and equipment based on government affair service data search and medium

Abstract

The invention discloses a data synchronization method, system and device based on government affair service data search and a medium, belongs to the technical field of data search, and aims to solve the technical problem of how to avoid data expiration in index data and provide better and more timely search experience for users. Synchronizing the total data to Elasticsearch through kafka, and synchronously updating corresponding data according to the content of the received data; wherein the data updating mode specifically comprises the following steps: updating data through a data synchronization interface, and synchronizing the data to Elasticsearch through kafka; the method comprises the following steps of: regularly acquiring data update through a timer mode, and synchronizing the data update to Elasticsearch through kafka; and updating the data according to a manual specified mode, and synchronizing the data to Elasticsearch through kafka.

Description

Technical Field

[0001] The present invention relates to the field of data search technology, and in particular to a data synchronization method, system, device and medium based on government service data search. Background Art

[0002] In the field of government services, the Elasticsearch search engine is currently used to develop and implement search functions, relying on the inverted indexing capabilities it provides to supplement the problem of insufficient fuzzy search capabilities in the database. For example, in the past, searching for "a certain insurance card" could not retrieve the "a certain association insurance card collection" item, which was solved through the search engine. However, the problem of expired data in the indexed data has not been well solved, and the retrieved content has expired and become obsolete.

[0003] Therefore, how to avoid data expiration in index data and provide users with a better and more timely search experience is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a data synchronization method, system, device and medium based on government service data search to solve the problem of how to avoid data expiration in index data and provide users with a better and more timely search experience.

[0005] The technical task of the present invention is achieved in the following way: a data synchronization method based on government service data search, which synchronizes the full amount of data to Elasticsearch through kafka through http request, and then synchronously updates the corresponding data according to the content of the received data; wherein the method of updating data is as follows:

[0006] Update data through the data synchronization interface and synchronize it to Elasticsearch through Kafka;

[0007] Get data updates regularly through a timer and synchronize them to Elasticsearch through Kafka;

[0008] Update data according to manually specified methods and synchronize it to Elasticsearch through Kafka.

[0009] As a preference, updating data through the data synchronization interface is as follows:

[0010] When the business system changes data, push the data for update synchronization;

[0011] After the interface receives the data, it writes it to Kafka;

[0012] Synchronize consumption data to Elasticsearch through service subscription.

[0013] Preferably, the data is updated by timing with a timer as follows:

[0014] Request the business party interface for incremental update every hour, and request the business party interface for full update every day;

[0015] Among them, the incremental update is processed by the update time or the total number of records.

[0016] More preferably, the data is updated according to the manually specified method as follows:

[0017] Provide a graphical page for manual data addition, deletion, editing operations and immediately initiate a timed task for update.

[0018] A data synchronization system based on government service data search, the system includes:

[0019] A full data synchronization module, used to synchronize the full data to Elasticsearch through kafka by means of an http request;

[0020] An update synchronization module, used to synchronize and update the corresponding data according to the content of the received data; among them, the method for the update synchronization module to update the data is as follows:

[0021] ① Update the data through the data synchronization interface and synchronize it to Elasticsearch through kafka;

[0022] ② Obtain data updates regularly through the timer method and synchronize them to Elasticsearch through kafka;

[0023] ③ Update the data according to the manually specified method and synchronize it to Elasticsearch through kafka.

[0024] Preferably, the data is updated through the data synchronization interface as follows:

[0025] When the business system changes the data, push the data for update synchronization;

[0026] After the interface receives the data, write it into kafka;

[0027] Consume the data through service subscription and synchronize it to Elasticsearch.

[0028] More preferably, the data is updated by timing with a timer as follows:

[0029] Request the business party interface for incremental update every hour, and request the business party interface for full update every day;

[0030] Among them, incremental updates are processed by update time or total number of records.

[0031] More preferably, the data is updated according to the manually specified method as follows:

[0032] A graphical page is provided for manual data addition, deletion, editing operations, and immediate initiation of a scheduled task for updating.

[0033] Elasticsearch is a distributed open-source search and analytics engine that uses an inverted index and is suitable for fast full-text search, providing near-real-time data search and analysis.

[0034] Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action flow data of consumers on a website. It is used to synchronize data into Elasticsearch.

[0035] An electronic device includes: a memory and at least one processor;

[0036] Wherein, a computer program is stored on the memory;

[0037] The at least one processor executes the computer program stored in the memory, enabling the at least one processor to execute the data synchronization method based on government service data search as described above.

[0038] A computer-readable storage medium stores a computer program that can be executed by a processor to implement the data synchronization method based on government service data search as described above.

[0039] The data synchronization method, system, device, and medium based on government service data search of the present invention have the following advantages:

[0040] (1) The present invention has the advantage of relatively high data real-time performance: If the third-party system can notify change information, near-real-time data synchronization can be achieved; if the third-party system does not support notification, data synchronization is achieved through a timer method or staff processing, maintaining data real-time performance to a certain extent;

[0041] (2) The present invention is based on the retrieval service model of Elasticsearch to provide a retrieval service for users. The retrieval content can include types such as single matters, joint handling of one thing, consultations, complaints, and government affairs public information, providing a better search experience for users to handle affairs, solving the problem of expired data in the index data, and the data synchronization method is relatively simple and has low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings.

[0043] Appendix Figure 1 It is a schematic flow diagram of a data synchronization method based on government service data search. Specific implementation manners

[0044] With reference to the accompanying drawings of the specification and specific embodiments, the data synchronization method, system, device and medium of the present invention based on government service data search are described in detail as follows.

[0045] Embodiment 1:

[0046] As shown in the appendix Figure 1 This embodiment provides a data synchronization method based on government service data search. This method synchronizes all data to Elasticsearch through kafka via an http request, and then synchronizes and updates the corresponding data according to the content of the received data. Among them, the method of updating data is specifically as follows:

[0047] Update data through a data synchronization interface and synchronize it to Elasticsearch through kafka;

[0048] Regularly obtain data updates through a timer method and synchronize them to Elasticsearch through kafka;

[0049] Update data according to the manually specified method and synchronize it to Elasticsearch through kafka.

[0050] In this embodiment, updating data through a data synchronization interface is specifically as follows:

[0051] When the business system changes data, push the data for update synchronization;

[0052] After the interface receives the data, write it into kafka;

[0053] Consume data synchronization to Elasticsearch through service subscription.

[0054] In this embodiment, regularly obtaining data updates through a timer is specifically as follows:

[0055] Request the business party interface for incremental updates every hour and request the business party interface for full updates every day;

[0056] Among them, incremental updates are processed by update time or total number of records.

[0057] In this embodiment, updating data according to the manually specified method is specifically as follows:

[0058] Provide a graphical page for manual data addition, deletion, editing operations and immediately initiate a timed task for update.

[0059] Example 2:

[0060] This embodiment provides a data synchronization system based on government service data search. The system includes:

[0061] A full - volume data synchronization module, which is used to synchronize full - volume data to Elasticsearch through kafka via an http request;

[0062] An update synchronization module, which is used to synchronize and update corresponding data according to the content of the received data. Among them, the specific ways for the update synchronization module to update data are as follows:

[0063] ① Update data through a data synchronization interface and synchronize it to Elasticsearch through kafka;

[0064] ② Regularly obtain data updates in a timer mode and synchronize them to Elasticsearch through kafka;

[0065] ③ Update data according to the manually specified method and synchronize it to Elasticsearch through kafka.

[0066] In this embodiment, updating data through a data synchronization interface is specifically as follows:

[0067] When the business system changes data, push the data for update synchronization;

[0068] After the interface receives the data, write it into kafka;

[0069] Consume data through service subscription and synchronize it to Elasticsearch.

[0070] In this embodiment, regularly obtaining data updates in a timer mode is specifically as follows:

[0071] Request the business party interface for incremental updates every hour and request the business party interface for full - volume updates every day;

[0072] Among them, incremental updates are processed by update time or total number of items.

[0073] In this embodiment, updating data according to the manually specified method is specifically as follows:

[0074] Provide a graphical page for manual data addition, deletion, editing operations and immediately initiate a timed task for update.

[0075] Example 3:

[0076] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0077] Wherein, the memory stores computer-executable instructions;

[0078] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data synchronization method based on government service data search in any embodiment of the present invention.

[0079] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0080] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory may also include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, at least one magnetic disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0081] Embodiment 4:

[0082] This embodiment also provides a computer-readable storage medium, which stores multiple instructions. The instructions are loaded by the processor to make the processor execute the data synchronization method based on government service data search in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided. The software program code for realizing the functions of any one of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0083] In this case, the program code read from the storage medium itself can realize the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0084] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0085] In addition, it should be clear that not only can the actual operations be completed in part or in whole by executing the program code read by the computer, but also by means of instructions based on the program code to cause an operating system operating on the computer, etc., so as to implement the functions of any one of the above embodiments.

[0086] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU, etc. installed on the expansion board or the expansion unit are caused to execute part and all of the actual operations, so as to implement the functions of any one of the above embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data synchronization method based on government service data search, characterized in that: This method uses http requests to synchronize all data to Elasticsearch through kafka, and then synchronously updates the corresponding data based on the content of the received data. The specific method of updating data is as follows: Update data through the data synchronization interface and synchronize it to Elasticsearch through Kafka; Get data updates regularly through a timer and synchronize them to Elasticsearch through Kafka; Update data according to manually specified methods and synchronize it to Elasticsearch through Kafka.

2. The data synchronization method based on government service data search according to claim 1 is characterized in that: The details of updating data through the data synchronization interface are as follows: When the business system changes data, push the data for update synchronization; After the interface receives the data, it writes it to Kafka; Synchronize consumption data to Elasticsearch through service subscription.

3. The data synchronization method based on government service data search according to claim 1 is characterized in that: The details of obtaining data updates regularly through the timer are as follows: Request the business side interface for incremental updates every hour and request the business side interface for full updates every day; Among them, incremental updates are processed by update time or total number of entries.

4. The data synchronization method based on government service data search according to any one of claims 1 to 3, characterized in that: The details of updating data according to manual designation are as follows: Provides a graphical page for manual data addition, deletion, and editing operations, as well as immediate initiation of scheduled tasks for updates.

5. A data synchronization system based on government service data search, characterized in that: The system includes: The full data synchronization module is used to synchronize the full data to Elasticsearch through kafka through http requests; The update synchronization module is used to synchronously update the corresponding data according to the content of the received data; wherein, the update synchronization module updates the data in the following manner: ① Update data through the data synchronization interface and synchronize it to Elasticsearch through Kafka; ② Get data updates regularly through a timer and synchronize them to Elasticsearch through Kafka; ③Update data according to manually specified methods and synchronize it to Elasticsearch through Kafka.

6. The data synchronization system based on government service data search according to claim 5 is characterized in that: The details of updating data through the data synchronization interface are as follows: When the business system changes data, push the data for update synchronization; After the interface receives the data, it writes it to Kafka; Synchronize consumption data to Elasticsearch through service subscription.

7. The data synchronization system based on government service data search according to claim 5 or 6, characterized in that: The details of obtaining data updates regularly through the timer are as follows: Request the business side interface for incremental updates every hour and request the business side interface for full updates every day; Among them, incremental updates are processed by update time or total number of entries.

8. The data synchronization system based on government service data search according to claim 7 is characterized in that: The details of updating data according to manual designation are as follows: Provides a graphical page for manual data addition, deletion, and editing operations, as well as immediate initiation of scheduled tasks for updates.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data synchronization method based on government service data search as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data synchronization method based on government service data search as described in any one of claims 1 to 5.

Images