ES and applet-based data entry and loading method and system

Abstract

The invention relates to the technical field of data interaction, discloses a data entry and loading method and system based on an ES and an applet, and aims to solve the problem that the existing applet is low in speed when processing mass data, and the scheme mainly comprises the following steps: receiving a data entry request sent by using the applet; reading and verifying input data in the data input request by utilizing a Redis cache, and asynchronously dropping the input data to a MySQL database by utilizing a kafka message queue if verification is passed; and carrying out change monitoring on the binary log of the MySQL database, identifying corresponding input data according to change information in the binary log, and transferring the corresponding input data to the ES database. Receiving a data query request sent by utilizing the applet; querying corresponding data in an ES database according to the data query request; and rendering the acquired data to a page in a list form by the applet. According to the method, the data entry and loading speed is increased, the data interaction experience of the user is improved, and the method is particularly suitable for mass data interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of data interaction, and in particular to a data entry and loading method and system based on ES and mini-programs. Background Art

[0002] Mini programs are applications that can be used without downloading or installing. Users can open the application by scanning or searching, without worrying about whether to install too many applications. With the rapid development of mobile Internet, mini programs have become an important window for enterprises to serve users. However, when processing massive amounts of data, mini programs often face problems such as slow query and loading speeds and poor interactive experience. Summary of the invention

[0003] The present invention aims to solve the problem that existing mini-programs are slow in processing massive data, and proposes a data entry and loading method and system based on ES and mini-programs.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] In a first aspect, the present invention provides a data entry method based on ES and a mini-program, the method comprising:

[0006] Step 101, the microservice cluster receives a data entry request sent by the mini program;

[0007] Step 102: Use Redis cache to read and verify the input data in the data input request from the microservice cluster. If the verification passes, use the kafka message queue to asynchronously write the input data to the MySQL database.

[0008] Step 103: Monitor changes in the binary log of the MySQL database, identify corresponding input data according to the change information in the binary log, and transfer the corresponding input data to the ES database.

[0009] Furthermore, the step 103 specifically includes:

[0010] Create the link-canal middleware and preset the transfer configuration table that stores the data tables that need to be monitored;

[0011] A consumption topic is dynamically generated according to the transfer configuration table. The link-canal middleware pulls binary logs from the consumption topic through Kafka concurrent consumption and parses the change information in the binary logs.

[0012] The input data corresponding to the change information is transferred to the ES database.

[0013] In a second aspect, the present invention provides a data loading method based on ES and a mini-program, the method comprising:

[0014] Step 201: The microservice cluster receives a data query request sent by a mini program;

[0015] Step 202: query the corresponding data in the ES database according to the data query request, and return the queried data to the mini program; the data in the ES database is entered by the data entry method based on ES and the mini program as described in claim 1 or 2;

[0016] Step 203: The applet renders the acquired data to the page in the form of a list.

[0017] Furthermore, the method further comprises:

[0018] Monitor the system load and data delay of the ES database in real time. If the system load or data delay is abnormal, query the corresponding data in the MySQL database according to the query request, and return the queried data to the mini program.

[0019] Furthermore, the mini program uses lazy loading to load data, including delaying data query and data loading for non-essential data, and querying and loading data according to the visible area of ​​the current page.

[0020] Furthermore, the method further comprises:

[0021] The number of queries for the data queried by the mini program is counted. If the number of queries is greater than a preset number, the corresponding data is regarded as hot data and stored in the Redis cache;

[0022] The corresponding data is preferentially queried in the Redis cache according to the data query request. If the corresponding data is not found, the corresponding data is queried in the ES database according to the data query request.

[0023] In a third aspect, the present invention provides a data entry system based on ES and a mini-program, wherein the system is used to implement the steps of the data entry method based on ES and a mini-program as described in the first aspect.

[0024] In a fourth aspect, the present invention provides a data loading system based on ES and a mini-program, wherein the system is used to implement the steps of the data loading method based on ES and a mini-program as described in the second aspect.

[0025] The beneficial effects of the present invention are as follows: the data entry and loading method and system based on ES and mini-programs provided by the present invention, the mini-programs enter and load big data based on ES (Elasticsearch), use cache to verify data during data entry, use kafka message queue to asynchronously drop data to disk, and use monitoring binary logs to synchronize the business narrow table data of the MySQL database to the wide table of the ES database. Due to the use of ES cluster deployment and sharding index technology, the speed of data entry, query and loading can be significantly improved, and users can quickly obtain the required information to optimize the query performance of massive data on the mini-program side and the user's data interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a data entry method based on ES and mini-programs provided in an embodiment of the present invention;

[0027] Figure 2 A flowchart of another data entry method based on ES and mini-programs provided in an embodiment of the present invention;

[0028] Figure 3 A flowchart of a data loading method based on ES and mini-programs provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] In some of the processes described in the specification of the present invention and the above-mentioned figures, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0031] The technical solution of the embodiment of the present invention is applicable to application scenarios in which mini programs are used to interact with massive data, such as WeChat mini programs, Alipay mini programs, etc. The inventors have found through research that mini programs currently face problems such as slow query and loading speeds and poor interactive experience when processing massive data.

[0032] In order to improve the efficiency of data entry, query and loading, the technical solution of the present invention is proposed. In the embodiment of the present invention, when using the applet for data entry, the microservice cluster receives the data entry request sent by the applet; the Redis cache is used to read and verify the entry data in the data entry request from the microservice cluster. If the verification passes, the kafka message queue is used to asynchronously drop the entry data to the MySQL database; the binary log of the MySQL database is monitored for changes, the corresponding entry data is identified according to the change information in the binary log, and the corresponding entry data is transferred to the ES database. When using the applet for data loading, the microservice cluster receives the data query request sent by the applet; the corresponding data is queried in the ES database according to the data query request, and the queried data is returned to the applet; the data in the ES database is entered by the data entry method based on ES and applet as described in claim 1 or 2; the applet renders the acquired data to the page in the form of a list. The present invention utilizes the cluster deployment and sharding indexing technology of ES to significantly improve the speed of data entry, query and loading, and users can quickly obtain the required information.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0034] Figure 1 A flow chart showing a data entry method based on ES and applet is shown in FIG. Figure 1 , the method comprises the following steps:

[0035] Step 101: The microservice cluster receives a data entry request sent by a mini program.

[0036] See also Figure 2 In actual applications, for the data that needs to be entered in real time generated by the mini program, a corresponding data entry request is generated according to the entered data, and then sent to the ES (Elasticsearch) microservice cluster.

[0037] Elasticsearch is a real-time distributed storage, search and analysis engine that stores data on different nodes of the cluster through a sharding mechanism to achieve distributed data processing and scalability. In ES, a microservice cluster usually refers to deploying Elasticsearch nodes on different machines and communicating with each other through Elasticsearch's default discovery mechanism (such as Zen Discovery). Such a cluster can provide advantages such as high availability, high scalability and load balancing.

[0038] Step 102: Use Redis cache to read and verify the input data in the data input request from the microservice cluster. If the verification passes, use the kafka message queue to asynchronously write the input data to the MySQL database.

[0039] Among them, Redis cache is a buffer for data exchange with high read and write performance. Using Redis cache for data verification can improve data access speed and reduce database pressure.

[0040] Kafka message queue is a distributed messaging system that is mainly used to process high-throughput data streams. Kafka message queue is designed to handle large amounts of data while ensuring the persistence, reliability, and order of messages.

[0041] The embodiment of the present invention uses Redis cache to perform validity verification on the input data in the data entry request. After the validity verification passes, the kafka message queue is used to asynchronously write the input data to the MySQL database, thereby improving the efficiency and reliability of data entry.

[0042] Step 103: Monitor changes in the binary log of the MySQL database, identify corresponding input data according to the change information in the binary log, and transfer the corresponding input data to the ES database.

[0043] In the embodiment of the present invention, step 103 specifically includes:

[0044] Step 1031: create link-canal middleware, and preset a transfer configuration table that stores the data table that needs to be monitored.

[0045] Step 1032: dynamically generate a consumption topic according to the transfer configuration table, and the link-canal middleware pulls the binary log from the consumption topic through Kafka concurrent consumption and parses the change information in the binary log.

[0046] Step 1033: transfer the input data corresponding to the change information to the ES database.

[0047] Specifically, the business narrow table data of the MySQL database is synchronized to the wide table of the ES database in real time through the middleware link-canal. In this process, the canal server is used to monitor the binary log (binlog) of the MySQL database, and the canal client is reconstructed to process the monitored data. Among them, the binary log is a log file in the MySQL database, which records all statements that change data in the MySQL database and is used for replication and data recovery.

[0048] In practical applications, firstly, the embodiment of the present invention designs a transfer configuration table, and only monitors the data table in the transfer configuration table for binary log changes, thereby reducing the binary log monitoring of irrelevant business tables.

[0049] Secondly, the Redis technology is used to cache the conversion configuration table, and the binary log content is directly parsed through the cache, thereby improving the efficiency of data transfer. In addition, for binary log data processing, multi-partition technology is used to process data concurrently through Kafka consumption. That is, according to the conversion configuration table, consumption topics are dynamically generated to achieve dynamic expansion processing of different subject data, further improving the efficiency of data transfer.

[0050] Among them, Kafka concurrent consumption refers to the use of multiple threads or processes to simultaneously process data from different partitions of Kafka to improve the efficiency and throughput of message processing. In Kafka, multiple consumer instances are configured to process different partitions in parallel to achieve concurrent consumption.

[0051] In addition, for data processing, the main table data and the related table data are logically distinguished, and different consumption themes are used when consuming, which can greatly improve the real-time transfer efficiency of the main table data and achieve almost zero delay. For the related table data, the consumers can be horizontally expanded to improve the real-time transfer capability.

[0052] Figure 3 A flow chart showing a data loading method based on ES and applet is shown in Figure 3 , the method comprises the following steps:

[0053] Step 201: The microservice cluster receives a data query request sent by a mini program.

[0054] Step 202: query the corresponding data in the ES database according to the data query request, and return the queried data to the mini program.

[0055] See also Figure 2 In actual applications, the ES database encapsulates a set of standard list query interfaces. List query, export, report, and analysis for users are all implemented through the ES database, thereby achieving fast query functions. When the microservice cluster receives a data query request, it queries the ES database by default. If data is found, the queried data is returned to the mini program.

[0056] In an embodiment of the present invention, the method further comprises:

[0057] Monitor the system load and data delay of the ES database in real time. If the system load or data delay is abnormal, query the corresponding data in the MySQL database according to the query request, and return the queried data to the mini program.

[0058] Specifically, for massive data queries, the default is to query the ES database. The backend service will monitor the system load and ES latency in real time. If the system detects an abnormality, the query will automatically switch to the MySQL database query and send a message notification to the system administrator, thereby reducing the risk of system abnormalities.

[0059] Step 203: The applet renders the acquired data to the page in the form of a list.

[0060] In an embodiment of the present invention, the mini program loads data in a lazy loading manner, specifically including delaying data query and data loading for non-essential data, and querying and loading data according to the visible area of ​​the current page.

[0061] Lazy loading is an optimization strategy based on lazy loading technology, which is mainly used to improve web page performance and user experience. It reduces the initial page loading time and network traffic by delaying loading or loading on demand. Lazy loading mainly targets images and other media resources in web pages, and optimizes page performance by delaying or loading non-essential resources on demand. The implementation principle of front-end lazy loading is to load only necessary resources when the page is first loaded, and delay or load other non-essential resources on demand. For image resources, lazy loading technology can place them near the visible area of ​​the page, or load them according to the user's scrolling behavior to reduce initial loading time and network traffic.

[0062] In actual applications, lazy loading technology is used on the mini program side to delay data loading. First, data is loaded only when the user needs it, reducing the initial loading time and bandwidth usage. Second, in the list page, only the data items in the user's current visible area are loaded. That is, after the user slides the page, a request is sent to load subsequent batches of data, but only the data items in the current visible area are loaded to the user. When the user operates the mini program interface to scroll down the list, the new list content is loaded.

[0063] In an embodiment of the present invention, it also includes counting the number of queries for the data queried by the mini program. If the number of queries is greater than the preset number, the corresponding data is used as hot data and stored in the Redis cache; the corresponding data is preferentially queried in the Redis cache according to the data query request. If the corresponding data is not found, the corresponding data is queried in the ES database according to the data query request. By querying the hot data in the Redis cache, the query efficiency of the hot data can be improved, and the efficiency of data loading can be further improved.

[0064] In summary, the data entry method based on ES and mini-programs provided in the embodiment of the present invention uses cache for data verification during data entry, uses kafka message queues for asynchronous data storage, and synchronizes the business narrow table data of the MySQL database to the wide table of the ES database by designing a transfer configuration table and using Kafka concurrent consumption to monitor binary logs, thereby improving data transfer efficiency. When loading data, the database is switched by monitoring the system status to ensure the reliability of data query, and the data is loaded in a lazy loading manner, which reduces the initial loading time and bandwidth usage. In addition, the embodiment of the present invention can significantly improve the data entry, query and loading speed due to the use of ES cluster deployment and sharding indexing technology, and users can quickly obtain the required information to optimize the query performance and user experience of massive data on the mini-program side.

[0065] Based on the above technical solution, this embodiment also proposes a data entry system based on ES and mini-programs, and the system is used to implement the steps of the data entry method based on ES and mini-programs described in the embodiment of the present invention.

[0066] Based on the above technical solution, this embodiment also proposes a data loading system based on ES and mini-programs, and the system is used to implement the steps of the data loading method based on ES and mini-programs described in the embodiment of the present invention.

[0067] It can be understood that since the data entry and loading system based on ES and mini-programs described in the embodiment of the present invention is a system for implementing the data entry and loading method based on ES and mini-programs described in the embodiment, for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For relevant matters, please refer to the partial description of the method, which will not be repeated here.

Claims

1. A data entry method based on ES and applet, characterized in that: The method comprises: Step 101, the microservice cluster receives a data entry request sent by the mini program; Step 102: Use Redis cache to read and verify the input data in the data input request from the microservice cluster. If the verification passes, use the kafka message queue to asynchronously write the input data to the MySQL database. Step 103: Monitor changes in the binary log of the MySQL database, identify corresponding input data according to the change information in the binary log, and transfer the corresponding input data to the ES database.

2. The data entry method based on ES and applet according to claim 1, characterized in that: The step 103 specifically includes: Create the link-canal middleware and preset the transfer configuration table that stores the data tables that need to be monitored; A consumption topic is dynamically generated according to the transfer configuration table. The link-canal middleware pulls binary logs from the consumption topic through Kafka concurrent consumption and parses the change information in the binary logs. The input data corresponding to the change information is transferred to the ES database.

3. A data loading method based on ES and applet, characterized in that: The method comprises: Step 201: The microservice cluster receives a data query request sent by a mini program; Step 202: query the corresponding data in the ES database according to the data query request, and return the queried data to the mini program; the data in the ES database is entered by the data entry method based on ES and the mini program as described in claim 1 or 2; Step 203: The applet renders the acquired data to the page in the form of a list.

4. The data loading method based on ES and applet according to claim 2 is characterized in that: The method further comprises: Monitor the system load and data delay of the ES database in real time. If the system load or data delay is abnormal, query the corresponding data in the MySQL database according to the query request, and return the queried data to the mini program.

5. The data loading method based on ES and applet according to claim 2 is characterized in that: The mini program uses lazy loading to load data, including delayed data query and data loading for non-essential data, and data query and data loading based on the visible area of ​​the current page.

6. The data loading method based on ES and applet according to claim 2 is characterized in that: The method further comprises: The number of queries for the data queried by the mini program is counted. If the number of queries is greater than a preset number, the corresponding data is regarded as hot data and stored in the Redis cache; The corresponding data is preferentially queried in the Redis cache according to the data query request. If the corresponding data is not found, the corresponding data is queried in the ES database according to the data query request.

7. A data entry system based on ES and applet, characterized in that: The system is used to implement the steps of the data entry method based on ES and applet as described in claim 1 or 2.

8. A data loading system based on ES and applet, characterized in that: The system is used to implement the steps of the data loading method based on ES and applet as described in any one of claims 3 to 6.

Images