Real-time index updating method and updating device thereof

Abstract

The application discloses a real-time index updating method and an updating device thereof, belongs to the technical field of real-time calculation, can effectively solve the data jitter problem in the real-time index updating process, can effectively solve the data delay problem in the real-time index updating process, is completely transparent to the end user in the process of real-time index updating, and realizes decoupling of the index updating and index use processes; a metadata storage unit is used for maintaining a registration information table of an index set, the registration information table comprises the following fields: a master unit is used for receiving an index set updating request, registering the index set to the metadata storage unit, and submitting a report data calculation job to a Flink cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time computing, and in particular to a real-time index updating method and an updating device thereof. BACKGROUND

[0002] With the development of big data technology, many companies have higher and higher requirements for the timeliness of data processing, so there are real-time computing frameworks such as Flink, which allow data development engineers to submit Flink jobs to Flink clusters, and the computing results of the Flink jobs can be output to external storage systems in real time.

[0003] Here, the above-mentioned Flink job is referred to as an index computing job, and the above-mentioned storage system is referred to as an index database, which contains a plurality of index sets; each index set contains a plurality of real-time indexes, simply referred to as indexes; if the index database is a relational database, then one index set corresponds to one table, and the index is a field in the table. The observer (also known as the user) of the index is referred to as an end user, who usually observes the changes of the index by means of a visual report tool such as DataV or FanSoft.

[0004] The following is a specific embodiment:

[0005] Index set: corresponding to a table in a relational database mysql, assuming that the name of the table is index_table, and the field definition is as follows: each field corresponds to an index;

[0006] Field name Field type Meaning Indicator 1 Any An indicator from the set of indicators Indicator 2 Any An indicator from the set of indicators Indicator 3 Any An indicator from the set of indicators

[0007] Index computing job: corresponding to a Flink SQL job, the content of the job is a sql file, and the job is submitted to a Flink cluster to output the result to index_table in real time;

[0008] Index updating includes two cases:

[0009] First, the index set adds a new index, and correspondingly, the computing logic of the new index needs to be added and the historical data of the index needs to be calculated (this needs to be);

[0010] Second, the index set modifies the computing logic of a certain index, and correspondingly, the historical data of the index needs to be backflushed.

[0011] When updating the index, if the original index computing job is stopped, modified, and then resubmitted, two problems will be caused:

[0012] One is data delay: the starting, modifying and stopping of the index calculation job consumes several minutes, which means that the real-time index cannot be updated and the users of the real-time report cannot obtain the latest value of the index they care about in time during the several minutes;

[0013] Two is data jitter: after the index calculation job is modified, if we start the index calculation job from a new state (for example, let the job start from the initial point of the message middleware and consume from the beginning), the historical data of the real-time index can be recalculated, and we call this process index backflushing. At this time, the real-time index output by Flink will experience a dramatic change from history to now, and the duration is several seconds to several minutes. We call this dramatic change of the index in a short time data jitter, which will confuse the end users of the index. SUMMARY

[0014] The present application is to solve the problem of data jitter and data delay in real-time index updating, and to provide a real-time index updating method and updating device which can effectively solve the problem of data jitter in real-time index updating, can effectively solve the problem of data delay in real-time index updating, is completely transparent to the end users in the process of real-time index updating, decouples the processes of index updating and index use, and has high reliability.

[0015] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0016] The real-time index updating device comprises:

[0017] A metadata storage unit: used for maintaining a registration information table of the index set, wherein the registration information table comprises the following fields:

[0018] (1.1) table: the name of the index set;

[0019] (1.2) flink_sql_script: the content of the real-time index calculation job;

[0020] (1.3) db_url: the connection address of the index database;

[0021] (1.4) status: the status of the index set, including the two statuses of updating and updating;

[0022] (1.5) view: the name of the index view; here, the index view exists in the index database and is used for pointing to the corresponding index set;

[0023] (1.6) start_time: the time when the real-time index starts to update;

[0024] (1.7) end_time: the time when the real-time index update ends;

[0025] The master control unit is configured to receive the index set update request, register the index set to the metadata storage unit, and submit a report data calculation job to the Flink cluster.

[0026] Preferably, the real-time index update device further comprises a Flink cluster, which provides a running environment for the execution of the Flink job; and a plurality of Flink jobs can be submitted to the Flink cluster by a user.

[0027] Preferably, the real-time index update device further comprises a report display tool, which is configured to configure and display a visual report.

[0028] Preferably, the report display tool comprises DATAV and FANQI.

[0029] Preferably, if the end user observes an index view rather than directly using the index set, the purpose of “updating the index” can be achieved by changing the index view.

[0030] Preferably, each report corresponds to a Flink job in the Flink cluster.

[0031] Preferably, the real-time index update device further comprises a job monitoring unit, which is configured to complete the index job in the update of all statuses in the report registration information table of the metadata storage unit in a fixed period.

[0032] A real-time index update method, the implementation steps are as follows:

[0033] Step 101, a data developer initiates an update request to the real-time index update device;

[0034] Step 102, the master control unit of the real-time index update device receives the update request, extracts the table parameter from the request parameter, and queries whether the table exists in the index registration information table of the metadata storage unit;

[0035] Step 102-1, if it does not exist, it means that it is a newly online index set, the master control unit inserts a new record into the report registration information table, and sets the status of the record to updating;

[0036] Step 102-2, if it exists, it means that the report has been online, the master control unit updates the record in the report registration information table, and sets the status to updating;

[0037] Step 103, submit a Flink SQL job to the Flink cluster, and the job content is flink_sql_script;

[0038] Step 201, the job monitoring unit obtains all records with status as updating from the report registration information table of the metadata storage unit in a fixed period, extracts the table field, and puts it into the to-be-updated set; each element in the set corresponds to an index set name;

[0039] Step 202, traverse the to-be-updated set, and find the corresponding record in the index set registration information table for the index set name in the set;

[0040] Step 203, if the current system time is greater than or equal to end_time, connect the index database using the db_url extracted from the record;

[0041] Step 204, extract the index view name view from the record; execute the data operation statement in the index database to point view to table;

[0042] Step 205, set the status field of the record to updated, and update it to the report registration information table.

[0043] As preferred, the parameters of the update request include:

[0044] 1) table: the name of the index set;

[0045] 2) flink_sql_script: the content of the real-time index calculation job, which is a string, and usually the content contains an sql script executed on the flink cluster;

[0046] 3) db_url: the connection address of the index database, which is a string;

[0047] 4) status: the status of the index set, including "updating" and "updated";

[0048] 5) view: the name of the index view; here, the index view exists in the index database, and is used to point to the corresponding index set; if the end user observes the index view instead of directly using the index set, the purpose of "updating the index" can be achieved by changing the index view pointing method;

[0049] 6) end_time: the time when the real-time index ends updating.

[0050] As preferred, the parameters of the update request include: the record inserted in step 102-2, wherein step 102-1 and step 102-2 are as follows:

[0051] table, flink_sql_script, db_url, status = updating, end_time.

[0052] The present application can achieve the following effects:

[0053] The device and method of the present application can effectively solve the data jitter problem in the real-time index updating process, can effectively solve the data delay problem in the real-time index updating process, and is completely transparent to the end user in the real-time index updating process, decouples the index updating and index using processes, and has high reliability. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 It is a block diagram of the real-time index updating device of the present application.

[0055] Figure 2 It is a schematic diagram of the system interaction diagram of the present application. DETAILED DESCRIPTION

[0056] The technical solutions of the present application will be further specifically described below by examples in combination with the drawings.

[0057] Example: real-time index updating device, see Figure 1 As shown in the figure, the real-time index updating device 6 includes:

[0058] Metadata storage unit 1: used for maintaining the registration information table of the index set, and the registration information table includes the following fields:

[0059] (1.1) table: name of the index set;

[0060] (1.2) flink_sql_script: content of the real-time index calculation job; type is string; usually, the content contains the sql script executed on the flink cluster;

[0061] (1.3) db_url: connection address of the index database; string type;

[0062] (1.4) status: status of the index set, including the two status of the index set, updating and updated;

[0063] (1.5) view: The name of the indicator view; here, the indicator view exists in the indicator database and is used to point to the corresponding indicator set; almost all relational databases support the concept of view, and many non-relational databases (such as HBase) also support it; if the end user observes the indicator view instead of directly using the indicator set, the purpose of "updating the indicator" can be achieved by changing the indicator view to point to.

[0064] (1.6) start_time: The time when the real-time metric starts updating;

[0065] (1.7) end_time: The time when the real-time metric ends its update;

[0066] Main control unit 2: Used to receive indicator set update requests, register indicator sets to the metadata storage unit, and submit report data calculation jobs to the Flink cluster; each report corresponds to a Flink job in the Flink cluster.

[0067] The real-time indicator update device also includes:

[0068] Flink Cluster 3: Provides a runtime environment for executing Flink jobs; users can submit multiple Flink jobs to the Flink cluster.

[0069] Reporting tool 4: Used for configuring and displaying visual reports; reporting tools include DATAV and FineReport.

[0070] The operation monitoring unit 5 is a timer that performs the following operations within a fixed period.

[0071] A real-time indicator update method, the steps of which are as follows: See Figure 1 As shown,

[0072] Step 101: The data developer sends an update request to the real-time indicator update device;

[0073] The parameters of the update request include:

[0074] 1) table: The name of the indicator set;

[0075] 2) flink_sql_script: The content of the real-time metric calculation job, which is a string. This content usually contains the SQL script that is executed on the Flink cluster.

[0076] 3) db_url: The connection address of the indicator database, a string type;

[0077] 4) status: The status of the indicator set, including the two statuses of "updating" and "updated";

[0078] 5) view: name of the metric view; here, the metric view exists in the metric database, and is used to point to the corresponding metric set; almost all relational databases support the concept of views, and many non-relational databases (such as hbase) also support it; if the end user observes the metric view instead of directly using the metric set, the purpose of "updating the metric" can be achieved by changing the way the metric view points;

[0079] 6) end_time: time at which the real-time metric ends updating;

[0080] Step 102, the main control unit of the real-time metric updating device receives an updating request, extracts the table parameter from the parameters of the request, and queries the metric registration information table of the metadata storage unit to query whether the table exists;

[0081] Step 102-1, if it does not exist, it means that it is a newly online metric set, and the main control unit inserts a new record into the report registration information table (inserts the corresponding value into the corresponding field according to the field name), and sets the status of the record to updating;

[0082] Step 102-2, if it exists, it means that the report has been online, and the main control unit updates the record in the report registration information table and sets the status to updating;

[0083] Among them, the record inserted in step 102-1 and step 102-2 is as follows:

[0084] table, flink_sql_script, db_url, status = updating, end_time;

[0085] Step 103, submit a Flink SQL job to the Flink cluster, and the job content is flink_sql_script;

[0086] Step 201, the job monitoring unit obtains all records with status updating in the report registration information table of the metadata storage unit in a fixed period, extracts the table field, and puts it into the updating set; each element in the set corresponds to a metric set name;

[0087] Step 202, traverse the updating set, and find the corresponding record in the metric set registration information table for the metric set name in it;

[0088] Step 203, if the current system time is greater than or equal to end_time, connect the metric database using the db_url extracted from the record;

[0089] Step 204, extracting the index view name view from the record; executing a data operation statement in the index database to point view to table;

[0090] Step 205, setting the status field of the record as updated, and updating to the report registration information table.

[0091] The device and method of the embodiment can effectively solve the data jitter problem in the real-time index updating process, can effectively solve the data delay problem in the real-time index updating process, is completely transparent to the end user in the process of real-time index updating, achieves decoupling of the index updating and index using processes, and has high reliability.

Claims

1. A real-time indicator update method, characterized in that, The implementation steps of the real-time indicator update method are as follows: Step 101: The data developer sends an update request to the real-time indicator update device; Step 102: The main control unit of the real-time indicator update device receives the update request, extracts the table parameter from the parameters of the request, and queries the indicator registration information table of the metadata storage unit to see if this table exists. Step 102-1: If it does not exist, it means that this is a newly launched indicator set. The main control unit inserts a new record into the report registration information table and sets the status of the record to "updating". Step 102-2: If it exists, it means that the report has been launched. The main control unit updates the record in the report registration information table and sets the status to "updating". Step 103: Submit a Flink SQL job to the Flink cluster. The job content is flink_sql_script. Step 201: Within a fixed period, the job monitoring unit retrieves all records with the status "updating" from the report registration information table of the metadata storage unit, extracts the "table" field, and puts them into the set to be updated; each element in this set corresponds to an indicator set name. Step 202: Traverse this set to be updated, and for each indicator set name, find the corresponding record in the indicator set registration information table; Step 203: If the current system time is greater than or equal to end_time, connect to the metrics database using the db_url extracted from the relevant record; Step 204: Extract the indicator view name (view) from the record; execute data manipulation statements in the indicator database to point the view to the table; Step 205: Set the status field of the record to updated and update it in the report registration information table.

2. The real-time indicator update method according to claim 1, characterized in that, The parameters of the update request include: 1) table: The name of the indicator set; 2) flink_sql_script: The content of the real-time metric calculation job, which is a string and contains the SQL script executed on the Flink cluster; 3) db_url: The connection address of the indicator database, a string type; 4) status: The status of the indicator set, including the two statuses of "updating" and "updated"; 5) view: The name of the indicator view; here, the indicator view exists in the indicator database and is used to point to the corresponding indicator set; if the end user observes the indicator view instead of directly using the indicator set, the purpose of "updating the indicator" can be achieved by changing the indicator view to point to. 6) end_time: The time when the real-time metric ends its update.

3. The real-time indicator update method according to claim 2, characterized in that, The parameters of the update request include: in step 102-2, where the records inserted in steps 102-1 and 102-2 are in the following form: table, flink_sql_script, db_url, status=updating, end_time.

4. A real-time index updating apparatus for executing the real-time index updating method of claim 1, characterized in that, include: Metadata storage unit (1): Used to maintain the registration information table of the indicator set, the registration information table containing the following fields: (1.1) table: The name of the index set; (1.2) flink_sql_script: The content of the real-time metric calculation job; (1.3) db_url: The connection address of the indicator database; (1.4) status: The status of the indicator set, including the status of indicator sets that are being updated and those that have been updated; (1.5) view: The name of the indicator view; here, the indicator view exists in the indicator database and is used to point to the corresponding indicator set; (1.6) start_time: The time when the real-time metric starts updating; (1.7) end_time: The time when the real-time metric ends its update; Main control unit (2): Used to receive indicator set update requests, register indicator sets to the metadata storage unit, and submit report data calculation jobs to the Flink cluster.

5. The real-time indicator update device according to claim 4, characterized in that, The real-time metrics update device also includes: Flink cluster (3): providing a runtime environment for the execution of Flink jobs; users can submit multiple Flink jobs to the Flink cluster.

6. The real-time indicator update device according to claim 4 or 5, characterized in that, The real-time indicator update device also includes: a report display tool (4): used to configure and display visual reports.

7. The real-time indicator update device according to claim 6, characterized in that, The real-time indicator update device also includes: the report display tools include DATAV and FineReport.

8. The real-time indicator update device according to claim 4, characterized in that, If the end user is observing a metric view instead of directly using a set of metrics, the purpose of "updating metrics" can be achieved by changing the target of the metric view.

9. The real-time indicator update device according to claim 4, 5, or 8, characterized in that, Each report corresponds to a Flink job in the Flink cluster.

10. The real-time indicator update device according to claim 4, characterized in that, The real-time indicator update device also includes: a job monitoring unit (5), which is used to obtain all indicator jobs with status as being updated from the report registration information table of the metadata storage unit within a fixed period.

Images