Real-time data monitoring method and device, electronic equipment and storage medium
By standardizing data and real-time data processing in the business data process, the problem of long-term data conversion and distribution steps in the existing technology is solved, efficient real-time data monitoring is achieved, and business scenario requirements with high timeliness requirements are met. .
Patent Information
- Application Number
- CN202411403693.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-02-11
AI Technical Summary
In the existing business process monitoring methods, the data conversion and data distribution steps are slow to process large amounts of business data in batches, which take a long time, making it difficult to meet the monitoring needs in business scenarios with high timeliness requirements.
Get real-time data of the business data system in the preset data portal and standardize it to obtain standard real-time data. Then, the standard real-time data is returned to the business data system, extracted to the preset database, and a model table of standard real-time data is built based on the database to achieve real-time monitoring.
By standardizing data before business data is extracted into the system, and using database technology to extract and store standard real-time data in real time, batch processing delays of traditional monitoring methods are avoided, monitoring timeliness is improved, and services with high timeliness requirements are met Monitoring requirements in scenarios.
Smart Images
Figure CN119415581A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data monitoring technology, and in particular to a real-time data monitoring method, device, electronic device and storage medium. Background Art
[0002] In the existing business process monitoring scenario, the steps of the monitoring operation mainly include five steps: data extraction, data conversion, data distribution, model loading and monitoring notification. The data extraction step is mainly to extract data from various sources into the system, the data conversion step is to convert different data into standard data that the system can process, the data distribution step is to synchronize the processed data to the clickhouse database, the model loading step is to integrate and calculate the data according to business rules to obtain the indicator data that needs to be monitored, and the monitoring step is to issue early warning notifications for the indicator data. At present, the entire operation chain is relatively long, among which the data conversion and data distribution steps are slow and time-consuming due to batch processing of large amounts of business data. In business scenarios with high timeliness requirements, it is difficult to meet the requirements for efficient monitoring of business data. Summary of the invention
[0003] The main purpose of this application is to provide a real-time data monitoring method, device, electronic device and storage medium, aiming to solve the technical problem of low efficiency in monitoring business data.
[0004] To achieve the above objectives, the present application proposes a real-time data monitoring method, which includes:
[0005] Acquire real-time data of a business data system in a preset data portal, and perform standardization processing on the real-time data to obtain standard real-time data;
[0006] Returning the standard real-time data to the business data system through the preset data portal;
[0007] Extracting standard real-time data from the business data system into a preset database, and constructing a model table of the standard real-time data based on the preset database;
[0008] In the process of monitoring the real-time data, the monitoring index data is obtained by querying the model table.
[0009] In one embodiment, the step of standardizing the real-time data to obtain standard real-time data includes:
[0010] According to the primary key of the real-time data, obtaining the original version number and the original deletion field of the real-time data;
[0011] The original version number is mapped to a standard version number according to a preset mapping method, and the original deletion field is mapped to a standard deletion field to obtain the standard real-time data.
[0012] In one embodiment, the preset database includes a first database and a second database, and the step of extracting the standard real-time data in the business data system to the preset database and constructing a model table of the standard real-time data based on the preset database includes:
[0013] Extracting the standard real-time data to a base table in the first database and the second database, wherein the second database backs up the standard real-time data to obtain historical standard real-time data stored in the second database;
[0014] Based on the business process end field configured in the first database, the historical standard real-time data in the second database is screened to obtain target historical standard real-time data that does not match the business process end field;
[0015] The target historical standard real-time data is sent to the base table, and a model table of the standard real-time data is constructed based on the base table.
[0016] In one embodiment, the step of constructing the model table of the standard real-time data based on the base table includes:
[0017] According to the primary keys of the standard real-time data and the target historical standard real-time data, the base table is screened to obtain the target table-building data in the base table, wherein the target table-building data is the standard real-time data newly added to the base table before the standard real-time data is received;
[0018] Constructing a model temporary table based on the target table building data, and obtaining the original model table;
[0019] The original model table is screened based on the target table building data to obtain the original table building data in the original model table that does not match the target table building data;
[0020] The original table building data is input into the model temporary table to obtain the model table, and the original model table is deleted.
[0021] In one embodiment, the step of screening the base table according to the primary keys of the standard real-time data and the target historical standard real-time data to obtain the target table building data in the base table includes:
[0022] Determine the reference update time of the base table according to the update time field corresponding to the primary key, wherein the update time field is recorded after the standard real-time data is extracted;
[0023] The base table is screened based on the reference update time, and the data in the base table whose update time is later than the reference update time is used as the target table creation data.
[0024] In one embodiment, the step of constructing a model temporary table based on the target table building data includes:
[0025] Remove the start time field and the end time field of the target table creation data, and obtain the time field corresponding to the current time;
[0026] The time field is appended to the target table creation data, and the model temporary table is constructed based on the target table creation data to which the time field is appended.
[0027] In one embodiment, the real-time data monitoring method further includes:
[0028] Detecting the time field of each table creation data in the model table based on a preset effective time;
[0029] When the duration between the current moment and the historical moment corresponding to any target time field in the time field is greater than the preset effective duration, the table creation data corresponding to the target time field is removed.
[0030] In addition, to achieve the above purpose, the present application also proposes a real-time data monitoring device, the real-time data monitoring device comprising:
[0031] A standardization module is used to obtain real-time data of a business data system in a preset data portal, and perform standardization processing on the real-time data to obtain standard real-time data;
[0032] A returning module, used for returning the standard real-time data to the business data system through the preset data portal;
[0033] A construction module, used to extract the standard real-time data in the business data system to a preset database, and construct a model table of the standard real-time data based on the preset database;
[0034] The monitoring module is used to obtain monitoring indicator data by querying the model table during the process of monitoring the real-time data.
[0035] In addition, to achieve the above objectives, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the real-time data monitoring method as described above.
[0036] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the real-time data monitoring method described above are implemented.
[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the real-time data monitoring method described above are implemented.
[0038] One or more technical solutions proposed in this application have at least the following technical effects:
[0039] The present application first obtains the real-time data of the business data system in the preset data portal, and standardizes the real-time data to obtain standard real-time data, so as to perform standardization in advance before the business data is extracted into the system, thereby avoiding the cumbersome data processing after the data from different channels are connected, which affects the execution efficiency; the standard real-time data is returned to the business data system through the preset data portal to realize the data closed loop, and provide an accurate and reliable data basis for subsequent monitoring; the standard real-time data in the business data system is extracted to the preset database, and a model table of the standard real-time data is constructed based on the preset database, so as to provide reliable data support for business monitoring; in the process of monitoring the real-time data, the monitoring indicator data is obtained by querying the model table, so as to realize real-time monitoring of the business indicator data by querying the model table, thereby quickly discovering abnormal situations, providing a basis for business decision-making, and ensuring the stable operation of business processes.
[0040] In summary, the present application standardizes the data before the business data is extracted into the system, and uses database technology in the system to extract and store standard real-time data in real time. This avoids the need for traditional monitoring methods to process data in batches only after a certain amount of business data has been accumulated in the business data system. This enables fast and efficient data processing after real-time data is acquired, thereby improving the timeliness of monitoring and effectively meeting the monitoring needs in business scenarios with high timeliness requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 A flowchart of the first embodiment of the real-time data monitoring method of the present application is provided;
[0044] Figure 2 A flowchart of the second embodiment of the real-time data monitoring method of the present application is provided;
[0045] Figure 3 A brief flowchart of the real-time data monitoring method provided in Example 2 of the present application;
[0046] Figure 4 This is a schematic diagram of the module structure of the real-time data monitoring device according to an embodiment of the present application;
[0047] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the real-time data monitoring method in the embodiment of the present application.
[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0051] The main solution of the embodiment of the present application is: obtaining real-time data of the business data system in a preset data portal, and standardizing the real-time data to obtain standard real-time data; returning the standard real-time data to the business data system through the preset data portal; extracting the standard real-time data in the business data system to a preset database, and constructing a model table of the standard real-time data based on the preset database; in the process of monitoring the real-time data, obtaining monitoring indicator data by querying the model table.
[0052] In existing business process monitoring scenarios, the steps of monitoring operations mainly include data extraction, data conversion, data distribution, model loading and monitoring notification. The entire operation chain is relatively long. The data conversion and data distribution steps are slow and time-consuming due to batch processing of large amounts of business data. In business scenarios with high timeliness requirements, it is difficult to efficiently monitor business data.
[0053] The present application provides a solution, which standardizes the business data before it is extracted into the system, and uses database technology in the system to extract and store standard real-time data in real time. This avoids the need for traditional monitoring methods to process data in batches only after a certain amount of business data has been accumulated in the business data system, and achieves fast and efficient data processing after real-time data is acquired, thereby improving the timeliness of monitoring and effectively meeting the monitoring needs in business scenarios with high timeliness requirements.
[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.
[0055] Based on this, the present application embodiment provides a real-time data monitoring method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the real-time data monitoring method of the present application.
[0056] In this embodiment, the real-time data monitoring method includes steps S10 to S40:
[0057] Step S10, acquiring real-time data of a business data system in a preset data portal, and performing standardization processing on the real-time data to obtain standard real-time data;
[0058] It should be noted that the preset data portal refers to a specially designed data access and processing platform, which can be an application software developed based on FlinkSQL; the business data system refers to a system used to store, process and manage business process data (such as Kafka), which is a key information source for business processes; real-time data refers to data extracted immediately when a business process occurs. This data reflects the latest status of business activities and is highly timely; standard real-time data refers to real-time data that has been standardized and meets specific format and specification requirements. This real-time data can be processed and recognized uniformly by the monitoring system.
[0059] It is understandable that since the real-time data generated by the business data system has various formats, direct use for monitoring may lead to data inconsistency and low processing efficiency. Therefore, performing step S10 can avoid monitoring errors and data processing delays caused by inconsistent data formats, reduce poor monitoring timeliness caused by low data conversion efficiency, ensure that the monitoring system can receive data with unified format and reliable quality, improve the accuracy and timeliness of monitoring, and provide data support for rapid response to business changes.
[0060] In a feasible implementation manner, the step of performing standardization processing on the real-time data to obtain standard real-time data in step S10 may include steps S11 to S12:
[0061] Step S11, acquiring the original version number and the original deletion field of the real-time data according to the primary key of the real-time data;
[0062] It should be noted that the primary key refers to the field used to uniquely identify real-time data; the original version number refers to the field used to identify the data record version, which can help track the data change history; the original deletion field refers to the field used to mark whether the data is new or deleted.
[0063] It is understandable that since different real-time data need to use different fields to identify the version and deletion status of the data, step S11 is performed to locate and extract the version and deletion status of the data through the primary key of the real-time data, thereby avoiding parsing errors in the data processing process due to not knowing the version and deletion status of the data, and ensuring that the monitoring system can accurately identify and process the version and deletion status of each data record, providing a reliable foundation for subsequent data monitoring.
[0064] Step S12, mapping the original version number to a standard version number and mapping the original deletion field to a standard deletion field according to a preset mapping method to obtain the standard real-time data.
[0065] It should be noted that the standard version number refers to the field used to uniformly identify the data record version after standardization processing. This field complies with the version management specifications of the base table of the monitoring system database; the standard deletion field refers to the field used to uniformly mark the deletion status of the data record after standardization processing. This field complies with the monitoring system's identification standards for the data deletion status.
[0066] It is understandable that since the real-time data generated by the business data system is directly used for monitoring, it will lead to data inconsistency and low processing efficiency. Therefore, step S12 is performed to make the real-time data of all business data systems conform to the standard format of the monitoring system through mapping conversion, thereby improving the data processing capability of the monitoring system and the accuracy of the monitoring results.
[0067] Exemplarily, according to a preset mapping method, the original version number is unified into a standard version number version, and the original deletion field is unified into a standard deletion field sign, wherein, for the first data insertion of the same primary key, the value of version is 0, and if data is subsequently inserted, the value of version increases by 1, and for the sign field, if the value of sign is 1, it indicates that the real-time data is newly added data, and if the value of sign is -1, it indicates that the real-time data is deleted data.
[0068] In this implementation, by standardizing the version number and deletion field of real-time data, the problem of data inconsistency caused by different real-time business data using different version numbers and deletion field identification methods is avoided, thereby ensuring the accurate identification and processing of data by the monitoring system and improving the data processing efficiency of the monitoring system.
[0069] Step S20, returning the standard real-time data to the business data system through the preset data portal;
[0070] It is understandable that since the monitoring system can only extract business data from the business data system, step S20 is performed to return the standardized data to the business data system through a data closed loop, ensuring that the monitoring system can obtain reliable standardized data from the business data system, thereby improving the efficiency and response speed of the overall business process.
[0071] Step S30, extracting standard real-time data from the business data system to a preset database, and constructing a model table of the standard real-time data based on the preset database;
[0072] It should be noted that the model table refers to a structured data table designed in the database according to business rules and monitoring requirements. The data table is used to store processed and calculated monitoring indicator data to facilitate data analysis and generate monitoring reports.
[0073] It is understandable that, since in the process of monitoring business process data, the acquired real-time data needs to be converted into a data form that is convenient for monitoring and query, step S30 is performed. By storing the standard real-time data in the business data system in the model table for effective monitoring and analysis, data loss, errors in the calculation of monitoring indicators, and low monitoring efficiency due to data dispersion are avoided, thereby improving the reliability of monitoring data and the degree of automation of the monitoring process, thereby achieving fast and accurate data monitoring and analysis.
[0074] In a feasible implementation manner, the preset database includes a first database and a second database, and step S30 may include steps S31 to S33:
[0075] Step S31, extracting the standard real-time data to the base table in the first database and the second database, wherein the second database backs up the standard real-time data to obtain the historical standard real-time data stored in the second database;
[0076] It should be noted that both the first database and the second database use Gaussian table structure for field mapping; the base table refers to the table in the first database used to store the latest standard real-time data. This table is the basis for building the model table and contains the unfinished historical data records and real-time data records of the business processes required for real-time monitoring. The base table is established using the version folding table engine VersionedCollapsingMergeTree.
[0077] It is understandable that, since business process data requires persistent storage and backup, performing step S31 can avoid the problem of incomplete monitoring data due to the inability to trace historical information caused by data loss, thereby ensuring redundant storage of data and traceability of historical data.
[0078] Exemplarily, a base table of a version folding table structure is established by the following SQL (Structured Query Language server database) statement:
[0079] CREATE TABLE[IF NOT EXISTS][db.]table_name[ON CLUSTER cluster] (
[0081] name1[type1][[DEFAULT|MATERIALIZED|ALIAS expr1]],
[0082] name2[type2][[DEFAULT|MATERIALIZED|ALIAS expr2]], ...
[0084] )ENGINE=VersionedCollapsingMergeTree(sign,version)
[0085] [PARTITION BY expr]
[0086] [ORDER BY expr]
[0087] [SAMPLE BY expr]
[0088] [SETTINGS name=value,...]
[0089] Among them, CREATE TABLE is used to create a new database table, [IF NOT EXISTS] means that if this option is specified, no error will be thrown when trying to create an existing table, [db.] table_name is used to specify the database name (optional) and table name, and is used for [ON CLUSTER cluster]: specifies the cluster name, which is used for multi-node deployment in distributed storage systems. (name1[type1][[DEFAULT|MATERIALIZED|ALIAS expr1]]) is used to define the name, type, and possible default value or expression of the column in the table. For example, name1 INT DEFAULT 0 means that name1 is an integer type column, and its default value is 0. ENGINE=VersionedCollapsingMergeTree(sign,version) means that the engine for selecting the table is VersionedCollapsingMergeTree, a specially designed merge tree engine that supports data version control and deletion operations, where the sign and version parameters are the deletion field and version number, which are associated with logical deletion. [PARTITION BY expr] means that data can be partitioned according to the result of an expression, which helps improve query performance. [ORDER BY expr] means that data is sorted according to the result of an expression, which can optimize certain queries. [SAMPLE BY expr] means that sampling technology can be used in the query to speed up the query when processing large amounts of data.
[0090] Step S32, based on the business process end field configured in the first database, the historical standard real-time data in the second database is screened to obtain target historical standard real-time data that does not match the business process end field;
[0091] It should be noted that the business process end field refers to a field configured in the first database, which is used to indicate that the business process has ended; the target historical standard real-time data refers to the historical standard real-time data screened out from the second database that does not match the business process end field, and these data represent business processes that have not yet been completed.
[0092] It is understandable that since the process of monitoring business processes requires identifying business process data that have not yet been completed, performing step S32 can avoid monitoring blind spots due to the lack of unfinished business process data, thereby causing the accuracy of monitoring business process data to decrease, thereby improving the accuracy and completeness of monitoring.
[0093] Step S33: sending the target historical standard real-time data to the base table, and constructing a model table of the standard real-time data based on the base table.
[0094] It can be understood that since complete monitoring indicator data requires the screening of unfinished business process data to be combined with the latest extracted business process data, step S33 is performed to avoid inaccurate monitoring or deviation in monitoring results due to incomplete data, thereby improving the data integrity of the monitoring system and the reliability of the monitoring results.
[0095] In this implementation, by performing data backup, data screening and data integration, monitoring deviations caused by data loss, neglect of incomplete business process data or incomplete monitoring data are avoided, and the effects of ensuring data redundancy, improving data processing accuracy and building a complete monitoring model are achieved, thereby providing a stable and reliable data foundation for real-time data monitoring and enhancing the robustness of the monitoring system and the continuity of business processes.
[0096] In a feasible implementation manner, the step of constructing the model table of the standard real-time data based on the base table in step S33 may include steps S331 to S334:
[0097] Step S331, screening the base table according to the primary keys of the standard real-time data and the target historical standard real-time data to obtain target table building data in the base table, wherein the target table building data is the standard real-time data newly added to the base table before the standard real-time data is received;
[0098] It should be noted that the target table creation data refers to the standard real-time data that is newly added after the last receipt of standard real-time data, which is filtered out in the base table according to the primary key.
[0099] It is understandable that since the model table needs to be updated in time to achieve real-time monitoring of business process data, step S331 is performed to avoid resource consumption caused by repeated processing of existing data in the process of updating the model table by identifying and extracting new data since the last data update. At the same time, by extracting only the newly added standard real-time data, the efficiency of subsequently establishing the model temporary table is improved, thereby improving the efficiency of the overall business process.
[0100] Step S332, constructing a model temporary table based on the target table building data, and obtaining the original model table;
[0101] It should be noted that the model temporary table refers to an intermediate table used to temporarily store and process the target table building data before building the final model table. The intermediate table is used to help maintain data consistency and integrity when updating the model table; the original model table refers to the model table that exists before the update operation, which contains all previous monitoring indicator data and business rule calculation results.
[0102] It can be understood that since the operation of directly changing data in the model table based on ClickHouse is inefficient, step S332 is performed. By creating an intermediate table before updating the model table, so that the intermediate table can be directly used as the model table in the future, inefficient modification operations on the original model table can be avoided, thereby improving the efficiency of business process data processing.
[0103] Step S333, screening the original model table based on the target table building data to obtain the original table building data in the original model table that does not match the target table building data;
[0104] It should be noted that the original table creation data refers to the data in the original model table that does not match the target table creation data. This data is still applicable to current monitoring needs.
[0105] It can be understood that since the original model table contains monitoring indicator data that have been calculated by the model, performing step S333 can avoid repeated calculation of the already calculated monitoring indicator data in the process of building a new model table, thereby further improving the efficiency of business process data processing.
[0106] Step S334, input the original table building data into the model temporary table to obtain the model table, and delete the original model table.
[0107] It is understandable that when using the CK (ClickHouse) base table based on ClickHouse, the efficiency of modifying the data in the CK base table is low, so step S334 is performed. By directly deleting the original model table and starting a new model table containing the newly added business process data, the time for updating the model table is shortened, thereby meeting the monitoring needs in business scenarios with high timeliness requirements.
[0108] In this implementation, by constructing a temporary table to replace the model table, the problem of difficulty in meeting monitoring needs in business scenarios with high timeliness requirements due to untimely data updates is avoided. While ensuring the accuracy of the model table data, real-time updating of the model table data is achieved, ensuring that the monitoring system can effectively monitor and analyze based on the latest business process data status, thereby improving the data integrity and monitoring efficiency of the monitoring system.
[0109] Step S40, in the process of monitoring the real-time data, obtaining monitoring index data by querying the model table.
[0110] It should be noted that monitoring indicator data refers to indicator data obtained from the model table and used to evaluate the status and performance of the business process. These data usually include information such as working hours.
[0111] This embodiment provides a real-time data monitoring method, which standardizes the data before the business data is extracted into the system, and uses database technology in the system to extract and store standard real-time data in real time. This avoids the need for traditional monitoring methods to process data in batches only after a certain amount of business data has been accumulated in the business data system, and achieves fast and efficient data processing after the real-time data is acquired, thereby improving the timeliness of monitoring and effectively meeting the monitoring needs in business scenarios with high timeliness requirements.
[0112] In a feasible implementation, step S331 may include steps S301 to S302:
[0113] Step S301, determining the reference update time of the base table according to the update time field corresponding to the primary key, wherein the update time field is recorded after the standard real-time data is extracted;
[0114] It should be noted that the update time field refers to the field in the base table used to record the completion time of each standard real-time data extraction. This field is used to help track data changes and updates; the reference update time refers to the time point determined based on the update time field and used to filter the base table data to identify new or changed data since the last update.
[0115] Exemplarily, after each extraction is completed, the latest '_update_at' field value in the extracted data is recorded, and the reference update time of the CK base table is determined according to the field value.
[0116] Step S302: Screen the base table based on the reference update time, and use the data in the base table whose update time is later than the reference update time as the target table creation data.
[0117] It is understandable that, since there is a problem of data redundancy in the process of establishing a temporary model table, performing step S302 can avoid unnecessary processing of redundant data, ensure that the temporary model table only contains the latest business data, and improve the timeliness of monitoring data and the overall performance of the monitoring system.
[0118] In this implementation, by adopting the update time screening method, the accuracy and timeliness of data updates in the base table are achieved, thereby ensuring that the monitoring system can effectively monitor and analyze based on the latest business data status, thereby improving the data quality and monitoring efficiency of the monitoring system.
[0119] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the real-time data monitoring method of the present application.
[0120] The real-time data monitoring method further includes steps S01-S02:
[0121] Step S01, detecting the time field of each table building data in the model table based on a preset effective time;
[0122] It should be noted that the time field refers to the field in the model table used to record the creation or start time of business data. This field is used to help track the newness and validity of the data.
[0123] It is understandable that since storing a large amount of business process data consumes a lot of resources and the model table needs to maintain a certain data quality, step S01 is performed to provide a reliable data basis for removing expired data by checking the start time of the data in the model table.
[0124] Step S02: when the duration between the current moment and the historical moment corresponding to any target time field in the time field is greater than the preset effective duration, the table creation data corresponding to the target time field is removed.
[0125] It should be noted that the target time field refers to the time field compared with the current moment during the detection process. This time field is used to determine whether the data exceeds the preset validity period, that is, whether it is expired; the historical moment refers to a point in the past recorded in the time field, usually the specific time when the business data was created or started.
[0126] It is understandable that since the model table needs to maintain a certain data quality, performing step S02 can avoid inaccurate monitoring results and decision support caused by retaining outdated data, so that the model table remains streamlined and efficient, ensuring that the monitoring system can provide accurate monitoring information based on the latest business data.
[0127] In a feasible implementation manner, the real-time data monitoring method further includes step S100:
[0128] Step S100, filtering the table building data in the model table based on the business process end field at a preset time, and removing the target filtered table building data matching the business process end field in the table building data.
[0129] It should be noted that the target screening table data refers to the business process data that has been completed.
[0130] It is understandable that, since there is a need to re-monitor the business process data that was not completed on the previous day during the process of business process monitoring, step S100 is performed to clean up the completed data in the model table to retain the uncompleted data, thereby meeting the actual monitoring needs and improving the monitoring accuracy of the monitoring system.
[0131] In this embodiment, by performing data timeliness detection and data cleaning, the decrease in monitoring efficiency due to the accumulation of outdated data in the model table is avoided, the effectiveness and real-time nature of the model table data are improved, thereby ensuring that the monitoring system can provide accurate monitoring results based on the latest business data, and improving the response speed and data management efficiency of the monitoring system.
[0132] For example, to help understand the implementation process of the real-time data monitoring method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flow chart of a real-time data monitoring method is provided, specifically:
[0133] In DDH, the preset data portal, the real-time data input by Kafka, the business data system, is obtained, and the real-time data is converted, i.e., standardized, and the converted data is output to GaussDB (Gaussian database, i.e., the second database) and the CK base table of the first database through Kafka, wherein GaussDB is used for data backup, and the data is distributed to the CK base table to generate an offline model through the CK base table for offline analysis. In the CK base table, case_id represents the primary key, sign represents the standard deletion field, and version represents the standard version number. After the CK base table receives the standard real-time data sent by Kafka, i.e., the target table building data: "case_id: 10001, sign: 1, version: 2" and "case_id: 10003, sign: 1, version: 1", a real-time model temporary table is built based on the target table building data. The real-time model generated by the CK base table contains a real-time model table, i.e., the original model table. The original table-building data in the implementation model table that does not match the target table-building data is inserted into the real-time model temporary table to obtain the real-time model table, and the original model table is deleted. The data insertion can also be inserted into the local table. Monitoring alarms and real-time analysis are performed by querying the real-time model table in the real-time model.
[0134] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the real-time data monitoring method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0135] This application also provides a real-time data monitoring device, please refer to Figure 4 , the real-time data monitoring device comprises:
[0136] The standardization module 10 is used to obtain the real-time data of the business data system in the preset data portal, and perform standardization processing on the real-time data to obtain standard real-time data;
[0137] A returning module 20, configured to return the standard real-time data to the business data system through the preset data portal;
[0138] A construction module 30 is used to extract the standard real-time data in the business data system to a preset database, and construct a model table of the standard real-time data based on the preset database;
[0139] The monitoring module 40 is used to obtain monitoring indicator data by querying the model table during the process of monitoring the real-time data.
[0140] Optionally, the standardization module 10 is further used for:
[0141] According to the primary key of the real-time data, obtaining the original version number and the original deletion field of the real-time data;
[0142] The original version number is mapped to a standard version number according to a preset mapping method, and the original deletion field is mapped to a standard deletion field to obtain the standard real-time data.
[0143] Optionally, the preset database includes a first database and a second database, and the construction module 30 is further used for:
[0144] Extracting the standard real-time data to a base table in the first database and the second database, wherein the second database backs up the standard real-time data to obtain historical standard real-time data stored in the second database;
[0145] Based on the business process end field configured in the first database, the historical standard real-time data in the second database is screened to obtain target historical standard real-time data that does not match the business process end field;
[0146] The target historical standard real-time data is sent to the base table, and a model table of the standard real-time data is constructed based on the base table.
[0147] Optionally, the building module 30 is further used for:
[0148] According to the primary keys of the standard real-time data and the target historical standard real-time data, the base table is screened to obtain the target table-building data in the base table, wherein the target table-building data is the standard real-time data newly added to the base table before the standard real-time data is received;
[0149] Constructing a model temporary table based on the target table building data, and obtaining the original model table;
[0150] The original model table is screened based on the target table building data to obtain the original table building data in the original model table that does not match the target table building data;
[0151] The original table building data is input into the model temporary table to obtain the model table, and the original model table is deleted.
[0152] Optionally, the building module 30 is further used for:
[0153] Determine the reference update time of the base table according to the update time field corresponding to the primary key, wherein the update time field is recorded after the standard real-time data is extracted;
[0154] The base table is screened based on the reference update time, and data in the base table whose update time is later than the reference update time is used as the target table creation data.
[0155] Optionally, the building module 30 is further used for:
[0156] Remove the start time field and the end time field of the target table creation data, and obtain the time field corresponding to the current time;
[0157] The time field is appended to the target table creation data, and the model temporary table is constructed based on the target table creation data to which the time field is appended.
[0158] Optionally, the time control module 50 in the real-time data monitoring device is used to:
[0159] Detecting the time field of each table creation data in the model table based on a preset effective time;
[0160] When the duration between the current moment and the historical moment corresponding to any target time field in the time field is greater than the preset effective duration, the table creation data corresponding to the target time field is removed.
[0161] The real-time data monitoring device provided by the present application adopts the real-time data monitoring method in the above embodiment, which can solve the technical problem of low efficiency in monitoring business data. Compared with the prior art, the beneficial effects of the real-time data monitoring device provided by the present application are the same as the beneficial effects of the real-time data monitoring method provided by the above embodiment, and other technical features in the real-time data monitoring device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0162] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time data monitoring method in the above-mentioned embodiment 1.
[0163] Reference below Figure 5, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0164] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0165] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0166] The electronic device provided by the present application adopts the real-time data monitoring method in the above embodiment to solve the technical problem of low efficiency in monitoring business data. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the real-time data monitoring method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be described in detail here.
[0167] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0169] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the real-time data monitoring method in the above-mentioned embodiment.
[0170] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0171] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0172] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains real-time data of a business data system in a preset data portal, and standardizes the real-time data to obtain standard real-time data; returns the standard real-time data to the business data system through the preset data portal; extracts the standard real-time data in the business data system to a preset database, and constructs a model table of the standard real-time data based on the preset database; and in the process of monitoring the real-time data, obtains monitoring indicator data by querying the model table.
[0173] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0175] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0176] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned real-time data monitoring method, and can solve the technical problem of low efficiency in monitoring business data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the real-time data monitoring method provided by the above-mentioned embodiment, and will not be repeated here.
[0177] The present application also provides a computer program product, including a computer program, which implements the steps of the real-time data monitoring method as described above when executed by a processor.
[0178] The computer program product provided by this application can solve the technical problem of low efficiency in monitoring business data. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the real-time data monitoring method provided by the above embodiment, which will not be repeated here.
[0179] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A real-time data monitoring method, characterized in that: The real-time data monitoring method comprises: Acquire real-time data of a business data system in a preset data portal, and perform standardization processing on the real-time data to obtain standard real-time data; Returning the standard real-time data to the business data system through the preset data portal; Extracting standard real-time data from the business data system into a preset database, and constructing a model table of the standard real-time data based on the preset database; In the process of monitoring the real-time data, the monitoring index data is obtained by querying the model table.
2. The real-time data monitoring method according to claim 1, characterized in that: The step of performing standardization processing on the real-time data to obtain standard real-time data comprises: According to the primary key of the real-time data, obtaining the original version number and the original deletion field of the real-time data; The original version number is mapped to a standard version number according to a preset mapping method, and the original deletion field is mapped to a standard deletion field to obtain the standard real-time data.
3. The real-time data monitoring method according to claim 1, characterized in that: The preset database includes a first database and a second database, and the step of extracting the standard real-time data in the business data system to the preset database and constructing a model table of the standard real-time data based on the preset database includes: Extracting the standard real-time data to a base table in the first database and the second database, wherein the second database backs up the standard real-time data to obtain historical standard real-time data stored in the second database; Based on the business process end field configured in the first database, the historical standard real-time data in the second database is screened to obtain target historical standard real-time data that does not match the business process end field; The target historical standard real-time data is sent to the base table, and a model table of the standard real-time data is constructed based on the base table.
4. The real-time data monitoring method according to claim 3, characterized in that: The step of constructing the model table of the standard real-time data based on the base table comprises: According to the primary keys of the standard real-time data and the target historical standard real-time data, the base table is screened to obtain the target table-building data in the base table, wherein the target table-building data is the standard real-time data newly added to the base table before the standard real-time data is received; Constructing a model temporary table based on the target table building data, and obtaining the original model table; The original model table is screened based on the target table building data to obtain the original table building data in the original model table that does not match the target table building data; The original table building data is input into the model temporary table to obtain the model table, and the original model table is deleted.
5. The real-time data monitoring method according to claim 4, characterized in that: The step of screening the base table according to the primary keys of the standard real-time data and the target historical standard real-time data to obtain the target table building data in the base table includes: Determine the reference update time of the base table according to the update time field corresponding to the primary key, wherein the update time field is recorded after the standard real-time data is extracted; The base table is screened based on the reference update time, and data in the base table whose update time is later than the reference update time is used as the target table creation data.
6. The real-time data monitoring method according to claim 4, characterized in that: The step of constructing a model temporary table based on the target table building data includes: Remove the start time field and the end time field of the target table creation data, and obtain the time field corresponding to the current time; The time field is appended to the target table creation data, and the model temporary table is constructed based on the target table creation data to which the time field is appended.
7. The real-time data monitoring method according to claim 1, characterized in that: The real-time data monitoring method further comprises: Detecting the time field of each table creation data in the model table based on a preset effective time; When the duration between the current moment and the historical moment corresponding to any target time field in the time field is greater than the preset effective duration, the table creation data corresponding to the target time field is removed.
8. A real-time data monitoring device, characterized in that: The real-time data monitoring device comprises: A standardization module is used to obtain real-time data of a business data system in a preset data portal, and perform standardization processing on the real-time data to obtain standard real-time data; A returning module, used for returning the standard real-time data to the business data system through the preset data portal; A construction module, used to extract standard real-time data from the business data system to a preset database, and construct a model table of the standard real-time data based on the preset database; The monitoring module is used to obtain monitoring indicator data by querying the model table during the process of monitoring the real-time data.
9. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the real-time data monitoring method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the real-time data monitoring method according to any one of claims 1 to 7 are implemented.