A data monitoring method and device, a computing device, and a storage medium

Abstract

The application discloses a data monitoring method and device, a computing device and a storage medium. The method is suitable for being executed in a computing device, and the computing device is in communication connection with a first server, a first database server and a second database server respectively. The first server, the first database server and the second database server are respectively provided with a configuration platform, a first database and a second database. The configuration platform is pre-configured with data synchronization configuration and task configuration. The method comprises the following steps: reading the data synchronization configuration from the configuration platform at a fixed time, and synchronizing the business data in the corresponding data source to the first database according to the data synchronization configuration; receiving the task configuration issued by the configuration platform, and generating a to-be-executed timing task based on the task configuration; executing the timing task to perform timing monitoring and statistics on the data in the first database, and obtaining a statistical result, wherein the data in the first database comprises business data and log data; and saving the statistical result to the second database.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a data monitoring method, apparatus, computing device, and storage medium. Background Technology

[0002] In production and operations, monitoring log data and business data allows for timely understanding of the latest changes in various business metrics, analysis of business operation status, and prompt identification of business opportunities and problems. However, with continuous business development, the number of users, business scale, and business complexity are all growing rapidly, and the volume of various data is also expanding at an accelerated pace. When the daily increase in data volume reaches hundreds of millions, data monitoring and statistics become quite difficult.

[0003] Currently, two types of methods are commonly used to monitor data. The first type is a technical solution based on Elasticsearch (a distributed search and analysis engine). Logstash (a lightweight data collection engine) is used to collect log data and store it in Elasticsearch so that Elasticsearch can monitor and statistically analyze the log data. Kibana (an open-source data analysis and visualization platform) is then used to visualize and analyze the log data.

[0004] This type of method can easily handle massive amounts of data, provides simple and intuitive log analysis, and can be used in conjunction with ElastAlert (a framework for monitoring and alerting) to discover anomalies in Elasticsearch data for data alerting. However, because Elasticsearch can only retrieve data within the same index, data from different indexes cannot be linked. Data that needs to be linked must be aggregated before an index is generated and then queried together. This makes it difficult to meet the requirement of statistical analysis of multiple linked data, thus impacting monitoring performance.

[0005] The second type is a technical solution based on a streaming processing framework. This framework can monitor data according to predefined processing rules and quickly compile real-time data statistics, providing excellent support for real-time batch data analysis and meeting the requirements of efficient data pipelines. However, the data processing rules rely on predefined rules, making it difficult to respond promptly to frequently changing needs, and tracing back to historical data is also inconvenient.

[0006] Therefore, a new data monitoring method is needed to optimize the above processing. Summary of the Invention

[0007] Therefore, the present invention provides a data monitoring scheme in an attempt to solve or at least alleviate the problems mentioned above.

[0008] According to one aspect of the present invention, a data monitoring method is provided, suitable for execution in a computing device. The computing device is communicatively connected to a first server, a first database server, and a second database server. A configuration platform, a first database, and a second database are respectively deployed on the first server, the first database server, and the second database server. The configuration platform is pre-configured with data synchronization configuration and task configuration. The method includes the following steps: First, periodically reading the data synchronization configuration from the configuration platform and synchronizing business data from the corresponding data source to the first database according to the data synchronization configuration; receiving the task configuration issued by the configuration platform and generating a scheduled task to be executed based on the task configuration; executing the scheduled task to periodically monitor and statistically analyze the data in the first database and obtain the statistical results. The data in the first database includes business data and log data; and saving the statistical results to the second database.

[0009] Optionally, in the data monitoring method according to the present invention, the task configuration is associated with a corresponding task identifier. The task configuration includes a monitoring and statistics template and an execution time rule. The method of generating a scheduled task to be executed based on the task configuration includes: generating a task to perform data monitoring and statistics according to the execution time rule based on the monitoring and statistics template and the execution time rule; and using the generated task as a scheduled task to be executed and associating it with the task identifier.

[0010] Optionally, in the data monitoring method according to the present invention, a scheduled task is executed to perform scheduled monitoring and statistics on the data in the first database and obtain statistical results, including: acquiring a distributed lock through a task identifier; after successfully acquiring the distributed lock, using a monitoring and statistics template to periodically monitor and count the data in the first database to obtain statistical results, and associating the statistical results with the task identifier.

[0011] Optionally, in the data monitoring method according to the present invention, the first server and the first database server are also connected to the second server, and the second server is also connected to the third server. The second server and the third server are respectively equipped with a stream processing engine and a message system, and the log data is consumed from the message system by the stream processing engine and sent to the first database.

[0012] Optionally, in the data monitoring method according to the present invention, the message system synchronously has system logs, and the configuration platform is pre-configured with log classification configuration. The log data is consumed from the message system to the first database by the stream processing engine, including: reading the log classification configuration from the configuration platform periodically through the stream processing engine; consuming the message queue in the message system according to the log classification configuration, filtering out target logs from the system logs as log data, and writing them to the first database.

[0013] Optionally, in the data monitoring method according to the present invention, the configuration platform is further configured with alarm configurations in advance. The alarm configurations are associated with corresponding task identifiers. The alarm configurations include alarm templates. The method further includes: reading all alarm configurations associated with the task identifiers from the configuration platform; comparing the statistical results based on the read alarm configurations to determine whether an alarm is needed; if an alarm is needed, triggering an alarm using the corresponding alarm template.

[0014] Optionally, in the data monitoring method according to the present invention, the alarm configuration further includes a time difference value for data comparison and an alarm threshold. Statistical results are compared based on the read alarm configuration to determine whether an alarm is needed. This includes: traversing the read alarm configurations; obtaining statistical results prior to the scheduled task from the second database as a first result based on the time difference value for data comparison in the traversed alarm configurations, and using the statistical results associated with the task identifier as a second result; if the absolute value of the difference between the first result and the second result is greater than the alarm threshold, then it is determined that an alarm is needed; otherwise, it is determined that no alarm is needed.

[0015] According to another aspect of the present invention, a data monitoring device is provided, suitable for residing in a computing device, the computing device being communicatively connected to a first server, a first database server, and a second database server. A configuration platform, a first database, and a second database are respectively deployed on the first server, the first database server, and the second database server. The configuration platform is pre-configured with data synchronization configuration and task configuration. The device includes a synchronization module, a generation module, an execution module, and a storage module. The synchronization module is adapted to periodically read the data synchronization configuration from the configuration platform and synchronize business data from the corresponding data source to the first database according to the data synchronization configuration. The generation module is adapted to receive task configurations issued by the configuration platform and generate scheduled tasks to be executed based on the task configurations. The execution module is adapted to execute the scheduled tasks to periodically monitor and statistically analyze the data in the first database and obtain statistical results. The data in the first database includes business data and log data. The storage module is adapted to save the statistical results to the second database.

[0016] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the data monitoring method as described above.

[0017] According to another aspect of the present invention, a readable storage medium storing program instructions is provided, which, when read and executed by a computing device, causes the computing device to perform the data monitoring method described above.

[0018] According to the data monitoring scheme of the present invention, data synchronization configuration is read from the configuration platform at regular intervals, and business data in the corresponding data source is synchronized to the first database according to the data synchronization configuration. Based on the task configuration issued by the configuration platform, a scheduled task to be executed is generated and executed to perform timed monitoring and statistics on the data in the first database, obtain the statistical results and save them to the second database. Through simple task configuration, the scheme responds to changes in requirements in a timely manner, realizing fast, flexible and real-time monitoring of massive data.

[0019] In the above technical solution, the data in the first database includes business data and log data. The log data comes from the stream processing engine. The stream processing engine consumes the corresponding system logs from the message system into the first database according to the log classification configuration pre-configured by the configuration platform, thus completing the classification and storage of data. It supports cross-database and cross-table data joint retrieval within the cluster without adjusting the data structure, making it convenient to use.

[0020] In addition, the configuration platform is also equipped with alarm configurations. If it is determined that the current task requires an alarm, an alarm will be triggered according to the alarm configuration, easily meeting alarm requirements at the minute or even second level. Since the configuration platform stores various different configurations, and the configurations take effect in real time, the monitoring, statistics, and alarm tasks managed through configuration can easily cope with changing needs, and the response time of the business is greatly shortened. Attached Figure Description

[0021] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of this disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0022] Figure 1 A schematic diagram of a data monitoring system 100 according to an embodiment of the present invention is shown;

[0023] Figure 2 A structural block diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0024] Figure 3 A flowchart of a data monitoring method 300 according to an embodiment of the present invention is shown; and

[0025] Figure 4 A schematic diagram of a data monitoring device 400 according to an embodiment of the present invention is shown. Detailed Implementation

[0026] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] Figure 1 A schematic diagram of a data monitoring system 100 according to an embodiment of the present invention is shown. It should be noted that... Figure 1 The data monitoring system 100 described herein is merely an example. In specific implementations, the data monitoring system 100 may contain varying numbers of computing devices, a first server, a second server, a third server, a first database server, and a second database server, depending on the actual situation. There is no limitation on this. The computing devices are typically implemented as servers.

[0028] like Figure 1 As shown, the data monitoring system 100 includes a computing device 110, a first server 121, a second server 122, a third server 123, a first database server 131, and a second database server 132. The computing device 110 is communicatively connected to the first server 121, the first database server 131, and the second database server 132. The first server 121 and the first database server 131 are also communicatively connected to the second server 122, and the second server 122 is also communicatively connected to the third server 123.

[0029] According to one embodiment of the present invention, the computing device 110 can be implemented as a server, with a configuration platform, a first database, and a second database respectively deployed on the first server 121, the first database server 131, and the second database server 132, and a stream processing engine and a message system respectively deployed on the second server 122 and the third server 123.

[0030] In this implementation, the configuration platform can be implemented using Redis (Remote Dictionary Server), Nacos (a dynamic service discovery, configuration management, and service management platform), ZooKeeper (a distributed application coordination service), etc. The stream processing engine, messaging system, first database, and second database can respectively employ Flink (a framework and distributed processing engine for stateful computation on unbounded and bounded data streams), Kafka (a high-throughput distributed publish-subscribe messaging system), ClickHouse (a column-oriented relational database for data management), and MySQL (a row-oriented relational database for data management).

[0031] The following is a brief description of the operation process of the data monitoring system 100. In practical applications, it is necessary to pre-configure log classification, data synchronization, task, and alarm configurations on the configuration platform according to data monitoring and statistical requirements, so that the required configurations can be read from the configuration platform for corresponding processing later. The task and alarm configurations are associated with corresponding task identifiers. In addition, the message system synchronizes system logs, which are generally collected by the log collection module from the deployed devices (such as servers and clients) and then synchronized to the message system.

[0032] The stream processing engine periodically reads log classification configurations from the configuration platform. Based on these configurations, it consumes message queues from the message system, filters target logs from the system logs as log data, and writes them to the first database. Computing device 110 periodically reads data synchronization configurations from the configuration platform and synchronizes business data from the corresponding data source to the first database according to these configurations. Thus, the data in the first database comprises two parts: business data and log data.

[0033] Then, in the computing device 110, the task configuration issued by the configuration platform is received, a scheduled task to be executed is generated based on the task configuration, the scheduled task is executed to perform timed monitoring and statistics on the data in the first database, the statistical results are obtained, the statistical results are associated with the corresponding task identifier, and saved to the second database.

[0034] After the statistical results are successfully saved, the computing device 110 reads all alarm configurations associated with the task identifier from the configuration platform. Based on the read alarm configurations, it retrieves the statistical results of the scheduled task from the second database and compares them with the statistical results associated with the task identifier. Based on the comparison results, it determines whether an alarm is needed. If needed, an alarm is triggered, for example, by sending an alarm message via email, SMS, or instant messaging software to indicate that an anomaly has occurred.

[0035] Figure 2 A structural block diagram of a computing device 200 according to an embodiment of the present invention is shown.

[0036] like Figure 2 As shown, in the basic configuration 202, the computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 can be used for communication between the processors 204 and the system memory 206.

[0037] Depending on the desired configuration, processor 204 can be any type of processor, including but not limited to: microprocessor (UP), microcontroller (UC), digital information processor (DSP), or any combination thereof. Processor 204 may include one or more levels of cache such as L1 cache 210 and L2 cache 212, processor core 214, and registers 216. Example processor core 214 may include an arithmetic logic unit (ALU), floating-point unit (FPU), digital signal processing core (DSP core), or any combination thereof. Example memory controller 218 may be used with processor 204, or in some implementations, memory controller 218 may be an internal part of processor 204.

[0038] Depending on the desired configuration, system memory 206 can be any type of memory, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include operating system 220, one or more applications 222, and program data 224. In some embodiments, applications 222 may be arranged to execute instructions on the operating system using program data 224 by one or more processors 204.

[0039] The computing device 200 also includes a storage device 232, which includes a removable storage device 236 and a non-removable storage device 238.

[0040] The computing device 200 may also include a storage interface bus 234. The storage interface bus 234 enables communication from storage devices 232 (e.g., removable storage 236 and non-removable storage 238) to the basic configuration 202 via the bus / interface controller 230. At least a portion of the operating system 220, applications 222, and program data 224 may be stored on the removable storage 236 and / or the non-removable storage 238, and loaded into system memory 206 via the storage interface bus 234 when the computing device 200 is powered on or when the application 222 is to be executed, and executed by one or more processors 204.

[0041] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via a bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices such as displays or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link through one or more communication ports 264.

[0042] A network communication link can be an example of a communication medium. A communication medium can typically be embodied in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A “modulated data signal” can be a signal whose data set, or its modifications, can be encoded as information within the signal. As a non-limiting example, a communication medium can include wired media such as wired networks or leased lines, and various wireless media including sound, radio frequency (RF), microwave, infrared (IR), or other wireless media. The term “computer-readable medium” as used herein can include both storage media and communication media.

[0043] The computing device 200 can be implemented as a personal computer, including desktop and laptop computer configurations. Of course, the computing device 200 can also be implemented as part of a small-sized portable (or mobile) electronic device, such as a cellular phone, digital camera, personal digital assistant (PDA), personal media player device, wireless network browsing device, personal head-mounted device, application-specific device, or a hybrid device that may include any of the above functions. It can even be implemented as a server, such as a file server, database server, application server, and web server. The embodiments of the present invention do not limit this.

[0044] In an embodiment of the present invention, the computing device 200 may be implemented as follows: Figure 1The computing device 110 shown (specifically implemented as a server) is configured to execute the data monitoring method 300 according to the present invention. The application 222, arranged on the operating system, contains multiple program instructions for executing the data monitoring method 300 of the present invention. These program instructions can instruct the processor 204 to execute the method 300 of the present invention, so that the computing device 200 monitors data by executing the method 300 of the present invention.

[0045] According to one embodiment of the present invention, application 222 arranged on an operating system includes a data monitoring device 400, which contains multiple program instructions for executing the data monitoring method 300 of the present invention, so that the method 300 of the present invention can be executed in the device 400.

[0046] Figure 3 A flowchart of a data monitoring method 300 according to an embodiment of the present invention is shown. The data monitoring method 300 can be executed in a data monitoring device 400 of a computing device (e.g., the aforementioned computing device 200). In this embodiment, the computing device 200 can be implemented as... Figure 1 The computing device 110 shown is specifically implemented as a server.

[0047] like Figure 3 As shown, method 300 begins with step S310. In step S310, data synchronization configuration is periodically read from the configuration platform, and business data in the corresponding data source is synchronized to the first database according to the data synchronization configuration. The configuration platform is pre-configured with data synchronization configuration and task configuration.

[0048] According to one embodiment of the present invention, if the first database uses ClickHouse, then through data synchronization configuration, the data engine provided by ClickHouse can be used to connect to the corresponding data source and synchronize the required business data to ClickHouse. In this embodiment, the data synchronization configuration includes an application identifier, a connection string of the data source to be synchronized, a username, a read password, a source table name, a source field set, a synchronization frequency, a target table name, and a target field set, etc. The data source includes various databases that store business-related data.

[0049] For the first database, the data includes not only business data synchronized from other data sources by computing device 110, but also log data from the stream processing engine. Specifically, the log data is consumed by the stream processing engine from the message system and then fed into the first database. The stream processing engine uses Flink, and the message system uses Kafka.

[0050] According to one embodiment of the present invention, the messaging system synchronizes system logs, which include attributes such as unique log identifier, log type, trigger time, application identifier, page identifier, click identifier, user identifier, IP (Internet Protocol Address), version, and custom dataset. In this embodiment, the system logs are synchronized to Kafka via a log collection module. The log collection module is typically deployed on the server or client side. If deployed on the server side, it collects the server-side system logs; if deployed on the client side, it collects the client-side system logs.

[0051] According to one embodiment of the present invention, the configuration platform is pre-configured with log classification configuration. The log data consumed from the message system by the stream processing engine and written to the first database can be achieved in the following manner. In this embodiment, the stream processing engine periodically reads the log classification configuration from the configuration platform, consumes the message queue in the message system according to the log classification configuration, filters the target logs from the system logs as log data, and writes them to the first database.

[0052] The log classification configuration includes log classification information and log storage information. For example, based on the log classification information, system logs can be divided into application PV (Page View) logs, HTML5 (Hyper Text Markup Language 5) page click logs, etc., with the corresponding log classification information for application PV logs and HTML5 page click logs denoted as app_pv_log and h5_click_log, respectively. The log storage information indicates the location of the table where the system logs are stored. For example, the log storage information for application PV logs of application 1 and HTML5 page click logs of application 2 are db1.project1_app_pv_log and db1.project2_h5_click_log, respectively.

[0053] Flink periodically reads log classification configurations from the configuration platform. For example, it reads the log classification configuration corresponding to the click logs of H5 pages in application 2, consumes the message queue in Kafka, filters the target logs that match the log classification information "h5_click_log" from the system logs as log data, and writes the log data to the corresponding table in ClickHouse according to the log storage information "db1.project2_h5_click_log".

[0054] In step S320, the task configuration issued by the configuration platform is received, and a scheduled task to be executed is generated based on the task configuration.

[0055] According to one embodiment of the present invention, a task configuration is associated with a corresponding task identifier. The task configuration includes a monitoring and statistics template and execution time rules. A scheduled task to be executed can be generated based on the task configuration in the following manner: In this embodiment, based on the monitoring and statistics template and execution time rules, a task is generated that performs data monitoring and statistics at regular intervals according to the execution time rules. The generated task is then used as a scheduled task to be executed and associated with the task identifier.

[0056] Execution time rules typically have different dimensions, such as user, page, and entry point dimensions. They can be set to execute tasks monthly, weekly, daily, hourly, or at specific intervals (e.g., every few minutes), depending on actual needs. Combined, execution time rules could include tasks such as executing page view (PV) statistics hourly, monitoring and counting entry points hourly, and compiling overall data daily.

[0057] Monitoring and statistics templates refer to SQL (Structured Query Language) statement templates used for monitoring and statistics. Below is a template example:

[0058] SELECT pageid AS ckey1,COUNT(DISTINCT logkey)AS stat_val,'${yyyy-MM-dd[-1]}'

[0059] AS stat_date,0AS stat_hour,0AS stat_minute FROM log_pv_app1

[0060] Where createTime>='${yyyy-MM-dd[-1]}'AND createTime<'${yyyy-MM-dd}'

[0061] AND custargv.key = 'entry_id'

[0062] AND notEmpty(EXTRACT(custargv.val,'pageid__([0-9]+)'))=1)AS T

[0063] GROUP BY pageid

[0064] In addition, it should be noted that during the initialization phase, the computing device 110 will actively read the task configuration from the configuration platform and generate scheduled tasks to be executed based on the task configuration. When the task configuration changes, the configuration platform will send the changed task configuration to the computing device 110 so that it can readjust the scheduled tasks.

[0065] Then, proceed to step S330 to execute a scheduled task to periodically monitor and statistically analyze the data in the first database and obtain the statistical results. The data in the first database includes business data and log data.

[0066] According to one embodiment of the present invention, a scheduled task can be executed in the following manner to perform scheduled monitoring and statistics on data in a first database and obtain statistical results. In this embodiment, a distributed lock is first acquired through a task identifier. After successfully acquiring the distributed lock, the data in the first database is monitored and statistically analyzed periodically using a monitoring and statistics template to obtain statistical results, and the statistical results are associated with the task identifier.

[0067] Assuming the configuration platform is built using Redis, the distributed lock can be implemented using the Redis SETNX command. SETNX is short for "SET if Not eXists," meaning it is only set if the lock does not exist. It returns 1 on success and 0 on failure. After successfully acquiring the distributed lock, the monitoring and statistics templates in the task configuration, such as the SQL statement template provided in the example above, are used to periodically monitor and analyze the data in ClinkHouse to obtain statistical results.

[0068] Specifically, replace the time parameter "${yyyy-MM-dd[-1]}" in the above SQL statement template with the current date minus one day. For example, if today is January 3, 2020, then replace the time parameter in the template with January 2, 2020, that is, set the time parameter to 2020-01-02. After the replacement is completed, execute the template and associate the execution result with the task identifier as a statistical result.

[0069] Finally, step S340 is executed to save the statistical results to the second database. According to one embodiment of the present invention, if the second database is MySQL, the statistical results are ultimately saved to MySQL for later use.

[0070] After the statistical results are saved, an alarm scan is triggered using the task identifier to determine if an alarm is needed, and alarm processing is performed if necessary. According to one embodiment of the present invention, the configuration platform also pre-configures alarm configurations, which are associated with corresponding task identifiers. The alarm configuration includes an alarm template, which is generally in Markdown (a lightweight markup language) format and carries replaceable parameters. The following is an example of an alarm template:

[0071] #PV Statistics (every 20 minutes)\n>Statistical Time: {statTime},\n{#LoopBegin}-{ckey}, Statistics: {statVal}, Baseline: {standardVal}, {trend}{diffResult};{#LoopEnd}

[0072] In this embodiment, method 300 further includes: reading all alarm configurations associated with the task identifier from the configuration platform, comparing the statistical results based on the read alarm configurations to determine whether an alarm is needed, and if an alarm is needed, triggering an alarm using the corresponding alarm template, such as sending an email, SMS, or instant message to prompt the alarm.

[0073] The alarm configuration also includes the time difference for data comparison and the alarm threshold, both of which can have different dimensions. For example, the time difference for data comparison could be one hour ago, one day ago, etc. The alarm threshold for a page's PV value change range could be 10%, indicating that an alarm should be issued when the page's PV value changes by more than 10%. The alarm threshold for an entry point click volume change range could be 15%, indicating that an alarm should be issued when the entry point click volume changes by more than 15%, and so on. In addition, the alarm configuration can also include the alarm name, etc.

[0074] When comparing statistical results based on the read alarm configuration to determine whether an alarm is needed, the read alarm configuration can be traversed first. Then, based on the time difference of the data comparison in the traversed alarm configuration, the statistical results before the scheduled task are obtained from the second database as the first result, and the statistical results associated with the task identifier are used as the second result. If the absolute value of the difference between the first result and the second result is greater than the alarm threshold, it is determined that an alarm is needed; otherwise, it is determined that no alarm is needed.

[0075] Figure 4 A schematic diagram of a data monitoring device 400 according to an embodiment of the present invention is shown. The data monitoring device 400 may reside in a computing device (e.g., the aforementioned computing device 200) and monitor data by executing the data monitoring method 300 of the present invention. In this embodiment, the computing device 200 may be implemented as... Figure 1 The computing device 110 shown is specifically implemented as a server.

[0076] like Figure 4 As shown, the data monitoring device 400 includes a synchronization module 410, and a generation module 420, an execution module 430, and a storage module 440 connected in sequence.

[0077] The synchronization module 410 periodically reads data synchronization configuration from the configuration platform and synchronizes the business data from the corresponding data source to the first database according to the data synchronization configuration. The generation module 420 receives task configuration from the configuration platform and generates scheduled tasks to be executed based on the task configuration. Subsequently, the execution module 430 executes the scheduled tasks to perform periodic monitoring and statistics on the data in the first database, which includes business data and log data, and obtains the statistical results. Finally, the saving module 440 saves the statistical results to the second database.

[0078] It should be noted that the synchronization module 410 is used to execute the aforementioned step S310, the generation module 420 is used to execute the aforementioned step S320, the execution module 430 is used to execute the aforementioned step S330, and the saving module 440 is used to execute the aforementioned step S340. Here, the execution logic of the synchronization module 410, the generation module 420, the execution module 430, and the saving module 440 can be found in the detailed description of steps S310 to S340 in method 300 above, and will not be repeated here.

[0079] According to the data monitoring scheme of this invention, data synchronization configuration is read from the configuration platform at regular intervals, and business data in the corresponding data source is synchronized to the first database according to the data synchronization configuration. Based on the task configuration issued by the configuration platform, a scheduled task to be executed is generated and executed to perform timed monitoring and statistics on the data in the first database, obtain the statistical results and save them to the second database. Through simple task configuration, the scheme can respond to changes in requirements in a timely manner and realize fast, flexible and real-time monitoring of massive data.

[0080] The data in the first database includes business data and log data. The log data comes from the stream processing engine. The stream processing engine consumes the corresponding system logs from the message system into the first database according to the log classification configuration pre-configured on the configuration platform, thus completing the classification and storage of data. It supports cross-database and cross-table data joint retrieval within the cluster without adjusting the data structure, making it convenient to use.

[0081] In addition, the configuration platform is also equipped with alarm configurations. If it determines that the current task requires an alarm, it will trigger an alarm according to the alarm configuration, easily meeting alarm requirements at the minute or even second level. Because the configuration platform stores various different configurations, and these configurations take effect in real time, the configuration-managed monitoring, statistics, and alarm tasks can easily handle changing needs, significantly reducing business response time. If there is a need to review historical data for statistical purposes, the task start time can be adjusted directly, making it simple and convenient.

[0082] In practical applications, the above technical solutions can monitor the year-on-year and month-on-month comparisons of various indicators, real-time page traffic, and real-time user conversion rates in real time. They provide strong technical support for judging the operational status of business and the effectiveness of newly added modules. They can also realize real-time statistics of event tracking data and joint statistics of event tracking data with other business data. By comparing real-time event tracking data with historical data from the same period, problems can be detected in a timely manner, quickly addressed, and the problem handling time can be shortened.

[0083] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the present invention.

[0084] When the program code is executed on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the data monitoring method of the present invention according to instructions in the program code stored in the memory.

[0085] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0086] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0087] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0088] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0089] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0090] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0091] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0092] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the methods or method elements forms means for implementing the methods or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purposes of carrying out the invention.

[0093] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0094] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A data monitoring method, suitable for execution in a computing device, wherein the computing device is communicatively connected to a first server, a first database server, and a second database server, respectively, wherein a configuration platform, a first database, and a second database are respectively deployed on the first server, the first database server, and the second database server, and the configuration platform is pre-configured with data synchronization configuration and task configuration, the method comprising: The system periodically reads the data synchronization configuration from the configuration platform and synchronizes the business data from the corresponding data source to the first database according to the data synchronization configuration. Receive the task configuration issued by the configuration platform, and generate a scheduled task to be executed based on the task configuration; The scheduled task is executed to perform scheduled monitoring and statistics on the data in the first database and obtain the statistical results. The data in the first database includes the business data and log data. The statistical results are saved to the second database; The first server and the first database server are also connected to the second server, and the second server is also connected to the third server. The second server and the third server are respectively deployed with a stream processing engine and a message system. The log data is consumed from the message system by the stream processing engine and then sent to the first database. The messaging system synchronizes system logs, and the configuration platform is pre-configured with log classification settings. The log data is consumed from the messaging system by the stream processing engine and fed into the first database, including: The stream processing engine periodically reads the log classification configuration from the configuration platform. According to the log classification configuration, the message queue in the message system is consumed, the target log is selected from the system log as log data, and written to the first database; The task configuration is associated with a corresponding task identifier, and includes monitoring and statistics templates and execution time rules. The generation of scheduled tasks to be executed based on the task configuration includes: Based on the monitoring and statistics template and the execution time rule, a task is generated to perform data monitoring and statistics on a regular schedule according to the execution time rule; The generated task is set as a scheduled task to be executed and associated with the task identifier.

2. The method as described in claim 1, wherein, The execution of the scheduled task to periodically monitor and statistically analyze the data in the first database and obtain the statistical results includes: Obtain the distributed lock using the task identifier; After successfully acquiring the distributed lock, the monitoring and statistics template is used to periodically monitor and statistically analyze the data in the first database to obtain statistical results, and the statistical results are associated with the task identifier.

3. The method as described in claim 1 or 2, wherein, The configuration platform is also pre-configured with alarm configurations, each alarm configuration being associated with a corresponding task identifier. The alarm configuration includes alarm templates. The method further includes: Read all alarm configurations associated with the task identifier from the configuration platform; The statistical results are compared based on the read alarm configuration to determine whether an alarm is needed. If an alarm is required, trigger the alarm using the corresponding alarm template.

4. The method of claim 3, wherein, The alarm configuration also includes a time difference value for data comparison and an alarm threshold. The step of comparing the statistical results based on the read alarm configuration to determine whether an alarm is needed includes: Iterate through the read alarm configurations; Based on the time difference value of the data comparison in the traversed alarm configuration, the statistical results before the scheduled task are obtained from the second database as the first result, and the statistical results associated with the task identifier are used as the second result. If the absolute value of the difference between the first result and the second result is greater than the alarm threshold, then it is determined that an alarm is required; otherwise, it is determined that no alarm is required.

5. A data monitoring device, suitable for residing in a computing device, the computing device being communicatively connected to a first server, a first database server, and a second database server, wherein a configuration platform, a first database, and a second database are respectively deployed on the first server, the first database server, and the second database server, the configuration platform being pre-configured with data synchronization configuration and task configuration, the device comprising: The synchronization module is adapted to periodically read the data synchronization configuration from the configuration platform and synchronize the business data in the corresponding data source to the first database according to the data synchronization configuration. The generation module is adapted to receive the task configuration issued by the configuration platform and generate a scheduled task to be executed based on the task configuration; An execution module is adapted to execute the scheduled task to perform scheduled monitoring and statistics on the data in the first database and obtain the statistical results. The data in the first database includes the business data and log data. A storage module is adapted to save the statistical results to the second database; The first server and the first database server are also connected to the second server, and the second server is also connected to the third server. The second server and the third server are respectively deployed with a stream processing engine and a message system. The log data is consumed from the message system by the stream processing engine and then sent to the first database. The messaging system synchronizes system logs, and the configuration platform is pre-configured with log classification settings. The log data is consumed from the messaging system by the stream processing engine and fed into the first database, including: The stream processing engine periodically reads the log classification configuration from the configuration platform. According to the log classification configuration, the message queue in the message system is consumed, the target log is selected from the system log as log data, and written to the first database; The task configuration is associated with a corresponding task identifier, and includes monitoring and statistics templates and execution time rules. The generation of scheduled tasks to be executed based on the task configuration includes: Based on the monitoring and statistics template and the execution time rule, a task is generated to perform data monitoring and statistics on a regular schedule according to the execution time rule; The generated task is set as a scheduled task to be executed and associated with the task identifier.

6. A computing device, comprising: At least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the method as described in any one of claims 1-4.

7. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-4.

Images