Data monitoring method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data management, and discloses a data monitoring method and device suitable for multiple technology stacks, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a declarative language expression; analyzing the declarative language expression into a task description formula, wherein the declarative language expression comprises various kinds of technology stack data corresponding to a plurality of data sources needing to be monitored; generating various monitoring codes corresponding to the technology stack data according to the task description and the technology stack data; and running the plurality of monitoring codes to monitor the target data in the plurality of data sources. According to the technical scheme, the data of various technical stacks can be monitored.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data management technology, and specifically to a data monitoring method, device, electronic device, and computer-readable storage medium that are adaptable to multiple technology stacks. Background Art

[0002] In today's data-driven business environment, data quality management and monitoring have become the core of the daily work of data analysts and data developers. As the scale and complexity of enterprise data increase, data quality issues at any stage in the process from raw data collection, cleaning and conversion to final provision to business applications may have a serious impact on enterprise decision-making and operations.

[0003] The full data link of a modern enterprise usually includes multiple data pipelines and storage technologies, which are often provided by different suppliers and have different technology stacks. In order to ensure the high-quality operation of the entire data link, data analysts and data developers need to strictly monitor the quality of data in each link. However, existing data monitoring technology tools can usually only run in a single technology stack or a specific environment, which forces data developers to write monitoring code separately for each technology stack. This decentralized monitoring method not only increases the workload, but also makes it difficult to centrally manage data monitoring data. In other words, it is difficult for existing data monitoring platforms to centrally monitor data from multiple data sources based on different technology stacks.

[0004] Therefore, how to provide a data monitoring method that is adaptable to multiple technology stacks has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, an embodiment of the present application provides a data monitoring method that is adaptable to multiple technology stacks, which is used to solve the problem in the prior art that the current data monitoring platform is difficult to centrally monitor data from multiple data sources implemented based on different technology stacks.

[0006] According to one aspect of an embodiment of the present application, a data monitoring method is provided, the method comprising: obtaining a declarative language expression; parsing the declarative language expression into a task description, the declarative language expression comprising a plurality of technology stack data corresponding to a plurality of data sources to be monitored; generating a plurality of monitoring codes corresponding to the technology stack data according to the task description and the technology stack data; and running the plurality of monitoring codes to monitor target data in a plurality of data sources.

[0007] Preferably, before running multiple monitoring codes to monitor target data in multiple data sources, the data monitoring method also includes: matching adapters corresponding to the monitoring code data according to the technology stack data; running multiple monitoring codes to monitor target data in multiple data sources, and also includes: running multiple monitoring codes on multiple adapters respectively to monitor the target data in multiple data sources.

[0008] Preferably, matching the adapter corresponding to the monitoring code data according to the technology stack data includes: converting the declarative language expression into multiple imperative expressions according to multiple technology stack data; running multiple monitoring codes on multiple adapters to monitor the target data in multiple data sources, further including: configuring the environment variables corresponding to the technology stack data according to the imperative expressions, wherein multiple environment variables correspond to the monitoring code; configuring the running environment of the monitoring code according to the environment variables; and running the corresponding monitoring codes in multiple running environments to monitor the target data in multiple data sources.

[0009] Preferably, the data monitoring method also includes: obtaining an analysis task corresponding to the target data; after running multiple monitoring codes to monitor the target data in multiple data sources, the data monitoring method also includes: obtaining monitoring data for monitoring the target data, the monitoring data corresponding to the analysis task; executing the analysis task corresponding to the monitoring data to obtain the analysis data; and displaying the analysis data.

[0010] Preferably, the data monitoring method also includes: obtaining the warning indicator range and target user corresponding to the analysis task; after executing the corresponding analysis task on the monitoring data and obtaining the analysis data, the data monitoring method also includes: judging whether the analysis data exceeds the warning indicator range; if the analysis data exceeds the warning indicator range, generating a warning report based on the analysis data; and sending the warning report to the target user.

[0011] Preferably, the declarative language expression also includes original definition data corresponding to the target data; parsing the declarative language expression into a task description includes: converting the original definition data into multiple tags through a lexical analyzer; generating an abstract syntax tree based on the multiple tags through a syntax analyzer; and converting the abstract syntax tree into a task description through a task translator.

[0012] Preferably, a plurality of monitoring codes corresponding to the technology stack data are generated according to the task description formula and the technology stack data, including: obtaining a plurality of code templates corresponding to the plurality of technology stacks; filling in the code templates according to the task description formula; and compiling the filled in code templates to obtain the monitoring code.

[0013] According to another aspect of an embodiment of the present application, a data monitoring device is provided, including: an acquisition module for acquiring a declarative language expression; a parsing module for parsing the declarative language expression into a task description, the declarative language expression including a plurality of technology stack data corresponding to a plurality of data sources to be monitored; a generation module for generating a plurality of monitoring codes corresponding to the technology stack data based on the task description and the technology stack data; and a monitoring module for running a plurality of monitoring codes to monitor target data in a plurality of data sources.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor and a computer program stored in the memory. Preferably, the processor executes the computer program to implement the steps of the data monitoring method as described in any of the above embodiments.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data monitoring method described in any of the above embodiments are implemented.

[0016] The embodiment of the present application obtains a declarative language expression based on a declarative language to unify the input language into a declarative language, and generates code for monitoring different data sources based on the declarative language expression and multiple technology stack data to reduce the difficulty and intensity of the developer's work; then, the target data is monitored by running the code of different technology stacks to implement the monitoring of multiple technology stack data sources based on the same data monitoring system, thereby realizing unified management of data.

[0017] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings: Figure 1 A schematic diagram of an application scenario of the data monitoring method provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of a data monitoring method according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of the structure of a data monitoring device provided in an embodiment of the present application is shown; Figure 4A schematic diagram of the structure of a data monitoring device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0020] Figure 1 A schematic diagram of an application scenario of the data monitoring method provided in an embodiment of the present application is shown. Figure 1 As shown, the first server is a local server or a cloud server, which is used to run the monitoring code input by the developer, and the second server is the target server. The first server runs the monitoring code to monitor the data in the second server.

[0021] Figure 2 FIG. 1 shows a flow chart of a data monitoring method provided in an embodiment of the present application, wherein the method is executed by a data monitoring device. The data monitoring device is a first server. Figure 2 As shown, the method comprises the following steps: S110, obtaining a declarative language expression.

[0022] Among them, the declarative language expression is the expression input by the developer based on the declarative language. The declarative language is a programming paradigm that allows developers to write programs by describing the logical relationship of the problem and the results to be achieved, rather than through detailed control processes. Declarative languages ​​are widely used in database query languages ​​(such as SQL), configuration management (such as Ansible, Puppet), functional programming languages ​​(such as Haskell, Erlang), and template languages ​​in many modern frameworks, making it adaptable to most technology stacks.

[0023] Data developers use declarative languages ​​to define data quality indicators, such as data accuracy, completeness, distribution patterns, etc. Declarative languages ​​are developed based on domain-specific languages ​​(DSL), which are specially designed to describe data quality rules and have concise and efficient expression capabilities.

[0024] For example, the declarative language expression can be written as "data_source: "DeltaBillingData"" to specify that the monitored data source is "DeltaBillingData". The declarative language expression can also be written as "monitoring_interval : "0 3 * * 1"" to use the Corn task expression to specify that the execution time of the monitoring task is 03:00 every Monday.

[0025] The input language is unified into a declarative language by obtaining a declarative language expression based on the declarative language.

[0026] Among them, the input of declarative language expressions can be based on text window input or on low-code or no-code platforms (such as OutSystems, Mendix, and Appian).

[0027] S130, parsing a declarative language expression into a task description, wherein the declarative language expression includes a plurality of technology stack data corresponding to a plurality of data sources to be monitored.

[0028] The parsed task description undergoes detailed grammatical and semantic analysis to ensure logical correctness and is presented in a structured format (such as JSON or YAML).

[0029] For example, the declarative language expression is: "quality_expectations: - data_source: "DeltaBillingData" data_location: "delta: / / your_delta_table_location" data_format: "delta" column: "amount,invoice_id" metric: "duplicate_detection_check" checks: - type: "duplicate_check" action_if_found: - type: "log" log_level: "error" log_file: "data_quality.log" monitoring_interval: cron_expression: "0 3 * * 1" description: "Weekly check on Mondays at 03:00 AM" action_on_failure: notify: channels: - type: "email" recipient: email_address: "notify@example.com" - type: "slack" recipient: channel: "#data-quality-alerts" webhook_url: "https: / / hooks.slack.com / services / T00000000 / B00000000 / XXXXXXXXXXXXXXXXXXXXXXXX" message_template: "Duplicate values ​​detected in amountand invoice_id columns." data_connection: connection_type: "delta_tables" table_name: "billing_data" delta_tables_connection: "delta: / / your_delta_tables_connection_string".

[0030] The task description obtained by parsing the above declarative language expression can be: "{ "data_source": "DeltaBillingData", "data_location": "delta: / / your_delta_table_location", "data_format": "delta", "columns": ["amount", "invoice_id"], "metric": "duplicate_detection_check", "checks": [ { "type": "duplicate_check", "action_if_found": [ { "type": "log", "log_level": "error", "log_file": "data_quality.log" } ] } ], "monitoring_interval": { "cron_expression": "0 3 * * 1", "description": "Weekly check on Mondays at 03:00 AM" }, "action_on_failure": { "notify": { "channels": [ { "type": "email", "recipient": { "email_address": "notify@example.com" }".

[0031] It can be found that the task description obtained by parsing is highly structured, so that it is easy for computers to recognize and read. The following uses the declarative language expression example and task description example here for explanation.

[0032] Data from multiple technology stacks usually correspond to multiple data formats. For example, web applications (front-end technology stacks such as JavaScript, Node.js, React, Angular, and Vue) usually use the JSON format; YAML is usually used as the configuration file format in tools such as Kubernetes, Ansible, and Docker Compose; MongoDB databases use binary serialization formats such as BSON.

[0033] When the data source is complex and multiple data sources are implemented based on multiple technology stacks, it is necessary to read and process texts in multiple data formats, and the whole process is relatively cumbersome. However, by parsing the declarative language expression into a task description, the resulting file format is more unified, which facilitates subsequent reading and processing of the file.

[0034] S150, generating a plurality of monitoring codes corresponding to the technology stack data according to the task description formula and the technology stack data.

[0035] The monitoring code is a code that can execute monitoring tasks, for example, a code that monitors the addition and deletion of data of a certain attribute in SQL.

[0036] Specifically, S150 includes the following sub-steps: S151, obtaining multiple code templates corresponding to multiple technology stacks.

[0037] Based on the task description, the data monitoring device automatically generates the corresponding monitoring code. This process relies on a pre-defined code template library that covers the implementation of different technology stacks, including Python, Java, SQL, Shell scripts, etc.

[0038] S153, filling in the code template according to the task description.

[0039] The code template library stores code templates suitable for various tasks and technology stacks. The template library not only covers common monitoring scenarios, but also supports the expansion of custom templates to ensure that the system can cope with diverse monitoring needs.

[0040] The template library can dynamically fill and customize the template content by introducing template engines such as Jinja2 and Mustache. Among them, the template engine: using powerful template engines such as Jinja2 and Mustache, the module can select the appropriate template from the code template library based on the task description, and perform variable replacement and logic filling. The use of template engines ensures the flexibility and scalability of the code generation process.

[0041] S155, compile the filled code template to obtain the monitoring code.

[0042] The code generator, dynamic compilation and optimization tools, and code versioning and auditing dynamically compile and optimize the generated code to ensure that the monitoring code can run efficiently in different environments.

[0043] For scripting languages, the generated code can be passed directly to the execution engine; for compiled languages, dynamic compilation and execution can be achieved with the help of Just-In-Time Compilation (JIT), such as Python's exec and Java's JIT compiler.

[0044] Among them, the code generator supports the processing of complex logic and the splicing of code fragments, and performs preliminary verification of the generated code to ensure that it complies with grammatical rules; the dynamic compilation and optimization tool integrates the JIT compiler and other optimization tools to ensure that the generated code that needs to be executed immediately achieves optimal performance at runtime, thereby adapting to large-scale data monitoring tasks; in order to ensure that the generated code is traceable, the code version management and audit tool also integrates the version control tool and supports the code audit function to record detailed information of each code generation to ensure the compliance and security of the code.

[0045] S170, running multiple monitoring codes to monitor target data in multiple data sources.

[0046] The monitoring code generated by S110~S150 can adapt to different technology stacks, such as SQL, NoSQL, and big data processing frameworks (such as Hadoop, Spark), to achieve cross-platform data quality monitoring, thereby realizing agile data quality monitoring deployment and data management; and, executing S110 can unify the input language of the data monitoring device into a declarative language, and executing S130 can parse the declarative language expression into a highly structured and uniformly formatted task description to improve the execution speed of S150, so that the data monitoring device can automatically generate monitoring code that adapts to the data format of multiple data sources. Preferably, the declarative language expression also includes original definition data corresponding to the target data, and S130 includes the following sub-steps: S131, converting the original definition data into a plurality of tokens through a lexical analyzer.

[0047] The lexical analyzer (Lexer) may be a tool such as Flex, ANTLR or PLY (Python Lex-Yacc).

[0048] For example, "type: "duplicate_check"" in the declarative language expression instance is the original definition data, indicating that a duplicate data monitoring task needs to be performed, and the target data is the data that appears repeatedly in the data source.

[0049] For example, the declarative language expression instance in S130 may be converted into the following markup instance: “DATA_SOURCE("DeltaBillingData") DATA_LOCATION("delta: / / your_delta_table_location") DATA_FORMAT("delta") COLUMN("amount,invoice_id") METRIC("duplicate_detection_check") CHECKS CHECK_TYPE("duplicate_check") ACTION_IF_FOUND LOG LOG_LEVEL("error") LOG_FILE("data_quality.log") MONITORING_INTERVAL CRON_EXPRESSION("0 3 * * 1") DESCRIPTION("Weekly check on Mondays at 03:00 AM") ACTION_ON_FAILURE NOTIFY CHANNELS EMAIL EMAIL_ADDRESS("notify@example.com") SLACK CHANNEL("#data-quality-alerts") WEBHOOK_URL("https: / / hooks.slack.com / services / T00000000 / B00000000 / XXXXXXXXXXXXXXXXXXXXXXXX") MESSAGE_TEMPLATE("Duplicate values detected in amount and invoice_idcolumns.") DATA_CONNECTION CONNECTION_TYPE("delta_tables") TABLE_NAME("billing_data") DELTA_TABLES_CONNECTION("delta: / / your_delta_tables_connection_string")".

[0050] S133, generating an abstract syntax tree according to the multiple tags through a syntax analyzer.

[0051] Among them, the parser uses context-free grammar (CFG) to define the grammatical rules of DSL. The parser converts the token sequence into an abstract syntax tree (AST) to represent the structure and hierarchy of the code through AST.

[0052] The lexical analyzer converts the input declarative language expression into a series of tokens, while the syntax analyzer organizes these tokens into AST.

[0053] For example, the tag instance in S131 can be converted into the following abstract syntax tree instance: "DataExpectations ( DataSource("DeltaBillingData"), Metric("0 3 * * 1", "Weekly check on ...", ActionOnFailure( Notify( Email("notify@example.com", Action(Checks(DuplicateCheck(ActionIfFound(Log("error", "data_quality.log")))))), Slack("#data-quality-alerts", None) ) )), MonitoringInterval("0 3 * * 1") )”.

[0054] Furthermore, the generated AST can be passed to a semantic analyzer for semantic checking. The semantic analyzer traverses the AST to check type consistency, range resolution, and other semantic rules to ensure the logical correctness and consistency of the quality definition.

[0055] S135, converting the abstract syntax tree into a task description through a task translator.

[0056] Among them, the Task Translator is used to convert AST into a standardized task description of the system. The conversion involves pattern matching, tree traversal and other technologies to map the complex AST structure into a standardized task description format.

[0057] For example, the abstract syntax tree instance in S133 is converted into a task description instance in S130.

[0058] By executing S131 to S135, the data quality indicators defined by the developer using the declarative language can be parsed and converted into tasks that can be executed by the data monitoring device.

[0059] In order to ensure that when the data monitoring device executes S170, multiple types of monitoring codes can all run on the same data monitoring device, preferably, before S170, the data monitoring method further includes: S160, matching an adapter corresponding to the monitoring code data according to the technology stack data.

[0060] Execute S160 to automatically load the adapter module according to the system configuration and task requirements.

[0061] The choice of adapter depends on the technology stack and target execution environment specified in the task description. Common technology stacks include SQL databases (such as MySQL, PostgreSQL), NoSQL databases (such as MongoDB, Cassandra), big data platforms (such as Apache Hadoop, Spark), and cloud services (such as AWS, Azure).

[0062] The Adapter Library contains multiple pre-built adapters, each corresponding to a specific technology stack. The Adapter Library supports extensions, and new adapters can be added based on the needs of new technology stacks or platforms. The design of the adapter is based on design patterns (such as the strategy pattern), which enables it to flexibly handle different task descriptions and target environments.

[0063] Specifically, S160 includes: S161, converting declarative language expressions into multiple imperative expressions based on multiple technology stack data.

[0064] The adapter and instruction generator convert the received standardized task description into executable instructions for a specific technology stack. This process includes parsing and mapping the task description and converting it into the commands or scripts required by the target technology stack. For example, converting the data quality check in the task description into a SQL query, MapReduce job, or REST API call.

[0065] Among them, the instruction generator can convert the standardized task description into internal specific instructions. The instruction generator can parse the operation requirements in the task description and map them into operation instructions for the target environment through template matching or rule engine. Common technologies include regular expressions, AST parsing and pattern matching algorithms.

[0066] For example, convert the declarative language expression instance of S130 into the following imperative instance: "{ "conn_config": { "database_host": "your_server", "database_port": 1433, "database_user": "your_username", "database_password": "your_password", "database_name": "your_database_name", "s3_bucket": "your_s3_bucket", "s3_feedback_path": "your_feedback_path", "delta_table_location": "delta: / / your_delta_table_location" }, "tasks": [ { "command": "SELECT COUNT(*) FROM sales_data WHERE total_amountIS NULL;", "type": "SQL" }, { "command": "import os; df = spark.read.format('delta').load(os.environ['delta_table_location']); duplicate_count = df.groupBy('amount','invoice_id').count().filter('count>1').count()", "type": "Spark" }, { "command": "import os; import boto3; import json; s3 =boto3.client('s3'); response = s3.get_object(Bucket=os.environ['s3_bucket'],Key=os.environ['s3_feedback_path']); content = response['Body'].read().decode('utf-8'); feedback_data = json.loads(content); for feedback infeedback_data: if feedback['rating']<1 or feedback['rating']>5: out_range_rating = feedback['rating'] "".

[0067] Furthermore, the adapter interacts with the native interfaces of the target technology stack, such as database drivers (e.g., JDBC, ODBC), REST API clients, cloud service SDKs, etc., to ensure the reliability and compatibility of operations. Among them, the adapter interacts with the target environment through the technology stack interface. The interface layer encapsulates access to the underlying system, such as connecting to the database through the JDBC driver, calling cloud services through the RESTful API, or accessing distributed systems through RPC (remote procedure call). The interface layer ensures compatibility between the adapter and the target environment and handles common errors and exceptions.

[0068] That is, S170 also includes: S171, running a plurality of monitoring codes on a plurality of adapters respectively to monitor target data in a plurality of data sources.

[0069] Specifically, S171 further includes: S171a, configuring environment variables corresponding to the technology stack data according to the instructional method, wherein a plurality of environment variables correspond to the monitoring code; S171b, configuring the operating environment of the monitoring code according to the environment variables; S171c, respectively running corresponding monitoring codes in multiple operating environments to monitor target data in multiple data sources.

[0070] Tasks for monitoring code execution may include running SQL queries, starting big data processing jobs, calling REST API interfaces, etc. to verify the accuracy, completeness and consistency of data; during the execution process, the data monitoring device collects data quality indicators in real time and monitors the execution status; the data monitoring device records the execution logs, error information and performance data of all tasks to ensure that any anomalies can be quickly discovered and handled; after the execution is completed, the data monitoring device returns the results to the central management and visualization platform, including detailed records of task execution and data quality monitoring results.

[0071] When running monitoring codes for multiple data sources, you may be limited by the available resources of the data monitoring device. Therefore, you can set up a task scheduler to optimize the execution order of tasks based on task priority, resource availability, and task dependencies. The scheduler also supports distributed scheduling (Apache Airflow, Celery) to ensure efficient scheduling of tasks in large-scale data environments.

[0072] Furthermore, to ensure that the adapter's operation is transparent and traceable, the data monitoring device also integrates logging and monitoring functions. Each instruction generation and execution will be recorded in detail, and the adapter's operating status can be monitored in real time. Log information can be used for troubleshooting, performance analysis, and auditing.

[0073] Through S160~S171c, you can ensure that the task can run successfully in the specified environment.

[0074] After obtaining the monitoring data, the data monitoring device may also provide data analysis. Preferably, the data monitoring method further includes: S168, obtaining an analysis task corresponding to the target data.

[0075] The monitoring results of the target data in multiple data sources include detailed quality indicators, task execution logs, and exception reports. The data monitoring device can also store this data in a central database or data lake for subsequent analysis and visualization.

[0076] After the data is stored, the data monitoring device automatically runs the preset analysis tasks. These tasks include data quality trend analysis, anomaly detection, and predictive analysis based on historical data. The analysis module also compares the quality indicators with the preset quality thresholds and generates quality warnings. If an abnormality exceeding the threshold is detected, the system will trigger the warning mechanism.

[0077] After S170, the data monitoring method further includes: S181, acquiring monitoring data for monitoring target data, where the monitoring data corresponds to the analysis task.

[0078] S183, executing the analysis task corresponding to the monitoring data to obtain analysis data.

[0079] S185, displaying the analyzed data.

[0080] Through the integrated visualization engine, the data monitoring device also provides intuitive graphical displays to help data developers quickly understand the data quality status. Visualization content can include time series graphs, distribution graphs, heat maps, etc. Users can also deeply analyze data through an interactive interface to view detailed information or customized views at a specific time point.

[0081] Preferably, the data monitoring method further includes: S191, obtaining the warning indicator range and target users corresponding to the analysis task.

[0082] After S183, the data monitoring method further includes: S184, determining whether the analysis data exceeds the warning indicator range.

[0083] S186: If the analysis data exceeds the warning indicator range, a warning report is generated based on the analysis data.

[0084] S188, sending the warning report to the target user.

[0085] When data monitoring equipment detects potential problems or anomalies, it will automatically push warning information to relevant personnel. Warnings can be sent via email, SMS, or instant messaging tools (Slack, Teams), and provide detailed descriptions of the anomalies and possible solutions to ensure that the problem can be handled in a timely manner.

[0086] Figure 3 FIG. 1 shows a schematic diagram of the structure of a data monitoring device provided in an embodiment of the present application. Figure 3 As shown, the device 200 includes: an acquisition module 210, a parsing module 220, a generation module 230 and a monitoring module 240.

[0087] The acquisition module 210 is used to acquire a declarative language expression; The parsing module 220 is used to parse the declarative language expression into a task description, where the declarative language expression includes multiple technology stack data corresponding to multiple data sources to be monitored; The generation module 230 is used to generate a plurality of monitoring codes corresponding to the technology stack data according to the task description formula and the technology stack data; The monitoring module 240 is used to run various monitoring codes to monitor target data in multiple data sources.

[0088] The data monitoring device of the embodiment of the present application also includes other modules for executing the steps of the above-mentioned data monitoring method embodiment, which will not be described one by one here.

[0089] Figure 3 The schematic diagram of the structure of the data monitoring device provided by the embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the data monitoring device.

[0090] like Figure 3 As shown, the data monitoring device may include: a processor (processor) 302 and a memory (memory) 304.

[0091] The memory 304 is used to store a computer program 306. The memory 304 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The computer program 306 may include computer executable instructions.

[0092] The processor 302 is configured to execute the computer program 306 to implement the steps in the above data monitoring method embodiment.

[0093] The processor 302 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the data monitoring device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0094] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above data monitoring method embodiment are implemented.

[0095] An embodiment of the present application provides a computer program, which can be executed by a processor to implement the steps in the above data monitoring method embodiment.

[0096] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the above data monitoring method embodiment are implemented.

[0097] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0098] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0099] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed application requires more features than those explicitly recited in each claim.

[0100] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0101] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data monitoring method adapted to multiple technology stacks, characterized in that: The data monitoring method comprises: Get declarative language expressions; Parsing the declarative language expression into a task description, wherein the declarative language expression includes multiple technology stack data corresponding to multiple data sources to be monitored; Generate a plurality of monitoring codes corresponding to the technology stack data according to the task description formula and the technology stack data; A plurality of monitoring codes are run to monitor target data in the plurality of data sources.

2. The data monitoring method according to claim 1, characterized in that: Before running the plurality of monitoring codes to monitor the target data in the plurality of data sources, the data monitoring method further includes: Matching an adapter corresponding to the monitoring code data according to the technology stack data; The running of the plurality of monitoring codes to monitor the target data in the plurality of data sources also includes: A plurality of monitoring codes are respectively run on a plurality of the adapters to monitor the target data in the plurality of data sources.

3. The data monitoring method according to claim 2, characterized in that: The step of matching the adapter corresponding to the monitoring code data according to the technology stack data includes: Converting the declarative language expression into multiple imperative expressions according to the multiple technology stack data; The step of respectively running a plurality of monitoring codes on a plurality of the adapters to monitor the target data in the plurality of data sources further comprises: configuring environment variables corresponding to the technology stack data according to the imperative method, wherein a plurality of the environment variables correspond to the monitoring code; Configuring the operating environment of the monitoring code according to the environment variables; The corresponding monitoring codes are respectively run in the multiple operating environments to monitor the target data in the multiple data sources.

4. The data monitoring method according to claim 3, characterized in that: The data monitoring method further includes: Acquire an analysis task corresponding to the target data; After running the plurality of monitoring codes to monitor the target data in the plurality of data sources, the data monitoring method further includes: Acquire monitoring data for monitoring the target data, wherein the monitoring data corresponds to the analysis task; Executing an analysis task corresponding to the monitoring data to obtain analysis data; The analysis data are displayed.

5. The data monitoring method according to claim 4, characterized in that: The data monitoring method further includes: Obtaining the warning indicator range and target users corresponding to the analysis task; After performing the analysis task corresponding to the monitoring data to obtain the analysis data, the data monitoring method further includes: Determining whether the analysis data exceeds the warning indicator range; If the analysis data exceeds the warning indicator range, generating a warning report based on the analysis data; The early warning report is sent to the target user.

6. The data monitoring method according to claim 1, characterized in that: The declarative language expression also includes original definition data corresponding to the target data; The step of parsing the declarative language expression into a task description includes: Converting the original definition data into a plurality of tokens by a lexical analyzer; Generate an abstract syntax tree according to the multiple tags by a syntax analyzer; The abstract syntax tree is converted into a task description by a task translator.

7. The data monitoring method according to claim 1, characterized in that: The generating a plurality of monitoring codes corresponding to the technology stack data according to the task description formula and the technology stack data includes: Obtain multiple code templates corresponding to multiple technology stacks; Filling the code template according to the task description; The filled code template is compiled to obtain the monitoring code.

8. A data monitoring device, characterized in that: The data monitoring device comprises: The acquisition module is used to obtain declarative language expressions; A parsing module, used to parse the declarative language expression into a task description, wherein the declarative language expression includes multiple technology stack data corresponding to multiple data sources to be monitored; A generation module, used to generate a plurality of monitoring codes corresponding to the technology stack data according to the task description formula and the technology stack data; The monitoring module is used to run a plurality of the monitoring codes to monitor the target data in the plurality of data sources.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the data monitoring method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data monitoring method according to any one of claims 1 to 7 are implemented.

Images