A data monitoring method and device, a storage medium and an electronic device

Abstract

Embodiments of the present application disclose a data monitoring method and device, a storage medium and an electronic device. The method comprises: displaying a monitoring task creation interface, the monitoring task creation interface being an interface for a user to create a data monitoring task, the monitoring task creation interface comprising a plurality of editing controls; determining, based on an input operation on the editing controls, task attribute information of a target task and monitoring rule information of the target task; creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information; executing the target task; and performing real-time detection on task execution result data of the target task, executing the data monitoring task on the task execution result data when it is detected that the task execution result data exists, obtaining a data monitoring result, and displaying the data monitoring result in real time. This scheme enables a user to obtain a data monitoring result in a timely manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data monitoring method and device, a storage medium and an electronic device. BACKGROUND

[0002] Data monitoring is a means of timely and effective feedback on data abnormality. By monitoring data, whether the data is abnormal is analyzed so as to discover potential problems and give timely alarm. However, in the prior art, data monitoring is to output data to a business database and display in the form of a configured report. Therefore, when a user needs to monitor data, the user needs to actively check the report or wait for a scheduled push report, which undoubtedly affects the timeliness of the user obtaining the monitoring data. SUMMARY

[0003] The embodiments of the present application provide a data monitoring method and device, a storage medium and an electronic device, which enable a user to obtain data monitoring results in time.

[0004] The embodiments of the present application provide a data monitoring method, comprising:

[0005] displaying a monitoring task creation interface, the monitoring task creation interface being an interface for a user to create a data monitoring task, the monitoring task creation interface comprising a plurality of editing controls;

[0006] determining, based on an input operation on the editing controls, task attribute information of an input target task and monitoring rule information for the target task;

[0007] creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information;

[0008] executing the target task and performing real-time detection on task execution result data of the target task;

[0009] when it is detected that the task execution result data exists, executing the data monitoring task for the task execution result data to obtain a data monitoring result;

[0010] real-time display of the data monitoring result.

[0011] Correspondingly, the embodiments of the present application also provide a data monitoring device, comprising:

[0012] an interface display module configured to display a monitoring task creation interface, the monitoring task creation interface being an interface for a user to create a data monitoring task, the monitoring task creation interface comprising a plurality of editing controls;

[0013] determining, based on an input operation for the editing control, task attribute information of an input target task and monitoring rule information for the target task;

[0014] creating, based on the task attribute information and the monitoring rule information, a data monitoring task corresponding to the target task;

[0015] executing the target task and performing real-time detection on task execution result data of the target task;

[0016] monitoring, when it is detected that the task execution result data exists, the data monitoring task for the task execution result data to obtain data monitoring result;

[0017] displaying the data monitoring result in real time.

[0018] At this time, the creating module can specifically be configured to invoke a computing platform task interface, and when the invocation of the computing platform task interface is successful, create, based on the task attribute information and the monitoring rule information, the data monitoring task corresponding to the target task.

[0019] At this time, the executing module can specifically be configured to determine a target task that needs to be monitored from a plurality of candidate tasks, determine target to-be-processed data that needs to be processed from a plurality of to-be-processed data corresponding to the target task, execute the target task for the target to-be-processed data, and perform real-time detection on task execution result data of the target task.

[0020] Optionally, in some embodiments, the monitoring module can include an executing submodule and a writing submodule, as follows:

[0021] The executing submodule is configured to, when it is detected that the task execution result data exists, execute the data monitoring task for the task execution result data to obtain initial monitoring data.

[0022] The writing submodule is configured to write the initial monitoring data into a preset page template to obtain data monitoring result.

[0023] Optionally, in some embodiments, the executing submodule can include a second executing submodule, a format conversion submodule and a building submodule, as follows:

[0024] The second executing submodule is configured to, when it is detected that the task execution result data exists, execute the data monitoring task for a plurality of monitoring objects in the task execution result data to obtain initial monitoring sub-data corresponding to each monitoring object.

[0025] a format conversion submodule, configured to perform format conversion on each of the initial monitoring sub-data to obtain a plurality of converted monitoring sub-data;

[0026] a construction submodule, configured to construct initial monitoring data based on the plurality of converted monitoring sub-data.

[0027] At this time, the execution module can be specifically configured to parse the plurality of converted monitoring sub-data to obtain a plurality of parsed monitoring sub-data when it is detected that the data integrity of the plurality of converted monitoring sub-data meets a preset standard, and combine the plurality of parsed monitoring sub-data to obtain initial monitoring data.

[0028] At this time, the determination module can be specifically configured to determine task attribute information of a target task based on an input operation of the editing control in the first monitoring task creation interface, and determine monitoring rule information of the target task based on an input operation of the editing control in the second monitoring task creation interface.

[0029] In addition, an embodiment of the present application further provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute steps in any data monitoring method provided by the embodiments of the present application.

[0030] In addition, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement steps in any data monitoring method provided by the embodiments of the present application.

[0031] The embodiment of the present application can display a monitoring task creation interface, the monitoring task creation interface is an interface for a user to create a data monitoring task, the monitoring task creation interface includes a plurality of editing controls, task attribute information of a target task and monitoring rule information of the target task are determined based on an input operation of the editing control, a data monitoring task corresponding to the target task is created based on the task attribute information and the monitoring rule information, the target task is executed, and task execution result data of the target task is detected in real time, when it is detected that there is task execution result data, a data monitoring task is executed for the task execution result data to obtain data monitoring result, and the data monitoring result is displayed in real time. This scheme enables the user to obtain data monitoring result in time. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0033] Figure 1 is a scene schematic diagram of a data monitoring system provided by the embodiments of the present application;

[0034] Figure 2 is a first flow chart of a data monitoring method provided by the embodiments of the present application;

[0035] Figure 3 is a second flow chart of a data monitoring method provided by the embodiments of the present application;

[0036] Figure 4 is a first monitoring task creation interface provided by the embodiments of the present application;

[0037] Figure 5 is a second monitoring task creation interface provided by the embodiments of the present application;

[0038] Figure 6 is a third flow chart of a data monitoring method provided by the embodiments of the present application;

[0039] Figure 7 is a fourth flow chart of a data monitoring method provided by the embodiments of the present application;

[0040] Figure 8 is a fifth flow chart of a data monitoring method provided by the embodiments of the present application;

[0041] Figure 9 is a data monitoring result schematic diagram provided by the embodiments of the present application;

[0042] Figure 10 is a first task deployment schematic diagram provided by the embodiments of the present application;

[0043] Figure 11 is a second task deployment schematic diagram provided by the embodiments of the present application;

[0044] Figure 12 is a first task execution schematic diagram provided by the embodiments of the present application;

[0045] Figure 13 is a second task execution schematic diagram provided by the embodiments of the present application;

[0046] Figure 14 is a structure schematic diagram of a data monitoring device provided by the embodiments of the present application;

[0047] Figure 15 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0049] The present application provides a data monitoring method and device, a storage medium and an electronic device. Specifically, the data monitoring method of the present application can be executed by an electronic device, wherein the electronic device can be a terminal or a server, etc. The terminal can be a mobile phone, a tablet computer, a notebook computer, a smart television, a wearable smart device, a personal computer (PC), etc. The terminal can include a client, which can be a video client or a browser client, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0050] For example, referring to Figure 1 For example, referring to

[0051] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0052] The embodiment of the present application provides a data monitoring method, which can be executed by a terminal or a server, or executed by the terminal and the server together. The embodiment of the present application takes the data monitoring method executed by the server as an example to illustrate, as shown in the following Figure 2 , the specific process of the data monitoring method can be as follows:

[0053] 201, display a monitoring task creation interface.

[0054] The task is a working unit of a computer, and by using the task, scripts, programs or documents can be arranged to run at a preset time. For example, setting to count data on a fixed date of each month to obtain a monthly rebate amount is a task. The task attribute information of the task is information capable of pointing to a specific task, for example, the name of the task can be used as a kind of task attribute information.

[0055] The data monitoring is a means for timely and effectively feeding back data abnormal conditions, and by monitoring the data, whether the data is abnormal is analyzed, so that potential problems can be found and timely alarm can be given.

[0056] The monitoring task creation interface is an interface for a user to create a data monitoring task, and the monitoring task creation interface includes a plurality of editing controls. For example, as shown in the following Figure 4 and Figure 5 , the monitoring task creation interface is a broad concept, and as long as an interface in which data monitoring task related information is edited can be called a monitoring task creation interface. The monitoring task creation interface in the present application can include a first monitoring task creation interface as shown in the following Figure 4 , and a second monitoring task creation interface as shown in the following Figure 5 .

[0057] In actual application, for example, when a data monitoring task needs to be created, a developer can trigger a monitoring task creation request by clicking a "new monitoring task" button on a query interface, and display a monitoring task creation interface as shown in the following Figure 4 and Figure 5 according to the monitoring task creation request. The developer can edit information for a plurality of editing controls of the monitoring task creation interface.

[0058] 202, based on an input operation for the editing control, determine task attribute information of a target task and monitoring rule information of the target task.

[0059] Monitoring rule information refers to pre-set rules for monitoring data during the data monitoring process. Through this information, one can identify the objects to be monitored and the rules governing those monitoring. These rules can be represented in the form of SQL statements. For example, after a task is completed, a result table is generated. This table includes four fields: ID (number), username (username), age (age), and trade_cnt (number of transactions). If a user wants to check for duplicate usernames in the result table, they can write corresponding SQL code based on the monitoring rules. This edited SQL code is called the monitoring rule information. Structured Query Language (SQL) is a database query and programming language used for accessing, querying, updating, and managing relational database systems.

[0060] In practical applications, for example, the data monitoring method of this application targets a specific task. After the target task is completed, task execution result data is obtained. If it is necessary to monitor this task execution result data, then a data monitoring task targeting this task execution result data needs to be established. During the creation of the data monitoring task, it is first necessary to determine which task the task execution result data belongs to, i.e., the task attribute information of the target task. It is also necessary to determine how to monitor the task execution result data, i.e., the monitoring rule information for the target task.

[0061] In one embodiment, the necessary information for creating a data monitoring task can be obtained by editing the monitoring task creation interface. Specifically, the step "determining the task attribute information of the target task and the monitoring rule information for the target task based on the input operation to the editing control" may include:

[0062] Based on the input operation of the editing control in the first monitoring task creation interface, the task attribute information of the input target task is determined;

[0063] Based on the input operations of the editing controls in the second monitoring task creation interface, monitoring rule information for the target task is determined.

[0064] The first monitoring task creation interface is primarily for configuring basic monitoring information for data monitoring tasks. This interface can include multiple editing controls, each corresponding to a specific type of editing information. For example... Figure 4As shown in the figure, after the end of the target task execution, a result data table is obtained, which includes four fields: ID-number, username-user name, age-age, and trade_cnt-number of transactions. When a user wants to check whether the user name in the result data table is repeated, a corresponding data monitoring task can be established. At this time, the "business Chinese name" in the first monitoring task creation interface is the name of the data monitoring task "duplicate data checking monitoring"; the "notification type" refers to the way to notify the result receiver after obtaining the data monitoring result, which can be "email"; the task period refers to the period of executing the target task. If the target task is executed once a month, the task period can be edited as "month", etc.

[0065] The second monitoring task creation interface is mainly configured with monitoring rule information for the data monitoring task. The second monitoring task creation interface can include multiple editing controls, and each editing control corresponds to one type of editing information. For example, as shown in the figure, Figure 5 As shown in the figure, after the end of the target task execution, a result data table is obtained, which includes four fields: ID-number, username-user name, age-age, and trade_cnt-number of transactions. When a user wants to check whether the user name in the result data table is repeated, a corresponding data monitoring task can be established. At this time, the "title" in the second monitoring task creation interface can be "data duplication checking"; the "SQL code" fills in the corresponding SQL statement; since the user name needs to be checked for duplication, the "table header English field" is "username, repeat_cnt"; the "table header Chinese meaning" is the meaning of the query field "user name, number of repeated occurrences"; the "limit record number" indicates the maximum number of records that can be displayed when displaying the data monitoring result, which can be set to "200", etc.

[0066] In actual application, for example, when a data monitoring task needs to be created, the developer can trigger a monitoring task creation request by clicking the "add monitoring task" button on the query interface, and display the first monitoring task creation interface as shown in the figure. Figure 4 The developer can edit information for multiple editing controls of the first monitoring task creation interface. After editing is completed, the information can be saved by clicking the "save" button of the first monitoring task creation interface, and the second monitoring task creation interface as shown in the figure is displayed. Figure 5The second monitoring task creation interface is shown. Then the developer can edit information for a plurality of editing controls of the second monitoring task creation interface. After editing is completed, information can be saved by clicking a preset area of the second monitoring task creation interface. The clicking operation is an information editing completion operation for the second monitoring task creation interface. At this time, the task attribute information of the target task and the monitoring rule information for the target task can be obtained according to the editing operation of the developer. After editing of the monitoring task creation interface is completed, the edited information can be stored to facilitate subsequent operation processes.

[0067] 203. Create a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information.

[0068] In actual application, after the task attribute information and the monitoring rule information are obtained, a corresponding data monitoring task can be created. The data monitoring task is used to monitor task execution result data corresponding to the target task.

[0069] In an embodiment, specifically, the step of "creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information" can include:

[0070] calling a computing platform task interface;

[0071] When the calling of the computing platform task interface is successful, the data monitoring task corresponding to the target task is created based on the task attribute information and the monitoring rule information.

[0072] In actual application, for example, as Figure 6 shown, after information required for the data monitoring task is edited, the big data computing platform task interface can be called. When the calling of the big data computing platform task interface is successful, the data monitoring task corresponding to the target task is generated according to the task attribute information and the monitoring rule information. If the calling is unsuccessful, the step of calling the big data computing platform task interface is returned until the data monitoring task is generated. Figure 11 After the data monitoring task is created, a task deployment success interface as shown can be displayed. The "task 3" refers to the target task. After the data monitoring task is created successfully, the relationship between the target task and the data monitoring task is displayed in the form of an arrow on the interface.

[0073] This method can greatly improve the flexibility of the data monitoring method, so that the user can create a data monitoring task according to actual needs. When different data monitoring tasks need to be created, the corresponding editing information can be adjusted. Editing and modifying information on the interface can make the creation of the data monitoring task convenient and fast.

[0074] 204. execute the target task and perform real-time detection on task execution result data of the target task.

[0075] In actual applications, for example, the target task can be initiated for calculation or re-run. Since the data monitoring task needs to monitor the task execution result data generated after the execution of the target task, it is necessary to perform real-time detection on the data state of the task execution result data of the target task, that is, to detect whether the task execution result data that can be monitored is generated.

[0076] In an embodiment, since there are multiple tasks, the target task can be selected for task execution according to actual needs. Specifically, the step of “executing the target task and performing real-time detection on task execution result data of the target task” can include:

[0077] determining the target task to be monitored from multiple candidate tasks;

[0078] determining the target to-be-processed data to be processed from multiple to-be-processed data corresponding to the target task;

[0079] executing the target task for the target to-be-processed data and performing real-time detection on task execution result data of the target task.

[0080] In actual applications, for example, as shown in Figure 11 The multiple tasks “task 1”, “task 2”, “task 3”, “task 4”, and “task 5” in the left column of the figure are tasks to be monitored, and task 3 is selected as the target task from these tasks. The developer can determine task 3 as the target task by clicking the task area where task 3 is located. At this time, as shown in Figure 10 The developer can select “view instance” in the pop-up window, and at this time, the display as shown in Figure 12The interface shown displays a list of pending data for the target task. This list includes multiple pending data entries: "2020-04-01 00:00:00", "2020-03-01 00:00:00", and "2020-02-01 00:00:00". Since the target task is executed monthly, the pending data differs each time. Therefore, "2020-04-01 00:00:00" represents the pending data for April, "2020-03-01 00:00:00" for March, and "2020-02-01 00:00:00" for February. When executing the target task, you can select one pending data entry to execute. For example, you can select the pending data for February and then click the "Instance Rerun" button to execute the target task for that February data.

[0081] 205. When task execution result data is detected, a data monitoring task is executed on the task execution result data to obtain the data monitoring results.

[0082] In practical applications, for example, when the target task starts to initiate calculations and generate task execution result data, the task has entered the monitoring process. At this time, the calculation of the data monitoring task can be performed, and the data monitoring results can be obtained.

[0083] Specifically, the step "when the existence of the task execution result data is detected, execute the data monitoring task on the task execution result data to obtain the data monitoring result" may include:

[0084] When the task execution result data is detected, the data monitoring task is executed on the task execution result data to obtain initial monitoring data;

[0085] The initial monitoring data is written into a preset page template to obtain the data monitoring results.

[0086] In practical applications, for example, Figure 13 As shown, when task execution result data is detected, it indicates that the task has reached the data flow stage. A data monitoring task can then be executed based on this result, yielding initial monitoring data. At this point, the interface will display "Execution Successful" in the "Status" field of the corresponding target data to be processed, along with the start time and execution duration, allowing the user to understand the task's execution status. Then, based on this initial monitoring data, an HTML template page is populated to obtain the data monitoring results in page format.

[0087] Specifically, the step of "when it is detected that the task execution result data exists, performing the data monitoring task on the task execution result data to obtain initial monitoring data" can include:

[0088] when it is detected that the task execution result data exists, performing the data monitoring task on the plurality of monitoring objects in the task execution result data to obtain initial monitoring sub-data corresponding to each monitoring object;

[0089] performing format conversion on each initial monitoring sub-data to obtain a plurality of converted monitoring sub-data;

[0090] constructing initial monitoring data based on the plurality of converted monitoring sub-data.

[0091] In actual applications, for example, as shown in Figure 7 After the target task execution ends, a result data table is obtained, which includes a user name field. When a user wants to check whether the user names in the result data table are duplicated, a corresponding data monitoring task can be established. However, the result data table can include a plurality of user name fields, such as a user name "test1", a user name "test2", and the like. Therefore, the data monitoring task can monitor whether the user name "test1" is duplicated and whether the user name "test2" is duplicated, respectively, to obtain initial monitoring sub-data corresponding to a plurality of user names. Then, each initial monitoring sub-data is converted into JSON data by a udf function, that is, a plurality of converted monitoring sub-data. Then, initial monitoring data is constructed according to the plurality of converted monitoring sub-data.

[0092] Specifically, the step of "constructing initial monitoring data based on the plurality of converted monitoring sub-data" can include:

[0093] when it is detected that the data integrity of the plurality of converted monitoring sub-data meets a preset standard, parsing the plurality of converted monitoring sub-data to obtain a plurality of parsed monitoring sub-data;

[0094] merging the plurality of parsed monitoring sub-data to obtain initial monitoring data.

[0095] In actual applications, for example, as shown in Figure 8 After the plurality of converted monitoring sub-data is obtained, an external program can be used to check whether the plurality of converted monitoring sub-data is complete. If the plurality of converted monitoring sub-data is complete, each converted monitoring sub-data can be parsed to obtain a plurality of parsed monitoring sub-data, and the plurality of parsed monitoring sub-data can be merged to obtain initial monitoring data. Then, the initial monitoring data can be written into a preset page template to obtain a data monitoring result, and the data monitoring result can be output in real time.

[0096] In an embodiment, the initial monitoring data including the plurality of converted monitoring sub-data can also be stored, such as shown in Figure 7 The initial monitoring data can be stored into a unified intermediate temporary table, and finally the intermediate results are written into the monitoring result table and permanently retained.

[0097] In an embodiment, such as after the target task execution ends, a result data table is obtained, which includes four fields: ID-number, username-user name, age-age, and trade_cnt-transaction number. When a user wants to check whether the user name in the result data table is duplicated, a corresponding data monitoring task can be established. After the target task is executed and the result data table is obtained, the data monitoring task can be executed for the plurality of user names of the result data table, and the initial monitoring sub-data corresponding to each user name is obtained. Then, each initial monitoring sub-data is converted into a plurality of converted monitoring sub-data through a udf function. Then, the plurality of converted monitoring sub-data is checked by an external program to determine whether the plurality of converted monitoring sub-data is complete. If the plurality of converted monitoring sub-data is complete, the plurality of converted monitoring sub-data is parsed to obtain a plurality of parsed monitoring sub-data, and the plurality of parsed monitoring sub-data is merged to obtain initial monitoring data. Then, the initial monitoring data is used to fill in an HTML template page to obtain a page-form data monitoring result.

[0098] As shown in Figure 13 After the data monitoring task is executed successfully, the time of starting the task execution "2020-03-2019:46:41", the execution duration of the task "00:12:32", and the state of the task "execution success" are displayed in the area corresponding to the target to-be-processed data "2020-02-0100:00:00".

[0099] As shown in Figure 9 The page-form data monitoring result can include the push time, the data monitoring date "202002", the current display record number "2", the maximum display record number "200", and the list-form monitoring result, which includes the plurality of monitoring objects "test1" and "test2", and the repeated occurrence number of the user name "test1" "5" and the repeated occurrence number of the user name "test2" "3". The push time is the time of successfully executing the data monitoring task, that is, once the data monitoring task is executed successfully and the data monitoring result is obtained, the data monitoring result is immediately pushed.

[0100] In the embodiments of the present application, the data monitoring task and the target task are both computing tasks, the execution of the data monitoring task depends on the execution of the target task, in this way, bypass monitoring is achieved, and the execution of the data monitoring task does not affect the target task. At the same time, for the target task, each time the target task initiates computing again, the data monitoring task is also initiated to execute, thereby realizing re-running monitoring, so that the developer does not need to repeatedly execute the data monitoring work regularly, and the development efficiency is greatly improved.

[0101] 206, real-time display of the data monitoring result.

[0102] In actual application, since the developer actively and frequently checks the data monitoring result, the work efficiency is greatly affected, therefore, in order to ensure that the developer can receive the data monitoring result in time, after obtaining the data monitoring result, the data monitoring result can be pushed to the developer in real time, for example, the data monitoring result as shown in FIG. 6 can be pushed to the result receiving party, i.e. the developer, by the way of email, when the developer needs to check the data monitoring result, the developer only needs to query the pushed email, and does not need to run the program again to obtain the data monitoring result, so that the developer can discover the problem in time and follow up the problem in time. Figure 9

[0103] By using the data monitoring method of the embodiments of the present application, the developer only needs to set the data monitoring task on the data monitoring system, the system will automatically push the data monitoring result to the developer in time. Even if the computing logic of the target task is modified and needs to initiate computing again, the data monitoring method can automatically initiate task re-running and push the data monitoring result again.

[0104] ​As can be seen from the above, the embodiment of the present application can display a monitoring task creation interface, the monitoring task creation interface is an interface for a user to create a data monitoring task, the monitoring task creation interface includes a plurality of editing controls, based on an input operation on the editing controls, task attribute information of an input target task and monitoring rule information of the target task are determined, based on the task attribute information and the monitoring rule information, a data monitoring task corresponding to the target task is created, the target task is executed, and task execution result data of the target task is detected in real time, when it is detected that there is task execution result data, the data monitoring task is executed on the task execution result data, data monitoring result is obtained, and the data monitoring result is displayed in real time. Since the data monitoring result can be automatically and timely pushed after the task execution ends, the present application does not require a developer to frequently query data and task execution progress, so that the risk can be quickly monitored. Moreover, after the business computing logic is adjusted, the data monitoring task can also be reinitiated when the target task is reinitiated, so that the developer can still obtain accurate data monitoring task in time even after the computing logic is adjusted, so that problems can be found and followed up in time.

[0105] According to the method described in the foregoing embodiment, the following will be further described in detail by taking an example of the data monitoring method executed by an electronic device. Referring to Figure 3 , the specific process of the data monitoring method of the embodiment of the present application can be as follows:

[0106] 301. The electronic device obtains task information of a data monitoring task corresponding to a target task.

[0107] In actual application, for example, a result data table T_User_List is obtained after the target task is executed, the result data table includes four fields: ID-number, username-user name, age-age, and trade_cnt-transaction number, when a user wants to check whether the user name in the result data table is repeated, the data monitoring task corresponding to the target task can be established.

[0108] As shown in Figure 4 , "duplicate data check monitoring" can be edited at "business Chinese name" of the first monitoring task creation interface, "email" can be edited at "notification type", "month" can be edited at the task period, and so on. As shown in Figure 5As shown, "Data duplication check" can be edited at "Title" of the second monitoring task creation interface, the corresponding SQL statement can be edited at "SQL code", "Table header English field" is "username, repeat_cnt", "User name, number of repeated occurrences" is edited at "Table header Chinese meaning", "Limit record number" is set to "200", and the note is "Check data duplication", and the like. Among them, the information edited in the monitoring task creation interface is the task information of the data monitoring task.

[0109] In an embodiment, for example, the SQL statement edited at "SQL code" of the second monitoring task creation interface can be as follows:

[0110] select

[0111] username,count(*)repeat_cnt

[0112] from T_User_List

[0113] group by username

[0114] having count(*)>1

[0115] 302、When the call to the big data computing platform task interface is successful, the electronic device generates a data monitoring task.

[0116] In actual application, for example, when the call to the big data computing platform task interface is successful, a data monitoring task can be generated, as Figure 11 As can be seen from the figure, the target task "task 3" corresponds to the data monitoring task "data monitoring task".

[0117] 303、The electronic device executes the target task and generates task execution result data.

[0118] In actual application, for example, the target task "task 3" can be selected, and calculation or re-run can be initiated for the target task. As Figure 10 As shown, the area corresponding to the data monitoring task can be clicked, and the "view instance" option can be selected in the pop-up window to display the target task and the plurality of to-be-processed data corresponding to the target task, as Figure 12 As shown. Then, the "instance re-run" button on the interface is clicked, and the target task can be executed for the target to-be-processed data of the data time "2020-02-0100:00:00".

[0119] 304、When it is detected that the task execution result data is generated, the electronic device performs a data monitoring task on the plurality of monitoring objects in the task execution result data to obtain initial monitoring sub-data corresponding to the plurality of monitoring objects.

[0120] In actual applications, for example, when it is detected that the task execution result data is generated, it indicates that the task has been executed to the monitoring process, at this time, a data monitoring task can be performed on the plurality of user names in the task execution result data to obtain initial monitoring sub-data corresponding to the plurality of user names. As shown in FIG. 12, when the task execution is completed, the current state is displayed as "execution success" on the interface, and the execution duration "00:12:32" and the start time "2020-03-20 19:46:41" are displayed, so that the user can master the process of task execution. Figure 13

[0121] 305、The electronic device performs format conversion on each initial monitoring sub-data to obtain a plurality of converted monitoring sub-data.

[0122] 306、If it is detected that the plurality of converted monitoring sub-data is complete, the electronic device parses the plurality of converted monitoring sub-data to obtain a plurality of parsed monitoring sub-data.

[0123] 307、The electronic device merges the plurality of parsed monitoring sub-data to obtain initial monitoring data.

[0124] 308、The electronic device writes the initial monitoring data into a preset page template to obtain data monitoring results.

[0125] In actual applications, for example, after the initial monitoring data is written into the preset page template, the data monitoring results as shown in FIG. 13 can be obtained. The data monitoring results include the push time, the repeated user name "test1" and the corresponding repeated occurrence number "5", the repeated user name "test2" and the corresponding repeated occurrence number "3", the current display record number "2", the maximum display record number "200", and the like. Figure 9

[0126] 309、The electronic device pushes the data monitoring results in real time.

[0127] In actual applications, after the data monitoring results are obtained, the data monitoring results can be automatically pushed through the email mode.

[0128] ​​From the above, the embodiment of the application can obtain the task information of the data monitoring task corresponding to the target task through the electronic device, generate the data monitoring task when the calling of the big data computing platform task interface is successful, execute the target task, and generate the task execution result data, execute the data monitoring task on the multiple monitoring objects in the task execution result data when it is detected that the task execution result data is generated, obtain the initial monitoring sub-data corresponding to the multiple monitoring objects, perform format conversion on each initial monitoring sub-data to obtain multiple converted monitoring sub-data, analyze the multiple converted monitoring sub-data if it is detected that the multiple converted monitoring sub-data is complete to obtain multiple analyzed monitoring sub-data, merge the multiple analyzed monitoring sub-data to obtain the initial monitoring data, write the initial monitoring data into the preset page template to obtain the data monitoring result, and push the data monitoring result in real time. The scheme can automatically and timely push the data monitoring result after the end of the task execution, so that the developer does not need to frequently query the data and the task execution progress, thereby the risk can be quickly monitored. Moreover, after the business computing logic is adjusted, the data monitoring task can also be re-initiated for monitoring when the target task is re-initiated for computing, so that the developer can still obtain the accurate data monitoring task in time even after the computing logic is adjusted, thereby the problem can be found in time and the problem can be followed up in time.

[0129] To better implement the above method, correspondingly, the embodiment of the application further provides a data monitoring device which can be integrated in an electronic device, referring to Figure 14 The data monitoring device includes an interface display module 141, a determination module 142, a creation module 143, an execution module 144, a monitoring module 145, and a result display module 146, as follows:

[0130] The interface display module 141 is configured to display a monitoring task creation interface, wherein the monitoring task creation interface is an interface for a user to create a data monitoring task, and the monitoring task creation interface includes multiple editing controls.

[0131] The determination module 142 is configured to determine the task attribute information of an input target task and the monitoring rule information for the target task based on an input operation on the editing controls.

[0132] The creation module 143 is configured to create a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information.

[0133] The execution module 144 is configured to execute the target task and perform real-time detection on the task execution result data of the target task.

[0134] The monitoring module 145 is configured to perform the data monitoring task on the task execution result data when it is detected that the task execution result data exists, to obtain a data monitoring result.

[0135] The result display module 146 is configured to display the data monitoring result in real time.

[0136] In an embodiment, the creating module 143 can be specifically configured to:

[0137] invoke a computing platform task interface;

[0138] when the invocation of the computing platform task interface succeeds, create a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information.

[0139] In an embodiment, the executing module 144 can be specifically configured to:

[0140] determine a target task that needs to be monitored from a plurality of candidate tasks;

[0141] determine target to-be-processed data that needs to be processed from a plurality of to-be-processed data corresponding to the target task;

[0142] perform the target task on the target to-be-processed data, and perform real-time detection on task execution result data of the target task.

[0143] In an embodiment, the monitoring module 145 can include a first executing submodule and a writing submodule, as follows:

[0144] The first executing submodule is configured to perform the data monitoring task on the task execution result data when it is detected that the task execution result data exists, to obtain initial monitoring data;

[0145] The writing submodule is configured to write the initial monitoring data into a preset page template, to obtain a data monitoring result.

[0146] In an embodiment, the executing submodule can include a second executing submodule, a format conversion submodule, and a building submodule, as follows:

[0147] The second executing submodule is configured to perform the data monitoring task on a plurality of monitoring objects in the task execution result data when it is detected that the task execution result data exists, to obtain initial monitoring sub-data corresponding to each monitoring object;

[0148] The format conversion submodule is configured to perform format conversion on each initial monitoring sub-data, to obtain a plurality of converted monitoring sub-data;

[0149] The constructing submodule is configured to construct initial monitoring data based on the plurality of converted monitoring sub-data.

[0150] In an embodiment, the constructing submodule can be specifically configured to:

[0151] When the data integrity of the plurality of converted monitoring sub-data meets a preset criterion, the plurality of converted monitoring sub-data is parsed to obtain a plurality of parsed monitoring sub-data;

[0152] The plurality of parsed monitoring sub-data is merged to obtain initial monitoring data.

[0153] In an embodiment, the determining module 142 can be specifically configured to:

[0154] Based on the input operation on the editing control in the first monitoring task creation interface, the task attribute information of the input target task is determined;

[0155] Based on the input operation on the editing control in the second monitoring task creation interface, the monitoring rule information for the target task is determined.

[0156] In implementation, each of the above units can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each of the above units can refer to the method embodiments above, which will not be described here.

[0157] As can be seen from the above, the embodiments of the present application can display a monitoring task creation interface through the interface display module 141, the monitoring task creation interface is an interface for creating a data monitoring task by a user, the monitoring task creation interface includes a plurality of editing controls, the determining module 142 determines the task attribute information of the input target task and the monitoring rule information of the target task based on the input operation on the editing control, the creating module 143 creates a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information, the executing module 144 executes the target task, and the task execution result data of the target task is detected in real time, when it is detected that there is task execution result data, the monitoring module 145 executes the data monitoring task for the task execution result data to obtain a data monitoring result, and the result display module 146 displays the data monitoring result in real time. Since the data monitoring result can be automatically and timely pushed after the task execution ends, the present application does not need the developer to frequently query the data and the task execution progress, so that the risk can be quickly monitored. Moreover, after the business computing logic is adjusted, the data monitoring task can also be re-initiated for monitoring when the target task is re-initiated for computing, so that the developer can still obtain accurate data monitoring tasks in time even after the computing logic is adjusted, so that the problem can be found in time and followed up in time.

[0158] The embodiment of the present application further provides an electronic device, which can integrate any data monitoring device provided by the embodiment of the present application.

[0159] For example, as shown in Figure 15 Fig. 1 shows a structural schematic diagram of an electronic device related to the embodiment of the present application, in particular:

[0160] The electronic device can include a processor 151 with one or more processing cores, a memory 152 with one or more computer readable storage media, a power supply 153, an input unit 154, and the like. Those skilled in the art can understand that the structure of the electronic device shown in Figure 15 does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or arrange different components. Among them:

[0161] The processor 151 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 152, and calling data stored in the memory 152, thereby overall monitoring the electronic device. Optionally, the processor 151 can include one or more processing cores; preferably, the processor 151 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 151.

[0162] The memory 152 can be used to store software programs and modules, and the processor 151 executes various functions and data processing by running the software programs and modules stored in the memory 152. The memory 152 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one application program required by the function (such as sound playing function, image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 152 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 152 can also include a memory controller to provide access for the processor 151 to the memory 152.

[0163] The electronic device can further include a power supply 153 for supplying power to each component. Preferably, the power supply 153 can be logically connected to the processor 151 through a power management system, so that the power management system can be used to manage charging, discharging, power consumption management, and the like. The power supply 153 can further include one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

[0164] The electronic device can further include an input unit 154 for receiving input digital or character information, and generating keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0165] Although not shown, the electronic device can further include a display unit and the like, which will not be described here. In particular, in the present embodiment, the processor 151 in the electronic device loads one or more executable files corresponding to the processes of one or more application programs into the memory 152, and runs the application programs stored in the memory 152 by the processor 151, thereby implementing various functions, such as the following:

[0166] Displaying a monitoring task creation interface, the monitoring task creation interface being an interface for creating a data monitoring task by a user, the monitoring task creation interface including a plurality of editing controls, determining task attribute information of an input target task and monitoring rule information of the target task based on an input operation on the editing controls, creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information, executing the target task, and performing real-time detection on task execution result data of the target task, when it is detected that there is task execution result data, executing the data monitoring task on the task execution result data to obtain data monitoring result, and displaying the data monitoring result in real time.

[0167] The specific implementation of each of the above operations can be referred to the previous embodiments, which will not be described here.

[0168] From the above, the embodiment of the present application can display a monitoring task creation interface, the monitoring task creation interface is an interface for a user to create a data monitoring task, the monitoring task creation interface includes a plurality of editing controls, based on an input operation on the editing controls, task attribute information of an input target task and monitoring rule information of the target task are determined, based on the task attribute information and the monitoring rule information, a data monitoring task corresponding to the target task is created, the target task is executed, and task execution result data of the target task is detected in real time, when it is detected that there is task execution result data, the data monitoring task is executed on the task execution result data, data monitoring result is obtained, and the data monitoring result is displayed in real time. The scheme can automatically and timely push the data monitoring result after the task execution ends, so it is not necessary for a developer to frequently query data and task execution progress, thereby the risk can be quickly monitored. Moreover, after business computing logic is adjusted, the data monitoring task can also be re-initiated when the target task is re-initiated, so even after the computing logic is adjusted, the developer can still obtain accurate data monitoring task in time, thereby problems can be found in time and followed up in time.

[0169] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0170] To this end, an electronic device is provided, in which a plurality of instructions are stored, the instructions can be loaded by a processor to execute steps in any one of the data monitoring methods provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0171] A monitoring task creation interface is displayed, the monitoring task creation interface is an interface for a user to create a data monitoring task, the monitoring task creation interface includes a plurality of editing controls, based on an input operation on the editing controls, task attribute information of an input target task and monitoring rule information of the target task are determined, based on the task attribute information and the monitoring rule information, a data monitoring task corresponding to the target task is created, the target task is executed, and task execution result data of the target task is detected in real time, when it is detected that there is task execution result data, the data monitoring task is executed on the task execution result data, data monitoring result is obtained, and the data monitoring result is displayed in real time.

[0172] The specific implementation of each operation can be referred to the foregoing embodiments, which will not be described here.

[0173] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or the like.

[0174] Due to the instructions stored in the storage medium, the steps in any of the data monitoring methods provided by the embodiments of the present application can be executed, thus achieving the beneficial effects of any of the data monitoring methods provided by the embodiments of the present application. Details are shown in the foregoing embodiments, which will not be repeated here.

[0175] The foregoing describes in detail a data monitoring method, device, storage medium and electronic device provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The foregoing description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data monitoring method, characterized by, The method comprises the following steps: displaying a monitoring task creation interface, which is an interface for a user to create a data monitoring task, and the monitoring task creation interface comprises a plurality of editing controls; determining, based on an input operation on the editing controls, task attribute information of an input target task and monitoring rule information for the target task, wherein the target task is a task monitored by the data monitoring method; creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information, wherein the data monitoring task and the target task are in a monitoring and monitored relationship, the data monitoring task is used to monitor the target task, and the target task is monitored by the data monitoring task; displaying a bypass connection relationship between the data monitoring task and the target task in an interface in which task deployment is successful, wherein an execution result of the data monitoring task does not affect the target task; executing the target task and performing real-time detection on task execution result data of the target task, wherein the execution of the target task and the real-time detection on the task execution result data of the target task comprise: determining a target task that needs to be monitored from a plurality of candidate tasks; determining target to-be-processed data that needs to be processed from a plurality of to-be-processed data corresponding to the target task; executing the target task on the target to-be-processed data and performing real-time detection on task execution result data of the target task; when it is detected that the task execution result data exists, executing the data monitoring task on the task execution result data to obtain data monitoring result; real-time display of the data monitoring result.

2. The data monitoring method according to claim 1, characterized in that When it is detected that the task execution result data exists, executing the data monitoring task on the task execution result data to obtain data monitoring result, comprises: when it is detected that the task execution result data exists, executing the data monitoring task on the task execution result data to obtain initial monitoring data; writing the initial monitoring data into a preset page template to obtain data monitoring result.

3. The data monitoring method according to claim 2, characterized in that When it is detected that the task execution result data exists, executing the data monitoring task on the task execution result data to obtain initial monitoring data, comprises: when it is detected that the task execution result data exists, executing the data monitoring task on a plurality of monitoring objects in the task execution result data to obtain initial monitoring sub-data corresponding to each monitoring object; format-converting each initial monitoring sub-data to obtain a plurality of converted monitoring sub-data; constructing initial monitoring data based on the plurality of converted monitoring sub-data.

4. The data monitoring method according to claim 3, characterized in that Constructing initial monitoring data based on the plurality of converted monitoring sub-data, comprises: when it is detected that data integrity of the plurality of converted monitoring sub-data meets a preset standard, parsing the plurality of converted monitoring sub-data to obtain a plurality of parsed monitoring sub-data; merging the plurality of parsed monitoring sub-data to obtain initial monitoring data.

5. The data monitoring method according to claim 1, characterized in that create a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information, including: calling a computing platform task interface; when the computing platform task interface is successfully called, creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information.

6. The data monitoring method according to claim 1, characterized in that The monitoring task creation interface includes a first monitoring task creation interface and a second monitoring task creation interface. Based on the input operation on the editing control, the task attribute information of the input target task and the monitoring rule information for the target task are determined, including: Based on the input operation on the editing control in the first monitoring task creation interface, the task attribute information of the input target task is determined. Based on the input operation on the editing control in the second monitoring task creation interface, the monitoring rule information for the target task is determined.

7. A data monitoring device, characterized in that including: an interface display module for displaying a monitoring task creation interface, the monitoring task creation interface being an interface for a user to create a data monitoring task, the monitoring task creation interface including a plurality of editing controls; a determination module for determining, based on an input operation on the editing control, the task attribute information of the input target task and the monitoring rule information for the target task; wherein the target task is a task monitored by a data monitoring method implemented by the data monitoring device; a creation module for creating a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information; wherein the data monitoring task and the target task are in a monitoring and monitored relationship, the data monitoring task is used to monitor the target task, and the target task is monitored by the data monitoring task; The device is also used to display the bypass connection relationship between the data monitoring task and the target task in the task deployment success interface; wherein the execution result of the data monitoring task does not affect the target task; an execution module for executing the target task and performing real-time detection on the task execution result data of the target task; wherein the execution module is specifically used to determine the target task that needs to be monitored from a plurality of candidate tasks, determine the target to-be-processed data that needs to be processed from a plurality of to-be-processed data corresponding to the target task, execute the target task on the target to-be-processed data, and perform real-time detection on the task execution result data of the target task; a monitoring module for executing the data monitoring task on the task execution result data when it is detected that there is the task execution result data, to obtain a data monitoring result; a result display module for displaying the data monitoring result in real time.

8. The data monitoring device according to claim 7, characterized in that The creation module is specifically used to call a computing platform task interface, and when the computing platform task interface is successfully called, create a data monitoring task corresponding to the target task based on the task attribute information and the monitoring rule information.

9. The data monitoring device according to claim 7, characterized in that The monitoring module includes an execution submodule and a writing submodule, as follows: The execution sub-module is configured to execute the data monitoring task on the task execution result data when it is detected that the task execution result data exists, to obtain initial monitoring data. The writing sub-module is configured to write the initial monitoring data into a preset page template, to obtain data monitoring result.

10. The data monitoring device according to claim 9, characterized in that The execution sub-module includes a second execution sub-module, a format conversion sub-module and a construction sub-module, as follows: The second execution sub-module is configured to execute the data monitoring task on a plurality of monitoring objects in the task execution result data when it is detected that the task execution result data exists, to obtain initial monitoring sub-data corresponding to each monitoring object. The format conversion sub-module is configured to perform format conversion on each initial monitoring sub-data, to obtain a plurality of converted monitoring sub-data. The construction sub-module is configured to construct initial monitoring data based on the plurality of converted monitoring sub-data.

11. The data monitoring device according to claim 10, characterized in that The execution module is specifically configured to parse the plurality of converted monitoring sub-data when it is detected that data integrity of the plurality of converted monitoring sub-data meets a preset standard, to obtain a plurality of parsed monitoring sub-data, and to combine the plurality of parsed monitoring sub-data, to obtain initial monitoring data.

12. The data monitoring device of claim 7, wherein, The monitoring task creation interface includes a first monitoring task creation interface and a second monitoring task creation interface; the determination module is specifically configured to determine task attribute information of a target task based on input operation on the editing control in the first monitoring task creation interface, and to determine monitoring rule information for the target task based on input operation on the editing control in the second monitoring task creation interface.

13. A computer storage medium having stored thereon a computer program, characterized in that When the computer program runs on the computer, the computer is caused to execute the data monitoring method of any one of claims 1-6.

14. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the program to implement the steps of the method of any one of claims 1 to 6. The processor executes the program to implement the steps of the method of any one of claims 1 to 6.

Images