Data supervision method and device based on task scheduling platform, equipment and medium

Abstract

The invention provides a data supervision method and device based on a task scheduling platform, equipment and a medium, and relates to the technical field of data security. According to the method, automatic creation and scheduling of the data supervision task are realized through the task scheduling system pre-configured by the visual task scheduling platform, so that the supervision efficiency and accuracy are improved; through a supervision result obtained through data analysis, the collected first supervision data is deeply analyzed, and comprehensiveness and meticulousness of supervision are ensured; and when abnormal data is found in the data analysis result, abnormal alarm information is generated in time, and the information is sent to the server according to the preset alarm mode, so that the real-time performance of supervision is improved, the quick response capability to potential risks is enhanced, and the reliability and safety of data supervision are integrally improved.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to a data supervision method, device, equipment and medium based on a task scheduling platform. Background Art

[0002] With the continuous development of the medical industry, the demand for clinic supervision is increasing, and the user population of clinics is also expanding. In order to improve the overall product experience, meet the compliance of Internet medical security management, improve the traceability of the medical system life cycle, facilitate pre-, mid- and post-supervision, monitor and control public health events, while also analyzing and making decisions on data, providing medical information systems, providing management and operation services for private medical institutions, and improving the overall efficiency and quality of medical services.

[0003] Traditional clinic supervision methods often rely on manual operations, which have problems such as low efficiency and poor accuracy. In order to improve supervision efficiency and accuracy, an automated and intelligent clinic supervision platform system is urgently needed.

[0004] In the initial version, the main body of the regulatory system is composed of private medical institutions (outpatient clinics, clinics and medical rooms) service systems, health supervision systems and basic support platforms. The private medical institutions service system provides technical support services in clinical diagnosis and treatment, operation management and other aspects for private medical institutions, and reserves the ability to connect with the national health information platform to realize regular data reporting to the provincial and municipal medical institutions practice supervision platform; at the same time, it reserves the ability to connect with drug supervision, medical insurance and other systems. However, with the gradual increase in medical institutions, various HIS manufacturers have also emerged. The connection between manufacturer systems has become more and more cumbersome, the system connection is bloated, the data reported by the manufacturer system is diversified, and there is a risk of non-compliance, resulting in underreporting of related events in the supervision of institutions, personnel, outpatient cases, drugs, prescriptions, and qualifications. The alarm system cannot identify and prompt the manufacturer's data in real time, resulting in untimely supervision and increased regulatory safety and compliance risks.

[0005] Therefore, how to improve the reliability of data supervision has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a data supervision method, device, equipment and medium based on a task scheduling platform, aiming to improve the reliability of data supervision.

[0007] In a first aspect, the present application provides a data supervision method based on a task scheduling platform, the data supervision method based on the task scheduling platform comprising the following steps:

[0008] Create and schedule at least one data supervision task based on the preconfigured task scheduling system of the visual task scheduling platform;

[0009] Based on the data supervision task, performing data analysis on first supervision data collected from at least one data source to obtain a data analysis result;

[0010] When there is abnormal data in the data analysis result, abnormal alarm information is generated, and the abnormal alarm information is sent to the server based on a preset alarm method.

[0011] In a second aspect, the present application further provides a data supervision device based on a task scheduling platform, the data supervision device based on the task scheduling platform comprising:

[0012] A task scheduling module, used to create and schedule at least one data supervision task based on a task scheduling system preconfigured by a visual task scheduling platform;

[0013] A data supervision module, configured to perform data analysis on first supervision data collected from at least one data source based on the data supervision task to obtain a data analysis result;

[0014] The abnormal alarm module is used to generate abnormal alarm information when abnormal data exists in the data analysis result, and send the abnormal alarm information to the server based on a preset alarm method.

[0015] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the data supervision method based on the task scheduling platform as described above are implemented.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the data supervision method based on the task scheduling platform as described above are implemented.

[0017] The present application provides a data supervision method, device, computer equipment and storage medium based on a task scheduling platform. The method of the present application includes creating and scheduling at least one data supervision task based on a task scheduling system pre-configured by a visual task scheduling platform; based on the data supervision task, performing data analysis on the first supervision data collected from at least one data source to obtain data analysis results; when there is abnormal data in the data analysis results, generating abnormal alarm information, and sending the abnormal alarm information to the server based on a preset alarm method. Through the above-mentioned method, the present application realizes the automatic creation and scheduling of data supervision tasks through the task scheduling system pre-configured by the visual task scheduling platform, thereby improving the supervision efficiency and accuracy; through the supervision results obtained by data analysis, the first supervision data collected is deeply analyzed to ensure the comprehensiveness and meticulousness of supervision; when abnormal data is found in the data analysis results, abnormal alarm information is generated in time, and the information is sent to the server according to the preset alarm method, which not only improves the real-time nature of supervision, but also enhances the ability to respond quickly to potential risks, thereby improving the reliability and security of data supervision as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a first embodiment of a data supervision method based on a task scheduling platform provided by the present application;

[0020] Figure 2 It is a structural diagram of a first embodiment of a data supervision device based on a task scheduling platform provided by the present application;

[0021] Figure 3 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a data supervision method based on a task scheduling platform provided in this application.

[0027] like Figure 1 As shown, the data supervision method based on the task scheduling platform includes steps S101 to S103.

[0028] S101. Create and schedule at least one data supervision task based on a task scheduling system preconfigured on a visual task scheduling platform;

[0029] In one embodiment, the visual task scheduling platform can be a task scheduling tool such as Azkaban, oozie, and DolphinScheduler. The present application embodiment takes DolphinScheduler as an example for description.

[0030] Specifically, DolphinScheduler is a distributed, scalable, visual DAG workflow scheduling platform that is dedicated to solving the complex dependencies in data processing flows and making the scheduling system usable out of the box in data processing flows.

[0031] In a specific embodiment, the workflow of data supervision tasks can be defined in the UI interface of DolphinScheduler, including various links such as data collection, processing, analysis and alarm. Each link is a node in the workflow and can be different types of tasks, such as Shell, Python, SQL, etc.

[0032] For example, in the workflow of DolphinScheduler, the dependency between various tasks is clarified. For example, the data processing task can be executed only after the data collection task is completed.

[0033] Exemplarily, necessary input and output parameters may be configured for each task node. For example, a data collection task may require parameters such as a source data address and a target data storage location.

[0034] For example, the execution time and frequency of the task can be configured according to the needs of the clinic's supervision tasks. For example, you can set a scheduled task (daily, hourly, real-time, etc.), or you can use the system's built-in parameters, such as setting the date of the day before the scheduled time, to facilitate time-based scheduling.

[0035] For example, DolphinScheduler can provide a timing and event triggering mechanism, and can start tasks according to a preset schedule or a specific event. For example, it can be set to automatically execute data supervision tasks at dawn every day.

[0036] For example, after the task is scheduled, DolphinScheduler can provide real-time monitoring function, which can view the execution status and progress of the task. At the same time, the system will record detailed task logs to facilitate problem troubleshooting and performance optimization.

[0037] For example, DolphinScheduler can automatically retry failed tasks to ensure the reliability of tasks. Retry strategies such as the number of retries, retry interval, etc. can be configured according to actual needs.

[0038] For example, when an exception occurs during task execution, DolphinScheduler can send an alarm notification via email, text message, etc. to ensure that the problem can be discovered and handled in a timely manner.

[0039] In a specific embodiment, through the task scheduling and management of DolphinScheduler, the full process of clinic supervision tasks is automated, human intervention is reduced, and work efficiency is improved. Through data analysis and alarm functions, intelligent identification and processing of abnormal data and illegal behaviors are realized, and the timeliness and accuracy of clinic supervision are improved. It supports various types of clinic supervision tasks, and users can freely configure and adjust according to actual needs to adapt to different supervision scenarios. The system architecture is modularly designed, supports function expansion and upgrade, and can be flexibly adjusted as supervision needs change.

[0040] For example, the data collection task is scheduled regularly, for example, at 6 am every day, the clinic management system automatically collects the day's medical records and stores them in the data warehouse (Postgresql, Mysql). The task retry strategy is configured. If the collection task fails, it will be retried every 10 minutes, with a maximum of 3 retries.

[0041] For example, event-driven data processing tasks automatically trigger data processing tasks when new medical record data arrives to clean and convert data. Task dependencies are configured through the API interface to ensure that data processing tasks are executed after data collection tasks are completed.

[0042] For example, load-balanced data analysis tasks dynamically adjust the execution frequency of data analysis tasks according to system load conditions to avoid overloading system resources. Configure task priorities to ensure that emergency alarm tasks are executed first and abnormal situations in the clinic are discovered and handled in a timely manner.

[0043] For example, real-time monitoring and alarm notification, real-time monitoring of the execution status of data collection, processing, and analysis tasks, and viewing detailed execution records through logs. Configure alarm notification policies, and automatically send alarm emails to administrators when task execution fails or abnormal data is found.

[0044] Furthermore, based on data supervision requirements, at least one of the data supervision tasks is created, wherein the data supervision tasks include data collection tasks, data processing tasks, data analysis tasks, and abnormal alarm tasks; task execution attributes of each of the data supervision tasks are configured, wherein the task execution attributes include input data, output data, dependencies, execution time, and execution frequency of each of the data supervision tasks; based on the task execution attributes, each of the data supervision tasks is scheduled to make each of the data supervision tasks effective.

[0045] In one embodiment, the task scheduling function of DolphinScheduler is used to automatically manage and schedule clinic supervision tasks, define various types of clinic supervision tasks such as data collection, processing, analysis and alarm, specify the input and output and dependencies of the tasks, configure the execution time and frequency of the tasks (such as daily, hourly, real-time), and use the timing and event triggering mechanism of DolphinScheduler to implement task scheduling, monitor task execution in real time, record task logs and status, automatically retry failed tasks, and ensure that tasks are completed on time.

[0046] Specifically, use DolphinScheduler's DAG graphical interface to define the workflow of the above tasks, set the dependencies between tasks, ensure the correct flow of data, that is, the process from data collection to analysis to alarm is continuous, configure the timing trigger of each task, and set the specific execution time and frequency according to business needs. Enable DolphinScheduler's monitoring function to track the execution status of tasks in real time to ensure that tasks are executed as planned, and configure the abnormal alarm mechanism to automatically trigger an alarm notification when a task fails or an abnormality occurs in the data analysis results. In this way, the automated execution of data supervision tasks can be ensured, the efficiency and accuracy of data processing can be improved, and relevant personnel can be notified in time to handle abnormalities when they occur.

[0047] Exemplarily, a data collection task may include defining a data source (such as a database table, API interface, or other data storage system), specifying the storage location of the collected data (such as HDFS, a local file system, or a database), defining dependencies, setting the task execution time (the execution time can be set according to the frequency of data generation, such as a specific time point every hour or every day), and setting the execution frequency (for example, it can be set to scheduled execution, such as once an hour or once a day).

[0048] Similarly, a data processing task may include input data (output from the data collection task, i.e., the collected raw data), output data (processed data, which may include the results of data cleaning, transformation, and aggregation), dependencies (depending on the data collection task and can only be executed after data collection is completed), execution time (executed immediately after the data collection task is completed), and execution frequency.

[0049] For example, in a data analysis task, the input data can be the output from a data processing task, that is, the data that has been cleaned and converted, and the output data can be the analysis results, such as statistical reports, trend charts, etc. The data analysis task depends on the data processing task to ensure that the latest processed data is analyzed. The data analysis task can be executed immediately after the data processing is completed, or within a specific time window. According to business needs, the data analysis task can be executed periodically or regularly, such as once a day, week, or month.

[0050] For example, in an abnormal alarm task, an alarm message is generated according to the analysis result output by the data analysis task, and then sent to relevant personnel via email, SMS or application notification.

[0051] S102: Based on the data supervision task, perform data analysis on first supervision data collected from at least one data source to obtain a data analysis result;

[0052] In one embodiment, the first regulatory data can be collected from multiple data sources, which may include medical institutions, drug supply chains, patient information bases, etc. The collected data needs to be strictly cleaned, converted, and integrated. Key data including compliance indicator data, service quality indicators, patient safety indicators, operational efficiency indicators, and patient feedback indicators are cleaned to form a standardized data format. These indicators are the key to evaluating the quality of medical services, and can help regulatory agencies, clinic managers, and other relevant parties to comprehensively evaluate the operating conditions of clinics and ensure the safety, efficiency, and quality of medical services.

[0053] Furthermore, based on a preset data transmission method, the first regulatory data is collected from at least one data source; based on a data processing algorithm, the first regulatory data is processed to obtain standard regulatory data; based on a data analysis algorithm, the standard regulatory data is analyzed to generate data analysis results.

[0054] In one embodiment, based on a preset data transmission method, data is collected from data sources such as a medical staff database, a medical institution database, an electronic medical record database, etc. A data processing algorithm is used to clean, convert and integrate the collected raw data to form a standardized data format. A data analysis algorithm is used to conduct an in-depth analysis of the processed standard regulatory data to generate data analysis results, which may include service quality indicators, patient safety indicators, operational efficiency indicators, and patient feedback indicators, etc., to provide decision support for regulatory agencies.

[0055] Specifically, the accessed data is cleaned, converted and integrated to form a standardized data format to provide support for subsequent analysis. The basic data includes relevant basic data such as the medical staff database, medical institution database, electronic medical record database, drug information database, facial information database, qualification document database, monitoring indicator database, role authority database, etc., to support the formation of an interconnected medical and health business collaboration network, to achieve the sharing and exchange of diagnosis and treatment information between institutions, to complete the standardization of data exchange and data collection behaviors, and to clean up the compliance indicator data, service quality indicators, patient safety indicators, operational efficiency indicators, patient feedback indicators and other corresponding data through the basic data of the institution. Through these indicators, regulatory agencies, clinic managers and other relevant parties can comprehensively evaluate the operation of the clinic to ensure that the clinic provides safe, efficient and high-quality medical services while complying with all relevant laws and regulations.

[0056] By combining data transmission technology, data processing algorithms and data analysis algorithms, this embodiment enables regulatory agencies to more effectively collect, process and analyze data from data sources, thereby improving the quality and safety of medical services.

[0057] Furthermore, when the data source is a homogeneous data source, the first regulatory data is collected from the homogeneous data source based on a preset data transmission channel and an IP access whitelist; when the data source is a heterogeneous data source, the first regulatory data collected from the heterogeneous data source is encrypted and verified based on a preset data encryption method, and when the first regulatory data passes the verification, the first regulatory data is received.

[0058] In one embodiment, data is collected from multiple data sources (such as clinic management systems, electronic medical record systems, drug management systems, etc.) to ensure the real-time and accuracy of the data, provide platform data access to the Gateway system, and while connecting to the cloud clinic, also meet the data upload needs of heterogeneous clinic institutions.

[0059] Among them, the data access Gateway system can connect to different data sources, including homogeneous and heterogeneous clinic institution data.

[0060] Specifically, for homogeneous data sources such as cloud clinics, internal channels are used to report and upload data, and the IP access whitelist method is used to report clinic institution-related data to the supervision system. For heterogeneous data sources such as heterogeneous clinics, institutional access rights are added, and system access signature keys are issued. When heterogeneous clinics report supervision data, the corresponding data needs to be encrypted and uploaded. The supervision gateway system uses client keys and server keys for encryption authentication verification to ensure secure and effective access. At the same time, multi-threading and asynchronous programming technologies are used to ensure the efficiency of data collection, and the integrity of data collection access is detected, and abnormal reported data is promptly warned and processed.

[0061] Furthermore, when a business service request initiated by a client is received, the identity information and service requirements of the requester are obtained based on the business service request; the business service authority of the client is determined based on the identity information of the requester; when the business service authority verification is passed, the corresponding service task is executed based on the service requirement, and the business service result is output.

[0062] Generally speaking, regulatory system services include data collection, data conversion, data encapsulation, code analysis, data import, assembly services, orchestration services, business rule services, standardization services, data quality services, replication services, data warehouse services, key-value management services, data access services and other related services.

[0063] In one embodiment, the overall application function can be displayed through the WEB through data conversion mapping service. Different permissions are set for business personnel with different functions. By logging in, operations such as browsing and querying applications and resources of related systems can be achieved.

[0064] In one embodiment, when a client initiates a business service request to a server via a network protocol (such as HTTP, HTTPS), the server first needs to receive and parse the request, wherein the request content may include but is not limited to user ID, request type, parameters of the required service, etc.

[0065] In one embodiment, after receiving the business service request, the server verifies the identity information of the requester through an authentication mechanism, such as username and password, digital certificate, OAuth tokens, biometric information, etc. After successful identity authentication, the server will obtain detailed information of the requester, such as user role, account status, etc., and analyze the service requirements to determine the specific service content. For example, if the request is about a financial transaction, the system needs to understand the specific type and amount of the transaction.

[0066] In one embodiment, based on the obtained identity information, the client's business service permissions are determined according to access control lists (ACLs), role-based access control (RBAC) or other permission management strategies. Check whether the user has the right to perform the requested operation. If the user does not have sufficient permissions, the server will reject the request and return a corresponding error message. If the permission verification is passed, the server will perform the corresponding service task.

[0067] In one embodiment, after the service task is executed, the server outputs the business service results, including confirmation messages, operation results, query data, etc., and formats the business service results into a format understandable to the client, such as JSON, XML, etc., and sends them back to the client through the target communication bridge.

[0068] Furthermore, based on preset alarm rules, anomaly detection is performed on the data analysis result; when the data analysis result triggers the preset alarm rule, it is determined that abnormal data exists in the data analysis result.

[0069] In one embodiment, according to the problem type and data characteristics, a suitable anomaly detection algorithm is selected and trained. The anomaly detection algorithm may include clustering-based anomaly detection methods (such as K-means, DBSCAN), statistical-based methods (such as Z-score, IQR method), depth-based methods (such as Autoencoder), classification model-based methods (such as One class SVM), etc.

[0070] In one embodiment, the abnormal data is analyzed and marked according to the output result of the abnormal detection algorithm. According to the result of the abnormal detection, the abnormal data is processed, including deletion, correction, replacement, etc.

[0071] Exemplarily, anomaly detection can be accomplished by a variety of methods, including statistical methods, machine learning methods, deep learning methods, etc. For example, the Z-score method can be used to identify univariate outliers. The difference between the data point and the mean of the data set is calculated and divided by the standard deviation. The Z-score obtained is used to determine whether the data point is abnormal data. For multivariate outliers, the isolation forest algorithm can be used. This is an unsupervised machine learning algorithm based on random forests. It creates a decision tree by randomly selecting features and segmentation rules to detect abnormal data points.

[0072] S103: When there is abnormal data in the data analysis result, abnormal alarm information is generated, and the abnormal alarm information is sent to the server based on a preset alarm method.

[0073] In one embodiment, based on a preset data conversion method, the data analysis result is converted into a preset data format to display the data analysis result; wherein the preset data format includes a report format and a chart format.

[0074] In one embodiment, the data processed by the application system can be deeply analyzed to generate various statistical reports and charts to provide intuitive display of supervision results. These reports can be in text form or contain charts and graphs to intuitively display the data analysis results.

[0075] Specifically, according to preset data conversion rules, such as data normalization, discretization, logarithmic transformation, etc., data visualization tools such as ChartCool or LiveGap are used to convert data into data formats such as reports and charts, including but not limited to bar charts, line graphs, pie charts, scatter plots, etc., so that users can intuitively understand data distribution and abnormal patterns.

[0076] In one embodiment, alarm rules can be set (such as abnormal drug dosage, abnormal patient visit frequency, risk prescriptions, fever prescriptions, electronic medical records, antibacterial drugs, intravenous infusions). When the data analysis results meet the alarm conditions, the system sends an alarm notification via email, text message or instant messaging tool, and records the alarm log.

[0077] In one embodiment, the data analysis results can be detected by an anomaly detection algorithm. When abnormal data or illegal behavior is detected, an alarm is automatically triggered to notify relevant personnel to handle it, and regular institutional risk factor assessments are conducted, high-risk institutions are reported, and security supervision and other measures are implemented.

[0078] In one embodiment, when the data analysis results trigger a preset alarm rule, abnormal alarm information is automatically generated, wherein the abnormal alarm information may include the type, time, location, possible impact and recommended response measures of the abnormality.

[0079] In one embodiment, the preset alarm method may include email, text message, mobile application push, system notification, etc. The preset alarm method may be determined according to a preset alarm strategy to ensure that abnormal alarm information can be conveyed to relevant personnel in a timely manner.

[0080] This embodiment provides a data supervision method based on a task scheduling platform. The method of this application realizes the automatic creation and scheduling of data supervision tasks through the task scheduling system pre-configured by the visual task scheduling platform, thereby improving the supervision efficiency and accuracy; the supervision results obtained through data analysis are used to conduct in-depth analysis of the collected first supervision data to ensure the comprehensiveness and meticulousness of the supervision; when abnormal data is found in the data analysis results, abnormal alarm information is generated in a timely manner, and the information is sent to the server according to the preset alarm method, which not only improves the real-time nature of supervision, but also enhances the ability to quickly respond to potential risks, thereby improving the reliability and security of data supervision as a whole.

[0081] See also Figure 2 , Figure 2 It is a structural schematic diagram of a first embodiment of a data supervision device based on a task scheduling platform provided in the present application. The data supervision device based on the task scheduling platform is used to execute the aforementioned data supervision method based on the task scheduling platform.

[0082] like Figure 2 As shown, the data supervision device 200 based on the task scheduling platform includes: a task scheduling module 201 , a data supervision module 202 and an abnormality alarm module 203 .

[0083] The task scheduling module 201 is used to create and schedule at least one data supervision task based on the task scheduling system preconfigured by the visual task scheduling platform;

[0084] A data supervision module 202, configured to perform data analysis on first supervision data collected from at least one data source based on the data supervision task to obtain a data analysis result;

[0085] The abnormal alarm module 203 is used to generate abnormal alarm information when abnormal data exists in the data analysis result, and send the abnormal alarm information to the server based on a preset alarm method.

[0086] In one embodiment, the data supervision module 202 includes:

[0087] A data collection unit, configured to collect the first supervision data from at least one data source based on a preset data transmission method;

[0088] A data processing unit, configured to process the first regulatory data based on a data processing algorithm to obtain standard regulatory data;

[0089] The data analysis unit is used to perform data analysis on the standard regulatory data based on a data analysis algorithm to generate data analysis results.

[0090] In one embodiment, the data acquisition unit includes:

[0091] A homogeneous data collection subunit, configured to collect the first supervision data from the homogeneous data source based on a preset data transmission channel and an IP access whitelist when the data source is a homogeneous data source;

[0092] The heterogeneous data collection subunit is used to encrypt and verify the first supervision data collected by the heterogeneous data source based on a preset data encryption method when the data source is a heterogeneous data source, and receive the first supervision data when the first supervision data is verified.

[0093] In one embodiment, the data supervision device 200 based on the task scheduling platform further includes a task execution module, including:

[0094] A request parsing unit, configured to obtain the identity information and service requirements of the requesting party based on the business service request when receiving the business service request initiated by the client;

[0095] An authority determination unit, configured to determine the business service authority of the client based on the identity information of the requester;

[0096] The task execution unit is used to execute the corresponding service task and output the business service result based on the service demand when the business service authority verification is passed.

[0097] In one embodiment, the data supervision device 200 based on the task scheduling platform further includes an anomaly detection module, including:

[0098] An anomaly detection unit, used to perform anomaly detection on the data analysis result based on preset alarm rules;

[0099] The abnormal data locating unit is used to determine whether abnormal data exists in the data analysis result when the data analysis result triggers the preset alarm rule.

[0100] In one embodiment, the data supervision device 200 based on the task scheduling platform further includes a format conversion module, including:

[0101] A data format conversion unit, used to convert the data analysis result into a preset data format based on a preset data conversion method, so as to display the data analysis result;

[0102] Wherein, the preset data formats include report formats and chart formats.

[0103] In one embodiment, the task scheduling module 201 includes:

[0104] A task creation unit, configured to create at least one of the data supervision tasks based on data supervision requirements, wherein the data supervision tasks include data collection tasks, data processing tasks, data analysis tasks, and abnormal alarm tasks;

[0105] An attribute configuration unit, used to configure the task execution attributes of each of the data supervision tasks, wherein the task execution attributes include input data, output data, dependency, execution time and execution frequency of each of the data supervision tasks;

[0106] The task scheduling unit is used to schedule each of the data supervision tasks based on the task execution attributes so that each of the data supervision tasks takes effect.

[0107] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned data supervision method embodiment based on the task scheduling platform, and will not be repeated here.

[0108] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 3 Runs on the computer device shown.

[0109] See also Figure 3 , Figure 3 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0110] See also Figure 3 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0111] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any data supervision method based on the task scheduling platform.

[0112] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0113] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any data supervision method based on the task scheduling platform.

[0114] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0117] Create and schedule at least one data supervision task based on the preconfigured task scheduling system of the visual task scheduling platform;

[0118] Based on the data supervision task, performing data analysis on first supervision data collected from at least one data source to obtain a data analysis result;

[0119] When there is abnormal data in the data analysis result, abnormal alarm information is generated, and the abnormal alarm information is sent to the server based on a preset alarm method.

[0120] In one embodiment, when the processor performs data analysis on the first supervision data collected from at least one data source based on the data supervision task to obtain the data analysis result, it is configured to implement:

[0121] Based on a preset data transmission method, collecting the first supervision data from at least one data source;

[0122] Based on a data processing algorithm, the first regulatory data is processed to obtain standard regulatory data;

[0123] Based on the data analysis algorithm, data analysis is performed on the standard regulatory data to generate data analysis results.

[0124] In one embodiment, when the processor collects the first supervision data from at least one data source based on the preset data transmission mode, it is used to implement:

[0125] When the data source is a homogeneous data source, based on a preset data transmission channel and an IP access whitelist, collecting the first supervision data from the homogeneous data source;

[0126] When the data source is a heterogeneous data source, the first supervision data collected by the heterogeneous data source is encrypted and verified based on a preset data encryption method, and when the first supervision data passes the verification, the first supervision data is received.

[0127] In one embodiment, after performing data analysis on the first supervision data collected from at least one data source based on the data supervision task and obtaining the data analysis result, the processor is further configured to implement:

[0128] Upon receiving a business service request initiated by a client, obtaining the identity information and service requirements of the requesting party based on the business service request;

[0129] Determining the business service authority of the client based on the identity information of the requester;

[0130] When the business service authority verification is passed, the corresponding service task is executed based on the service demand and the business service result is output.

[0131] In one embodiment, the processor, before generating abnormal alarm information when abnormal data exists in the data analysis result, is further used to implement:

[0132] Based on preset alarm rules, perform anomaly detection on the data analysis results;

[0133] When the data analysis result triggers the preset alarm rule, it is determined that abnormal data exists in the data analysis result.

[0134] In one embodiment, after performing data analysis on the first supervision data collected from at least one data source based on the data supervision task and obtaining the data analysis result, the processor is further configured to implement:

[0135] Based on a preset data conversion method, convert the data analysis result into a preset data format to display the data analysis result;

[0136] Wherein, the preset data formats include report formats and chart formats.

[0137] In one embodiment, the processor, when implementing the task scheduling system preconfigured based on the visual task scheduling platform and creating and scheduling at least one data supervision task, is used to implement:

[0138] Based on the data supervision requirements, at least one of the data supervision tasks is created, wherein the data supervision tasks include data collection tasks, data processing tasks, data analysis tasks, and abnormal alarm tasks;

[0139] Configuring task execution attributes of each of the data supervision tasks, wherein the task execution attributes include input data, output data, dependency, execution time, and execution frequency of each of the data supervision tasks;

[0140] Based on the task execution attributes, each of the data supervision tasks is scheduled to make each of the data supervision tasks effective.

[0141] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any data supervision method based on a task scheduling platform provided in an embodiment of the present application.

[0142] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data supervision method based on a task scheduling platform, characterized in that: The method comprises: Create and schedule at least one data supervision task based on the preconfigured task scheduling system of the visual task scheduling platform; Based on the data supervision task, performing data analysis on first supervision data collected from at least one data source to obtain a data analysis result; When there is abnormal data in the data analysis result, abnormal alarm information is generated, and the abnormal alarm information is sent to the server based on a preset alarm method.

2. The data supervision method based on the task scheduling platform according to claim 1 is characterized in that: The step of performing data analysis on first supervision data collected from at least one data source based on the data supervision task to obtain data analysis results includes: Based on a preset data transmission method, collecting the first supervision data from at least one data source; Based on a data processing algorithm, the first regulatory data is processed to obtain standard regulatory data; Based on the data analysis algorithm, data analysis is performed on the standard regulatory data to generate data analysis results.

3. The data supervision method based on the task scheduling platform according to claim 2 is characterized in that: The collecting the first supervision data from at least one data source based on a preset data transmission method includes: When the data source is a homogeneous data source, based on a preset data transmission channel and an IP access whitelist, collecting the first supervision data from the homogeneous data source; When the data source is a heterogeneous data source, the first supervision data collected by the heterogeneous data source is encrypted and verified based on a preset data encryption method, and when the first supervision data passes the verification, the first supervision data is received.

4. The data supervision method based on the task scheduling platform according to claim 1 is characterized in that: After performing data analysis on the first supervision data collected from at least one data source based on the data supervision task and obtaining the data analysis result, the method further includes: Upon receiving a business service request initiated by a client, obtaining the identity information and service requirements of the requesting party based on the business service request; Determining the business service authority of the client based on the identity information of the requester; When the business service authority verification is passed, the corresponding service task is executed based on the service demand and the business service result is output.

5. The data supervision method based on the task scheduling platform according to claim 1 is characterized in that: When there is abnormal data in the data analysis result, before generating abnormal alarm information, the method further includes: Based on preset alarm rules, perform anomaly detection on the data analysis results; When the data analysis result triggers the preset alarm rule, it is determined that abnormal data exists in the data analysis result.

6. The data supervision method based on the task scheduling platform according to claim 1 is characterized in that: After performing data analysis on the first supervision data collected from at least one data source based on the data supervision task and obtaining the data analysis result, the method further includes: Based on a preset data conversion method, convert the data analysis result into a preset data format to display the data analysis result; Wherein, the preset data formats include report formats and chart formats.

7. The data supervision method based on the task scheduling platform according to claim 1 is characterized in that: The task scheduling system pre-configured based on the visual task scheduling platform creates and schedules at least one data supervision task, including: Based on the data supervision requirements, at least one of the data supervision tasks is created, wherein the data supervision tasks include data collection tasks, data processing tasks, data analysis tasks, and abnormal alarm tasks; Configuring task execution attributes of each of the data supervision tasks, wherein the task execution attributes include input data, output data, dependency, execution time, and execution frequency of each of the data supervision tasks; Based on the task execution attributes, each of the data supervision tasks is scheduled to make each of the data supervision tasks effective.

8. A data monitoring device based on a task scheduling platform, characterized in that: The data supervision device based on the task scheduling platform includes: A task scheduling module, used to create and schedule at least one data supervision task based on a task scheduling system preconfigured by a visual task scheduling platform; A data supervision module, configured to perform data analysis on first supervision data collected from at least one data source based on the data supervision task to obtain a data analysis result; The abnormal alarm module is used to generate abnormal alarm information when abnormal data exists in the data analysis result, and send the abnormal alarm information to the server based on a preset alarm method.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the data supervision method based on the task scheduling platform as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data supervision method based on the task scheduling platform as described in any one of claims 1 to 7 are implemented.

Images