Data monitoring method, task monitoring method, system and computer readable medium

Abstract

The present disclosure provides a data monitoring method, comprising: determining a data arrival detection strategy configured by a to-be-monitored data source; detecting whether data sent by the to-be-monitored data source has reached a predetermined level based on the data arrival detection strategy; when it is detected that the data sent by the to-be-monitored data source has not reached the predetermined level, continuing to perform the determination of the data arrival detection strategy configured by the to-be-monitored data source; and when it is detected that the data sent by the to-be-monitored data source has reached the predetermined level, marking the to-be-monitored data source as arrived. The present disclosure also provides a task monitoring method, a data monitoring module, a task monitoring system and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data monitoring method, a task monitoring method, a data monitoring module, a task monitoring system, and a computer-readable medium. Background Technology

[0002] In addition to possessing the general characteristics of big data (large data volume, data diversity, etc.), telecommunications operators' big data also has characteristics related to their business, such as data from multiple vendors, uncertainty in data transmission time, and uncertainty in data integrity.

[0003] If tasks are executed on a traditional schedule, the following problems may occur: 1) When the task starts, the data received from the data source may be incomplete, resulting in incomplete data used during task execution and incorrect processing results; 2) When the task starts, the complete data provided by the data source may have already arrived, leading to poor real-time data processing. The problem causing these problems is that existing technologies lack a mechanism for real-time monitoring of the data sent by the data source. Summary of the Invention

[0004] This disclosure aims to address at least one of the technical problems existing in the prior art, and proposes a data monitoring method, a task monitoring method, a data monitoring module, a task monitoring system, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a data monitoring method, including:

[0006] Determine the data arrival detection strategy configured for the data source to be monitored;

[0007] Based on the data arrival detection strategy, it is detected whether the data sent by the data source to be monitored has reached the predetermined level.

[0008] When it is detected that the data sent by the data source to be monitored has not reached the predetermined level, the data arrival detection strategy configured for the data source to be monitored will continue to be executed.

[0009] When it is detected that the data sent by the data source to be monitored has reached the predetermined level, the data source to be monitored is marked as having reached the predetermined level.

[0010] Secondly, embodiments of this disclosure provide a task monitoring method, wherein the task depends on data sent from multiple target data sources;

[0011] For each target data source, the target data source is taken as the data source to be monitored, and the data monitoring method provided in the above embodiments is used to monitor the data sent by the target data source;

[0012] The task can be started based on the marking results of each of the target data sources, wherein the marking results include: marking arrived or not marked.

[0013] Thirdly, this disclosure also provides a data monitoring module, which includes:

[0014] One or more processors;

[0015] A memory on which one or more programs are stored;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data monitoring method provided in the foregoing embodiments.

[0017] Fourthly, this disclosure also provides a task monitoring system, which includes: a data monitoring module and a detection module as provided in the foregoing embodiments;

[0018] The detection module is used to detect whether the task can be started based on the marking results of each target data source.

[0019] Fifthly, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps in the data monitoring method and the task monitoring method provided in the foregoing embodiments.

[0020] This disclosure provides a data monitoring method, a task monitoring method, a system, and a computer-readable medium. Compared with existing technologies, the disclosed technical solution can achieve real-time monitoring of the status of data sent by a data source, thereby ensuring the integrity and reliability of the received data. Furthermore, the technical solution of this disclosure can immediately begin executing the corresponding task when the monitored data obtained by the server is sufficient to ensure the effective execution of the task, effectively guaranteeing the real-time performance of data processing. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of the technical solution disclosed herein;

[0022] Figure 2 A flowchart of a data monitoring method provided in this disclosure embodiment;

[0023] Figure 3 This is a flowchart illustrating a specific implementation of step S1 in an embodiment of this disclosure;

[0024] Figure 4 A flowchart of another data monitoring method provided in this disclosure embodiment;

[0025] Figure 5 This is a flowchart illustrating a specific implementation of step S5 in an embodiment of this disclosure;

[0026] Figure 6 A flowchart illustrating yet another data monitoring method provided in this disclosure embodiment;

[0027] Figure 7 A flowchart of a task monitoring method provided in this embodiment of the disclosure;

[0028] Figure 8 This is a flowchart illustrating a specific implementation of step S12 in an embodiment of this disclosure;

[0029] Figure 9 This is a flowchart illustrating another specific implementation of step S12 in an embodiment of this disclosure. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions of this disclosure, the data monitoring method, task monitoring method, data monitoring module, task monitoring system, and computer-readable medium provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of a feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0033] It will be understood that while the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. Therefore, without departing from the teachings of this disclosure, the first element, first component, or first module discussed below may be referred to as a second element, second component, or second module.

[0034] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario of the technical solution disclosed herein, such as... Figure 1 As shown in the illustration, as one application scenario of this disclosure, multiple data sources report data to a server, and the server executes corresponding tasks based on the data reported by the data sources. The technical solution of this disclosure can achieve real-time monitoring of the status of the data sent by the data sources, thereby ensuring the integrity and reliability of the received data. Furthermore, the technical solution of this disclosure can also immediately begin executing the corresponding task when the monitored data obtained by the server is sufficient to ensure the effective execution of the task, effectively guaranteeing the real-time performance of data processing.

[0036] Figure 2 A flowchart of a data monitoring method provided in this disclosure embodiment is shown below. Figure 2 As shown, the data monitoring method includes:

[0037] Step S1: Determine the data arrival detection strategy configured for the data source to be monitored.

[0038] In step S1, different data arrival detection strategies can be pre-configured for different data sources to be monitored. The data arrival detection strategy can be used to detect whether the data reported by the corresponding data source has reached a predetermined level.

[0039] It should be noted that the "predetermined level state" in this disclosure is a state that is predefined by the user and can characterize the progress of the data reported by the data source. For example, if the progress corresponding to the data source reporting complete data is 100%, then the predetermined level state can refer to a certain level of the data reporting progress of the data source between 0% and 100%, and the specific value is not limited.

[0040] In some embodiments, "predetermined level state" can characterize the data reported by the data source as being in a state of full arrival or close to a state of full arrival.

[0041] In this embodiment of the disclosure, whether the data sent by the data source has reached the "predetermined level state" can be detected by a data arrival detection strategy pre-configured for the data source.

[0042] Figure 3 This is a flowchart illustrating a specific implementation of step S1 in an embodiment of this disclosure, as follows: Figure 3As shown, in some embodiments, step S1 includes:

[0043] Step S101: Obtain the value of the attribute of the data source to be monitored as the data inspection type from the data table.

[0044] In this embodiment of the disclosure, the data table records the values ​​of each data source for each attribute feature. The attributes recorded in the data table include at least: data source name (or ID) and data check type (Checktype). Of course, attributes such as "data type" and "data storage location" (storage location on the server) can also be configured as needed. Other configurable attributes in the data table will not be listed here.

[0045] Step S202: Query the data corresponding to the value from the pre-set correspondence table to obtain the detection strategy.

[0046] The correspondence table records different data inspection types and their corresponding data arrival detection strategies.

[0047] By using a combination of data tables and corresponding relationship tables, unified management of data arrival detection strategies for different data sources can be achieved, and it is also convenient to adjust the data arrival detection strategies corresponding to the data sources.

[0048] In some embodiments, the correspondence table is shown in Table 1 below.

[0049] Table 1. Correspondence between Data Inspection Types and Data Arrival Detection Strategies

[0050]

[0051] Strategy A (“Check Stream File” strategy): Checktype is configured as “streaming”, a file stream-based detection strategy. Specifically, it checks whether the stream file corresponding to the data sent by the corresponding data source and its subsequent files exist; if the stream file and subsequent files exist, it is determined that the data sent by the corresponding data source has reached a predetermined level; otherwise, it is determined that the data sent by the corresponding data source has not reached the predetermined level.

[0052] Strategy B (“Check File Existence” strategy): Checktype is configured as “fileexist”, a file existence detection strategy. Specifically, it queries the data storage location of the corresponding data source from the data table, and then checks whether a file exists in the partition corresponding to the queried data storage location (e.g., province partition, time partition, etc.). If the file exists, it is detected that the data sent by the corresponding data source has reached the predetermined level; otherwise, it is detected that the data sent by the corresponding data source has not reached the predetermined level.

[0053] Strategy C (“Check File Count” strategy): Checktype is configured as “filenum”, a file count-based detection strategy. Specifically, it obtains the number of files generated by the data sent from the corresponding data source and determines whether the number of files meets a preset condition (e.g., the number of files is between 5 and 10). If the preset condition is met, it is detected that the data sent by the corresponding data source has reached a predetermined level; otherwise, it is detected that the data sent by the corresponding data source has not reached the predetermined level.

[0054] Strategy D (“Check Hive Partition” strategy): Checktype is configured as “hivepartition”, a detection strategy based on Hive partitions. Specifically, it determines whether the Hive partition for the data sent by the corresponding data source exists; if it exists, it detects that the data sent by the corresponding data source has reached the predetermined level; otherwise, it detects that the data sent by the corresponding data source has not reached the predetermined level.

[0055] Strategy E (“Check Filename” strategy): Checktype is configured as “filename”, a filename-based detection strategy. Specifically, it scans all files in the table partition directory. Since this directory is already divided by time (e.g., days), it only needs to check if a filename corresponding to the data source (e.g., the data source ID) exists in this directory. If it exists, it is detected that the data sent by the corresponding data source has reached the predetermined level; otherwise, it is detected that the data sent by the corresponding data source has not reached the predetermined level.

[0056] Strategy F (“Check File Content” strategy): Checktype is configured as “filecontent”, a file content-based detection strategy. Specifically, based on different groups in the filename, the latest file in the corresponding group of the relevant data source is found. The filename is then parsed to obtain the ID of the relevant data source. The time in the first record of the file content (the column containing the time field needs to be configured) is used as the data time t. Based on the obtained ID and time t, it can be inferred that the data sent by the relevant data source before time t has arrived at the server. It is then determined whether time t meets a predetermined condition. If the preset condition is met, it is detected that the data sent by the relevant data source has reached the predetermined level; otherwise, it is detected that the data sent by the relevant data source has not reached the predetermined level.

[0057] Strategy G (“Check File Directory” strategy): Checktype is configured as “filedirexist”, a file directory-based detection strategy. Specifically, it scans the Hadoop Distributed File System (HDFS) to determine if the partition corresponding to the data source and a partition at a certain time granularity after that partition exist. If both exist, it is detected that the data sent by the corresponding data source has reached the predetermined level; otherwise, it is detected that the data sent by the corresponding data source has not reached the predetermined level.

[0058] Strategy H ("Check Filename and File Content" strategy): Checktype is configured as "filecontentandname", a detection strategy based on filename and file content. Specifically, it scans all files in the temporary directory, groups them according to different IDs in the filenames, determines the group corresponding to the data source, finds the latest file in that group, and uses the time in the first record of that file's content (the column containing the time field needs to be configured) as the data time t. If the file content is empty, the timestamp recorded in the filename is used as the data time t. At this point, it can be inferred that the data sent by the corresponding data source before time t has arrived at the server. It then determines whether time t meets a predetermined condition. If the preset condition is met, it is detected that the data sent by the corresponding data source has reached the predetermined level; otherwise, it is detected that the data sent by the corresponding data source has not reached the predetermined level.

[0059] Strategy I ("Default" Strategy): Checktype is configured as "nocheck". No checks are performed on the data sent by the corresponding data source; the data sent by that data source is directly assumed to have reached the predetermined level.

[0060] It should be noted that the correspondence table shown in Table 1 above is for illustrative purposes only and does not limit the technical solution of this disclosure. In practical applications, this correspondence table supports user-defined design. Users can set the possible value range of the data check type and the corresponding data arrival detection strategy for each value as needed. The specific algorithm of the data arrival detection strategy is also preset by the user as needed. Of course, this disclosure also supports the combined use of data arrival detection strategies. In this case, it is only necessary to configure a corresponding data check type value for the new data arrival detection strategy obtained by combining at least two data arrival detection strategies.

[0061] Step S2: Based on the data arrival detection strategy, detect whether the data sent by the data source to be monitored has reached the predetermined level.

[0062] In step S2, based on the data arrival detection strategy determined in step S1, it is detected whether the data sent by the monitored data source has reached a predetermined level. If it is detected that the data sent by the monitored data source has not reached the predetermined level, step S1 is continued to achieve real-time monitoring. If it is detected that the data sent by the monitored data source has reached the predetermined level, step S3 is executed.

[0063] Step S3: Mark the arrival of the data source to be monitored.

[0064] The technical solution disclosed herein can perform real-time monitoring of the data sent by the data source to be monitored, indicating that the data has reached a predetermined level.

[0065] In some application scenarios, users need to monitor whether data sent by a certain data source has completely arrived at the server, i.e., the predetermined level is a complete arrival state. When detecting whether the data sent by the data source is in a complete arrival state based on the data arrival detection strategy determined in step S1, it is possible that the detection result in step S2 shows that the data sent by the data source is in a complete arrival state, but in reality, the data sent by the data source is still being transmitted. For example, when detecting based on the "check the number of files" strategy, a certain number of files already exist on the server, but the data corresponding to some of these files is still being transmitted (e.g., the complete file is 100MB, but the file size generated on the server is 90MB, and 10MB is still in transmission and has not yet arrived at the server). In this case, to ensure the complete arrival of the data, a delay time (e.g., 50s) can be pre-configured; when step S2 detects that the data sent by the data source to be monitored is in a complete arrival state, after this delay time, step S3 is executed to ensure that the data has truly arrived completely. The delay time can be configured based on prior experiments.

[0066] Figure 4 A flowchart of another data monitoring method provided in this disclosure embodiment is shown below. Figure 4 As shown, Figure 4 The data monitoring method shown includes not only the steps S1 to S3 mentioned above, but also steps S4 to S6. Only steps S4 to S6 will be described in detail below.

[0067] Step S4: Obtain the arrival time and amount of data sent by the data source to be monitored.

[0068] Step S5: Detect whether the data transmission process of the monitored data source is abnormal based on the arrival time and the amount of data received.

[0069] Figure 5 This is a flowchart illustrating a specific implementation of step S5 in an embodiment of this disclosure, as follows: Figure 5As shown, in some embodiments, step S5 includes:

[0070] Step S501: Based on the historical arrival time and historical arrival data volume of the data sent by the data source to be monitored, predict the first predetermined time range and the first predetermined data volume range.

[0071] In step S501, based on the historical arrival time and historical arrival data volume of the data sent by the data source to be monitored, a first predetermined time range and a first predetermined data volume range are predicted using a BP neural network prediction model.

[0072] It should be noted that the specific process of learning and predicting based on the BP neural network prediction model is a conventional technique in this field and will not be described in detail here.

[0073] Step S502: Determine whether the arrival time is within the first predetermined time range and whether the amount of arriving data is within the first predetermined data amount range.

[0074] Specifically, if the arrival time is within the first predetermined time range and the amount of data received is within the first predetermined data volume range, then the data transmission process of the monitored data source is detected as normal; otherwise, the data transmission process of the monitored data source is detected as abnormal (e.g., the monitored data source reported incorrect data, the data was tampered with by others during transmission, or there are problems with the data communication network).

[0075] In some embodiments, the “first predetermined time range” and “first predetermined data volume” can be preset by the user based on experience, in which case step S501 does not need to be executed.

[0076] It should be noted that step S5, which includes steps S501 and S502, is merely an optional implementation scheme in this disclosure and does not limit the technical solution of this disclosure. In the embodiments of this disclosure, using arrival time and the amount of arriving data, and employing other algorithms, to detect whether the data transmission process of the monitored data source is abnormal should also fall within the protection scope of this disclosure.

[0077] In step S502, when an abnormality is detected in the data transmission process of the data source to be monitored, step S6 is then executed.

[0078] Step S6: Cancel the arrival of the target data source to be monitored.

[0079] Through the above steps S4 to S6, the security and accuracy of the marked data can be ensured.

[0080] Figure 6 A flowchart of yet another data monitoring method provided in this disclosure embodiment, such as... Figure 6 As shown, Figure 6 The data monitoring method shown includes not only steps S1 to S6 as described above, but also steps S7 to S9. Only steps S7 to S9 will be described in detail below.

[0081] Step S7: Determine whether the arrival time is within the second predetermined time range and whether the amount of data arriving is within the second predetermined data range.

[0082] The second predetermined time range completely covers the first predetermined time range but is not equal to the first predetermined time range, and the second predetermined data volume range completely covers the first predetermined data volume range but is not equal to the first predetermined data volume range.

[0083] In step S7, if it is determined that the arrival time is within the second predetermined time range and the amount of data arriving is within the second predetermined data range, then step S8 is executed; if it is determined that the arrival time is outside the second predetermined time range and / or the amount of data arriving is outside the second predetermined data range, then step S9 is executed.

[0084] The second predetermined time range and the second predetermined data volume range can be pre-configured by the user according to actual needs.

[0085] Step S8: Supplement the data sent by the data source to be monitored.

[0086] When the arrival time and the amount of data received are both within the second predetermined time range and the amount of data received are within the second predetermined data range, it indicates that although the data transmission process of the monitored data source is abnormal, the offsets of the arrival time and the amount of data received are within a reasonable range, and the integrity of the data can be recalibrated by data supplementation.

[0087] In some embodiments, step S1 is performed after step S8.

[0088] Step S9: Issue an alarm notification.

[0089] If the arrival time is outside the second predetermined time range and / or the amount of arrival data is outside the second predetermined data range, it indicates that the offset of at least one of the arrival time and the amount of arrival data is too large, and the user needs to be reminded to verify or adjust it.

[0090] After receiving the alert message, the user can verify the security of the data sent by the monitored data source (it may be that there is a problem with the data sent by the monitored data source itself), or adjust the data arrival detection strategy configured for the monitored data source (there is no problem with the data sent by the monitored data source itself, but the configured data arrival detection strategy is unreasonable).

[0091] Figure 7A flowchart of a task monitoring method provided in this disclosure embodiment is shown below. Figure 7 As shown, this task monitoring method relies on data sent from multiple target data sources. The task monitoring method includes:

[0092] Step S11: For each target data source, treat the target data source as the data source to be monitored and monitor the data sent by the target data source.

[0093] The method used to monitor the target data source can be the data monitoring method provided in the aforementioned embodiments. Each target data source has a corresponding marking result at any time. The marking result includes: marking arrived or not marked. The specific monitoring process will not be described here.

[0094] Step S12: Detect whether the task can be started based on the labeling results of each target data source.

[0095] Figure 8 This is a flowchart illustrating a specific implementation of step S12 in an embodiment of this disclosure, as follows: Figure 8 As shown, in some embodiments, step S12 includes:

[0096] Step S1201a: Determine whether the marking results of all target data sources are all marked as arrived.

[0097] When it is determined that the marking results of all target data sources are all marked, the task can be started; otherwise, the task cannot be started.

[0098] Figure 9 Here is another specific implementation flowchart of step S12 in this embodiment of the present disclosure, as follows: Figure 9 As shown, in some embodiments, step S12 includes:

[0099] Step S1201b: Calculate the task execution confidence score Q based on the labeling results of each target data source.

[0100]

[0101] Where n is the total number of target data sources, q i w represents the confidence score pre-configured for the i-th target data source. i This represents the coefficient corresponding to the labeling result of the i-th target data source, where the coefficient w is the coefficient when the labeling result is reached. i The value is 1, and the coefficient w is used when the labeling result is unlabeled. i The value is 0.

[0102] In this embodiment of the disclosure, an attribute "confidence score" can be set in the data table, and the confidence score corresponding to each data source can be configured by the user according to actual needs.

[0103] Step S1202b: Compare whether the task execution confidence score is greater than or equal to the confidence score threshold.

[0104] When the task execution confidence score is greater than or equal to the confidence score threshold, the task is detected as ready to start execution, and the task execution system automatically executes the corresponding task immediately. When the task execution confidence score is less than the confidence score threshold, the task is detected as not ready to start execution, and step S11 continues.

[0105] Of course, this disclosure can also use other algorithms based on the marking results of each target data source to detect whether the task can start execution. For example, if the result of the task execution is a reliable result based on more than 80% of the target data sources being sent, then in step S12, it can be set that when more than 80% of the target data sources are marked as arrived, it can be detected that the task can start execution. Other algorithms will not be described in detail here.

[0106] Based on the above steps S11 and S12, when the data received by the server meets the requirements to support the normal execution of the task, the task can start execution immediately, thereby improving the real-time performance of data processing.

[0107] This disclosure also provides a data monitoring module, including: one or more processors and a memory, wherein the memory stores one or more programs, and when the one or more programs are executed by one or more processors, the one or more processors implement the data monitoring method provided in the foregoing embodiments.

[0108] This disclosure also provides a task monitoring system, including the data monitoring module and detection module provided in the foregoing embodiments, wherein the detection module is used to detect whether a task can be started based on the marking results of each target data source.

[0109] The detection module can execute step S12 in the foregoing embodiments. For a description of the detection module, please refer to the corresponding content in the foregoing embodiments.

[0110] In some embodiments, the data task monitoring system may be configured on a server.

[0111] This disclosure also provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps in the data monitoring method and task monitoring method provided in the foregoing embodiments.

[0112] It will be understood by those skilled in the art that all or some of the steps in the methods disclosed above, and the functional modules / units in the apparatus, can be implemented as software, firmware, hardware, and suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0113] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A data monitoring method, wherein, include: Determine the data arrival detection strategy configured for the data source to be monitored; Based on the data arrival detection strategy, it is detected whether the data sent by the data source to be monitored has reached the predetermined level. When it is detected that the data sent by the data source to be monitored has not reached the predetermined level, the data arrival detection strategy configured for the data source to be monitored will continue to be executed. When it is detected that the data sent by the data source to be monitored has reached the predetermined level, the data source to be monitored is marked as having reached the predetermined level. The step of determining the data arrival detection strategy configured for the data source to be monitored includes: Obtain the data inspection type value of the attribute of the data source to be monitored from the data table; query the data arrival detection strategy corresponding to the value from the pre-set correspondence table, which records different data inspection types and their corresponding data arrival detection strategies. The data arrival detection strategy includes at least one of the following: checking stream files, checking file existence, checking the number of files, checking Hive partitions, checking file names, checking file content, checking file directories, checking both file names and file content, and a default strategy; the default strategy includes: not detecting the data sent by the corresponding data source, and directly assuming that the data sent by the data source has reached a predetermined level.

2. The data monitoring method according to claim 1, wherein, Following the step of marking the arrival of the data source to be monitored, the method further includes: Obtain the arrival time and amount of data sent by the data source to be monitored; Based on the arrival time and the amount of data received, detect whether the data transmission process of the monitored data source is abnormal.

3. The data monitoring method according to claim 2, wherein, The step of detecting whether the data transmission process of the monitored data source is abnormal based on the arrival time and the amount of data received includes: Determine whether the arrival time is within a first predetermined time range, and determine whether the amount of arriving data is within a first predetermined data amount range; When it is determined that the arrival time is within the first predetermined time range and the amount of arriving data is within the first predetermined data amount range, the data transmission process of the monitored data source is detected to be normal; otherwise, the data transmission process of the monitored data source is detected to be abnormal, and the marker for the monitored data source is canceled.

4. The data monitoring method according to claim 3, wherein, Before the steps of determining whether the arrival time is within a first predetermined time range and determining whether the amount of arriving data is within a first predetermined data range, the method further includes: Based on the historical arrival time and historical arrival data volume of the data sent by the data source to be monitored, the first predetermined time range and the first predetermined data volume range are predicted.

5. The data monitoring method according to claim 3, wherein, After the step of detecting an anomaly in the data transmission process of the monitored data source, the method further includes: Determine whether the arrival time is within a second predetermined time range, and determine whether the amount of arriving data is within a second predetermined data range; wherein, the second predetermined time range completely covers the first predetermined time range and is not equal to the first predetermined time range, and the second predetermined data range completely covers the first predetermined data range and is not equal to the first predetermined data range; When it is determined that the arrival time is within the second predetermined time range and the amount of arriving data is within the second predetermined data range, the data sent by the data source to be monitored is supplemented and then the step of determining the data arrival detection strategy configured for the data source to be monitored continues to be executed. When it is determined that the arrival time is outside the second predetermined time range, and / or the amount of arriving data is outside the second predetermined data amount range, an alarm message is issued.

6. The data monitoring method according to claim 1, wherein, When it is detected that the data sent by the data source to be monitored has reached the predetermined level, the step of marking the data source to be monitored as reached is executed after a pre-configured delay time.

7. A task monitoring method, wherein, The task depends on data sent from multiple target data sources; For each of the target data sources, the target data source is taken as the data source to be monitored, and the data sent by the target data source is monitored using any of the data monitoring methods described in claims 1-6; The task is detected as having started execution based on the marking results of each of the target data sources, wherein the marking results include: marking has arrived or not being marked.

8. The task monitoring method according to claim 7, wherein, The step of detecting whether the task has started execution based on the marking results of each of the target data sources specifically includes: Determine whether the marking results of all the target data sources are all marked as arrived; When it is determined that the marking results of all the target data sources are all marked as arrived, the task is detected as ready to start execution, and the task is executed immediately; otherwise, the task is detected as not ready to start execution.

9. The task monitoring method according to claim 7, wherein, The step of detecting whether the task has started execution based on the marking results of each of the target data sources specifically includes: Based on the labeling results of each target data source, the task execution confidence score Q is calculated: n is the total number of target data sources, represents the confidence score pre-configured for the i-th target data source, and represents the coefficient corresponding to the labeling result of the i-th target data source. The coefficient is 1 when the labeling result is "labeled" and 0 when the labeling result is "not labeled". Compare whether the task execution confidence score is greater than or equal to the confidence score threshold; When the confidence score for task execution is found to be greater than or equal to the confidence score threshold, it is detected that the task can begin execution, and the task is executed immediately. If the confidence score for task execution is found to be less than the confidence score threshold, then the task is detected as not being able to start execution.

10. A data monitoring module, wherein, include: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-6.

11. A task monitoring system, wherein, include: The data monitoring module and detection module as described in claim 10 above; The detection module is used to detect whether the task has started execution based on the marking results of each target data source.

12. A computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9.

Images