Data acquisition fault identification method and device, medium and program product

Abstract

The embodiment of the invention provides a data acquisition fault identification method and device, a medium and a program product, and relates to the technical field of data acquisition. The method comprises the following steps: acquiring log information of each data acquisition event in a data acquisition system in real time, and storing the log information in a buried point log database; identifying abnormal collection events with abnormal logs in the burying point log database, and obtaining data collection templates corresponding to the abnormal collection events; performing retry collection based on the data collection template, and obtaining retry collection log information as retry log information of the data collection template; and determining the fault condition of the data acquisition template by combining the retry log information and the original log information of the abnormal acquisition event. According to the embodiment of the invention, the log information of data acquisition is recorded through the burying point database, retry acquisition is carried out when acquisition is abnormal, and the fault condition of the data acquisition template is judged in combination with the retry log and the original log, so that the efficiency and accuracy of data acquisition fault recognition are effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of data acquisition. Specifically, it relates to a method, device, medium, and program product for identifying data acquisition failures. Background Art

[0002] In the field of data aggregation, platforms for data aggregation and management usually use web crawler technology to obtain target data such as enterprise bid winning and losing, government procurement bidding, etc. from source websites. Due to factors such as the revision of source websites and the replacement of website addresses, the situation where the web crawler technology fails to obtain target data may occur. Therefore, it is necessary to periodically verify and confirm the effectiveness of the data obtained by the web crawler technology.

[0003] Currently, the existing technology mainly uses manual verification. By conducting data acquisition tests one by one based on each source website, the failure situation of the acquired data is identified. Since manual troubleshooting requires a large amount of manpower and time costs, and it is prone to missed inspections and misjudgments, the efficiency and accuracy of manually identifying data acquisition failures are not high. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method, device, medium, and program product for identifying data acquisition failures, so as to improve the efficiency and accuracy of identifying data acquisition failures.

[0005] In a first aspect, the embodiments of this application provide a method for identifying data acquisition failures, including:

[0006] Real-time obtain the log information of each data acquisition event in the data acquisition system, and store each log information in the buried point log database;

[0007] Identify the abnormal acquisition events with log anomalies in the buried point log database, and obtain the data acquisition template corresponding to the abnormal acquisition events;

[0008] Based on the data acquisition template, perform a retry acquisition, and obtain the log information of the retry acquisition as the retry log information of the data acquisition template;

[0009] Combine the retry log information and the original log information of the abnormal acquisition event to determine the failure situation of the data acquisition template.

[0010] In the embodiments of this application, the log information of data acquisition is recorded through the buried point database. When the acquisition is abnormal, a retry acquisition is performed, and the failure situation of the data acquisition template is judged by combining the retry log and the original log, thereby effectively improving the efficiency and accuracy of data acquisition failure identification.

[0011] In some possible embodiments, the log information of each of the data collection events includes at least one of request node log information, parsing node log information, and storage node log information;

[0012] The identified abnormal collection events with log anomalies in the buried point log database include:

[0013] When it is determined that at least one type of log information of a data collection event in the buried point log database is abnormal, the data collection event is identified as an abnormal collection event.

[0014] In the embodiments of the present application, by separately recording the log information of different nodes during the data collection process and comprehensively judging whether the data collection is abnormal based on the abnormal conditions of the log information of each node, the flexibility of data collection fault identification is further improved.

[0015] In some possible embodiments, the determining the fault condition of the data collection template by combining the retry log information and the original log information of the abnormal collection event includes:

[0016] When the result of the retry collection is normal, it is determined that the data collection template is a normal template;

[0017] When the result of the retry collection is abnormal, the fault condition of the data collection template is determined according to the retry log information and the original log information of the abnormal collection event.

[0018] In the embodiments of the present application, by combining the abnormal conditions of the retry collection to judge the fault condition of the data collection template, the accuracy of data collection fault identification is further improved.

[0019] In some possible embodiments, the determining the fault condition of the data collection template according to the retry log information and the original log information of the abnormal collection event includes:

[0020] Judge whether the retry log information is consistent with the original log information of the abnormal collection event;

[0021] If so, it is determined that the data collection template is a faulty template, and the fault cause of the abnormal collection event is determined according to the retry log information or the original log information;

[0022] If not, retry the collection based on the data collection template again, update the retry log information, and re-determine the fault condition of the data collection template according to the retry log information and the original log information of the abnormal collection event.

[0023] In the embodiments of the present application, by judging whether the data collection template is abnormal according to the consistency between the retry log information and the original log information, and determining the cause of the failure of the abnormal collection event in the case of the retry log information and the original log information, the efficiency of subsequent repair of the data collection template can be further improved.

[0024] In some possible embodiments, the re-determining the failure condition of the data collection template according to the retry log information and the original log information of the abnormal collection event includes:

[0025] If the result of the re-retry collection is normal, it is determined that the data collection template is a normal template;

[0026] If the result of the re-retry collection is abnormal, it is judged whether there is a situation where the log information of at least two retry collections in the retry log information is consistent;

[0027] If so, it is determined that the data collection template is a faulty template, and the cause of the failure of the abnormal collection event is determined according to the consistent log information;

[0028] If not, the retry collection is re-performed based on the data collection template and the retry log information is updated until the result of the retry collection is normal, or until there is a situation where the log information of at least two retry collections in the retry log information is consistent.

[0029] In the embodiments of the present application, by judging the failure condition of the data collection template according to the log consistency of multiple retry collections, and determining the cause of the failure of the abnormal collection event according to the consistent log information, the efficiency of subsequent repair of the data collection template can be further improved.

[0030] In some possible embodiments, the performing a retry collection based on the data collection template and obtaining the log information of the retry collection as the retry log information of the data collection template includes:

[0031] Performing a retry collection based on the data collection template in a preset cycle, and obtaining the log information of each retry collection as the retry log information of the data collection template until the result of the retry collection is normal or the number of retry collections reaches a preset number threshold.

[0032] In the embodiments of the present application, by performing cyclic retry collections on the abnormal data collection template at a set cycle, the failure condition of the data collection template can be comprehensively judged according to the abnormal conditions of multiple retry collections, thereby further improving the accuracy of data collection failure identification.

[0033] In some possible embodiments, determining the fault condition of the data collection template by combining the retry log information and the original log information of the exception collection event includes:

[0034] Obtain the data collection log information corresponding to the data collection template; wherein, the data collection log information includes all log information of the data collection system for data collection based on the data collection template within a preset historical period;

[0035] According to the data collection log information and the retry log information, obtain the real-time metric parameters corresponding to the data collection template;

[0036] Based on the real-time metric parameters, use a preset calculation model to obtain the fault degree index corresponding to the data collection template;

[0037] When it is determined that the fault degree index exceeds the preset index tolerance, determine that the data collection template is a faulty template;

[0038] When it is determined that the fault degree index does not exceed the preset index tolerance, determine that the data collection template is a minor fault template; wherein, the repair priority corresponding to the faulty template is configured to be higher than the repair priority corresponding to the minor fault template.

[0039] In the embodiments of the present application, by obtaining multiple data collection logs within a preset historical period corresponding to the same data collection template and calculating the fault degree index based on the relevant data of these collection logs as the basis for determining whether the data collection template is abnormal, the accuracy and flexibility of fault identification of the data collection template are further improved.

[0040] In a second aspect, an embodiment of the present application provides a data collection fault identification device, including:

[0041] A log acquisition module, configured to acquire the log information of each data collection event in the data collection system in real time and store each piece of the log information in the buried point log database;

[0042] An exception identification module, configured to identify an exception collection event with log exceptions in the buried point log database and obtain the data collection template corresponding to the exception collection event;

[0043] A retry collection module, configured to perform retry collection based on the data collection template and obtain the log information of the retry collection as the retry log information of the data collection template;

[0044] A fault identification module, configured to determine the fault condition of the data collection template by combining the retry log information and the original log information of the exception collection event.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any embodiment of the first aspect can be implemented.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the method described in any embodiment of the first aspect can be implemented.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. Description of the Drawings

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 It is a schematic flowchart of a data acquisition fault identification method provided by an embodiment of the present application;

[0050] Figure 2 It is an overall flowchart block diagram of the data acquisition fault identification method provided by an embodiment of the present application;

[0051] Figure 3 It is a schematic structural diagram of a data acquisition fault identification device provided by an embodiment of the present application;

[0052] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0053] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0054] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0055] It should be noted that data aggregation platforms generally use web crawler technology to obtain target data from source websites. When there are certain changes to the source website, such as website revisions or URL replacements, it may lead to failures in obtaining target data through web crawler technology. Therefore, it is necessary to regularly verify and confirm the validity of the acquired data. Currently, the faults in the data collection process are mainly identified through manual verification and repaired according to requirements. However, when there are a large number of data collection templates configured in the template library, there may be a large number of website revisions every day, resulting in the existing data collection templates being unable to parse the data to be collected. In this case, the manual verification method one by one requires excessive human and time costs, and there may be a certain probability of false detection and missed detection. In addition, some automatic identification technologies directly judge the faults of data collection templates based on the result of whether valid data is collected, ignoring the collection anomalies caused by factors such as network fluctuations, which may misdetect a large number of normal data collection templates and also increase the workload of manual verification and repair.

[0056] In view of the problems existing in the above-mentioned prior art, the embodiment of the present application provides a method for identifying data collection faults. By recording the logs of collection events, retrying the collection events with anomalies, and finally comprehensively determining the faults of data collection templates by combining the log information of the retry collection and the original log information, the efficiency and accuracy of data collection fault identification are effectively improved.

[0057] As Figure 1 shown, the embodiment of the present application provides a method for identifying data collection faults, which may include the steps:

[0058] S1. Real-time obtain the log information of each data collection event in the data collection system, and store each log information in the buried point log database.

[0059] First, based on the pre-configured template library, the data collection system (a collection system based on web crawler technology) will obtain a data collection template from it each time and execute the data collection task. After the collection task is completed, the status information of the execution process of the collection task can be sent to the message queue, and then the log consumer service consumes it from the message queue and stores it in the buried point log database. Among them, the complete process of a collection task (which may be a successful collection or a failed collection) is regarded as a data collection event. It should be noted that a data collection event can correspond to one or more log information. For example, a collection task may include multiple links (nodes), and each link corresponds to one log information, then a data collection event corresponds to the log information of multiple collection links corresponding to the collection task.

[0060] S2. Identify the abnormal collection events with log anomalies in the buried point log database, and obtain the data collection templates corresponding to the abnormal collection events.

[0061] Based on the buried point log database, one or more abnormal collection events with log anomalies can be identified therefrom. Since one data collection event may correspond to multiple log messages, an abnormal collection event with a log anomaly refers to a data collection event that includes at least one abnormal log message. Then, obtain the data collection template corresponding to the abnormal collection event, where the data collection event and the data collection template have a one-to-one correspondence.

[0062] S3. Retry the collection based on the data collection template, and obtain the log messages of the retry collection as the retry log messages of the data collection template.

[0063] Based on the data collection templates corresponding to each abnormal collection event, the retry collection system can be separately called for retry collection. Among them, the retry collection system can adopt a collection system deployed independently of the data collection system, but the collection logic and configuration of the two need to be kept consistent. The retry collection can be regarded as another collection task of the same data collection template. The process of the retry collection is also a data collection event, and a series (or one) of log messages will also be formed during the retry collection process. Record these log messages of the retry collection process as the retry log messages of the data collection template.

[0064] S4. Determine the fault situation of the data collection template by combining the retry log messages and the original log messages of the abnormal collection event.

[0065] Finally, the fault situation of the data collection template can be comprehensively judged by combining the retry log messages and the original log messages of the abnormal collection event (the log messages of the data collection event of the same data collection template stored in step S1). Exemplarily, it can be judged whether there is a fault in the data collection template, the level of the fault degree, etc. according to the success or failure of the retry collection, the consistency between the retry log messages and the original log messages, etc.

[0066] Based on this, by recording the log messages of the data collection in the buried point database, retrying the collection when the collection is abnormal, and combining the retry log and the original log to judge the fault situation of the data collection template, the efficiency and accuracy of data collection fault identification can be effectively improved.

[0067] Exemplarily, the execution flowchart of the data collection fault identification method provided by the embodiments of the present application can be as Figure 2 shown. It can be understood that since the process of fault identification has little impact on the original data collection system and only a small number of log buried point rules need to be set, by handing over the template fault problem to a dedicated fault identification system for processing, the data collection system does not need to care about the identification rules and subsequent processing methods of the template fault. The optimization and online of the fault identification rules have no impact on the original data collection system.

[0068] In some possible embodiments, the log information of each data collection event includes at least one of request node log information, parsing node log information, and storage node log information;

[0069] In step S2, identifying an abnormal collection event with log anomalies in the buried point log database may include:

[0070] When it is determined that at least one type of log information of a data collection event in the buried point log database is abnormal, the data collection event is identified as an abnormal collection event.

[0071] It should be noted that the process of successfully collecting data each time sequentially includes a request link, a parsing link, and a storage link; for the same data collection process, when an abnormality occurs in the previous link, the subsequent links may not be executed. Among them, the log information of each link can be recorded when each link ends. Therefore, the log information of each data collection event may include one or more of request node log information, parsing node log information, and storage node log information, and each type of node log information can include two states: normal and abnormal.

[0072] For example, if an abnormality occurs in the parsing link of a data collection event, the log information corresponding to the data collection event will include request node log information and parsing node log information; when a data collection event successfully collects valid data, the log information corresponding to the data collection event will include request node log information, parsing node log information, and storage node log information.

[0073] It should be noted that in the request link, it means that the crawler collection system sends an access request to the source website URL of the template. If the request does not time out, the return status code is a predefined value (such as 200), and if the return result is not empty, the request is normal; otherwise, the request is abnormal. For the parsing link, it means that the crawler collection system parses the target data according to different parsing rules for different templates (such as including regular expression matching parsing, lxml parsing, BeautifulSoup parsing, requests-html parsing, etc.). If the necessary fields (such as title, publication time, detail page URL, page content, etc.) are parsed, it is determined that the parsing is normal; otherwise, the parsing is abnormal. For the storage link, it means that the target data parsed is stored in the database based on the storage rules corresponding to different templates. If all the necessary fields required are successfully stored, it is determined that the storage is normal; otherwise, it is determined that the storage is abnormal.

[0074] In step S2, when it is determined that at least one type of log information of the same data collection event in the buried point log database is abnormal, then this data collection event is identified as an abnormal collection event. It can be understood that when an abnormality occurs in the previous link of the same data collection event, it may not include the node log information of the subsequent links. Therefore, the log information identified as an abnormal collection event may all be only one abnormal node log information.

[0075] Based on this, by separately recording the log information of different nodes in the data collection process and comprehensively judging whether the data collection is abnormal according to the abnormality of each node's log information, the flexibility of data collection fault identification is further improved.

[0076] In some possible embodiments, step S4, determining the fault situation of the data collection template by combining the retry log information and the original log information of the abnormal collection event, may include:

[0077] S401. When the result of the retry collection is normal, it is determined that the data collection template is a normal template;

[0078] S402. When the result of the retry collection is abnormal, the fault situation of the data collection template is determined according to the retry log information and the original log information of the abnormal collection event.

[0079] It should be noted that after retrying the collection of the data collection template with abnormal collection, if the result of the retry collection is normal (referring to successfully obtaining the valid data corresponding to the template and successfully storing it in the database), it is directly determined that the corresponding data collection template is a normal template. For the case of determining a normal template, information can be pushed to the data collection system for re - collection.

[0080] For the case where the result of the retry collection is abnormal, further judge the fault situation of the data collection template according to the retry log information and the original log information of the abnormal collection event. Exemplarily, it can be judged whether there is the same abnormal information in the retry log information and the original log information. If there is, it is highly probable that the abnormal collection is caused by the fault of the data collection template. Therefore, it can be determined that the data collection template has a fault.

[0081] Based on this, by combining the abnormal situation of the retry collection to judge the fault situation of the data collection template, the accuracy of data collection fault identification is further improved.

[0082] In some possible embodiments, in step S402, determining the fault situation of the data collection template according to the retry log information and the original log information of the abnormal collection event may include:

[0083] S4021. Judge whether the retry log information is consistent with the original log information of the abnormal collection event;

[0084] S4022. If so, determine that the data collection template is a faulty template, and determine the cause of the failure of the abnormal collection event based on the retry log information or the original log information;

[0085] S4023. If not, retry the collection based on the data collection template again, update the retry log information, and re-determine the fault condition of the data collection template according to the retry log information and the original log information of the abnormal collection event.

[0086] It should be noted that determining whether the retry log information is consistent with the original log information of the abnormal collection event can be to determine whether there is the same abnormal information in the retry log information and the original log information. For example, if both pieces of log information indicate an abnormality in the request link, it can be determined that the data collection template has failed, and the cause of the failure can be located in the request link; similarly, for example, if both pieces of log information indicate an abnormality in the warehousing link, it can be determined that the data collection template has failed, and the cause of the failure can be located in the warehousing link. Accordingly, the subsequent template repair system can repair the rules of the corresponding link according to the cause of the failure.

[0087] For the case where the template is determined to be a faulty template, the fault-related information can be sent to the manual confirmation and repair system, where the fault-related information can include the number of the data collection template with the fault, and the cause of the failure, etc. By automatically locating the cause of the failure, it helps the manual confirmation and repair system quickly understand the fault information of the faulty template and take corresponding repair measures, effectively improving the efficiency of repairing the faulty template.

[0088] If the retry log information is inconsistent with the original log information of the abnormal collection event, the retry collection can be performed again based on the same data collection template, and the retry log information can be updated according to the status of the second retry collection, and then the fault condition of the data collection template can be re-determined according to the updated retry log information and the original log information.

[0089] It should be noted that after the second retry collection, updating the retry log information can be using the log information of the last retry collection as the retry log information of the data collection template, or adding a new set of retry collection log information on the basis of the log information of the first retry collection.

[0090] Based on this, by judging whether the data collection template is abnormal according to the consistency between the retry log information and the original log information, and determining the cause of the failure of the abnormal collection event in the case of the retry log information and the original log information, the efficiency of subsequent data collection template repair can be further improved.

[0091] In some possible embodiments, in step S4023, re-determining the failure condition of the data acquisition template according to the retry log information and the original log information of the exception collection event may include:

[0092] S40231. If the result of the re-retry acquisition is normal, it is determined that the data acquisition template is a normal template;

[0093] S40232. If the result of the re-retry acquisition is abnormal, it is determined whether there is a situation where the log information of at least two retry acquisitions in the retry log information is consistent;

[0094] S40233. If so, it is determined that the data acquisition template is a faulty template, and the cause of the exception collection event is determined according to the consistent log information;

[0095] S40234. If not, re-retry the acquisition based on the data acquisition template and update the retry log information until the result of the retry acquisition is normal, or until there is a situation where the log information of at least two retry acquisitions in the retry log information is consistent.

[0096] It should be noted that if the result of the re-retry acquisition is normal, it can be directly determined that the data acquisition template is a normal template.

[0097] In the case where the result of the retry acquisition is still abnormal, it is possible to determine whether there is a situation where the log information of at least two retry acquisitions (including the log information of at least two retry acquisitions) is consistent according to the updated retry log information. If so, it can be determined that the data acquisition template is a faulty template, and the cause of the current exception collection event is determined according to the consistent log information; if not, it is necessary to re-retry the acquisition based on the data acquisition template until the result of the retry acquisition is normal, or until there is a situation where the log information of at least two retry acquisitions in the retry log information is consistent, or the number of retry acquisitions reaches the preset upper limit.

[0098] Based on this, by judging the failure condition of the data acquisition template according to the log consistency of multiple retry acquisitions and determining the cause of the exception collection event according to the consistent log information, the efficiency of subsequent repair of the data acquisition template can be further improved.

[0099] In some possible embodiments, step S3, performing a retry acquisition based on the data acquisition template and obtaining the log information of the retry acquisition as the retry log information of the data acquisition template, may include:

[0100] S301. Retry the acquisition in a preset cycle based on the data acquisition template, and obtain the log information of each retry acquisition as the retry log information of the data acquisition template until the result of the retry acquisition is normal or the number of retry acquisitions reaches the preset number threshold.

[0101] Specifically, in step S3, for the identified abnormal acquisition event, multiple cyclic retry acquisitions can be performed based on its corresponding data acquisition template at a preset cycle. During this period, if the retry acquisition is normal, the retry acquisition is ended and the data acquisition template is directly determined to be normal; otherwise, the retry acquisition is cycled until the number of retry acquisitions reaches the preset number threshold.

[0102] After the number of retry acquisitions reaches the preset number threshold, the log information of each retry acquisition is used as the retry log information of the data acquisition template, which is one of the judgment bases for fault identification in step S4. During the fault identification process in step S4, according to the log information of multiple retry acquisitions, the abnormal information with the highest occurrence frequency can be used as the fault cause corresponding to the data acquisition template.

[0103] Based on this, by cyclically retrying the acquisition of the abnormal data acquisition template at a set cycle, the fault situation of the data acquisition template can be comprehensively judged according to the abnormal situations of multiple retry acquisitions, avoiding the situation where the basis for fault identification is too thin due to a single retry acquisition, and effectively eliminating abnormal acquisition situations caused by factors such as network fluctuations, thereby further improving the accuracy of data acquisition fault identification.

[0104] In some possible embodiments, step S4, determining the fault situation of the data acquisition template by combining the retry log information and the original log information of the abnormal acquisition event, may include:

[0105] S411. Obtain the data acquisition log information corresponding to the data acquisition template; wherein, the data acquisition log information includes all log information of the data acquisition system for data acquisition based on the data acquisition template within a preset historical period;

[0106] S412. According to the data acquisition log information and the retry log information, obtain the real-time index parameters corresponding to the data acquisition template;

[0107] S413. Based on the real-time index parameters, use a preset calculation model to obtain the fault degree index corresponding to the data acquisition template;

[0108] S414. In the case where it is determined that the fault degree index exceeds the preset index tolerance, determine that the data acquisition template is a faulty template;

[0109] S415. When it is determined that the fault degree index does not exceed the preset index tolerance, it is determined that the data acquisition template is a secondary fault template; wherein, the repair priority corresponding to the fault template is configured to be higher than the repair priority corresponding to the secondary fault template.

[0110] It should be noted that in practical applications, the data acquisition system may perform data acquisition multiple times based on the data acquisition template. Before the data acquisition template is updated or adjusted, the same data acquisition template may form multiple log messages in the buried point log database (including log messages characterized as normal and log messages characterized as abnormal). Since the identification of data acquisition events with abnormal log messages has been performed before step S4, the data acquisition log messages corresponding to the data acquisition template here must include at least one abnormal log message (i.e., the original log message corresponding to the abnormal acquisition event).

[0111] By obtaining all the log messages of the data acquisition system based on the data acquisition template within a preset historical period, and then combining the retry log messages formed during the retry acquisition process, relevant parameters can be statistically calculated or directly obtained according to requirements as real-time index parameters.

[0112] Then, based on the current real-time index parameters, calculations are performed using a pre-configured calculation model to obtain the fault degree index corresponding to the data acquisition template, which represents the probability (or severity of the fault) that the current data acquisition template to be identified has a fault within a preset historical period. The specific calculation model and the real-time index parameters used for inputting into the model can be set according to requirements.

[0113] Exemplarily, the data buried point situation of a certain data acquisition template within a preset historical period can be obtained, including the number of normal and abnormal requests, the number of normal and abnormal parses, the number of normal and abnormal storages, the number of normal and abnormal retry acquisitions, etc., and further statistically calculated or directly used as the real-time index parameters corresponding to the data acquisition template, such as the number of normal times, the number of abnormal times, the proportion of abnormal times to the total number of times, the mean, variance, standard deviation, etc. corresponding to the number of normal or abnormal times within a unit period. Then, using a preset calculation model, such as an AI small model (e.g., a model that iteratively adjusts calculation parameters using fewer training samples), a pre-designed calculation formula (e.g., Z-score), etc., the above-obtained real-time index parameters are input into the calculation model to obtain the fault degree index.

[0114] Finally, the fault condition of the data collection template is determined based on whether the fault degree index exceeds the preset index tolerance. Specifically, if the fault degree index exceeds the preset index tolerance, the data collection template is determined to be a faulty template. At this time, it is generally believed that the data collection template has a major fault and needs to be manually confirmed and repaired first; if the fault degree index does not exceed the preset index tolerance, the data collection template is determined to be a minor fault template. At this time, it can be considered that this type of data collection template has a certain fault, but it has little impact on the overall data collection work, and it can be configured to be manually confirmed and repaired with a relatively low priority.

[0115] Based on this, by classifying and judging the degree of fault of data collection templates, faulty templates can be repaired according to priority. When a large number of collection abnormalities occur due to massive templates or network fluctuations, the pressure of manual intervention can be reduced, thereby improving the accuracy and flexibility of data collection fault identification.

[0116] Please refer to Figure 3 , Figure 3 FIG. 1 is a block diagram showing the composition of a data acquisition fault identification device provided by some embodiments of the present application. It should be understood that the data acquisition fault identification device is similar to the above-mentioned Figure 1 Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the data acquisition fault identification device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0117] Figure 3 The data acquisition fault identification device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the data acquisition fault identification device, and the data acquisition fault identification device includes:

[0118] The log acquisition module 310 is used to acquire the log information of each data acquisition event in the data acquisition system in real time and store each log information in the embedding point log database;

[0119] The abnormality identification module 320 is used to identify abnormal collection events with log abnormalities in the embedding point log database, and obtain the data collection template corresponding to the abnormal collection event;

[0120] A retry collection module 330 is used to perform retry collection based on the data collection template, and obtain log information of the retry collection as retry log information of the data collection template;

[0121] The fault identification module 340 is used to determine the fault condition of the data collection template by combining the retry log information and the original log information of the abnormal collection event.

[0122] It can be understood that the above device item embodiments correspond to the method item embodiments of the present invention. A data acquisition fault identification device provided by the embodiments of the present invention can implement the data acquisition fault identification method provided by any method item embodiment of the present invention.

[0123] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described device can refer to the corresponding process in the foregoing method, and will not be elaborated herein.

[0124] As Figure 4 shown, some embodiments of the present application provide an electronic device 400, which includes: a memory 410, a processor 420, and a computer program stored on the memory 410 and executable on the processor 420. Among them, when the processor 420 reads the program from the memory 410 through the bus 430 and executes the program, it can implement the method of any embodiment included in the above data acquisition fault identification method.

[0125] The processor 420 can process digital signals and can include various computing architectures. For example, a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, the processor 420 can be a microprocessor.

[0126] The memory 410 can be used to store instructions executed by the processor 420 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 420 of the present disclosure embodiment can be used to execute the instructions in the memory 410 to implement the method shown above. The memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0127] Some embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, it executes the method described in the method embodiment.

[0128] Some embodiments of the present application also provide a computer program product, which when run on a computer causes the computer to execute the method described in the method embodiment.

[0129] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the similarities and common parts among the embodiments, reference can be made to each other. For device embodiments, since they are basically similar to method embodiments, they are described relatively simply. For the relevant parts, reference can be made to the corresponding descriptions in the method embodiments.

[0130] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] In addition, in each embodiment of the present application, the functional modules can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0132] If the above functions are implemented in the form of software function modules and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0133] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application. It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0134] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all of them should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claimed rights.

[0135] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

Claims

1. A data acquisition fault identification method, characterized in that: include: Obtain log information of each data collection event in the data collection system in real time, and store each log information in the embedding point log database; Identify abnormal collection events with log anomalies in the tracking point log database, and obtain a data collection template corresponding to the abnormal collection event; Retrying collection based on the data collection template, and obtaining log information of the retry collection as retry log information of the data collection template; The failure condition of the data collection template is determined by combining the retry log information and the original log information of the abnormal collection event.

2. The data acquisition fault identification method according to claim 1, characterized in that: The log information of each data collection event includes at least one of request node log information, parsing node log information and storage node log information; The identifying the abnormal collection event of the log abnormality in the embedding log database includes: When it is determined that at least one log information of a data collection event in the tracking log database is abnormal, the data collection event is identified as an abnormal collection event.

3. The data acquisition fault identification method according to claim 1, characterized in that: The determining the failure condition of the data collection template by combining the retry log information and the original log information of the abnormal collection event includes: If the result of the retry acquisition is normal, the data acquisition template is determined to be a normal template; In the case where the result of the retry collection is abnormal, the failure condition of the data collection template is determined according to the retry log information and the original log information of the abnormal collection event.

4. The data acquisition fault identification method according to claim 3, characterized in that: The determining the failure condition of the data collection template according to the retry log information and the original log information of the abnormal collection event includes: Determine whether the retry log information is consistent with the original log information of the abnormal collection event; If so, the data collection template is determined to be a fault template, and the fault cause of the abnormal collection event is determined according to the retry log information or the original log information; If not, retry the data collection based on the data collection template, update the retry log information, and re-determine the failure status of the data collection template based on the retry log information and the original log information of the abnormal collection event.

5. The data acquisition fault identification method according to claim 4, characterized in that: The re-determining the failure condition of the data collection template according to the retry log information and the original log information of the abnormal collection event includes: If the result of the retry acquisition is normal, the data acquisition template is determined to be a normal template; If the result of the retry collection is abnormal, it is determined whether there is a situation in the retry log information where the log information of at least two retry collections is consistent; If so, the data collection template is determined to be a fault template, and the fault cause of the abnormal collection event is determined according to the consistent log information; If not, retry the collection based on the data collection template and update the retry log information until the result of the retry collection is normal, or until the log information of at least two retry collections is consistent in the retry log information.

6. The data acquisition fault identification method according to claim 1, characterized in that: The retrying collection based on the data collection template and obtaining log information of the retrying collection as the retry log information of the data collection template includes: Based on the data collection template, retry collection is performed in a preset periodic cycle, and log information of previous retry collections is obtained as the retry log information of the data collection template, until the result of the retry collection is normal or the number of retry collections reaches a preset threshold number.

7. The data acquisition fault identification method according to claim 1, characterized in that: The determining the failure condition of the data collection template by combining the retry log information and the original log information of the abnormal collection event includes: Acquire data collection log information corresponding to the data collection template; wherein the data collection log information includes all log information of data collection performed by the data collection system within a preset historical period based on the data collection template; Acquire real-time indicator parameters corresponding to the data collection template according to the data collection log information and the retry log information; Based on the real-time index parameter, a fault degree index corresponding to the data acquisition template is obtained using a preset calculation model; In the case where it is determined that the fault degree index exceeds a preset index tolerance, determining that the data collection template is a fault template; When it is determined that the fault degree index does not exceed the preset index tolerance, the data collection template is determined to be a secondary fault template; wherein the repair priority corresponding to the fault template is configured to be higher than the repair priority corresponding to the secondary fault template.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can implement the data acquisition fault identification method described in any one of claims 1 to 7 when executing the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data acquisition fault identification method according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the data acquisition fault identification method according to any one of claims 1 to 7 is implemented.

Images