Real-time monitoring and early warning method and device based on incremental crawler data

Abstract

The embodiment of the present invention provides a real-time monitoring and early warning method, device and electronic device based on incremental crawler data, belonging to the field of data processing technology, the method comprising: adding the project tasks to be monitored to the data record table T1; reading the data record table T1, querying the data update of each crawler project on the previous day according to the incremental mode; reading the latest incremental data from the real-time monitoring table T2 and adding it to the monitoring email attachment table T4; checking whether the current time is the first day of the month, if so, reading the data volume of the whole month of the previous month from the monitoring history table T3 and adding it to the monitoring email attachment table T4; labeling the abnormal projects and outputting them to the monitoring email alarm attachment table T5 to send the alarm email attachment, and sending the normal email for the non-abnormal situation. The present invention improves the efficiency and flexibility of crawler data monitoring.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time monitoring and early warning method, device and electronic equipment based on incremental crawler data. Background Art

[0002] Crawler data refers to information or data captured from the Internet through web crawling technology. A web crawler is an automated program or script that collects and organizes information through the Internet according to certain rules. These crawlers can simulate user behavior, access web pages and extract the required data content, and then store it in a structured form for further analysis and use.

[0003] Crawler data is widely used. Search engine companies use crawlers to index web page content in order to provide users with relevant search results. Scholars may need a large amount of data for research, and web crawlers can help them quickly obtain the required data. In the field of big data, crawler technology is one of the important means of obtaining raw data.

[0004] As the number of crawler projects increases, it becomes difficult or even impossible to manually monitor whether the incremental scripts are running normally and whether the data is being stored normally. Therefore, it is necessary to write scripts to monitor and alarm all crawler projects.

[0005] The above problems become technical problems that need to be solved. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a real-time monitoring and early warning method, device and electronic device based on incremental crawler data, which at least partially solves the problems existing in the prior art.

[0007] In a first aspect, an embodiment of the present invention provides a real-time monitoring and early warning method based on incremental crawler data, comprising:

[0008] Add the project tasks that need to be monitored to the data record table T1, which contains the project, website name, website link, sql library name, sql table name, incremental mode and alarm days;

[0009] Read the data record table T1, query the data update status of each crawler project on the previous day according to the incremental mode, and after obtaining the latest amount of data added to the database, update it to the real-time monitoring table T2 according to the situation of each project, and update the latest incremental data to the monitoring history table T3 at the same time;

[0010] Read the latest incremental data from the real-time monitoring table T2 and add it to the monitoring email attachment table T4;

[0011] Check the current time to see if it is the first day of the month. If so, read the data volume of the entire month of the previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4;

[0012] After obtaining the newly added data volume of all crawler projects, according to the daily warning limit values ​​and no-increment days limit values ​​set for different projects, the abnormal projects are labeled and output to the monitoring email alarm attachment table T5 to send the alarm email attachment. For normal situations, a normal email is sent and the monitoring email attachment table T4 is attached to the email so that the attachment can be directly opened to view the daily increment of all projects when needed.

[0013] According to a specific implementation of the embodiment of the present disclosure, adding the project tasks to be monitored to the data recording table T1 includes:

[0014] Define data record table T1. Record table T1 is a two-dimensional table. Each row represents a project task and each column represents the attribute of the project. Table T1 is represented as a matrix Mt1. Each row of the matrix is ​​a vector Ri, which is used to represent a project record.

[0015] For a new project, construct a vector Rnw=(P,W,L,D,T,M,A), where P represents the project, W represents the website name, L represents the website link, D represents the SQL library name, T represents the SQL table name, M represents the incremental mode, and A represents the number of days to alarm.

[0016] Add the vector Rnw to the matrix Mt1 to form a new matrix Mt1':

[0017]

[0018] Represents the union operation of sets;

[0019] based on Update and save the modified table T1 so that the new record is persistently stored.

[0020] According to a specific implementation of the embodiment of the present disclosure, the data record table T1 is read, and the data update status of each crawler project on the previous day is queried according to the incremental mode. After obtaining the latest amount of data added to the database, it is updated to the real-time monitoring table T2 according to the status of each project, including:

[0021] The data record table T1 is represented as:

[0022]

[0023] Pi is the name of the i-th project, Wi is the website name corresponding to the i-th project, Li is the website link corresponding to the i-th project, Di is the data repository corresponding to the i-th project, Ti is the data table corresponding to the i-th project, Mi is the incremental mode of the i-th project, and Ai is the number of alarm days for the i-th project;

[0024] The real-time monitoring table T2 is expressed as:

[0025]

[0026] Pi is the name of the i-th project, △Qi is the most recent data increment of the i-th project;

[0027] Read all project records from the data record table T1. For each project i, query the data update status of the crawler project on the previous day according to the incremental mode Mi. The query result is △Qi;

[0028] Update the incremental data volume △Qi obtained from the query to the corresponding item Pi record in the real-time monitoring table T2, expressed as:

[0029]

[0030]

[0031]

[0032] The Qry function indicates that the data update status of the previous day is queried from the table Ti in the database Di according to the incremental mode Mi, and T2i indicates the i-th item record in the updated T2 table;

[0033] Update the incremental data of all projects to T2 to form a complete real-time monitoring table T2:

[0034] .

[0035] According to a specific implementation of the embodiment of the present disclosure, the updating of the latest incremental data amount to the monitoring history table T3 at the same time includes:

[0036] Define T3 as follows:

[0037]

[0038] Pi is the name of the i-th item, It is a timestamp, indicating the time point at which the incremental data is recorded. is the i-th item at time The amount of incremental data;

[0039] For each item i, add the latest queried incremental data volume △Qi to the monitoring history table T3;

[0040] For each item i, construct a new record (Pi, t, △Qi) and add it to T3, where t is the current time, expressed as:

[0041]

[0042]

[0043]

[0044] The CTm() function means getting the current timestamp;

[0045] Update the latest incremental data of all projects to T3 to form a complete monitoring history table T3:

[0046] .

[0047] According to a specific implementation of the embodiment of the present disclosure, the latest incremental data is read from the real-time monitoring table T2 and added to the monitoring email attachment table T4; including:

[0048] Read the records of all items from the real-time monitoring table T2. For each item i, read the latest incremental data volume ΔQi from T2, construct this information into a new record, and add the newly constructed record to the monitoring email attachment table T4.

[0049] According to a specific implementation of the embodiment of the present disclosure, the data volume of the entire month of the previous month is read from the monitoring history table T3 and added to the monitoring email attachment table T4, including:

[0050] According to the current time t, calculate the start time and end time of last month;

[0051] For each item i, read all incremental data from T3 between the start time and the end time. ;

[0052] Sum up all incremental data in the time period to get the data volume of project i last month ;

[0053] For each item i, construct a new record And add it to T4.

[0054] According to a specific implementation manner of an embodiment of the present disclosure, the daily warning limit value and the non-increment days limit value set according to different projects are used to tag abnormal projects and output them to the monitoring email alarm attachment table T5 to send an alarm email attachment, including:

[0055] Read the latest incremental data volume △Qi of each project from the real-time monitoring table T2, and at the same time read the relevant historical data from the monitoring history table T3;

[0056] For each project Pi, judge according to the daily warning limit value Li and the non-increment days limit value Ai set in the data record table T1 to determine the warning limit value;

[0057] For each project, check whether its latest incremental data volume △Qi is lower than the set daily warning limit value Li. If △Qi < Li, it is considered that the project has a risk of insufficient data update;

[0058] Obtain the data update records of project Pi within the most recent preset time from the monitoring history table T3, calculate the consecutive non-increment days. If the consecutive non-increment days exceed the non-increment days limit value Ai of the project, it is considered that the project has a risk of not being updated for a long time;

[0059] If project Pi meets △Qi < Li or the consecutive non-increment days exceed Ai, mark the project as abnormal and record it in the monitoring email alarm attachment table T5;

[0060] Output all the project records (Pi, △Qi) marked as abnormal to the monitoring email alarm attachment table T5 and prepare to send an alarm email.

[0061] According to a specific implementation manner of an embodiment of the present disclosure, for the case of no abnormality, send a no-abnormality email and attach the monitoring email attachment table T4 to the email, including:

[0062] Confirm that all monitored projects have not reached the warning limit value or the non-increment days limit value;

[0063] Compose the content of the email body, indicating that all projects are currently in a normal state and no abnormalities are found;

[0064] Add the monitoring email attachment table T4 as an email attachment.

[0065] In a second aspect, an embodiment of the present invention provides a real-time monitoring and warning device based on incremental crawler data, including:

[0066] Add module, add the project tasks to be monitored to the data record table T1, the data record table T1 contains the project, website name, website link, sql library name, sql table name, incremental mode and alarm days;

[0067] The update module reads the data record table T1, queries the data update status of each crawler project yesterday according to the incremental mode, obtains the latest amount of data added to the database, and updates it to the real-time monitoring table T2 according to the situation of each project. At the same time, the latest incremental data is updated to the monitoring history table T3;

[0068] The reading module reads the latest incremental data from the real-time monitoring table T2 and adds it to the monitoring email attachment table T4;

[0069] Check the module to see if the current time is the first day of the month. If so, read the data volume of the entire month of the previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4;

[0070] After obtaining the newly added data volume of all crawler projects, the sending module labels the abnormal projects and outputs them to the monitoring email alarm attachment table T5 according to the daily warning limit values ​​and no-increment days limit values ​​set for different projects to send alarm email attachments. For normal situations, a normal email is sent and the monitoring email attachment table T4 is attached to the email so that the attachment can be directly opened to view the daily increments of all projects when needed.

[0071] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0072] at least one processor; and,

[0073] a memory communicatively connected to the at least one processor; wherein,

[0074] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time monitoring and early warning method based on incremental crawler data in any of the aforementioned first aspects or any of the implementations of the first aspect.

[0075] In the fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the real-time monitoring and early warning method based on incremental crawler data in the aforementioned first aspect or any implementation method of the first aspect.

[0076] In the fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the real-time monitoring and early warning method based on incremental crawler data in the aforementioned first aspect or any implementation of the first aspect.

[0077] The real-time monitoring and early warning scheme based on incremental crawler data in the embodiment of the present invention includes: adding the project tasks to be monitored to the data recording table T1, wherein the data recording table T1 contains the project to which it belongs, the website name, the website link, the sql library name, the sql table name, the incremental mode and the number of days for alarm; reading the data recording table T1, querying the data update status of each crawler project on the previous day according to the incremental mode, and after obtaining the latest amount of data added to the database, updating them to the real-time monitoring table T2 according to the status of each project, and updating the latest incremental data to the monitoring history table T3 at the same time; reading the latest incremental data from the real-time monitoring table T2 The amount of data is read and added to the monitoring email attachment table T4; check the current time to see if it is the first day of the month. If so, read the data volume of the entire month of the previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4; after obtaining the newly added data volume of all crawler projects, according to the daily warning limit value and the limit value of the number of days without increment set for different projects, label the abnormal projects and output them to the monitoring email alarm attachment table T5 to send the alarm email attachment. For the case of no abnormality, send a no abnormality email and attach the monitoring email attachment table T4 to the email, so that you can directly open the attachment to see the daily increment of all projects when needed. The solution of the present application does not require any access to any software interface, can be adapted to any system, has high adaptability, and has no other costs. It can meet most of the crawler task monitoring needs of general small enterprises, can adjust the table fields and script parameters according to actual conditions, and quickly and efficiently realize the monitoring of crawler tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A flowchart of a real-time monitoring and early warning method based on incremental crawler data provided by an embodiment of the present invention;

[0080] Figure 2 A flowchart of another real-time monitoring and early warning method based on incremental crawler data provided by an embodiment of the present invention;

[0081] Figure 3 A schematic diagram of the structure of a real-time monitoring and early warning device based on incremental crawler data provided by an embodiment of the present invention;

[0082] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0084] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0085] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0086] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0087] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0088] The disclosed embodiment provides a real-time monitoring and early warning method based on incremental crawler data. The real-time monitoring and early warning method based on incremental crawler data provided in this embodiment can be executed by a computing device, which can be implemented as software, or as a combination of software and hardware, and the computing device can be integrated in a server, terminal device, etc.

[0089] See also Figure 1 The present disclosure provides a real-time monitoring and early warning method based on incremental crawler data, including:

[0090] S101, adding the project tasks to be monitored to the data record table T1, wherein the data record table T1 includes the project, website name, website link, sql library name, sql table name, incremental mode and alarm days.

[0091] First, you can collect detailed information for each project that needs to be monitored, including:

[0092] - Project Name

[0093] -WebsiteName

[0094] -Website URL

[0095] -SQL database name (DatabaseName)

[0096] -SQL table name (TableName)

[0097] -IncrementalMode

[0098] -AlarmDays

[0099] Create a new record for each project, make sure all the information is correct, and enter the above information into the data recording table T1. Each row represents a project, and each column represents a different attribute of the project. T1 can exist in a spreadsheet format (for example, an Excel spreadsheet), and you can simply add this new record in the last row.

[0100] After completing the above steps, the monitoring information of the new project is successfully added to the data record table T1. Next, the system can automatically perform monitoring tasks based on the information in T1, such as data capture, data update status check, and sending monitoring emails.

[0101] S102, read the data record table T1, query the data update status of each crawler project on the previous day according to the incremental mode, and after obtaining the latest amount of data added to the database, update it to the real-time monitoring table T2 according to the situation of each project, and update the latest incremental data to the monitoring history table T3 at the same time.

[0102] To read information from the data record table T1, query the data update status of each crawler project on the previous day according to the incremental mode, and then update this information to the real-time monitoring table T2, and update the monitoring history table T3 at the same time, you can follow the steps below:

[0103] S1021: Read detailed information of each crawler project from the data record table T1, including the project, website name, website link, SQL library name, SQL table name, incremental mode and alarm days.

[0104] S1022: Configure the corresponding data query strategy according to the incremental mode of each project. For example, if the incremental mode is "daily", it is necessary to query the data update status of the previous day.

[0105] S1023: For each project, a corresponding database query operation is performed to obtain the data update status of the previous day. This usually involves connecting to the corresponding SQL library and querying the incremental data in the specified table.

[0106] S1024: Calculate the latest amount of data added to the database for each project based on the query results.

[0107] S1025: Update the latest data increment of each project to the real-time monitoring table T2. For each project, create or update a record including the project name and the latest incremental data amount.

[0108] S1026: At the same time, the latest incremental data volume is updated to the monitoring history table T3. For each project, a record is created or updated, including the project name, the query timestamp (usually the current time) and the latest incremental data volume.

[0109] Specifically, you can read the records of all projects from T1. For each project Pi, determine the date range of the query based on the incremental mode (such as daily). Use SQL query statements to obtain the data update status of the previous day from the corresponding SQL library Di and table Ti, and calculate the latest amount of data added to the library △Qi.

[0110] For each project Pi, find or create a record in T2, including the project name Pi and the incremental data amount △Qi. Similarly, for each project Pi, find or create a record in T3, including the project name Pi, the current timestamp t and the incremental data amount △Qi.

[0111] In this way, it can be ensured that both the real-time monitoring table T2 and the monitoring history table T3 can accurately reflect the latest data updates of each crawler project. This is helpful for subsequent monitoring, data analysis, and troubleshooting.

[0112] S103, reading the latest incremental data from the real-time monitoring table T2 and adding it to the monitoring email attachment table T4.

[0113] The latest incremental data volume of each project can be read from the real-time monitoring table T2, and the records of all projects can be obtained from T2. Each record contains the project name and the latest incremental data volume. For each project, a new record is created, including the project name and the latest incremental data volume. These newly constructed records are added to the monitoring email attachment table T4.

[0114] Specifically, we can read the records of all projects from T2, obtain the latest incremental data volume ΔQi of each project, and for each project Pi, obtain the latest incremental data volume ΔQi from T2, create a new record containing the project name Pi and the latest incremental data volume ΔQi. Add these records to T4 to form a table containing the latest incremental data volume of all projects.

[0115] For example, if there are two projects in T2, namely "ProjectA" and "ProjectB", and their latest incremental data volumes are 1000 and 1500 respectively, then:

[0116] The incremental data volume of "ProjectA" read from T2 is 1000.

[0117] The incremental data volume of "ProjectB" read from T2 is 1500.

[0118] Create two records in T4:

[0119] One record is "ProjectA,1000".

[0120] Another record is "ProjectB,1500".

[0121] In this way, T4 will contain the latest incremental data of all projects, which can be sent to relevant personnel as attachments to monitoring emails to let them know the latest status of each project. This helps maintain transparency and ensures that all relevant personnel can get the information they need in a timely manner.

[0122] S104, check whether the current time is the first day of the month, if so, read the data volume of the entire month of last month from the monitoring history table T3 and add it to the monitoring email attachment table T4.

[0123] First, obtain the current date and time to determine whether it is the first day of the month. Check whether the current date is the first day of the current month. This can be achieved by comparing whether the day part of the current date is 1. If it is indeed the first day of the month, then read the data volume for the entire previous month from the monitoring history table T3. This involves querying all data records within the time range corresponding to the previous month in T3. For each project, summarize the data volume for the entire previous month and create new records. Add these records to the monitoring email attachment table T4.

[0124] For example, if the current time is September 6, 2024, it means it is not the first day of the month, so there is no need to perform the operation of reading the data of the previous month. But if the current time is September 1, 2024, the above steps should be executed to read the data volume of August from T3 and add these data volumes to T4.

[0125] Doing so can ensure that at the beginning of the month, all relevant personnel can receive a report containing the data volume of each project in the previous month for further analysis and decision-making.

[0126] S105. After obtaining the newly added data volume of all crawler projects, according to the daily warning limit values and the limit values of the number of days without increment set for different projects, tag the abnormal projects and output them to the monitoring email alarm attachment table T5 to send the alarm email attachment. For the situation without abnormalities, send a no-abnormality email and attach the monitoring email attachment table T4 to the email so that the daily increment situation of all projects can be directly viewed by opening the attachment when needed.

[0127] The latest increment data volume of each crawler project can be read from the real-time monitoring table T2, and the daily warning limit value and the limit value of the number of days without increment of each project can be read from the data record table T1. According to the daily warning limit value and the limit value of the number of days without increment, judge whether there is an abnormal situation for each project. If there is an abnormal situation for the project, mark the project in the monitoring email alarm attachment table T5. According to whether there is an abnormal situation, decide whether to send an alarm email or a no-abnormality email and attach T5 or T4 accordingly.

[0128] Specifically, the latest increment data volume △Qi of each project can be obtained from T2, and the daily warning limit value Li and the limit value of the number of days without increment Ai of each project can be obtained from T1.

[0129] For each project Pi, if △Qi < Li, it is considered that the project has a risk of insufficient data update. If the number of consecutive days without increment of the project exceeds Ai, it is considered that the project has a risk of not being updated for a long time.

[0130] If any of the above abnormal situations exist in project Pi, the name of the project and the description of the abnormal situation shall be recorded in T5.

[0131] If there are any abnormal items, an alarm email will be sent with T5 as an attachment to notify relevant personnel that there are abnormal items that need attention.

[0132] If there are no abnormal items, a normal email will be sent with T4 as an attachment to inform all items that the daily increments are normal.

[0133] Through such a process, it can ensure that relevant personnel are notified in time when there are abnormal situations, and a detailed monitoring report is provided when everything is normal, thus ensuring the stable operation of the project and continuous monitoring of data.

[0134] See also Figure 2 Another real-time monitoring and early warning method based on incremental crawler data provided by the present invention includes:

[0135] 1) First, add the project tasks to be monitored to the spiders_monitoring.xlsx table, which includes the project / website name / website link / sql library name / sql table name / incremental mode (1 without 2-3_) / alarm days, etc.

[0136] 2) Read spiders_monitoring.xlsx table, and query the data update of each crawler project yesterday according to the incremental mode. Many incremental tasks are run all day or multiple times a day. It is not suitable to check today's increments uniformly. Important project scripts have real-time separate alarm systems, which are not specifically explained in this system. After getting the latest amount of data added to the database, update it to the monitoring table monitoring according to the situation of each project.

[0137] 3) The latest incremental data is updated to the crawler monitoring history table monitoring_log at the same time

[0138] 4) Read the latest increment from the monitoring table monitoring to the monitoring email attachment.xlsx

[0139] 5) Check if it is the first day of the month. If it is, read the total data volume for the whole month from the monitoring history table monitoring_log and save it in monitoring email_attachment.xlsx

[0140] 6) Now that the amount of new data for all crawler projects has been obtained, according to the daily warning limit and no-increment day limit set for different projects, the abnormal projects are labeled and output to the monitoring email_alarm attachment.xlsx to send the alarm email attachment. For normal situations, a no-abnormal email is sent and the monitoring email_attachment.xlsx is attached to the email, so that you can directly open the attachment to view the daily increment of all projects when needed.

[0141] According to a specific implementation of the embodiment of the present disclosure, adding the project tasks to be monitored to the data recording table T1 includes:

[0142] Define data record table T1. Record table T1 is a two-dimensional table. Each row represents a project task and each column represents the attribute of the project. Table T1 is represented as a matrix Mt1. Each row of the matrix is ​​a vector Ri, which is used to represent a project record.

[0143] For a new project, construct a vector Rnw=(P,W,L,D,T,M,A), where P represents the project, W represents the website name, L represents the website link, D represents the SQL library name, T represents the SQL table name, M represents the incremental mode, and A represents the number of days to alarm.

[0144] Add the vector Rnw to the matrix Mt1 to form a new matrix Mt1':

[0145]

[0146] Represents the union operation of sets;

[0147] based on Update and save the modified table T1 so that the new record is persistently stored.

[0148] According to a specific implementation of the embodiment of the present disclosure, the data record table T1 is read, and the data update status of each crawler project on the previous day is queried according to the incremental mode. After obtaining the latest amount of data added to the database, it is updated to the real-time monitoring table T2 according to the status of each project, including:

[0149] The data record table T1 is represented as:

[0150]

[0151] Pi is the name of the i-th project, Wi is the website name corresponding to the i-th project, Li is the website link corresponding to the i-th project, Di is the data repository corresponding to the i-th project, Ti is the data table corresponding to the i-th project, Mi is the incremental mode of the i-th project, and Ai is the number of alarm days for the i-th project;

[0152] The real-time monitoring table T2 is expressed as:

[0153]

[0154] Pi is the name of the i-th project, △Qi is the most recent data increment of the i-th project;

[0155] Read all project records from the data record table T1. For each project i, query the data update status of the crawler project on the previous day according to the incremental mode Mi. The query result is △Qi;

[0156] Update the incremental data volume △Qi obtained from the query to the corresponding item Pi record in the real-time monitoring table T2, expressed as:

[0157]

[0158]

[0159]

[0160] The Qry function indicates that the data update status of the previous day is queried from the table Ti in the database Di according to the incremental mode Mi, and T2i indicates the i-th item record in the updated T2 table;

[0161] Update the incremental data of all projects to T2 to form a complete real-time monitoring table T2:

[0162] .

[0163] According to a specific implementation of the embodiment of the present disclosure, the updating of the latest incremental data amount to the monitoring history table T3 at the same time includes:

[0164] Define T3 as follows:

[0165]

[0166] Pi is the name of the i-th item, It is a timestamp, indicating the time point at which the incremental data is recorded. is the i-th item at time The amount of incremental data;

[0167] For each project i, add the latest queried incremental data volume △Qi to the monitoring history table T3;

[0168] For each item i, construct a new record (Pi, t, △Qi) and add it to T3, where t is the current time, expressed as:

[0169]

[0170]

[0171]

[0172] The CTm() function means getting the current timestamp;

[0173] Update the latest incremental data of all projects to T3 to form a complete monitoring history table T3:

[0174] .

[0175] According to a specific implementation of the embodiment of the present disclosure, the latest incremental data is read from the real-time monitoring table T2 and added to the monitoring email attachment table T4; including:

[0176] Read the records of all items from the real-time monitoring table T2. For each item i, read the latest incremental data volume ΔQi from T2, construct this information into a new record, and add the newly constructed record to the monitoring email attachment table T4.

[0177] According to a specific implementation of the embodiment of the present disclosure, the data volume of the entire month of the previous month is read from the monitoring history table T3 and added to the monitoring email attachment table T4, including:

[0178] According to the current time t, calculate the start time and end time of last month;

[0179] For each item i, read all incremental data from T3 between the start time and the end time. ;

[0180] Sum up all incremental data in the time period to get the data volume of project i last month ;

[0181] For each item i, construct a new record And add it to T4.

[0182] According to a specific implementation of the embodiment of the present disclosure, the daily warning limit value and the no-increment day limit value set according to different projects are used to label the abnormal projects and output them to the monitoring email alarm attachment table T5 to send the alarm email attachment, including:

[0183] Read the latest incremental data volume △Qi of each project from the real-time monitoring table T2, and read the relevant historical data from the monitoring history table T3;

[0184] For each project Pi, determine the warning limit according to the daily warning limit value Li and the non-increment days limit value Ai set in the data record table T1.

[0185] For each project, check whether its latest incremental data volume △Qi is lower than the set daily warning limit value Li. If △Qi < Li, it is considered that the project has a risk of insufficient data update.

[0186] Obtain the data update records of project Pi within the recent preset time from the monitoring history table T3, calculate the consecutive non-increment days. If the consecutive non-increment days exceed the non-increment days limit value Ai of the project, it is considered that the project has a risk of not being updated for a long time.

[0187] If project Pi satisfies △Qi < Li or the consecutive non-increment days exceed Ai, mark the project as abnormal and record it in the monitoring email alarm attachment table T5.

[0188] Output all the project records (Pi, △Qi) marked as abnormal to the monitoring email alarm attachment table T5 and prepare to send an alarm email.

[0189] According to a specific implementation manner of the embodiment of the present disclosure, for the case of no abnormality, send a no-abnormality email and attach the monitoring email attachment table T4 to the email, including:

[0190] Confirm that all monitored projects have not reached the warning limit value or the non-increment days limit value.

[0191] Compose the content of the email body, indicating that all projects are currently in a normal state and no abnormalities are found.

[0192] Add the monitoring email attachment table T4 as an email attachment.

[0193] Corresponding to the above method embodiment, see Figure 3 , the embodiment of the present invention also discloses a real-time monitoring and warning device 30 based on incremental crawler data, including:

[0194] An adding module 301 adds the project tasks to be monitored to the data record table T1. The data record table T1 includes the affiliated project, website name, website link, sql library name, sql table name, incremental mode, and alarm days.

[0195] An updating module 302 reads the data record table T1, queries the data update situation of each crawler project yesterday according to the incremental mode, obtains the latest incrementally stored data volume, and then updates it to the real-time monitoring table T2 according to the situation of each project. At the same time, update the latest incremental data volume to the monitoring history table T3.

[0196] The reading module 303 reads the latest incremental data from the real-time monitoring table T2 and adds it to the monitoring email attachment table T4;

[0197] Check module 304, check whether the current time is the first day of the month, if yes, read the data volume of the whole month of last month from monitoring history table T3 and add it to monitoring email attachment table T4;

[0198] The sending module 305, after obtaining the newly added data volume of all crawler projects, labels the abnormal projects according to the daily warning limit values ​​and the limit values ​​of no-increment days set for different projects, and outputs them to the monitoring email alarm attachment table T5 to send the alarm email attachment. For normal situations, a normal email is sent, and the monitoring email attachment table T4 is attached to the email, so that the attachment can be directly opened to view the daily increments of all projects when needed.

[0199] See also Figure 4 The embodiment of the present invention further provides an electronic device 60, the electronic device comprising:

[0200] at least one processor; and,

[0201] a memory communicatively connected to the at least one processor; wherein,

[0202] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time monitoring and early warning method based on incremental crawler data in the aforementioned method embodiment.

[0203] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned method embodiment.

[0204] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the real-time monitoring and early warning method based on incremental crawler data in the aforementioned method embodiment.

[0205] Reference below Figure 4, which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0206] like Figure 4 As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0207] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gage, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0208] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0209] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A real-time monitoring and early warning method based on incremental crawler data, characterized in that: include: Add the project tasks that need to be monitored to the data record table T1, which contains the project, website name, website link, sql library name, sql table name, incremental mode and alarm days; Read the data record table T1, query the data update status of each crawler project on the previous day according to the incremental mode, and after obtaining the latest amount of data added to the database, update it to the real-time monitoring table T2 according to the situation of each project, and update the latest incremental data to the monitoring history table T3 at the same time; Read the latest incremental data from the real-time monitoring table T2 and add it to the monitoring email attachment table T4; Check the current time to see if it is the first day of the month. If so, read the data volume of the entire month of the previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4; After obtaining the newly added data volume of all crawler projects, according to the daily warning limit values ​​and no-increment days limit values ​​set for different projects, the abnormal projects are labeled and output to the monitoring email alarm attachment table T5 to send the alarm email attachment. For normal situations, a normal email is sent and the monitoring email attachment table T4 is attached to the email so that the attachment can be directly opened to view the daily increment of all projects when needed.

2. The method according to claim 1, characterized in that The step of adding the project tasks to be monitored to the data record table T1 includes: Define data record table T1. Record table T1 is a two-dimensional table. Each row represents a project task and each column represents the attribute of the project. Table T1 is represented as a matrix Mt1. Each row of the matrix is ​​a vector Ri, which is used to represent a project record. For a new project, construct a vector Rnw=(P,W,L,D,T,M,A), where P represents the project, W represents the website name, L represents the website link, D represents the SQL library name, T represents the SQL table name, M represents the incremental mode, and A represents the number of days to alarm. Add the vector Rnw to the matrix Mt1 to form a new matrix Mt1': Represents the union operation of sets; based on Update and save the modified table T1 so that the new record is persistently stored.

3. The method according to claim 2, characterized in that The data record table T1 is read, and the data update status of each crawler project on the previous day is queried according to the incremental mode. After obtaining the latest amount of data added to the database, it is updated to the real-time monitoring table T2 according to the status of each project, including: The data record table T1 is represented as: Pi is the name of the i-th project, Wi is the website name corresponding to the i-th project, Li is the website link corresponding to the i-th project, Di is the data repository corresponding to the i-th project, Ti is the data table corresponding to the i-th project, Mi is the incremental mode of the i-th project, and Ai is the number of alarm days for the i-th project; The real-time monitoring table T2 is expressed as: Pi is the name of the i-th project, △Qi is the most recent data increment of the i-th project; Read all project records from the data record table T1. For each project i, query the data update status of the crawler project on the previous day according to the incremental mode Mi. The query result is △Qi; Update the incremental data volume △Qi obtained from the query to the corresponding item Pi record in the real-time monitoring table T2, expressed as: The Qry function represents querying the data update situation of the previous day from the table Ti in the database Di according to the incremental mode Mi, and T2i represents the i-th item record in the updated T2 table; Update the incremental data volume of all items to T2 to form a complete real-time monitoring table T2: 。 4. The method according to claim 3, characterized in that The above-mentioned simultaneously update the latest incremental data volume to the monitoring history table T3, including: Define T3 as follows: Pi is the name of the i-th item, It is a timestamp, indicating the time point at which the incremental data is recorded. is the i-th item at time The amount of incremental data; For each item i, add the latest queried incremental data volume △Qi to the monitoring history table T3; For each item i, construct a new record (Pi, t, △Qi) and add it to T3, where t is the current time, expressed as: The CTm() function represents obtaining the current timestamp; Update the latest incremental data volume of all items to T3 to form a complete monitoring history table T3: 。 5. The method according to claim 4, characterized in that The above-mentioned read the latest incremental data from the real-time monitoring table T2 and add it to the monitoring email attachment table T4, including: Read the records of all items from the real-time monitoring table T2. For each item i, read the latest incremental data volume ΔQi from T2, and construct these information into a new record, and add the newly constructed record to the monitoring email attachment table T4.

6. The method according to claim 5, characterized in that The above-mentioned read the data volume of the entire previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4, including: According to the current time t, calculate the start time and end time of the previous month; For each item i, read all incremental data from T3 between the start time and the end time. ; Sum up all incremental data in the time period to get the data volume of project i last month ; For each item i, construct a new record And add it to T4.

7. The method according to claim 6, characterized in that The above-mentioned according to the daily warning limit value and the non-incremental days limit value set for different items, tag the abnormal items and output them to the monitoring email alarm attachment table T5 to send the alarm email attachment, including: Read the latest incremental data volume △Qi of each item from the real-time monitoring table T2, and at the same time read the relevant historical data from the monitoring history table T3; For each item Pi, judge according to the daily warning limit value Li and the non-incremental days limit value Ai set in the data record table T1 to determine the warning limit value; For each item, check whether its latest incremental data volume △Qi is lower than the set daily warning limit value Li. If △Qi < Li, it is considered that the item has a risk of insufficient data update; Obtain the data update records of item Pi within the recent preset time from the monitoring history table T3, calculate the consecutive non-incremental days, and if the consecutive non-incremental days exceed the non-incremental days limit value Ai of the item, it is considered that the item has a risk of not being updated for a long time; If item Pi meets △Qi < Li or the consecutive non-incremental days exceed Ai, mark the item as abnormal and record it in the monitoring email alarm attachment table T5; Output all the item records (Pi, △Qi) marked as abnormal to the monitoring email alarm attachment table T5 and prepare to send the alarm email.

8. The method according to claim 7, characterized in that The above-mentioned for the situation without abnormalities, send an email indicating no abnormalities, and attach the monitoring email attachment table T4 to the email, including: Confirm that none of the monitored items have reached the warning limit value or the non-incremental days limit value; Compose the content of the email body, indicating that all items are currently in a normal state and no abnormalities are found; Add the monitoring email attachment table T4 as an email attachment.

9. A real-time monitoring and early warning device based on incremental crawler data, characterized in that: Including: Add module, add the project tasks to be monitored to the data record table T1, the data record table T1 contains the project, website name, website link, sql library name, sql table name, incremental mode and alarm days; The update module reads the data record table T1, queries the data update status of each crawler project yesterday according to the incremental mode, obtains the latest amount of data added to the database, and updates it to the real-time monitoring table T2 according to the situation of each project. At the same time, the latest incremental data is updated to the monitoring history table T3; The reading module reads the latest incremental data from the real-time monitoring table T2 and adds it to the monitoring email attachment table T4; Check the module to see if the current time is the first day of the month. If so, read the data volume of the entire month of the previous month from the monitoring history table T3 and add it to the monitoring email attachment table T4; After obtaining the newly added data volume of all crawler projects, the sending module labels the abnormal projects and outputs them to the monitoring email alarm attachment table T5 according to the daily warning limit values ​​and no-increment days limit values ​​set for different projects to send alarm email attachments. For normal situations, a normal email is sent and the monitoring email attachment table T4 is attached to the email so that the attachment can be directly opened to view the daily increments of all projects when needed.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time monitoring and early warning method based on incremental crawler data as described in any one of claims 1 to 8.

Images