Data early warning method and device

Abstract

The invention discloses a data early warning method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: obtaining target data through a data acquisition code, writing the target data into a message queue through a data insertion message queue code, obtaining message queue data, obtaining the target data, carrying out calculation statistics to obtain a calculation result, and if the calculation result comprises early warning data, sending the early warning data to a server; and sending the early warning information to an early warning center. According to the embodiment, the timeliness of the collected data can be guaranteed, and the efficiency and accuracy of early warning data determination are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data early warning. Background Art

[0002] At present, in the field of big data, flink is an important processing engine, mainly for distributed processing of stream data and batch data. Flink tasks are the basic components of flink programs, with characteristics such as high performance and fault tolerance. However, the data accuracy of flink tasks after they are launched cannot be monitored and warned in real time. We can only passively receive inaccurate data feedback from users, and then conduct data troubleshooting, locate problems, and solve them. At the same time, the data freshness of flink tasks after they are launched cannot be monitored and warned in real time. We can only obtain information through task backlog alarms and user feedback data update delays. However, judging from the backlog of flink tasks, we cannot give accurate data delay time and cannot handle data warnings in a timely manner. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for data early warning, which can ensure the real-time nature of collected data and achieve timely early warning of problematic data.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a data method is provided, comprising:

[0005] During the execution of the data stream processing task, the target data is obtained through the data acquisition code, and the data acquisition code is generated based on the annotation information pre-configured in the data stream processing task execution code;

[0006] Writing target data into the first message queue through a data insertion message queue code, wherein the data insertion message queue code is generated based on the annotation information;

[0007] Obtain target data from the first message queue, and perform calculations and statistics on the target data according to the configured statistical rule information to obtain calculation results;

[0008] In response to determining that the calculation result includes warning data based on the warning threshold condition, warning information is sent to the warning center.

[0009] Optionally, the data collection code and the data insertion message queue code are generated by:

[0010] When compiling the data stream processing task execution code, the data stream processing task execution code is parsed to obtain the operator types included in the data stream processing task execution code and annotation information corresponding to each operator type;

[0011] For each operator type, the annotation information corresponding to the operator type is processed through aspect-oriented programming technology to generate data collection code and data insertion message queue code corresponding to the operator type.

[0012] Optionally, after obtaining target data from the message queue and calculating and counting the target data according to the configured statistical rule information, and obtaining the calculation result, the method further includes:

[0013] The calculation results are stored regularly according to the time interval set by the user.

[0014] Optionally, the calculation results include a statistical analysis report, and the method further includes:

[0015] Send the statistical analysis report corresponding to the warning data to the warning center so that the warning center can locate the problem based on the statistical analysis report.

[0016] Optionally, the data collection code includes a code for obtaining task information, and the method further includes:

[0017] Obtain result data of the data stream processing task by obtaining the task information code, and write the result data into the second message queue;

[0018] Calculate and count the target data based on the calculation and statistics code to obtain the statistical result, and write the statistical result into the first message queue, wherein the calculation and statistics code is generated based on the annotation information;

[0019] The result data of the second message queue and the Statistical results Compare and get the accuracy index;

[0020] A statistical analysis report including the accuracy index is generated based on the accuracy index and the statistical analysis report.

[0021] According to another aspect of an embodiment of the present invention, a data early warning device is provided, including:

[0022] An acquisition module is used to acquire target data through a data acquisition code during the execution of a data stream processing task, where the data acquisition code is generated based on annotation information pre-configured in the data stream processing task execution code;

[0023] A writing module, used for writing target data into the first message queue through a data insertion message queue code, wherein the data insertion message queue code is generated based on the annotation information;

[0024] The acquisition module is further used to acquire target data from the first message queue, and perform calculation and statistics on the target data according to the configured statistical rule information to obtain a calculation result;

[0025] The sending module is used to send warning information to the warning center in response to determining that the calculation result includes warning data based on the warning threshold condition.

[0026] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data warning method provided by an embodiment of the present invention.

[0027] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for data early warning provided by an embodiment of the present invention is implemented.

[0028] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data early warning method provided by an embodiment of the present invention is implemented.

[0029] An embodiment of the above invention has the following advantages or beneficial effects: it is able to collect target data with heavy tasks through the data collection code generated by annotation parsing, and write the target data into the message queue through the generated data insertion message code. Furthermore, the target data is calculated and counted according to the configured statistical rule information to obtain the calculation result, and an early warning message is sent when the early warning data is included, thereby ensuring the real-time and freshness of data acquisition, and further ensuring the accuracy of the early warning data.

[0030] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0032] Figure 1 This is a schematic diagram of the main steps of the data early warning method according to an embodiment of the present invention;

[0033] Figure 2 It is an example of a statistical analysis report;

[0034] Figure 3 The overall flow chart of data processing;

[0035] Figure 4 The figure shows the annotation collection and analysis process;

[0036] Figure 5 This is a schematic diagram of main modules of a data early warning device according to an embodiment of the present invention;

[0037] Figure 6is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0038] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0040] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0041] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0042] Figure 1 FIG. 1 is a schematic diagram of the main steps of the method for data early warning according to an embodiment of the present invention. Figure 1 As shown, including:

[0043] Step S101, during the execution of the data stream processing task, target data is acquired through a data acquisition code, where the data acquisition code is generated based on annotation information pre-configured in the data stream processing task execution code.

[0044] After the flink (data stream processing) task is started, during its execution, data is automatically acquired through the pre-set data acquisition code, and the data is parsed according to the format of the acquired data to finally obtain the target data. The number of acquired target data can be set according to actual needs, and the number of corresponding data acquisition codes also needs to be set accordingly. For example, in the logistics scenario, the collected target data includes the original waybill information, package information and outbound information, the intermediate waybill package splicing data and outbound information, and the final output data. For the original, intermediate and final data, 3 data acquisition codes need to be set for collection.

[0045] The data collection code is generated based on the annotation information set when the code is compiled. The annotation information is the annotation information set in the code by the R&D personnel when developing the Flink task, and the data collection code is generated by compiling later.

[0046] Optionally, the data collection code includes a data parsing code and a flink task information obtaining code. Specifically, the flink task data is obtained through the flink task information code, and then the data to be collected is obtained and parsed based on the flink task data and the data parsing code to obtain the target data.

[0047] Step S102, writing the target data into the first message queue through the data insertion message queue code, where the data insertion message queue code is generated based on the annotation information.

[0048] After the data analysis operation, the target data is written into the preset first message queue for storage according to the set data insertion message queue code, so as to be used for subsequent data acquisition from the first message queue for calculation and statistics. Among them, the data insertion message queue code is also a code generated based on the annotation information.

[0049] Step S103, obtaining target data from the first message queue, and calculating and counting the target data according to the configured statistical rule information to obtain a calculation result.

[0050] The target data is obtained from the first message queue, and the target data is calculated according to the pre-configured statistical rule information to obtain statistical data. The statistical rule can be determined according to the configured analytical statistical annotation information, or determined to be based on the lineage information between fields. It can include relational operations on fields, such as the total number of parcels in a waybill, which can be set according to actual conditions.

[0051] Optionally, when obtaining target data from the first message queue, data can be obtained from a message queue containing information such as the id (identity identifier) ​​and name of the target flink task according to the statistical task type, and added to the calculation result in subsequent calculation and statistical operations, for example, the calculation result includes a statistical report with the flink task id.

[0052] Step S104: in response to determining that the calculation result includes warning data based on the warning threshold condition, sending warning information to the warning center.

[0053] After obtaining the calculation results, it is necessary to make an early warning judgment. Specifically, the data corresponding to the field in the calculation result is compared with the preset early warning threshold. If the preset early warning threshold is not met, it means that there is early warning data. Early warning information is generated based on the early warning data and sent to the early warning center. Among them, the early warning information can be information containing early warning data, or generated according to the association between the preset early warning information and the field.

[0054] Specifically, for example, the total number of waybills in the calculation result is 105, and the preset warning threshold is 100, then a warning will appear for the total number of waybills, and the generated warning information can be in the form of "Warning on total number of waybills: the total number of waybills is 105" or "Warning: the total number of waybills exceeds the limit".

[0055] According to the data warning method provided by the embodiment of the present invention, the data collection code generated by annotation parsing can collect target data with heavy tasks, and the target data can be written into the message queue through the generated data insertion message code. Furthermore, the target data is calculated and counted according to the configured statistical rule information to obtain the calculation result, and the warning information is sent when the warning data is included, thereby ensuring the real-time and freshness of data acquisition, and further ensuring the accuracy of the warning data determination.

[0056] Optionally, the data acquisition code and the data insertion message queue code are generated in the following manner: when compiling the data flow processing task execution code, the data flow processing task execution code is parsed to obtain the operator types included in the data flow processing task execution code and the annotation information corresponding to each operator type; for each operator type, the annotation information corresponding to the operator type is processed through aspect-oriented programming technology to generate the data acquisition code and the data insertion message queue code corresponding to the operator type.

[0057] The data collection code and data insertion message queue code are implemented through the following steps: First, the annotation information is set by the R&D personnel during the code development stage. When the flink task is executed, the task execution code needs to be compiled. During the compilation, the configured collection annotation information is obtained, which marks different operator types and corresponding annotation information. The annotation information includes the entity class, field name, field type, etc. of the data. For each operator, it is necessary to automatically generate the data collection code and data insertion message queue code corresponding to each operator type based on the annotation information through AOP technology (aspect-oriented programming technology).

[0058] In one embodiment, the collection annotation includes three types: data source collection, data output operator collection, and calculation operator collection. The configuration process is as follows:

[0059] Data source collection: This annotation marks the source operator, which is used to parse the source data according to the structure and write it into the message queue. This annotation needs to mark the entity class of the data corresponding to the data source or the data source name, field name, field Chinese name, and field type.

[0060] Data output operator collection: This annotation marks the sink operator, which is used to parse the sink data according to the structure and write it into the message queue. This annotation not only requires the annotation of the entity class of the data corresponding to the data source or the data source name, field name, field Chinese name, field type, but also the upstream field of the field.

[0061] Computing operator collection: Except for the data source and sink operators, the other operators collect the calculation data of the intermediate process. It is necessary to define the table name, field name, field Chinese name, field type, source field, etc. for parsing. Similar to the sink output operator collection, you can set the upstream field or set the source to the upstream field.

[0062] Optionally, for simple statistical reports, you can also configure task annotations without configuring them. Through the jar package, the method type of the Flink code is automatically parsed, the source, sink, and other three modes are automatically parsed, the data structure is automatically parsed, and the data is written to the message queue.

[0063] Optionally, after obtaining the target data from the message queue and calculating and counting the target data according to the configured statistical rule information, the method further includes: storing the calculation results regularly according to the time interval set by the user. After obtaining the calculation results, they can be stored regularly according to the time interval preset by the user. At the same time, other calculation data can be deleted to avoid occupying more memory.

[0064] Optionally, the stored calculation results include statistical analysis reports. At the same time, users are supported to view and download statistical analysis reports based on Flink task ID, name and other information through a visual interface.

[0065] Optionally, the calculation result includes a statistical analysis report, and the method further includes: sending the statistical analysis report corresponding to the warning data to the warning center, so that the warning center locates the problem based on the statistical analysis report. When sending the warning information to the warning center, the warning report containing the warning data can be sent at the same time.

[0066] Statistical analysis reports can be Figure 2 As shown, the output attributes include waybill number, package data volume and delivery time; the data corresponding to each field includes output field, source field, accuracy, freshness (seconds), random sampling maximum value, random sampling minimum value and random sampling data (default 10); specific statistical values, such as the accuracy of package data volume is 80%.

[0067] Optionally, the information available for user setting in the above process is as follows: Support users to set configurations for analysis of collected information, for example, data output operator collection, the waybill number field of the data object is equal to the waybill number field of the waybill information table data, the number of waybill packages is equal to the sum of the package details table data, etc., relational operations support sum, max, =, >, <, concat, etc.; Support users to set the accuracy of the flink task level, freshness threshold, time interval for writing statistical results to storage, groups and people who receive alarm information, and other information.

[0068] Optionally, the data acquisition code includes a task information acquisition code, and the method also includes: acquiring result data of the data flow processing task by acquiring the task information code, and writing the result data into the second message queue; performing calculation statistics on the target data based on the calculation statistics code to obtain statistical results, and writing the statistical results into the first message queue, wherein the calculation statistics code is generated based on the annotation information; comparing the result data of the second message queue with the statistical results of the first message queue to obtain an accuracy index; generating a statistical analysis report containing the accuracy index based on the accuracy index and the statistical analysis report.

[0069] The result data of the flink task is obtained by parsing the task information code, such as the total number of waybills, and the result data is written to the second message queue. The target data is calculated and counted based on the calculation statistics code, that is, the original data in the target data is calculated and counted based on the calculation statistics code to obtain the statistical results, and the statistical results are written to the first message queue. Among them, the calculation statistics code is also generated based on the annotation information and can be pre-configured by the user, such as the code for calculating the total number of waybills. The result data obtained is compared with the statistical results to obtain the accuracy index, and the accuracy index is added to the statistical analysis report.

[0070] According to the data warning method provided by the embodiment of the present invention, the flink task data can be obtained through annotation information, and the data is automatically parsed and written into the message queue, and the calculation statistics are calculated according to the statistical rules to obtain the calculation results, and it is determined whether there is warning data. If so, the warning information is sent, which ensures the real-time and freshness of data acquisition and helps to calculate the accuracy of statistical results. At the same time, the warning data is judged according to the statistical rules and the warning threshold conditions, which ensures the accuracy and timeliness of the warning data.

[0071] like Figure 3 The figure shows the overall flow chart of data processing. R&D personnel conduct flink development and set statistical rules to send to the task statistical analysis system. R&D personnel start or automatically parse flink tasks by adding annotations. Flink tasks are written to the message queue after data analysis. The task statistics task obtains the statistical information of the message queue data and the task statistical analysis system, calculates and counts, and pushes the statistical results to the task statistical analysis system at a fixed time. Among them, the statistical information is obtained according to the flink task dimension. The task statistical analysis system determines whether to trigger an early warning based on the statistical results. If so, an alarm is sent to the R&D personnel or user side. If not, no alarm is triggered. R&D personnel or users can query and download the statistical report of the task statistical analysis system.

[0072] like Figure 4 The figure shows the annotation collection and parsing process. The original data of the flink task includes waybill information, package information, and outbound information. The flink task needs to merge the three types of data and output the merged data. For the original data, the intermediate merged data, and the final output data, three annotation parsing or parsing according to the source and sink method types are set to obtain parsed data, and the parsed data is written to the message queue. The parsed data includes, for example, sinkTable (output table), which specifically includes waybill number, total number of packages, and outbound time. The parsed data is the data obtained by simple calculation based on the flink task data. For example, the flink task data is the waybill number, package information, and other data of all waybills, and the parsed data is the total number of waybills for all waybills.

[0073] Figure 5 is a schematic diagram of a data early warning device 500 provided in an embodiment of the present invention, such as Figure 5 As shown, the data early warning device 500 of the embodiment of the present invention mainly includes:

[0074] An acquisition module 501 is used to acquire target data through a data acquisition code during the execution of a data stream processing task, where the data acquisition code is generated based on annotation information pre-configured in the data stream processing task execution code;

[0075] A writing module 502, used to write target data into a message queue through a data insertion first message queue code, wherein the data insertion message queue code is generated based on annotation information;

[0076] The acquisition module 501 is further used to acquire target data from the first message queue, and perform calculation and statistics on the target data according to the configured statistical rule information to obtain a calculation result;

[0077] The sending module 503 is used to send warning information to the warning center in response to determining that the calculation result includes warning data based on the warning threshold condition.

[0078] The data warning device provided in an embodiment of the present invention can collect target data with heavy tasks through the data collection code generated by annotation parsing, and write the target data into the message queue through the generated data insertion message code. Furthermore, the target data is calculated and counted according to the configured statistical rule information to obtain the calculation result, and the warning information is sent when the warning data is included, thereby ensuring the real-time and freshness of data acquisition, and further ensuring the accuracy of the warning data determination.

[0079] Optionally, the data collection code and the data insertion message queue code are generated by:

[0080] When compiling the data stream processing task execution code, the data stream processing task execution code is parsed to obtain the operator types included in the data stream processing task execution code and annotation information corresponding to each operator type;

[0081] For each operator type, the annotation information corresponding to the operator type is processed through aspect-oriented programming technology to generate data collection code and data insertion message queue code corresponding to the operator type.

[0082] Optionally, the device further includes: a storage module 504 (not shown in the figure), which is used to store the calculation results at regular intervals according to a time interval set by a user.

[0083] Optionally, the calculation result includes a statistical analysis report, and the device further includes: a sending module 505 (not shown in the figure), which is used to send the statistical analysis report corresponding to the warning data to the warning center, so that the warning center can locate the problem based on the statistical analysis report.

[0084] Optionally, the data acquisition code includes a task information acquisition code, and the device also includes: acquiring result data of the data flow processing task by acquiring the task information code, and writing the result data into a second message queue; performing calculation statistics on the target data based on a calculation statistics code to obtain statistical results, and writing the statistical results into the first message queue, wherein the calculation statistics code is generated based on annotation information; comparing the result data of the second message queue with the statistical results of the first message queue to obtain an accuracy index; generating a statistical analysis report containing the accuracy index based on the accuracy index and the statistical analysis report.

[0085] According to the data warning device provided by the embodiment of the present invention, it is possible to obtain flink task data through annotation information, automatically parse and write into the message queue, perform calculation statistics according to statistical rules, obtain calculation results, and determine whether there is warning data. If so, a warning message is sent, which ensures the real-time and freshness of data acquisition and helps to calculate the accuracy of statistical results. At the same time, the warning data is judged according to the statistical rules and warning threshold conditions, which ensures the accuracy and timeliness of the warning data.

[0086] Figure 6 An exemplary system architecture 600 is shown to which the method for data early warning or the apparatus for data early warning according to the embodiment of the present invention can be applied.

[0087] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0088] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0089] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0090] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the data warning request, and feed back the processing result (such as warning information - only an example) to the terminal device.

[0091] It should be noted that the data warning method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data warning device is generally set in the server 605 .

[0092] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0093] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 7 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0094] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0095] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0096] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0097] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be arranged in a processor. For example, they may be described as: a processor includes an acquisition module, a writing module, and a sending module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the acquisition module may also be described as "a module for acquiring target data through data acquisition code during the execution of a data stream processing task."

[0100] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0101] During the execution of the data stream processing task, the target data is obtained through the data acquisition code, and the data acquisition code is generated based on the annotation information pre-configured in the data stream processing task execution code;

[0102] Writing target data into the first message queue through a data insertion message queue code, wherein the data insertion message queue code is generated based on the annotation information;

[0103] Obtain target data from the first message queue, and perform calculations and statistics on the target data according to the configured statistical rule information to obtain calculation results;

[0104] In response to determining that the calculation result includes warning data based on the warning threshold condition, warning information is sent to the warning center.

[0105] According to the technical solution of the embodiment of the present invention, the target data of the heavy task can be collected through the data collection code generated by annotation parsing, and the target data can be written into the message queue through the generated data insertion message code. Furthermore, the target data is calculated and counted according to the configured statistical rule information to obtain the calculation result, and the warning information is sent when the warning data is included, thereby ensuring the real-time and freshness of data acquisition, and further ensuring the accuracy of the warning data determination.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data early warning method, characterized in that: include: During the execution of the data stream processing task, target data is acquired through a data acquisition code, wherein the data acquisition code is generated based on annotation information pre-configured in the data stream processing task execution code; Writing the target data into the first message queue through a data insertion message queue code, wherein the data insertion message queue code is generated based on the annotation information; Acquire the target data from the first message queue, and perform calculation and statistics on the target data according to the configured statistical rule information to obtain a calculation result; In response to determining that the calculation result includes warning data based on the warning threshold condition, warning information is sent to a warning center.

2. The method according to claim 1, characterized in that The data collection code and the data insertion message queue code are generated in the following way: When compiling the data stream processing task execution code, parsing the data stream processing task execution code to obtain operator types included in the data stream processing task execution code and annotation information corresponding to each operator type; For each operator type, the annotation information corresponding to the operator type is processed by aspect-oriented programming technology to generate data collection code and data insertion message queue code corresponding to the operator type.

3. The method according to claim 1, characterized in that The target data is obtained from the message queue, and the target data is calculated and counted according to the configured statistical rule information. After the calculation result is obtained, the method further includes: The calculation results are stored regularly according to the time interval set by the user.

4. The method according to any one of claims 1 to 3, characterized in that The calculation results include a statistical analysis report, and the method further includes: The statistical analysis report corresponding to the warning data is sent to the early warning center, so that the early warning center can locate the problem based on the statistical analysis report.

5. The method according to claim 4, characterized in that The data collection code includes a code for obtaining task information, and the method further includes: Obtaining result data of the data stream processing task through the obtaining task information code, and writing the result data into the second message queue; Performing calculation statistics on the target data based on a calculation statistics code to obtain a statistical result, and writing the statistical result into the first message queue, wherein the calculation statistics code is generated based on the annotation information; Compare the result data of the second message queue with the statistical results of the first message queue to obtain an accuracy index; A statistical analysis report including the accuracy indicator is generated based on the accuracy indicator and the statistical analysis report.

6. A data early warning device, characterized in that: include: An acquisition module, used to acquire target data through a data acquisition code during the execution of a data stream processing task, wherein the data acquisition code is generated based on annotation information pre-configured in the data stream processing task execution code; A writing module, used for writing the target data into the first message queue through a data insertion message queue code, wherein the data insertion message queue code is generated based on the annotation information; The acquisition module is further used to acquire the target data from the first message queue, and perform calculation and statistics on the target data according to the configured statistical rule information to obtain a calculation result; The sending module is used to send warning information to the warning center in response to determining that the calculation result includes warning data based on the warning threshold condition.

7. The device according to claim 6, characterized in that The data collection code and the data insertion message queue code are generated in the following way: When compiling the data stream processing task execution code, parsing the data stream processing task execution code to obtain operator types included in the data stream processing task execution code and annotation information corresponding to each operator type; For each operator type, the annotation information corresponding to the operator type is processed by aspect-oriented programming technology to generate data collection code and data insertion message queue code corresponding to the operator type.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images