Stream data processing method, device and equipment and storage medium

Abstract

The application provides a stream data processing method, device, equipment and storage medium. The method comprises the following steps: receiving stream data, a label of the stream data and time information of the stream data sent by a stream processing platform; determining a historical stream data table from a historical stream database according to the label of the stream data; updating target historical stream data in the historical stream data table to the stream data according to the historical time information and the time information, to obtain an updated historical stream data table; determining a first data volume of the stream data and a second data volume of the historical stream data in the updated historical stream data table; determining a data processing rule corresponding to the label according to the label of the stream data; and obtaining a processing result according to the data processing rule, the stream data and the historical stream data in the updated historical stream data table. The method improves the processing effect of the stream data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a stream data processing method and device, equipment and a storage medium. BACKGROUND

[0002] Data alarm refers to that when data is abnormal or exceeds a preset threshold in a data processing process, the system automatically sends a warning to inform relevant personnel to process. Data alarm can help enterprises to discover data abnormalities in time, avoid data leakage, data loss and other risks, and protect data security. At the same time, data alarm can also help enterprises to discover business problems in time, improve business efficiency and quality. Common data alarms include system performance alarms, network anomaly alarms, security alarms, etc.

[0003] At present, when the existing data processing method performs data alarm, the current period statistical value and the last period statistical value are compared to obtain the growth or decline percentage, so as to know whether the data has a sudden increase or sudden decrease change, and then data alarm is performed.

[0004] However, the existing data alarm has the problem of poor data processing effect when performing data alarm. SUMMARY

[0005] The present application provides a stream data processing method, device, equipment and storage medium to solve the problem of poor data processing effect in the prior art.

[0006] In a first aspect, the present application provides a stream data processing method applied to a stream processing system, and the method comprises the following steps:

[0007] Receiving stream data, a label of the stream data and time information of the stream data sent by a stream processing platform;

[0008] According to the label of the stream data, a historical stream data table is determined from a historical stream database, wherein the historical stream data table is a data table corresponding to the label, and the historical stream data table stores historical stream data and historical time information of the historical stream data;

[0009] If the data amount of the historical stream data in the historical stream data table meets a preset data amount requirement, the target historical stream data in the historical stream data table is updated to the stream data according to the historical time information and the time information, to obtain an updated historical stream data table;

[0010] The first data amount of the stream data and the second data amount of the historical stream data in the updated historical stream data table are determined;

[0011] If the first data amount and the second data amount satisfy a preset data amount ratio requirement, a data processing rule corresponding to the label of the stream data is determined according to the label of the stream data.

[0012] The stream data and the historical stream data in the historical stream data table are updated according to the data processing rule, so that a processing result is obtained.

[0013] In the present application, the historical stream data table is determined from the historical stream data database according to the label of the stream data, comprising:

[0014] The historical label matching the label of the stream data is queried from the historical stream data database according to the label of the stream data.

[0015] If the historical label exists, the historical stream data table corresponding to the historical label is determined.

[0016] In the present application, after the historical label matching the label of the stream data is queried from the historical stream data database according to the label of the stream data, the method further comprises:

[0017] If the historical label does not exist, a target historical stream data table is constructed in the historical stream data database, wherein the target historical stream data table is a data table corresponding to the label of the stream data.

[0018] The stream data and the time information of the stream data are written into the target historical stream data table, so that the step of receiving the stream data, the label of the stream data and the time information of the stream data is executed.

[0019] In the present application, after the historical stream data table is determined from the historical stream data database according to the label of the stream data, the method further comprises:

[0020] If the data amount of the historical stream data does not satisfy the preset data amount requirement, the stream data and the time information of the stream data are written into the historical stream data table, and the step of receiving the stream data, the label of the stream data and the time information of the stream data is executed until the data amount in the historical stream data table satisfies the preset data amount requirement.

[0021] In the present application, the first data amount of the stream data and the second data amount of the historical stream data in the historical stream data table are determined, comprising:

[0022] The stream data and the historical stream data in the historical stream data table are updated according to the historical time information of the historical stream data in the historical stream data table.

[0023] The stream data and the historical stream data are statistically processed, so that the first data amount of the stream data and the second data amount of the historical stream data in the historical stream data table are determined.

[0024] In the present application, after determining the first data amount of the stream data in the updated historical stream data table and the second data amount of the historical stream data, the method further comprises:

[0025] If the first data amount and the second data amount do not meet the preset data amount comparison requirement, returning to perform the steps of receiving the stream data, the label of the stream data, and the time information of the stream data until the first data amount and the second data amount meet the preset data amount comparison requirement, obtaining the updated historical stream data table.

[0026] In the present application, before determining the data processing rule corresponding to the label according to the label of the stream data if the first data amount and the second data amount meet the preset data amount comparison requirement, the method further comprises:

[0027] Receiving the stream data, the label type, and the initial data processing rule corresponding to the label type of the stream data required by the user for data processing;

[0028] According to the label type and the initial data processing rule, constructing a rule database, the rule database being used to determine the data processing rule corresponding to the label.

[0029] In the present application, according to the data processing rule, the stream data and the historical stream data in the updated historical stream data table, obtaining a processing result, comprising:

[0030] According to the data processing rule, respectively performing data processing on the stream data and the historical stream data, obtaining stream processing data and historical stream processing data;

[0031] Performing difference processing on the stream processing data and the historical stream processing data, obtaining a data difference value;

[0032] According to the data difference value and the stream processing data, obtaining a processing result representing a rate of change;

[0033] In the present application, after obtaining the processing result representing the rate of change according to the first data amount and the historical stream data if the first data amount and the second data amount meet the preset data amount comparison requirement, the method further comprises:

[0034] According to the label and the data processing rule, obtaining a preset alarm threshold value;

[0035] According to the processing result and the alarm threshold value, generating an alarm message;

[0036] Sending the alarm message to a stream data processing platform to display the alarm message to the user.

[0037] In a second aspect, the present application provides a stream data processing device, comprising:

[0038] The receiving module is configured to receive the streaming data, the label of the streaming data, and the time information of the streaming data sent by the streaming platform;

[0039] The first determining module is configured to determine a historical streaming data table from the historical streaming database according to the label of the streaming data, wherein the historical streaming data table is a data table corresponding to the label, and the historical streaming data table stores historical streaming data and historical time information of the historical streaming data;

[0040] The updating module is configured to update target historical streaming data in the historical streaming data table to the streaming data according to the historical time information and the time information to obtain an updated historical streaming data table, if a data amount of the historical streaming data in the historical streaming data table meets a preset data amount requirement.

[0041] The second determining module is configured to determine a first data amount of the streaming data and a second data amount of the historical streaming data in the updated historical streaming data table.

[0042] The third determining module is configured to determine a data processing rule corresponding to the label according to the label of the streaming data, if the first data amount and the second data amount meet a preset data amount ratio requirement.

[0043] The obtaining module is configured to obtain a processing result according to the data processing rule, the streaming data, and the historical streaming data in the updated historical streaming data table.

[0044] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication;

[0045] The memory stores computer execution instructions.

[0046] The processor executes the computer execution instructions stored in the memory to implement the method of the present application.

[0047] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of the present application.

[0048] The stream data processing method, device, equipment and storage medium provided by the present application, by obtaining the label of the stream data, determining the historical stream data which has recorded the label, and by updating the historical stream data, obtaining the means of the stream data and the historical stream data for obtaining the processing result, so that when receiving the massive stream data, the stream data of different labels can be accurately saved, and the saved stream data is processed according to the data processing rule corresponding to the label, so that the massive real-time stream data can be accurately and efficiently processed to obtain the processing result of the massive real-time stream data, thereby facilitating the required alarm work, and realizing the effect of improving the processing of stream data. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 A flowchart of a stream data processing method provided by an embodiment of the present application;

[0051] Figure 2 A flowchart of another stream data processing method provided by an embodiment of the present application;

[0052] Figure 3 A structural diagram of a stream data processing device provided by an embodiment of the present application;

[0053] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application.

[0054] The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is only one of the specific embodiments consistent with the present application, and is not intended to represent all the embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] In the prior art, when a large amount of real-time streaming data is processed to perform alarm, a timing task triggering alarm strategy is generally used, that is, when the strategy is triggered, expressions under each rule are traversed to perform one-by-one judgment, an opentsdb query statement is constructed to call an API to query a current period value and a base period value, and a same / compared value is calculated. However, due to a large amount of real-time streaming data, a large amount of data and a large number of corresponding rules are used, so that a long time is consumed in processing, and a calculation performance problem may occur.

[0057] The embodiment of the present application determines the historical streaming data recorded with the label by acquiring the label of the streaming data, and obtains the means of the streaming data and the historical streaming data used to obtain the processing result by updating the historical streaming data, so that when a large amount of streaming data is received, the streaming data of different labels can be accurately saved, and the saved streaming data is processed according to the data processing rule corresponding to the label, so that the large amount of real-time streaming data can be accurately and efficiently processed to obtain the processing result of the large amount of real-time streaming data, thereby facilitating the required alarm work, and improving the effect of processing the streaming data.

[0058] Therefore, the embodiment of the present application provides a streaming data processing method, device, equipment and storage medium. The technical scheme of the present application and how the technical scheme of the present application solves the above technical problems will be described in detail below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0059] The execution subject of the stream data processing method provided in the embodiments of the present application can be a server. The server can be a mobile phone, a tablet computer, a computer or the like. The embodiments of the present application do not particularly limit the implementation manner of the execution subject, as long as the execution subject can receive the stream data, the label of the stream data and the time information of the stream data sent by the stream processing platform, determine a historical stream data table from a historical stream data database according to the label of the stream data, wherein the historical stream data table is a data table corresponding to the label, and the historical stream data table stores historical stream data and historical time information of the historical stream data, update target historical stream data in the historical stream data table to the stream data according to the historical time information and the time information if the data amount of the historical stream data in the historical stream data table meets a preset data amount requirement, obtain an updated historical stream data table, determine a first data amount of the stream data and a second data amount of the historical stream data in the updated historical stream data table, and determine a data processing rule corresponding to the label according to the label of the stream data if the first data amount and the second data amount meet a preset data amount ratio requirement, and obtain a processing result according to the data processing rule, the stream data in the updated historical stream data table and the historical stream data.

[0060] The big data processing refers to a technology and a method for processing a large-scale data set. The big data processing technology includes data collection, data storage, data processing and data analysis, and the data processing refers to processing data stored in a big data platform, including data cleaning, data conversion, data aggregation and the like. Currently, commonly used big data processing technologies include Hadoop, Spark, Kafka, Hive, Pig and the like.

[0061] Figure 1 A flowchart of a stream data processing method provided in the embodiments of the present application is shown. The execution subject of the method can be a server, and the embodiments of the present application do not particularly limit the execution subject, for example, as shown in FIG. 1, the method can include the following steps. Figure 1

[0062] S101, receiving stream data, a label of the stream data and time information of the stream data sent by a stream processing platform.

[0063] The stream processing platform can refer to a platform for collecting stream data, and the stream processing platform can be a distributed message system platform. In the embodiments of the present application, the stream processing platform can be a Kafka cluster, and the Kafka cluster can be a distributed system composed of multiple Kafka Broker nodes, used for processing a large amount of real-time stream data flow. The Kafka cluster can have the characteristics of high horizontal expansion and high throughput, and is suitable for a massive stream data monitoring scenario.

[0064] ​The streaming data can refer to a data processing manner of processing and analyzing data in a data stream that is continuously generated and changed in real time. For example, the streaming data can be CPU (Central Processing Unit) usage rate data of a server in a normal state.

[0065] The label of the streaming data can represent the type of the streaming data and the source of the streaming data. The type of the streaming data can include CPU usage rate, device temperature, and the like. The source of the streaming data can include the CPU usage rate of which server.

[0066] The time information of the streaming data can refer to a time at which the underlying component pushes the streaming data to the streaming processing platform.

[0067] In S102, a historical streaming data table is determined from a historical streaming database according to a label of the streaming data. The historical streaming data table is a data table corresponding to the label. The historical streaming data table stores historical streaming data and historical time information of the historical streaming data.

[0068] The historical streaming database can refer to a database that stores various historical streaming data tables.

[0069] The historical streaming data table can refer to a table for storing historical streaming data. The historical streaming data table can include stored historical streaming data and historical time information. The data label of the historical streaming data is the same as the label of the streaming data.

[0070] The historical streaming data can refer to streaming data that has been received and stored.

[0071] The historical time information can refer to a historical time at which the underlying component pushes the historical streaming data to the streaming processing platform.

[0072] In the embodiments of the present application, the method of determining the historical streaming data table from the historical streaming database according to the label of the streaming data can include:

[0073] According to the label of the streaming data, a historical label matching the label of the streaming data is queried from the historical streaming database.

[0074] If the historical label exists, a historical streaming data table corresponding to the historical label is determined.

[0075] The historical label can refer to a historical label of the historical streaming data. In the embodiments of the present application, the historical labels of the historical streaming data in one historical streaming data table are the same. The historical label of the historical streaming data in the historical streaming data table can be used as the label of the historical streaming data table.

[0076] In the embodiments of the present application, the historical tags matching the tag of the stream data can be queried from the historical stream database by a state function, or can be queried from the historical stream database by a tag comparison manner.

[0077] In the embodiments of the present application, after the historical tags matching the tag of the stream data are queried from the historical stream database according to the tag of the stream data, the method can further include:

[0078] If the historical tags do not exist, a target historical stream data table corresponding to the tag of the stream data is constructed in the historical stream database;

[0079] The stream data and the time information of the stream data are written into the target historical stream data table, so as to return to execute the step of receiving the stream data, the tag of the stream data and the time information of the stream data.

[0080] In the embodiments of the present application, the historical tags do not exist can mean that the historical stream database does not store the same type of data as the received stream data, that is, the stream data of a new type or a new device.

[0081] The target historical stream data table can mean that when the historical tags matching the tag of the stream data do not exist in the historical stream database, a target historical stream data table is newly constructed in the historical stream database, and the target historical stream data table is used to store the stream data corresponding to the tag.

[0082] The stream data and the time information of the stream data are written into the target historical stream data table can mean a process of recording the stream data and the time information of the stream data in the target historical stream data table.

[0083] In the embodiments of the present application, after the historical stream data table is determined from the historical stream database according to the tag of the stream data, the method can further include:

[0084] If the data amount of the historical stream data does not satisfy the preset data amount requirement, the stream data and the time information of the stream data are written into the historical stream data table, and the step of receiving the stream data, the tag of the stream data and the time information of the stream data is returned to be executed until the data amount in the historical stream data table satisfies the preset data amount requirement.

[0085] In the embodiments of the present application, the data amount can mean the total data number of the historical stream data.

[0086] The preset data amount requirement can refer to whether the data amount of the historical streaming data meets the data amount for data processing. In the embodiment of the present application, in order to improve the calculation effect, the streaming data of the current period and the previous period need to be counted, wherein the data amount in the current period and the previous period can be set to 5 data, that is, the preset data amount requirement can be whether the historical streaming data meets 10.

[0087] When the data amount of the historical streaming data does not meet the preset data amount requirement, after the streaming data and the time information of the streaming data are written into the historical streaming data table, the data is continuously acquired until the data amount in the historical streaming data table meets the preset data amount requirement.

[0088] S103, if the data amount of the historical streaming data in the historical streaming data table meets the preset data amount requirement, the target historical streaming data in the historical streaming data table is updated to the streaming data according to the historical time information and the time information, to obtain an updated historical streaming data table.

[0089] The data amount of the historical streaming data in the historical streaming data table meeting the preset data amount requirement can refer to that the data amount of the historical streaming data has reached the data amount for data processing.

[0090] The target historical streaming data can refer to the data with the earliest historical time information in the historical streaming data.

[0091] The updated historical streaming data table can refer to a data table for updating the streaming data.

[0092] In the embodiment of the present application, after the data amount of the historical streaming data in the historical streaming data table meets the preset data amount requirement, the target historical streaming data with the earliest historical time information in the historical streaming data table can be determined, and the streaming data and the time information are replaced with the target historical streaming data and the target historical time information respectively, thereby obtaining the updated historical streaming data table.

[0093] S104, determining a first data amount of the streaming data and a second data amount of the historical streaming data in the updated historical streaming data table.

[0094] The first data amount can refer to the data amount of the streaming data in the updated historical streaming data table.

[0095] The second data amount can refer to the data amount of the historical streaming data in the updated historical streaming data table.

[0096] In the embodiment of the present application, the method for determining the first data amount of the streaming data and the second data amount of the historical streaming data in the updated historical streaming data table can include:

[0097] According to historical time information of the historical stream data in the historical stream data table, stream data and historical stream data in the historical stream data table are determined to be updated;

[0098] The stream data and the historical stream data are statistically processed to determine a first data amount of the stream data and a second data amount of the historical stream data in the historical stream data table to be updated.

[0099] The statistical processing can refer to quantity statistics of the stream data and the historical stream data. In the embodiment of the application, the stream data can refer to current stream data in a current period, and the historical stream data can refer to last stream data in a last period.

[0100] S105, if the first data amount and the second data amount meet a preset data amount comparison requirement, a data processing rule corresponding to the tag is determined according to the tag of the stream data.

[0101] The data amount comparison requirement can refer to a comparison requirement preset by a user, that is, a required data amount of two data when data processing is performed. In the embodiment of the application, the data amount comparison requirement can refer to that the first data amount is equal to the second data amount, that is, the first data amount and the second data amount each account for half of a total data amount in the historical stream data table to be updated.

[0102] The data processing rule can refer to a rule of how to monitor the stream data according to a data type and equipment of the stream data represented by the tag. The data processing rule can be preset by a human being.

[0103] In the embodiment of the application, the method after the first data amount and the second data amount of the stream data and the historical stream data in the historical stream data table to be updated are determined can further include:

[0104] If the first data amount and the second data amount do not meet the preset data amount comparison requirement, the step of receiving the stream data, the tag of the stream data, and the time information of the stream data is returned to be executed until the first data amount and the second data amount meet the preset data amount comparison requirement, and the historical stream data table to be updated is obtained.

[0105] If the first data amount and the second data amount do not meet the preset data amount comparison requirement, the step of receiving the stream data, the tag of the stream data, and the time information of the stream data is returned to be executed until the first data amount and the second data amount meet the preset data amount comparison requirement, and the historical stream data table to be updated is obtained.

[0106] In the embodiment of the application, the method before the data processing rule corresponding to the tag is determined according to the tag of the stream data if the first data amount and the second data amount meet the preset data amount comparison requirement can further include:

[0107] receiving stream data processed by a user according to needs, sending a label type, and an initial data processing rule corresponding to the label type;

[0108] According to the label type and the initial data processing rule, a rule database is constructed, and the rule database is used to determine a data processing rule corresponding to the label.

[0109] The label type can refer to the type of the label of the stream data, and the initial data processing rule can refer to a processing rule set for different types of labels. Through different types of stream labels, a data processing rule corresponding to the label type can be determined.

[0110] The rule database can be a database for storing the label type and the initial data processing rule. After receiving the label of the stream data, the target label corresponding to the label is determined by pairing, and the target data processing rule corresponding to the target label is determined.

[0111] S106, according to the data processing rule, updating the stream data and the historical stream data in the historical stream data table to obtain a processing result.

[0112] The change rate can refer to the change rate of an index between two adjacent periods or two consecutive time periods.

[0113] In the embodiments of the present application, the method for obtaining a processing result according to the data processing rule, updating the stream data and the historical stream data in the historical stream data table can include:

[0114] According to the data processing rule, the stream data and the historical stream data are respectively processed to obtain stream processing data and historical stream processing data;

[0115] The stream processing data and the historical stream processing data are processed to obtain a data difference value;

[0116] According to the data difference value and the stream processing data, a processing result representing the change rate is obtained.

[0117] The data processing rule can refer to a process of pre-processing the stream data and the historical stream data. For example, when the average value is required to determine the change rate, the data processing rule can refer to obtaining the average value of the stream data and the average value of the historical stream data. When the maximum value is required to determine the change rate, the data processing rule can refer to obtaining the maximum value of the stream data and the maximum value of the historical stream data. Through the setting of the data processing rule, the stream data processing system can automatically switch the processing mode according to the different data types when processing a large amount of data.

[0118] The processing result can be a change rate result. The formula for obtaining the change rate result can be: change rate = (second statistical period statistical value - first statistical period statistical value) / first statistical period statistical value * 100%. The second statistical period statistical value can be the historical stream data after data processing, and the first statistical period statistical value can refer to the stream data after data processing.

[0119] In the embodiment of the present application, if the first data amount and the second data amount meet the preset data amount ratio requirement, the method of obtaining the processing result representing the change rate after the first data amount and the historical stream data can further include:

[0120] According to the label and the data processing rule, a preset alarm threshold is obtained.

[0121] According to the processing result and the alarm threshold, an alarm message is generated.

[0122] The alarm message is sent to the stream data processing platform to show the alarm message to the user.

[0123] The alarm threshold can refer to a critical value when the processing result is in an abnormal state. The alarm threshold can be determined according to the data processing rule. In the embodiment of the present application, after the data processing rule is determined according to the label, the corresponding alarm threshold can be obtained according to the data processing rule. Thus, the judgment in the process of processing a large amount of stream data can be more efficient.

[0124] The alarm message can refer to the alarm message sent or shown to the user.

[0125] The stream data processing method provided by the embodiments of the present application can accurately classify and save stream data of different labels when receiving massive stream data, acquire the classified and saved stream data according to the data processing rules corresponding to the labels, and finally process the classified and saved stream data and the data processing rules, so that the massive real-time stream data can be accurately and efficiently processed to obtain the processing result of the massive real-time stream data, thereby facilitating the required alarm work and improving the processing effect of the stream data.

[0126] Figure 2 The flowchart of another stream data processing method provided by the embodiments of the present application is shown. The execution subject of the method can be a computer, which is not particularly limited in the embodiments, and as shown in the figure, the method can include: Figure 2

[0127] S201, the Kafka cluster collects stream data representing performance monitoring indicators from underlying components, and sends the stream data to Spark Streaming.

[0128] Kafka is a distributed stream processing platform for processing real-time data streams. The Kafka cluster is a collection of multiple Kafka broker nodes, and each node is an independent Kafka server that works together to provide high availability and scalability.

[0129] The underlying components can refer to various devices to be monitored.

[0130] Spark Streaming is a component of Apache Spark that provides real-time data processing and stream data analysis capabilities.

[0131] S202, after receiving the stream data, Spark Streaming queries the list of data values with the same state as the stream data in the memory through a state function.

[0132] Spark Streaming can receive multiple stream data and query the list of data values with the same state in the memory for different stream data.

[0133] The state of the stream data can refer to the identification (key) of the stream data.

[0134] S203, after querying the list of data values, if the historical stream data recorded in the list of data values has reached two statistical periods, the target historical data in the target statistical period representing an earlier time is replaced and updated according to the received stream data.

[0135] ​The replacement update can refer to replacing the target historical data representing the earliest time information in the target statistical period with the stream data. Or, the target historical data representing the earliest time information in the target statistical period is deleted, and the received stream data is written into the corresponding data value list according to the state (key), stream data and timestamp.

[0136] If the historical stream data recorded in the data value list does not reach two statistical periods, the received stream data is written into the corresponding data value list according to the state (key), stream data and timestamp.

[0137] If the data value list is not queried, a data value list is newly created in the memory, and the received stream data is written into the newly created data value list according to the state (key), stream data and timestamp.

[0138] S204, querying the user-preconfigured alarm strategy corresponding to the stream data state from Redis;

[0139] S205, repeating steps S202-S203 until all target historical data in the target statistical period are replaced by the received stream data to obtain an updated data value list;

[0140] S206, according to the alarm strategy, calculating the change rate of the stream data and the historical stream data in the two statistical periods in the updated data value list to obtain a change rate result.

[0141] The alarm strategy can determine the processing mode of the stream data and the historical stream data, so that when a large amount of data is processed, the processing mode of each stream data can be determined according to the state (key) of the received stream data, so that the stream data processing system can accurately process a large amount of stream data and improve the efficiency.

[0142] The change rate = (the statistical value of the second statistical period - the statistical value of the first statistical period) / the statistical value of the first statistical period * 100%.

[0143] S207, if the change rate result reaches the alarm threshold, triggering an alarm and pushing an alarm message to Kafka.

[0144] Another stream data processing method provided by the embodiment of the application can alarm the change rate of a large amount of real-time stream data based on the Kafka cluster and the sparkstreaming technology. Meanwhile, the state function of the sparkstreaming can be used to record the historical data value, and the complete data in the statistical period can be obtained while the stream data is processed.

[0145] Figure 3 A structural schematic diagram of a stream data processing apparatus provided in an embodiment of the present application is shown in FIG. 3. As shown in the figure, the stream data processing apparatus 30 comprises a receiving module 301, a first determining module 302, an updating module 303, a second determining module 304, a third determining module 305, and an obtaining module 306. Wherein: Figure 3

[0146] The receiving module 301 is configured to receive stream data, a tag of the stream data, and time information of the stream data sent by a stream processing platform.

[0147] The first determining module 302 is configured to determine a historical stream data table from a historical stream database according to the tag of the stream data, wherein the historical stream data table is a data table corresponding to the tag, and the historical stream data table stores historical stream data and historical time information of the historical stream data.

[0148] The updating module 303 is configured to update target historical stream data in the historical stream data table to the stream data according to the historical time information and the time information, to obtain an updated historical stream data table, if a data amount of the historical stream data in the historical stream data table meets a preset data amount requirement.

[0149] The second determining module 304 is configured to determine a first data amount of the stream data and a second data amount of the historical stream data in the updated historical stream data table.

[0150] The third determining module 305 is configured to determine a data processing rule corresponding to the tag according to the tag of the stream data, if the first data amount and the second data amount meet a preset data amount ratio requirement.

[0151] The obtaining module 306 is configured to obtain a processing result according to the data processing rule, the stream data and the historical stream data in the updated historical stream data table.

[0152] In the embodiment of the present application, the first determining module 302 is further configured to:

[0153] query a historical tag matching the tag of the stream data from the historical stream database according to the tag of the stream data.

[0154] If the historical tag exists, determine a historical stream data table corresponding to the historical tag.

[0155] In the embodiment of the present application, the first determining module 302 is further configured to:

[0156] If the historical tag does not exist, construct a target historical stream data table in the historical stream database, wherein the target historical stream data table is a data table corresponding to the tag of the stream data. ​

[0157] write the streaming data and the time information of the streaming data into the target historical streaming data table, and return to perform the step of receiving the streaming data, the label of the streaming data, and the time information of the streaming data.

[0158] In the embodiment of the present application, the first determining module 302 is further specifically configured to:

[0159] If the data amount of the historical streaming data does not satisfy the preset data amount requirement, the streaming data and the time information of the streaming data are written into the historical streaming data table, and the step of receiving the streaming data, the label of the streaming data, and the time information of the streaming data is returned to be performed until the data amount in the historical streaming data table satisfies the preset data amount requirement.

[0160] In the embodiment of the present application, the second determining module 304 is further specifically configured to:

[0161] According to the historical time information of the historical streaming data in the historical streaming data table, the streaming data and the historical streaming data in the historical streaming data table are determined to be updated;

[0162] The streaming data and the historical streaming data are statistically processed, and the first data amount of the streaming data and the second data amount of the historical streaming data in the historical streaming data table are determined to be updated.

[0163] In the embodiment of the present application, the second determining module 304 is further specifically configured to:

[0164] If the first data amount and the second data amount do not satisfy the preset data amount ratio requirement, the step of receiving the streaming data, the label of the streaming data, and the time information of the streaming data is returned to be performed until the first data amount and the second data amount satisfy the preset data amount ratio requirement, and the updated historical streaming data table is obtained.

[0165] In the embodiment of the present application, the third determining module 305 is further specifically configured to:

[0166] receive the streaming data, the label type sent by the user according to the need for data processing, and the initial data processing rule corresponding to the label type;

[0167] According to the label type and the initial data processing rule, a rule database is constructed, and the rule database is used to determine the data processing rule corresponding to the label.

[0168] In the embodiment of the present application, the obtaining module 306 is further specifically configured to:

[0169] According to the data processing rule, the streaming data and the historical streaming data are respectively processed to obtain streaming processing data and historical streaming processing data;

[0170] The data difference is obtained by performing a difference operation on the stream data and the historical stream data.

[0171] According to the data difference and the stream data, a processing result representing a rate of change is obtained.

[0172] In the embodiments of the present application, the obtaining module 306 is further specifically configured to:

[0173] According to the label and the data processing rule, a preset alarm threshold is obtained;

[0174] According to the processing result and the alarm threshold, an alarm message is generated;

[0175] The alarm message is sent to the stream data processing platform to show the alarm message to a user.

[0176] The stream data processing device provided in the embodiments of the present application can facilitate the required alarm work and improve the effect of processing stream data.

[0177] Figure 4 The structure schematic diagram of the electronic device provided in the embodiments of the present application is shown in FIG. 4. As shown in FIG. 4, the electronic device 40 includes: Figure 4

[0178] The electronic device 40 can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a communication component 403, and the like. The processor 401, the memory 402, and the communication component 403 are connected through a bus 404.

[0179] In the specific implementation process, the at least one processor 601 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 performs the stream data processing method as described above.

[0180] The specific implementation process of the processor 401 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here again.

[0181] In the above Figure 4 ​In the illustrated embodiment, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0182] The memory can include random access memory (RAM), and can also include non-volatile memory (NVM), such as at least one disk memory.

[0183] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0184] In some embodiments, a computer program product is also provided, which includes a computer program or instructions, which, when executed by a processor, implements the steps of any of the above-mentioned stream data processing methods.

[0185] The specific implementation of each operation can refer to the foregoing embodiments, which will not be repeated here.

[0186] Those of ordinary skill in the art can understand that all or part of the steps of the various methods of the above-mentioned embodiments can be completed by instructions, or by relevant hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0187] To this end, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps of any of the stream data processing methods provided by the embodiments of the present application.

[0188] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0189] According to an aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium.

[0190] Due to the instructions stored in the storage medium, the steps in any of the stream data processing methods provided by the embodiments of the present application can be performed, thus the beneficial effects of any of the stream data processing methods provided by the embodiments of the present application can be achieved, which are described in detail in the foregoing embodiments and will not be repeated here.

[0191] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0192] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A streaming data processing method, characterized in that, Applied to a streaming processing system, the method includes: Receive streaming data, the tags of the streaming data, and the time information of the streaming data sent by the streaming processing platform; Based on the tags of the streaming data, a historical streaming data table is determined from the historical streaming database, wherein the historical streaming data table is a data table corresponding to the tags, and the historical streaming data table stores historical streaming data and historical time information of the historical streaming data; If the amount of historical streaming data in the historical streaming data table meets the preset data amount requirement, then according to the historical time information and the time information, the target historical streaming data in the historical streaming data table is updated to streaming data to obtain an updated historical streaming data table. Determine the first data volume of streaming data and the second data volume of historical streaming data in the updated historical streaming data table; If the first data volume and the second data volume meet the preset data volume comparison requirements, then the data processing rules corresponding to the tags are determined according to the tags of the streaming data; Based on the data processing rules, the streaming data in the updated historical streaming data table, and the historical streaming data, the processing result is obtained; The process of obtaining the processing result based on the data processing rules, the updated streaming data in the historical streaming data table, and the historical streaming data includes: According to the data processing rules, the streaming data and the historical streaming data are processed respectively to obtain streaming processed data and historical streaming processed data. The difference between the stream processing data and the historical stream processing data is calculated to obtain the data difference value. Based on the data difference and the streaming data, a processing result characterizing the month-on-month change rate is obtained.

2. The method according to claim 1, characterized in that, The step of determining the historical streaming data table from the historical streaming database based on the tags of the streaming data includes: Based on the tags of the streaming data, query the historical streaming database for historical tags that match the tags of the streaming data; If the historical tag exists, then the historical streaming data table corresponding to the historical tag is determined.

3. The method according to claim 2, characterized in that, After retrieving historical tags from the historical streaming database that match the tags of the streaming data based on the tags of the streaming data, the method further includes: If the historical tag does not exist, a target historical streaming data table is constructed in the historical streaming database, wherein the target historical streaming data table is a data table corresponding to the tag of the streaming data; Write the streaming data and the time information of the streaming data into the target historical streaming data table to return to the steps of receiving streaming data, the tags of the streaming data, and the time information of the streaming data.

4. The method according to claim 1, characterized in that, After determining the historical streaming data table from the historical streaming database based on the tags of the streaming data, the method further includes: If the amount of historical streaming data does not meet the preset data amount requirement, the streaming data and the time information of the streaming data are written into the historical streaming data table, and the steps of receiving streaming data, the tag of the streaming data, and the time information of the streaming data are returned until the amount of data in the historical streaming data table meets the preset data amount requirement.

5. The method according to claim 1, characterized in that, Determining the first data volume of streaming data and the second data volume of historical streaming data in the updated historical streaming data table includes: Based on the historical time information of the historical streaming data in the historical streaming data table, determine the streaming data and historical streaming data in the updated historical streaming data table; Statistical processing is performed on the streaming data and the historical streaming data to determine the first data volume of streaming data and the second data volume of historical streaming data in the updated historical streaming data table.

6. The method according to claim 1, characterized in that, After determining the first data volume of streaming data and the second data volume of historical streaming data in the updated historical streaming data table, the method further includes: If the first data volume and the second data volume do not meet the preset data volume comparison requirements, then return to the steps of receiving streaming data, the tags of the streaming data, and the time information of the streaming data, until the first data volume and the second data volume meet the preset data volume comparison requirements, and the updated historical streaming data table is obtained.

7. The method according to claim 1, characterized in that, Before determining the data processing rule corresponding to the tag based on the tag of the streaming data, if the first data volume and the second data volume meet the preset data volume comparison requirements, the method further includes: Receive streaming data from users as needed for data processing, send the tag type, and the initial data processing rules corresponding to the tag type; Based on the label type and the initial data processing rules, a rule database is constructed, which is used to determine the data processing rules corresponding to the label.

8. The method according to claim 1, characterized in that, If the first data volume and the second data volume meet the preset data volume comparison requirements, and after obtaining the processing result representing the month-on-month change rate based on the first data volume and the historical streaming data, the method further includes: Based on the tags and the data processing rules, obtain the preset alarm threshold; Based on the processing result and the alarm threshold, an alarm message is generated; The alarm message is sent to the streaming data processing platform to be displayed to the user.

9. A streaming data processing device, characterized in that, include: The receiving module is used to receive streaming data sent by the streaming processing platform, the tags of the streaming data, and the time information of the streaming data; The first determining module is used to determine a historical streaming data table from the historical streaming database based on the tags of the streaming data, wherein the historical streaming data table is a data table corresponding to the tags, and the historical streaming data table stores historical streaming data and historical time information of the historical streaming data; The update module is used to update the target historical streaming data in the historical streaming data table to streaming data according to the historical time information and the time information if the amount of historical streaming data in the historical streaming data table meets the preset data amount requirement, thereby obtaining an updated historical streaming data table. The second determining module is used to determine the first data volume of streaming data and the second data volume of historical streaming data in the updated historical streaming data table; The third determining module is used to determine the data processing rules corresponding to the tags based on the tags of the streaming data if the first data volume and the second data volume meet the preset data volume comparison requirements. The module is used to obtain the processing result based on the data processing rules, the streaming data in the updated historical streaming data table, and the historical streaming data; The process of obtaining the processing result based on the data processing rules, the updated streaming data in the historical streaming data table, and the historical streaming data includes: According to the data processing rules, the streaming data and the historical streaming data are processed respectively to obtain streaming processed data and historical streaming processed data. The difference between the stream processing data and the historical stream processing data is calculated to obtain the data difference value. Based on the data difference and the streaming data, a processing result characterizing the month-on-month change rate is obtained.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

Images