Real-time monitoring method, system, device and storage medium for massive data

Abstract

The present invention discloses a method, system, device and storage medium for real-time monitoring of massive data, the method comprising the following steps: configuring a field pool for access services, the field pool comprising a number of fields divided according to the services; combining a number of fields selected by a user from the field pool according to business requirements to generate a corresponding configuration; responding to a user's startup instruction, using a real-time computing framework to perform group computing on raw data according to the combination of the fields in each configuration to obtain a corresponding computing result, storing the computing result in a database; and outputting the computing result according to the user's requirements. The present invention is aimed at the scenario of real-time computing of massive data with high concurrency and high traffic. There is no need to repeatedly develop new user requirements. New group computing services can be quickly generated by simply dragging fields to add configurations, quickly adapting to changes in business requirements, and avoiding waste of human resources.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a real-time monitoring method, system, device and storage medium for massive data. Background Art

[0002] With the rapid development of big data, there are more and more demands for real-time computing and monitoring of massive data with high concurrency and high traffic. The main implementation method is that whenever a user, such as the business party, puts forward a monitoring requirement, such as the number of calls per minute for client and corresponding version group calculation, first embed a point in the system, then report the data to a message queue, and then use the real-time computing framework to write a topology program, upload it to the real-time computing framework cluster to start real-time computing and storage, and then develop an interface display. The problem with the existing implementation method is that when the business party puts forward a new monitoring requirement again, such as the number of calls per second for the computer room group calculation, the entire process mentioned above needs to be repeated. When the business party's needs continue to increase or change, huge and highly repetitive work will be generated, resulting in a waste of human resources. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the defect of waste of human resources caused by highly repetitive work in the existing technology for real-time calculation of massive data with high concurrency and large flow, when user needs are constantly changing, and to provide a real-time monitoring method, system, device and storage medium for massive data that can meet different monitoring needs of users with simple operations without repeated development.

[0004] The present invention solves the above technical problems through the following technical solutions:

[0005] The embodiment of the present invention provides a method for real-time monitoring of massive data, comprising the following steps:

[0006] Configuring a field pool for access services, the field pool comprising a plurality of fields divided according to the services;

[0007] Combining a plurality of the fields selected by the user from the field pool according to business requirements to generate a corresponding configuration;

[0008] In response to a user's start-up instruction, the raw data is grouped and calculated using a real-time calculation framework according to a combination of the fields in each of the configurations to obtain corresponding calculation results, and the calculation results are stored in a database;

[0009] The calculation result is output according to the user's requirements.

[0010] Preferably, the types of the fields include a time type, an enumeration type of a limited type, and a numerical type used for calculation;

[0011] The first field of each configuration is a field of the time type;

[0012] When the user's requirement is a statistical average or a statistical data sum, the last field of each configuration is a numerical field;

[0013] The other fields of each of the configurations are fields of the enumeration type.

[0014] Preferably, the real-time monitoring method further comprises the following steps:

[0015] Reporting business data to the message queue one by one according to the fields in the field pool to obtain each corresponding piece of original data;

[0016] Each piece of the business data has a corresponding timestamp;

[0017] Each piece of the original data includes a field of the time type generated according to the timestamp.

[0018] Preferably, the method of performing group calculation on the raw data using a real-time calculation framework according to the combination of the fields in each of the configurations to obtain corresponding calculation results, and storing the calculation results in a database, comprises the following steps:

[0019] Obtaining a plurality of configurations and storing them one by one in order into a first variable array; generating a corresponding SQL (Structured Query Language) insert statement for each configuration and storing them one by one in order into a second variable array;

[0020] Create an execution environment for the Stream in the real-time computing framework; set the event time to the timestamp field in the business data; set the message queue to be the data source of the Stream;

[0021] The first variable array is traversed, wherein each of the configurations forms a branch stream Stream, a group calculation operation is performed, and the corresponding calculation result and the corresponding group field are stored in the second variable array.

[0022] Preferably, the combining of the plurality of fields selected by the user from the field pool according to business requirements includes:

[0023] The multiple fields selected by the user from the field pool by dragging are combined according to business requirements.

[0024] Preferably, outputting the calculation result according to the user's needs includes:

[0025] The page is triggered to generate a corresponding menu according to the needs of the user;

[0026] In response to the user clicking on the menu, the corresponding calculation result is queried from the database to generate a line graph.

[0027] Preferably, the real-time monitoring method further comprises the following steps:

[0028] In response to a preview instruction from the user for each of the configurations, an example result of the group calculation corresponding to the configuration is output.

[0029] Preferably, the time type field includes seconds and / or minutes;

[0030] The limited types of enumerated fields include at least one of interface, client, exception code, domain name, version, operator, computer room and province;

[0031] The numerical fields used for calculation include at least one of inflow, time consumption and outflow.

[0032] Preferably, the real-time computing framework is Flink (a scalable open source batch processing and stream processing platform); the message queue is Kafka (a high-throughput distributed publish-subscribe message queue system).

[0033] Another embodiment of the present invention provides a real-time monitoring system for massive data, including:

[0034] A field pool module, used to configure a field pool for accessing a service, wherein the field pool includes a plurality of fields divided according to the service;

[0035] A group calculation configuration module, used for combining a plurality of fields selected by a user from the field pool according to business requirements to generate a corresponding configuration;

[0036] A group real-time calculation module, used to respond to a user's start-up instruction, use a real-time calculation framework to perform group calculations on the original data according to the combination of the fields in each of the configurations to obtain corresponding calculation results, and store the calculation results in a database;

[0037] The group calculation display module is used to output the calculation results according to the needs of the user.

[0038] Preferably, the types of the fields include a time type, an enumeration type of a limited type, and a numerical type used for calculation;

[0039] The first field of each configuration is a field of the time type;

[0040] When the user's requirement is a statistical average or a statistical data sum, the last field of each configuration is a numerical field;

[0041] The other fields of each of the configurations are fields of the enumeration type.

[0042] Preferably, the real-time monitoring system further includes:

[0043] The original data acquisition module is used to report the business data to the message queue one by one according to the fields in the field pool to obtain each corresponding original data;

[0044] Each piece of the business data has a corresponding timestamp;

[0045] Each piece of the original data includes a field of the time type generated according to the timestamp.

[0046] Preferably, the group real-time calculation module includes:

[0047] Initialize the configuration unit, used to obtain a plurality of the configurations and store them one by one in order into the first variable array, generate a corresponding SQL insert statement for each of the configurations and store them one by one in order into the second variable array;

[0048] Initialize a topology unit, which is used to create an execution environment for a Stream in the real-time computing framework, set the event time to the timestamp field in the business data, and set the message queue to be the data source of the Stream;

[0049] A cyclic group calculation storage unit is used to traverse the first variable array, wherein each of the configurations forms a branch stream Stream, performs a group calculation operation, and stores the corresponding calculation results and the corresponding group fields obtained into the second variable array.

[0050] Preferably, the group calculation configuration module is used to combine the several fields selected by the user from the field pool in a dragging manner according to business requirements.

[0051] Preferably, the group calculation display module includes:

[0052] A menu generating unit, used to trigger the page to generate a corresponding menu according to the needs of the user;

[0053] The result display unit is used to query the corresponding calculation result from the database to generate a line graph in response to the user clicking the menu.

[0054] Preferably, the real-time monitoring system further includes:

[0055] The preview module is used to respond to the user's preview instruction for each of the configurations and output an example result of the group calculation corresponding to the configuration.

[0056] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the real-time monitoring method for massive data provided in the above-mentioned embodiment when executing the computer program.

[0057] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for real-time monitoring of massive data provided in the above-mentioned embodiment are implemented.

[0058] The positive and progressive effect of the present invention is that the real-time monitoring method, system, device and storage medium of massive data provided by the present invention can form different field combinations by setting a field pool, and each group of field combinations corresponds to a configuration, and different configurations meet different monitoring needs of users. The generated new configuration can automatically trigger real-time calculation and storage according to user instructions, and output the calculation results. For the scenario of real-time calculation of massive data with high concurrency and large traffic, there is no need to repeatedly develop new needs of users. New group calculation services can be quickly generated by simply dragging fields to add configurations, quickly adapting to changes in business needs, and avoiding waste of human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a method for real-time monitoring of massive data according to Embodiment 1 of the present invention.

[0060] Figure 2 For Example 1 Figure 1 Flow chart of step 104.

[0061] Figure 3 This is a schematic diagram of a QPM (requests per minute) table generated based on the configuration “minutes|client|version” in Example 1.

[0062] Figure 4 This is a schematic diagram of a QPS (number of requests per second) table generated based on the configuration “seconds|computer room” in Example 1.

[0063] Figure 5 Based on Example 1 Figure 3 Select the trend chart generated by the Android client in the table.

[0064] Figure 6 In Example 1, Figure 5 On this basis, select the trend chart generated by version 2.0.0.

[0065] Figure 7 This is a module schematic diagram of a real-time monitoring system for massive data according to Example 2 of the present invention.

[0066] Figure 8 for Figure 7 Schematic diagram of the structure of the group real-time computing module.

[0067] Fig. 9 for Figure 7 Schematic diagram of the structure of the group calculation display module.

[0068] Fig.10 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment provides a real-time monitoring method for massive data, including the following steps:

[0072] Step 101: configure a field pool for access services, wherein the field pool includes a plurality of fields divided according to the services.

[0073] In this embodiment, the configuration of the field pool is divided by the user according to the business. The more fields that can be reported by the business, the more suitable the scenario is, which means that the demand changes more and a lot of repeated work can be saved.

[0074] Among them, each field is like a building block, which is divided into three types, including time type, limited type of enumeration type and numerical type for calculation. The time type is for the timestamp contained in the business data uploaded by the user. According to the timestamp, a field of the time type is generated, specifically generating two fields of seconds and minutes, which are used to calculate QPS and QPM respectively. The limited type of enumeration type fields include at least one of interface, client, exception code, domain name, version, operator, computer room and province. For example, the client includes android, apple, m (browser), etc., which are a limited number of fields. The numerical field used for calculation includes at least one of inflow, time consumption and outflow, such as the 'time consumption' field, which can range from 1 to infinity. This field can be used to perform avg (statistical average) operation to calculate the average time consumption, and sum (statistical data and) operation can calculate the total time consumption.

[0075] Step 102: Report the business data to the message queue Kafka in real time one by one according to the fields in the field pool to obtain the corresponding original data.

[0076] Each piece of the business data carries a corresponding timestamp; each piece of the original data includes a field of the time type generated according to the timestamp.

[0077] Step 103: Combine the plurality of fields selected by the user from the field pool by dragging according to business requirements to generate a corresponding configuration.

[0078] Among them, fields are dragged from the configured field pool for combination, and multiple combinations can be formed at the same time, corresponding to multiple configurations, and each combination can be multiple fields. The first field of each configuration is the time type field; when the user's demand is the statistical average avg or the statistical data and sum, the last field of each configuration is the numerical type field; the other fields of each configuration are the enumeration type fields.

[0079] In specific implementation, the combinational logic restrictions are as follows:

[0080] The first field is a time type field. Minutes are used to calculate QPM, and seconds are used to calculate QPS.

[0081] The last field: If it is a sum or avg operation, you need to select a numeric field. If it is a count (statistical data quantity) operation, you do not need to configure this field.

[0082] Other fields: These are finite enumeration fields, and you cannot select fields with continuous values.

[0083] Step 104, responding to the user's start instruction, using the real-time computing framework Flink to perform group computing on the original data according to the combination of the fields in each of the configurations to obtain corresponding computing results, and storing the computing results in a database.

[0084] Step 105: output the calculation result according to the user's needs.

[0085] Specifically, according to the needs of the user, the page is triggered to generate a corresponding menu, each menu corresponds to a configuration; in response to the user clicking the menu, the corresponding calculation result is queried from the database to generate a line graph. In specific implementation, the user can click the start button after the drag and drop combination is completed. This embodiment will respond to the user's start instruction and specifically trigger a topology daemon, that is, a daemon process to start one or more general topology programs. Each general topology program performs group calculations according to the configured field combination and stores the calculation results in the database; at the same time, the page can also be triggered to generate a corresponding menu. At this time, clicking the menu can see the calculation results in the form of a line graph.

[0086] like Figure 2As shown, step 104 in this embodiment specifically includes the following steps:

[0087] Step 1041 , respond to the user's start instruction and execute step 1042 .

[0088] During the specific implementation process, the subsequent steps can be implemented by a general topology program.

[0089] Step 1042, obtain a plurality of the configurations and store them one by one in order into the first variable array; generate a corresponding SQL insert statement for each of the configurations and store them one by one in order into the second variable array.

[0090] In specific implementation, this step corresponds to the initialization configuration operation performed when the general topology program is started, specifically, reading all field combinations of each configuration, and then storing all grouped configurations in order into the first variable array, such as configs, and generating a sql insert statement for each configuration and storing it in the second variable array, such as sqls.

[0091] Among them, an example of generating a SQL insert statement is as follows:

[0092] For a configuration of "minutes|client|version", the generated SQL insert statement can be:

[0093] replace into minute_client_version(minute,client,version,num)values(?,?,?,?)

[0094] For a configuration of "seconds|computer room", the generated SQL insert statement can be:

[0095] replace into second_idc(second idc,num)values(?,?,?)

[0096] These SQL insert statements will be used in the subsequent processing steps of the general topology program, and each combined calculation result will be inserted into its own table.

[0097] Step 1043: Create an execution environment for the Stream in the real-time computing framework; set the event time to the timestamp field in the business data; and set the message queue as the data source of the Stream.

[0098] In specific implementation, this step corresponds to the initialization topology operation performed when the general topology program is started, specifically, first creating a Stream execution environment, setting the event time to the timestamp of the message in the message queue, and then setting Kafka as the data source of the Stream. Among them, the timestamp field in the business data may be seconds or minutes, depending on the expression form of the timestamp in the business data reported in real time in step 102. This is common knowledge in the art and will not be repeated here.

[0099] Step 1044, traverse the first variable array, wherein each of the configurations forms a branch stream Stream, perform a group calculation operation, and store the corresponding calculation result and the corresponding group field obtained into the second variable array.

[0100] In specific implementation, this step corresponds to the loop group calculation storage operation of the general topology program, specifically, traverse the variable array configs, where each combination stored in configs[i] forms a branch stream Stream, perform a group by(configs[i]) count group calculation operation, and then use sqls[i] to store the group field and calculation results to the corresponding data table, i++, and then loop to perform the group calculation storage operation. Where i is a loop variable, one for each configuration.

[0101] In this embodiment, after a configuration is generated, a preview function can be implemented according to user needs, specifically, in response to a preview instruction from the user for each of the configurations, an example result of the group calculation corresponding to the configuration is output.

[0102] The final storage result generated based on this embodiment can be referred to Figure 3 and Figure 4 ,in Figure 3 The following is a diagram of the QPM table generated based on the configuration "minutes|clients|versions". Figure 4 This is a diagram of the QPS table generated based on the configuration "Seconds | Data Center". Figure 3 and Figure 4 In the calculation result table, perform a secondary summary query. The specific example is as follows:

[0103] 1) The user selects the 'Minute_Client_Version' menu and selects the Android client to generate the following SQL statement:

[0104] select minute,client,sum(num)from minute_client_version where client='android'group by minute,client;After querying the data, display it in a trend chart. The result is referenced Figure 5 shown.

[0105] 2) If the user selects a version 2.0.0, the following SQL statement will be generated:

[0106] select minute,client,version,sum(num)from minute_client_version whereclient='android'and version='1.0.0'group by minute,client,version;After querying the data, display it in a trend chart as follows Figure 6 shown.

[0107] The real-time monitoring method for massive data provided in this embodiment enables users to freely drag and drop field combinations to generate multiple configurations, just as simple as building blocks, and can automatically trigger the generation of real-time computing programs to calculate, store and display results according to user instructions. The present invention uses Flink branch flow and group computing technology to quickly generate new group computing services by adding configurations, and quickly adapt to changes in business needs.

[0108] Example 2

[0109] like Figure 7 As shown, this embodiment provides a real-time monitoring system for massive data, including a field pool module 1, an original data acquisition module 2, a group calculation configuration module 3, a preview module 4, a group real-time calculation module 5 and a group calculation display module 6.

[0110] The field pool module 1 is used to configure a field pool for accessing a service, and the field pool includes a number of fields divided according to the service.

[0111] In this embodiment, the configuration of the field pool is divided by the user according to the business. The more fields that can be reported by the business, the more suitable the scenario is, which means that the demand changes more and a lot of repeated work can be saved.

[0112] Among them, each field is like a building block, which is divided into three types, including time type, limited type of enumeration type and numerical type for calculation. The time type is for the timestamp contained in the business data uploaded by the user. The time type field is generated according to the timestamp, and specifically generates two fields of seconds and minutes, which are used to calculate QPS and QPM respectively. The limited type of enumeration type fields include at least one of interface, client, exception code, domain name, version, operator, computer room and province. For example, the client includes android, apple, m, etc., which are a limited number of fields. The numerical field used for calculation includes at least one of inbound flow, time consumption and outbound flow, such as the 'time consumption' field, which can range from 1 to infinity. This field can be used for avg operation to calculate the average time consumption, and the sum operation can calculate the total time consumption; if it is a group count operation, it is used to calculate the request volume.

[0113] The original data acquisition module 2 is used to report business data to the message queue Kafka one by one according to the fields in the field pool to obtain each corresponding original data; each business data has a corresponding timestamp; each original data includes a field of the time type generated according to the timestamp.

[0114] The group calculation configuration module 3 is used to combine several fields selected by the user from the field pool in a dragging manner according to business requirements to generate a corresponding configuration.

[0115] Among them, the fields are dragged from the configured field pool for combination, and multiple combinations can be formed at the same time, corresponding to multiple configurations, and each combination can be multiple fields; the first field of each configuration is a field of the time type; when the user's demand is a statistical average or a statistical data sum, the last field of each configuration is a numerical field; the other fields of each configuration are enumeration fields.

[0116] The preview module 4 is used to respond to the user's preview instruction for each of the configurations and output an example result of the group calculation corresponding to the configuration. In this embodiment, after a configuration is generated, the preview module can implement the preview function according to the user's needs.

[0117] The grouping real-time computing module 5 is used to respond to the user's startup instruction, use the real-time computing framework Flink to perform group computing on the original data according to the combination of the fields in each of the configurations, to obtain corresponding computing results, and store the computing results in the database.

[0118] like Figure 8As shown, in this embodiment, the group real-time computing module 5 includes: an initialization configuration unit 501, an initialization topology unit 502 and a cyclic group computing storage unit 503. Among them, the initialization configuration unit 501 is used to obtain a number of the configurations and store them one by one in order into the first variable array, generate a corresponding SQL insert statement for each configuration and store them one by one in order into the second variable array. The initialization topology unit 502 is used to create an execution environment for the Stream in the real-time computing framework, set the event time to the timestamp field in the business data, and set the message queue as the data source of the Stream. The cyclic group computing storage unit 503 is used to traverse the first variable array, wherein each of the configurations forms a branch stream Stream, performs a group computing operation, and stores the corresponding calculation results and the corresponding group fields obtained into the second variable array.

[0119] The group calculation display module 6 is used to output the calculation results according to the needs of the user.

[0120] like Fig. 9 As shown, in this embodiment, the group calculation display module 6 includes a menu generation unit 601 and a result display unit 602. The menu generation unit 601 is used to trigger the page to generate a corresponding menu according to the needs of the user. The result display unit 602 is used to query the corresponding calculation result from the database to generate a line chart in response to the user clicking the menu.

[0121] The real-time monitoring system for massive data provided by this embodiment uses the group computing configuration module to enable users to freely drag and drop field combinations to generate multiple configurations, just as simple as building blocks. The group real-time computing module can automatically trigger the generation of real-time computing program calculation storage according to user instructions, and the group computing display module can display the calculation results. The present invention uses Flink branch flow and group computing technology to quickly generate new group computing services by adding configurations, and quickly adapt to changes in business needs.

[0122] Example 3

[0123] Fig.10 This is a schematic diagram of the structure of an electronic device provided in Example 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the real-time monitoring method of mass data in Example 1 when executing the program. Figure 3 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0124] like Fig.10As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0125] The bus 33 includes a data bus, an address bus, and a control bus.

[0126] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0127] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0128] The processor 31 executes various functional applications and data processing by running the computer programs stored in the memory 32, such as the real-time monitoring method of massive data provided in Embodiment 1 of the present invention.

[0129] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model-generated device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. As shown, the network adapter 36 communicates with other modules of the model-generated device 30 via a bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0130] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0131] Example 4

[0132] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the real-time monitoring method for massive data provided in Embodiment 1 are implemented.

[0133] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0134] In a possible implementation manner, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the real-time monitoring method for massive data described in Example 1.

[0135] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0136] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A real-time monitoring method for massive data, characterized in that: The following steps are involved: Configuring a field pool for access services, the field pool comprising a plurality of fields divided according to the services; Combining a plurality of the fields selected by the user from the field pool according to business requirements to generate a corresponding configuration; In response to a user's start-up instruction, the raw data is grouped and calculated using a real-time calculation framework according to a combination of the fields in each of the configurations to obtain corresponding calculation results, and the calculation results are stored in a database; Outputting the calculation result according to the user's needs; The method of performing group calculation on the raw data using a real-time calculation framework according to the combination of the fields in each of the configurations to obtain corresponding calculation results, and storing the calculation results in a database, comprises the following steps: Obtain a number of configurations and store them one by one in order into the first variable array; generate a corresponding SQL insert statement for each configuration and store them one by one in order into the second variable array; Create an execution environment for the Stream in the real-time computing framework; set the event time to the timestamp field in the business data; set the message queue to be the data source of the Stream; The first variable array is traversed, wherein each of the configurations forms a branch stream Stream, a group calculation operation is performed, and the corresponding calculation result and the corresponding group field are stored in the second variable array.

2. The real-time monitoring method for mass data according to claim 1, characterized in that: The types of the fields include time types, limited types of enumeration types, and numerical types used for calculations; The first field of each configuration is a field of the time type; When the user's requirement is a statistical average or a statistical data sum, the last field of each configuration is a numerical field; The other fields of each of the configurations are fields of the enumeration type.

3. The real-time monitoring method for mass data according to claim 2, characterized in that: The real-time monitoring method further comprises the following steps: Reporting business data to the message queue one by one according to the fields in the field pool to obtain each corresponding piece of original data; Each piece of the business data has a corresponding timestamp; Each piece of the original data includes a field of the time type generated according to the timestamp.

4. The real-time monitoring method for mass data according to claim 1, characterized in that: The combining of the plurality of fields selected by the user from the field pool according to business requirements includes: The multiple fields selected by the user from the field pool by dragging are combined according to business requirements.

5. The real-time monitoring method for mass data according to claim 1, characterized in that: Outputting the calculation result according to the user's needs includes: The page is triggered to generate a corresponding menu according to the needs of the user; In response to the user clicking on the menu, the corresponding calculation result is queried from the database to generate a line graph.

6. The real-time monitoring method for mass data according to claim 1, characterized in that: The real-time monitoring method further comprises the following steps: In response to a preview instruction from the user for each of the configurations, an example result of the group calculation corresponding to the configuration is output.

7. The real-time monitoring method for mass data according to claim 2, characterized in that: The time type field includes seconds and / or minutes; The limited types of enumerated fields include at least one of interface, client, exception code, domain name, version, operator, computer room and province; The numerical fields used for calculation include at least one of inflow, time consumption and outflow.

8. The real-time monitoring method for mass data according to claim 1, characterized in that: The real-time computing framework is Flink; the message queue is Kafka.

9. A real-time monitoring system for massive data, characterized in that: include: A field pool module, used to configure a field pool for accessing a service, wherein the field pool includes a plurality of fields divided according to the service; A group calculation configuration module, used for combining a plurality of fields selected by a user from the field pool according to business requirements to generate a corresponding configuration; A group real-time calculation module, used to respond to a user's start-up instruction, use a real-time calculation framework to perform group calculations on the original data according to the combination of the fields in each of the configurations to obtain corresponding calculation results, and store the calculation results in a database; A group calculation display module is used to output the calculation results according to the needs of the user; The group real-time calculation module includes: Initialize the configuration unit, used to obtain a plurality of the configurations and store them one by one in order into the first variable array, generate a corresponding SQL insert statement for each of the configurations and store them one by one in order into the second variable array; Initialize the topology unit, which is used to create an execution environment for the Stream in the real-time computing framework, set the event time to the timestamp field in the business data, and set the message queue to the data source of the Stream; A cyclic group calculation storage unit is used to traverse the first variable array, wherein each of the configurations forms a branch stream Stream, performs a group calculation operation, and stores the corresponding calculation results and the corresponding group fields obtained into the second variable array.

10. The real-time monitoring system for massive data according to claim 9, characterized in that: The types of the fields include time types, limited types of enumeration types, and numerical types used for calculations; The first field of each configuration is a field of the time type; When the user's requirement is a statistical average or a statistical data sum, the last field of each configuration is a numerical field; The other fields of each of the configurations are fields of the enumeration type.

11. The real-time monitoring system for mass data according to claim 10, characterized in that: The real-time monitoring system also includes: The original data acquisition module is used to report the business data to the message queue one by one according to the fields in the field pool to obtain each corresponding original data; Each piece of the business data has a corresponding timestamp; Each piece of the original data includes a field of the time type generated according to the timestamp.

12. The real-time monitoring system for mass data according to claim 9, characterized in that: The group calculation configuration module is used to combine the multiple fields selected by the user from the field pool in a dragging manner according to business requirements.

13. The real-time monitoring system for mass data according to claim 9, characterized in that: The group calculation display module includes: A menu generating unit, used to trigger the page to generate a corresponding menu according to the needs of the user; The result display unit is used to query the corresponding calculation result from the database to generate a line graph in response to the user clicking the menu.

14. The real-time monitoring system for mass data according to claim 9, characterized in that: The real-time monitoring system also includes: The preview module is used to respond to the user's preview instruction for each of the configurations and output an example result of the group calculation corresponding to the configuration.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the real-time monitoring method for mass data according to any one of claims 1 to 8 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the real-time monitoring method for mass data according to any one of claims 1 to 8 are implemented.

Images