Data processing method, big data processing platform, electronic device and storage medium

Abstract

The application provides a data processing method, a big data processing platform, an electronic device and a storage medium, pre-processes obtained index data; determines a target open source event processing model matched with the index data from each open source event processing model pre-set according to data characteristics of the pre-processed index data and model characteristics of each open source event processing model pre-set; and performs alarm analysis processing on the index data by using a target CEP alarm analysis rule pre-set on the target open source event processing model to obtain corresponding alarm information. The application can solve the problems that there is no mature and efficient rule to process the monitored data when data is abnormal or a system fails, and corresponding alarm information is generated in time for an operation and maintenance personnel to perform corresponding emergency processing according to the alarm information, and that data formats are quite different and affect the operation performance of a production system.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a data processing method, a big data processing platform, an electronic device, and a storage medium. Background Technology

[0002] As major banks continue to expand their businesses and increase the number of systems, transaction volumes are experiencing explosive growth, leading to a rise in complex system problems. Therefore, a pressing issue is how to monitor the massive amounts of data generated by these systems to accurately, effectively, and quickly identify relevant alarm information so that maintenance personnel can take appropriate action.

[0003] However, in the course of operation and maintenance, the frequent comparison of monitored resources by various system monitoring methods and tools leads to significant differences in data formats, affecting the performance of the production system. On the other hand, when data is abnormal or the system fails, there are no mature and efficient rules to process the monitored data and generate corresponding alarm information in a timely manner so that operation and maintenance personnel can take appropriate emergency measures based on the alarm information. Summary of the Invention

[0004] In view of this, the present invention provides a data processing method, a big data processing platform, an electronic device, and a storage medium to solve the problems of lack of mature and efficient rules for processing monitored data and generating corresponding alarm information in a timely manner when data is abnormal or the system fails, so as to enable maintenance personnel to perform corresponding emergency handling based on the alarm information, and the problem of large differences in data format affecting the operating performance of the production system.

[0005] The first aspect of this invention discloses a data processing method applied to a big data processing platform, the method comprising:

[0006] The acquired indicator data is preprocessed;

[0007] Based on the characteristics of the preprocessed indicator data and the characteristics of each pre-set open-source event processing model, a target open-source event processing model matching the indicator data is determined from the pre-set open-source event processing models; wherein, the target open-source event processing model is configured with pre-set target CEP alarm analysis rules; the CEP rules of each open-source event processing model are configured based on historical information;

[0008] The indicator data is analyzed and processed using the pre-set target CEP alarm analysis rules configured on the target open-source event processing model to obtain the corresponding alarm information.

[0009] Optionally, the target open source event processing model is further configured with preset target CEP aggregation rules, and the method further comprises:

[0010] The target CEP aggregation rules configured on the target open source event processing model are used to aggregate the index data, and aggregated data is obtained.

[0011] Optionally, the process of configuring corresponding CEP rules for the open source event processing model according to historical information comprises:

[0012] A rule library is created in the message middleware, wherein the rule library comprises a plurality of initial CEP rules, and each initial CEP rule carries a corresponding rule weight;

[0013] Each initial CEP rule is updated according to the rule weight corresponding to each initial CEP rule, historical alarm information and historical aggregation information, and a CEP rule corresponding to each initial CEP rule is obtained; wherein each CEP rule comprises a plurality of CEP alarm analysis rules and a plurality of CEP aggregation rules; the historical aggregation information and the historical alarm information constitute historical information.

[0014] For each open source event processing model, the CEP alarm analysis rules and the CEP aggregation rules matched with the open source event processing model are determined from the plurality of CEP alarm analysis rules and the plurality of CEP aggregation rules according to the model characteristics of each open source event processing model, and are configured on the open source event processing model.

[0015] Optionally, the method further comprises:

[0016] The preset target CEP alarm analysis rules and the preset target CEP aggregation rules configured on the target open source event processing model are updated using the alarm information and the aggregation information.

[0017] Optionally, the method further comprises:

[0018] Index data is obtained, and a real-time processing task is created by a data processing component;

[0019] The index data is preprocessed using the real-time processing task.

[0020] The second aspect of the application discloses a big data processing platform, and the platform comprises:

[0021] A preprocessing unit is configured to preprocess the obtained index data;

[0022] A model matching unit is configured to determine a target open source event processing model matched with the index data from a plurality of pre-set open source event processing models according to data characteristics of the pre-processed index data and model characteristics of the pre-set open source event processing models; wherein the target open source event processing model is configured with a pre-set target CEP alarm analysis rule; the CEP rule on each target open source event processing model is configured by a configuration unit; and the CEP rule at least includes a CEP alarm analysis rule;

[0023] An alarm analysis processing unit is configured to perform alarm analysis processing on the index data by using the pre-set target CEP alarm analysis rule configured on the target open source event processing model to obtain corresponding alarm information.

[0024] Optionally, the target open source event processing model is further configured with a pre-set target CEP aggregation rule, and the platform further includes:

[0025] An aggregation processing unit is configured to perform aggregation processing on the index data by using the pre-set target CEP aggregation rule configured on the target open source event processing model to obtain aggregation data.

[0026] Optionally, the configuration unit includes:

[0027] A rule library creating unit is configured to create a rule library in the message middleware; wherein the rule library includes a plurality of initial CEP rules, and each initial CEP rule carries a corresponding rule weight;

[0028] A first rule updating unit is configured to update each initial CEP rule according to a corresponding rule weight, historical alarm information and historical aggregation information of each initial CEP rule to obtain a corresponding CEP rule of each initial CEP rule; wherein each CEP rule includes a plurality of CEP alarm analysis rules and a plurality of CEP aggregation rules; and the historical aggregation information and the historical alarm information constitute historical information.

[0029] A configuration sub-unit is configured to determine, for each open source event processing model, the CEP alarm analysis rule and the CEP aggregation rule matched with the open source event processing model from a plurality of CEP alarm analysis rules and a plurality of CEP aggregation rules according to model characteristics of each open source event processing model, and configure the CEP alarm analysis rule and the CEP aggregation rule to the open source event processing model.

[0030] The third aspect of the present application discloses an electronic device, characterized in that the electronic device comprises a processor and a memory, the memory is used for storing program codes and data of data processing, and the processor is used for calling program instructions in the memory to execute the data processing method disclosed in the first aspect of the present application.

[0031] The fourth aspect of the present application discloses a storage medium comprising a program, wherein the program controls a device where the storage medium is located to execute the data processing method disclosed in the first aspect of the present application when the program is running.

[0032] The present application provides a data processing method, a big data processing platform, an electronic device and a storage medium. A plurality of open source event processing models are pre-configured, and a corresponding CEP alarm analysis rule is configured for each pre-configured open source event processing model. After the big data processing platform obtains index data, the obtained index data is pre-processed to avoid affecting the production system operation performance due to large differences in data formats. According to the data characteristics of the pre-processed index data and the model characteristics of each pre-configured open source event processing model, a target open source event processing model that matches the index data is determined from each pre-configured open source event processing model. The target CEP alarm analysis rule configured on the target open source event processing model is used to perform alarm analysis processing on the index data to obtain corresponding alarm information. The obtained index data is processed by a mature and efficient rule to generate corresponding alarm information in time, so that the operation and maintenance personnel can perform corresponding emergency processing in time according to the alarm information. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0034] Figure 1 A structural diagram of a data processing system provided for the embodiments of the present application;

[0035] Figure 2 A flowchart of a data processing method provided for the embodiments of the present application;

[0036] Figure 3 A structural diagram of a big data processing platform provided for the embodiments of the present application;

[0037] Figure 4 A structural diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0039] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0040] cep: Complex Event Processing (cep), is a kind of event stream analysis technology based on dynamic environment, through the analysis of the management between events, using filtering, association, aggregation and other technologies, according to the time sequence relationship and aggregation relationship between events, formulating detection rules, continuously querying the event sequence meeting the requirements, and finally analyzing more complex composite events.

[0041] Referring to Figure 1 , a data processing system architecture diagram provided by an embodiment of the present application is shown, and the data processing system includes a message middleware, a big data processing platform, a data warehouse, an analysis learning system, a monitoring system and an alarm system.

[0042] In the embodiment of the present application, the index data is collected by an Agent end or a control tool, and the collected index data is stored into the message middleware, so that the big data processing platform obtains corresponding index data from the message middleware; a real-time processing task is created by a data processing component; and the index data is preprocessed by using the real-time processing task.

[0043] The big data processing platform determines the target open-source event processing model that matches the indicator data from the pre-set open-source event processing models based on the data characteristics of the pre-processed indicator data and the model characteristics of each pre-set open-source event processing model. In order to use the pre-set target CEP alarm analysis rules and indicator data configured on the target open-source event processing model to perform alarm analysis and processing, and obtain the corresponding alarm information.

[0044] Furthermore, in this embodiment of the application, the target open-source event processing model is also configured with pre-set target CEP aggregation rules. Therefore, the big data processing platform can also use the pre-set target CEP aggregation rules configured on the target open-source event processing model to aggregate the indicator data and obtain aggregated data.

[0045] In this embodiment, multiple open-source event processing models are pre-configured, and multiple corresponding CEP rules are configured for each open-source event processing model. These multiple CEP rules include CEP alarm analysis rules and CEP aggregation rules.

[0046] In this embodiment of the application, the process of configuring corresponding CEP rules for an open-source event processing model includes: creating a rule base in the message middleware; updating each initial CEP rule according to the rule weight, historical alarm information, and historical aggregation information corresponding to each initial CEP rule in the rule base to obtain the CEP rules corresponding to each initial CEP rule; wherein each CEP rule includes multiple CEP alarm analysis rules and multiple CEP aggregation rules; for each open-source event processing model, based on the model characteristics of each open-source event processing model, determining the CEP alarm analysis rules and CEP aggregation rules that match the open-source event processing model from the multiple CEP alarm analysis rules and multiple CEP aggregation rules, and configuring them on the open-source event processing model.

[0047] The rule base includes multiple initial CEP rules, each carrying a corresponding rule weight.

[0048] Furthermore, in this embodiment of the application, after the big data processing platform processes the indicator data to obtain the corresponding alarm information and aggregated information, it stores the obtained alarm information in the data warehouse so that the alarm system can subscribe to the corresponding alarm information from the data warehouse and output the corresponding alarm information.

[0049] Furthermore, in this embodiment of the application, after the big data processing platform processes the indicator data to obtain the corresponding alarm information and aggregated information, it stores the obtained alarm information in the data warehouse so that the monitoring system can subscribe to the corresponding alarm information and aggregated information from the data warehouse and store the subscribed alarm information and aggregated information.

[0050] Furthermore, in this embodiment of the application, after storing the obtained alarm information in the data warehouse, the corresponding alarm information and aggregation information can be subscribed to from the data warehouse through the analysis and learning system, and the subscribed alarm information and aggregation information can be used to update the pre-set target CEP alarm analysis rules and pre-set target CEP aggregation rules configured on the target open source event processing model.

[0051] Based on the above Figure 1 The present invention discloses a data processing method corresponding to the publicly available data processing system, such as... Figure 2 As shown, this data processing method is applied to Figure 1 The data processing method of the publicly available data processing system's big data processing platform specifically includes the following steps:

[0052] S201: Preprocess the acquired indicator data.

[0053] During the specific execution step S201, indicator data is collected through the Agent or data acquisition and control tools, and the collected indicator data is stored in the message middleware so that the big data processing platform can obtain the corresponding indicator data from the message middleware. After obtaining the indicator data, the big data processing platform can create real-time processing tasks through the data processing component and use the real-time processing tasks to preprocess the indicator data.

[0054] It should be noted that the message middleware can be Kafka, Flume, or other message middleware.

[0055] It should be noted that the data processing component can be a Spark data processing component. Specifically, the Spark data processing component can be used to convert the data format of indicator data into a standard format.

[0056] S202: Based on the characteristics of the preprocessed indicator data and the characteristics of each pre-set open-source event processing model, determine the target open-source event processing model that matches the indicator data from the pre-set open-source event processing models.

[0057] In this embodiment of the application, multiple open-source event processing models are pre-set. These pre-set open-source event processing models can be ESPER open-source event processing model, Drools open-source event processing model, and URULE open-source event processing model.

[0058] Among them, the ESPER open-source event processing model has a built-in SQL engine, which can process data in batches using SQL syntax. It is suitable for large amounts of data with simple processing logic, such as logs and performance metrics. The Drools open-source event processing model is an open-source rule engine produced by JBoss. It is entirely in Java and uses matching rules. When using it, the business logic needs to be abstracted and transformed into rule code. It is suitable for small amounts of data with complex business processing, such as alarms and log classification. The URULE open-source event processing model is a rule engine based on the Rete algorithm and implemented in pure Java. It provides wizard-style and script-style rule sets. Combined with a web designer, it can quickly define and publish rules. It is suitable for data with constantly changing rules and logic.

[0059] Therefore, we can see that the ESPER open-source event processing model is suitable for large amounts of data with simple processing logic; the Drools open-source event processing model is suitable for small amounts of data with complex business processing; and the URULE open-source event processing model is suitable for data with constantly changing rules and logic.

[0060] In this embodiment, the rich operational experience of the operations and maintenance personnel is converted into multiple initial CEP rules, and a corresponding rule weight is configured for each initial CEP rule. Based on the multiple initial CEP rules with configured rule weights, Guiying's rule base is created in the message middleware. This allows for the updating of each initial CEP rule according to its corresponding rule weight, historical alarm information, and historical aggregation information, resulting in the corresponding CEP rule. Each CEP rule includes multiple CEP alarm analysis rules and multiple CEP aggregation rules. Historical aggregation information and historical alarm information constitute historical information.

[0061] For each open-source event processing model, based on the model characteristics of each open-source event processing model, determine the CEP alarm analysis rules and CEP aggregation rules that match the open-source event processing model from multiple CEP alarm analysis rules and multiple CEP aggregation rules, and configure them on the open-source event processing model.

[0062] In the specific execution step S202, after obtaining the indicator data and preprocessing the indicator data, the target open source event processing model that matches the indicator data can be determined from the pre-set open source event processing models by the pre-set CEP engine adapter based on the data characteristics of the pre-processed indicator data and the model characteristics of each pre-set open source event processing model.

[0063] It should be noted that the CEP engine adapter can connect different predefined CEP engines, i.e., various open-source event processing models and big data processing platforms. When matching data and models, it adapts the target open-source event processing model that matches the indicator data by taking into account the characteristics of the indicator data and the characteristics of each open-source event processing model.

[0064] It should be noted that the characteristics of indicator data can be: large data volume and simple processing logic; small data volume but complex business processing; and large changes in rules and logic.

[0065] S203: Perform alarm analysis and processing using the pre-configured target CEP alarm analysis rule index data configured on the target open-source event processing model to obtain the corresponding alarm information.

[0066] In the specific execution step S203, after matching the target open-source event processing model that matches the indicator data, the target CEP alarm analysis rule indicator data configured on the target open-source event processing model is used to perform alarm analysis and processing to obtain the corresponding alarm information.

[0067] S204: Aggregate the indicator data using the pre-configured target CEP aggregation rules configured on the target open-source event processing model to obtain aggregated data.

[0068] In the specific execution step S204, after matching the target open-source event processing model that matches the indicator data, the indicator data is aggregated using the pre-set target CEP aggregation rules configured on the target open-source event processing model to obtain aggregated data.

[0069] Furthermore, in this embodiment of the application, after the big data processing platform processes the indicator data to obtain the corresponding alarm information and aggregated information, it stores the obtained alarm information in the data warehouse so that the alarm system can subscribe to the corresponding alarm information from the data warehouse and output the corresponding alarm information.

[0070] Furthermore, in this embodiment of the application, after the big data processing platform processes the indicator data to obtain the corresponding alarm information and aggregated information, it stores the obtained alarm information in the data warehouse so that the monitoring system can subscribe to the corresponding alarm information and aggregated information from the data warehouse and store the subscribed alarm information and aggregated information.

[0071] Furthermore, in this embodiment of the application, after storing the obtained alarm information in the data warehouse, the corresponding alarm information and aggregation information can be subscribed to from the data warehouse through the analysis and learning system, and the subscribed alarm information and aggregation information can be used to update the pre-set target CEP alarm analysis rules and pre-set target CEP aggregation rules configured on the target open source event processing model.

[0072] This invention provides a data processing method that pre-sets multiple open-source event processing models and configures corresponding CEP alarm analysis rules for each model. After acquiring indicator data, the big data processing platform first preprocesses the data to avoid impacting production system performance due to significant data format differences. Based on the characteristics of the pre-processed indicator data and the characteristics of the pre-set open-source event processing models, a target open-source event processing model matching the indicator data is determined. The indicator data is then analyzed using the pre-set target CEP alarm analysis rules configured on the target model to obtain corresponding alarm information. Mature and efficient rules are used to process the acquired indicator data and generate timely alarm information, allowing maintenance personnel to promptly handle emergencies based on the alarm information.

[0073] Corresponding to the data processing method disclosed in the above embodiments of the present invention, refer to Figure 3 This invention also provides a schematic diagram of the structure of a big data processing platform, which includes:

[0074] The preprocessing unit 31 is used to preprocess the acquired indicator data;

[0075] The model matching unit 32 is used to determine the target open-source event processing model that matches the indicator data from the pre-set open-source event processing models based on the data characteristics of the pre-processed indicator data and the model characteristics of each pre-set open-source event processing model. The target open-source event processing model is configured with pre-set target CEP alarm analysis rules. The CEP rules on each target open-source event processing model are configured through the configuration unit. The CEP rules include at least CEP alarm analysis rules.

[0076] The alarm analysis and processing unit 33 is used to perform alarm analysis and processing using the pre-set target CEP alarm analysis rule index data configured on the target open-source event processing model, and obtain the corresponding alarm information.

[0077] The specific principles and execution processes of each unit in the big data processing platform disclosed in the above embodiments of the present invention are similar to those in the above embodiments of the present invention. Figure 2The publicly disclosed data processing methods are the same, and can be found in the above embodiments of the present invention. Figure 2 The relevant parts of the publicly available data processing methods will not be elaborated here.

[0078] This invention provides a big data processing platform that pre-configures multiple open-source event processing models and sets corresponding CEP alarm analysis rules for each model. After acquiring indicator data, the platform preprocesses the data to avoid impacting production system performance due to significant data format differences. Based on the characteristics of the pre-processed indicator data and the characteristics of the pre-configured open-source event processing models, a target open-source event processing model matching the indicator data is selected. The platform then uses the pre-configured target CEP alarm analysis rules configured on the target model to perform alarm analysis and processing, obtaining corresponding alarm information. Mature and efficient rules are used to process the acquired indicator data and generate timely alarm information, enabling maintenance personnel to promptly handle emergencies based on the alarm information.

[0079] Optionally, the target open-source event processing model is also configured with pre-set target CEP aggregation rules. The big data processing platform provided in this embodiment of the invention further includes:

[0080] The aggregation processing unit is used to aggregate indicator data using the pre-configured target CEP aggregation rules configured on the target open-source event processing model to obtain aggregated data.

[0081] Optional configuration units include:

[0082] The rule base creation unit is used to create a rule base in the message middleware; the rule base includes multiple initial CEP rules, each of which carries a corresponding rule weight;

[0083] The first rule update unit is used to update each initial CEP rule according to the rule weight, historical alarm information and historical aggregation information corresponding to each initial CEP rule, so as to obtain the CEP rule corresponding to each initial CEP rule; wherein each CEP rule includes multiple CEP alarm analysis rules and multiple CEP aggregation rules; historical aggregation information and historical alarm information constitute historical information.

[0084] The configuration subunit is used to determine, based on the model characteristics of each open-source event processing model, the CEP alarm analysis rules and CEP aggregation rules that match the open-source event processing model from multiple CEP alarm analysis rules and multiple CEP aggregation rules, and configure them onto the open-source event processing model.

[0085] Optionally, the big data processing platform provided by the present invention further includes:

[0086] The second rule update unit is used to update the pre-configured target CEP alarm analysis rules and pre-configured target CEP aggregation rules configured on the target open-source event processing model using alarm information and aggregation information.

[0087] Optional, the preprocessing unit includes:

[0088] The acquisition unit is used to acquire indicator data and create real-time processing tasks through the data processing component.

[0089] The preprocessing subunit is used to preprocess the indicator data using real-time processing tasks.

[0090] The following is for reference. Figure 4 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present invention. The electronic devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments of the present invention.

[0091] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 406 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0092] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0093] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the data processing method shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by the processing device 401, it performs the functions defined in the data processing method of the embodiments of this invention.

[0094] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing a data processing method.

[0095] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: preprocessing the acquired indicator data; determining a target open-source event processing model matching the indicator data from among the pre-set open-source event processing models based on the data characteristics of the preprocessed indicator data and the model characteristics of each pre-set open-source event processing model; wherein the target open-source event processing model is configured with pre-set target CEP alarm analysis rules; the CEP rules of each open-source event processing model are configured based on historical information; and performing alarm analysis processing on the indicator data using the pre-set target CEP alarm analysis rules configured on the target open-source event processing model to obtain corresponding alarm information.

[0096] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] It should be noted that the computer-readable medium disclosed in this invention may be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0099] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. Units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0101] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0102] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, Applied to a big data processing platform, the method includes: The acquired indicator data is preprocessed; Based on the characteristics of the preprocessed indicator data and the characteristics of each pre-set open-source event processing model, a target open-source event processing model matching the indicator data is determined from the pre-set open-source event processing models; wherein, the target open-source event processing model is configured with pre-set target CEP alarm analysis rules; the CEP rules of each open-source event processing model are configured based on historical information; The indicator data is analyzed and processed using the pre-set target CEP alarm analysis rules configured on the target open-source event processing model to obtain the corresponding alarm information. The process of configuring corresponding CEP rules for the open-source event handling model based on historical information includes: A rule base is created in the message middleware; wherein, the rule base includes multiple initial CEP rules, and each initial CEP rule carries a corresponding rule weight; Each initial CEP rule is updated based on its corresponding rule weight, historical alarm information, and historical aggregation information to obtain the corresponding CEP rule; wherein each CEP rule includes multiple CEP alarm analysis rules and multiple CEP aggregation rules; the historical aggregation information and the historical alarm information constitute historical information; For each open-source event processing model, based on the model characteristics of each open-source event processing model, the CEP alarm analysis rule and the CEP aggregation rule that match the open-source event processing model are determined from multiple CEP alarm analysis rules and multiple CEP aggregation rules, and then configured onto the open-source event processing model.

2. The method according to claim 1, characterized in that, The target open-source event processing model is also configured with pre-set target CEP aggregation rules, and the method further includes: The indicator data is aggregated using the pre-set target CEP aggregation rules configured on the target open-source event processing model to obtain aggregated data.

3. The method according to claim 2, characterized in that, The method further includes: The alarm information and aggregation information are used to update the pre-set target CEP alarm analysis rules and the pre-set target CEP aggregation rules configured on the target open-source event processing model.

4. The method according to claim 1, characterized in that, The method for preprocessing the acquired indicator data further includes: Acquire indicator data and create real-time processing tasks using the data processing component; The real-time processing task is used to preprocess the indicator data.

5. A big data processing platform, characterized in that, The platform includes: The preprocessing unit is used to preprocess the acquired indicator data; A model matching unit is used to determine a target open-source event processing model that matches the indicator data from among the pre-set open-source event processing models, based on the data characteristics of the pre-processed indicator data and the model characteristics of each pre-set open-source event processing model. The target open-source event processing model is configured with pre-set target CEP alarm analysis rules. Each CEP rule on the target open-source event processing model is configured through a configuration unit. The CEP rules include at least CEP alarm analysis rules. The alarm analysis and processing unit is used to perform alarm analysis and processing on the indicator data using the pre-set target CEP alarm analysis rules configured on the target open-source event processing model, and obtain the corresponding alarm information. The configuration unit includes: A rule base creation unit is used to create a rule base in the message middleware; wherein, the rule base includes multiple initial CEP rules, and each initial CEP rule carries a corresponding rule weight; The first rule update unit is used to update each initial CEP rule according to the rule weight, historical alarm information and historical aggregation information corresponding to each initial CEP rule, so as to obtain the CEP rule corresponding to each initial CEP rule; wherein each CEP rule includes multiple CEP alarm analysis rules and multiple CEP aggregation rules; the historical aggregation information and the historical alarm information constitute historical information; The configuration subunit is used to determine, for each of the open-source event processing models, the CEP alarm analysis rules and the CEP aggregation rules that match the open-source event processing model from multiple CEP alarm analysis rules and multiple CEP aggregation rules based on the model characteristics of each open-source event processing model, and configure them onto the open-source event processing model.

6. The platform according to claim 5, characterized in that, The target open-source event processing model is also configured with pre-set target CEP aggregation rules, and the platform also includes: The aggregation processing unit is used to aggregate the indicator data using the pre-set target CEP aggregation rules configured on the target open-source event processing model to obtain aggregated data.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program code and data for data processing, and the processor being used to call program instructions in the memory to execute a data processing method as described in any one of claims 1-4.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform a data processing method as described in any one of claims 1-4.

Images