Data processing method, device, computer equipment and storage medium

Abstract

The present invention provides a data processing method, device, computer equipment and storage medium. The above method includes: obtaining target JSON data within a preset time period from a preset data source; connecting the target JSON data to a preset message queue; obtaining a preset target configuration file and alarm rules; based on a preset target processing engine, filtering out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue; performing alarm judgment on the abnormal object data based on the alarm rules to obtain an alarm judgment result corresponding to the abnormal object data; if the alarm judgment result is that the abnormal object data triggers an alarm, then executing the alarm processing corresponding to the abnormal object data. The present invention efficiently and intelligently implements quasi-real-time data quality monitoring and alarming for JSON data, effectively improving the processing efficiency and accuracy of data quality monitoring and alarming for JSON data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, computer equipment and storage medium. Background Art

[0002] In today's era of rapid development of information technology, unprecedented high standards have been put forward for the timeliness of data statistical calculations in the process of business operations. It is expected to quickly and accurately obtain business data and gain insights into the trends and patterns behind the data in order to quickly respond to market changes and optimize decision-making processes.

[0003] In order to meet this demand, data quality monitoring has become a key link in ensuring the accuracy and reliability of business data. It is necessary to conduct comprehensive inspections and assessments on multiple dimensions such as data integrity, accuracy, consistency, and timeliness, and promptly discover and warn of data quality issues to prevent them from having a negative impact on business decisions.

[0004] However, traditional data quality monitoring solutions often use offline parsing when facing JSON format data. That is, the JSON data is first parsed into structured data, and then formatted and stored in a distributed file system. On this basis, abnormal data is detected and alarms are triggered through offline batch processing. Although this processing method can meet the basic needs of data quality monitoring to a certain extent, the offline parsing and batch processing methods lead to obvious lags in the discovery and alarm of data quality problems. There is often a long time interval between the generation of data and the discovery of problems, which makes it difficult to obtain the true status of the data in a timely manner, and it is even more difficult to quickly take effective measures to correct data errors or optimize data processes. Therefore, how to improve the timeliness of JSON data quality monitoring and realize real-time parsing, monitoring and alarming of data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present invention is to provide a data processing method, apparatus, computer equipment and storage medium, aiming to solve the technical problem of insufficient timeliness in processing JSON data in the prior art.

[0006] To achieve the above object, the present invention provides a data processing method, which comprises:

[0007] Get the target JSON data within the preset time period from the preset data source;

[0008] Connect the target JSON data to a preset message queue;

[0009] Get the preset target configuration files and alarm rules;

[0010] Based on a preset target processing engine, filter out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue;

[0011] Performing an alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data;

[0012] If the alarm determination result is that the abnormal object data triggers an alarm, an alarm process corresponding to the abnormal object data is executed.

[0013] Optionally, the preset target processing engine based on which the abnormal object data not conforming to the target configuration file is screened out from the target JSON data in the message queue includes:

[0014] Invoking the target processing engine;

[0015] Based on the target processing engine, consume the target JSON data from the message queue;

[0016] Generate a corresponding target template based on the target configuration file;

[0017] Compare the target JSON data with the target template to identify whether the target JSON data conforms to the target configuration file and obtain a corresponding identification result;

[0018] Filter out, from the target JSON data, designated JSON data whose identification result is that it does not conform to the target configuration file;

[0019] The specified JSON data is used as the abnormal object data.

[0020] Optionally, performing alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data includes:

[0021] Counting the abnormal object data in groups, and calculating the number of the abnormal object data within the preset time period;

[0022] Obtaining an alarm quantity threshold corresponding to the alarm rule;

[0023] Determine whether the data quantity is greater than the alarm quantity threshold;

[0024] If so, generating a first alarm judgment result of the abnormal object data triggering an alarm;

[0025] If not, a second alarm judgment result indicating that the abnormal object data does not trigger an alarm is generated.

[0026] Optionally, the executing alarm processing corresponding to the abnormal object data includes:

[0027] Generate corresponding warning information based on the abnormal object data;

[0028] Get the preset target push method;

[0029] Identify designated personnel associated with alert push;

[0030] Based on the target push mode, the alarm information is sent to the designated person.

[0031] Optionally, the generating corresponding alarm information based on the abnormal object data includes:

[0032] Get the preset alarm template;

[0033] Based on the alarm template, obtaining associated information related to the abnormal object data;

[0034] Filling the abnormal object data and the associated information into corresponding positions in the alarm template to obtain a filled alarm template;

[0035] The filled alarm template is used as the alarm information.

[0036] Optionally, before obtaining the preset target configuration file and the alarm rule, the method further includes:

[0037] Determine whether a file configuration request corresponding to the JSON data triggered by the user is received;

[0038] If yes, display the preset file configuration interface;

[0039] Receiving an initial configuration file input by the user in the file configuration interface;

[0040] Performing content verification on the initial configuration file;

[0041] If the initial configuration file passes the content verification, the initial configuration file is used as the target configuration file, and the target configuration file is stored.

[0042] Optionally, before obtaining the preset target configuration file and the alarm rule, the method further includes:

[0043] Determine whether an alarm rule configuration request corresponding to the JSON data triggered by the user is received;

[0044] If yes, display the preset rule configuration interface;

[0045] Receiving the initial alarm rule input by the user in the rule configuration interface;

[0046] Performing rule verification on the initial alarm rule;

[0047] If the initial alarm rule passes the rule verification, the initial alarm rule is used as the alarm rule, and the alarm rule is stored.

[0048] In addition, to achieve the above object, the present invention further provides a data processing device, the data processing device comprising:

[0049] The first acquisition module is used to acquire target JSON data within a preset time period from a preset data source;

[0050] A first processing module, used for connecting the target JSON data to a preset message queue;

[0051] The second acquisition module is used to obtain a preset target configuration file and an alarm rule;

[0052] A screening module, used for screening out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue based on a preset target processing engine;

[0053] A second processing module, configured to perform an alarm judgment on the abnormal object data based on the alarm rule, and obtain an alarm judgment result corresponding to the abnormal object data;

[0054] The third processing module is used to execute alarm processing corresponding to the abnormal object data if the alarm judgment result is that the abnormal object data triggers an alarm.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0056] The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the data processing methods proposed in the embodiments of the present application are implemented.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0058] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of any one of the data processing methods proposed in the embodiments of the present application are implemented.

[0059] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0060] The present invention provides a data processing method, device, computer equipment and storage medium, the method comprising: firstly obtaining target JSON data within a preset time period from a preset data source; then connecting the target JSON data to a preset message queue; then obtaining a preset target configuration file and an alarm rule; subsequently, based on a preset target processing engine, filtering out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue; finally, based on the alarm rule, performing an alarm judgment on the abnormal object data to obtain an alarm judgment result corresponding to the abnormal object data; if the alarm judgment result is that the abnormal object data triggers an alarm, executing the alarm processing corresponding to the abnormal object data. The present invention obtains target JSON data within a preset time period from a preset data source, connects the target JSON data to a preset message queue, then obtains a preset target configuration file and an alarm rule, and then based on the use of a target processing engine, screens out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue, and then performs an alarm judgment on the abnormal object data based on the alarm rule, obtains an alarm judgment result corresponding to the abnormal object data, and when it is detected that the alarm judgment result is that the abnormal object data triggers an alarm, the alarm processing corresponding to the abnormal object data will be executed. The present invention connects JSON data to a real-time message queue and utilizes the real-time processing capability of the target processing engine to efficiently and intelligently realize quasi-real-time data quality monitoring and alarming for JSON data, effectively improving the processing efficiency and accuracy of data quality monitoring and alarming for JSON data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0064] Figure 3 is a schematic structural diagram of an embodiment of a data processing device according to the present application;

[0065] Figure 4 This is a basic structural block diagram of the computer device in this embodiment. DETAILED DESCRIPTION

[0066] The data processing method provided by the embodiment of the present invention is applied to a data processing device. Unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those generally understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0070] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social online platform software, etc.

[0071] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0072] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0073] It should be noted that the data processing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the data processing device is generally arranged in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0075] In today's era of rapid development of information technology, the continuous innovation of big data technology and the booming Internet have jointly promoted the rapid growth of data processing and analysis needs in all walks of life. Especially in the process of business operations, unprecedented high standards have been put forward for the timeliness of data statistical calculations. Enterprises not only require the ability to obtain business data quickly and accurately, but also expect to be able to gain real-time insights into the trends and laws behind the data, so as to quickly respond to market changes and optimize decision-making processes, thereby gaining an advantage in the fierce market competition.

[0076] In order to meet this demand, data quality monitoring has become a key link in ensuring the accuracy and reliability of business data. Data quality monitoring aims to comprehensively check and evaluate data integrity, accuracy, consistency, timeliness and other dimensions through a series of technical means, timely discover and warn of data quality problems, and prevent them from having a negative impact on business decisions.

[0077] JSON (JavaScript Object Notation) is a lightweight data exchange format that is widely used in data exchange in Web applications because it is easy for humans to read and write, and easy for machines to parse and generate. The JSON data format can flexibly represent various complex data structures through the combination of key-value pairs and arrays, which greatly facilitates data transmission and processing.

[0078] However, traditional data quality monitoring solutions often use offline parsing when facing JSON format data. That is, the JSON data is first parsed into structured data, and then formatted and stored in a distributed file system. On this basis, abnormal data is detected and alarms are triggered through offline batch processing. Although this processing method can meet the basic needs of data quality monitoring to a certain extent, the offline parsing and batch processing methods lead to a significant lag in the discovery and alarm of data quality problems. There is often a long time interval between the generation of data and the discovery of problems, which makes it difficult for enterprises to obtain the true status of the data in a timely manner, and it is even more difficult to quickly take effective measures to correct data errors or optimize data processes. Therefore, how to improve the timeliness of JSON data quality monitoring and realize real-time data parsing, monitoring and alarming has become a technical problem that needs to be solved urgently.

[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of the data processing method proposed in the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology.

[0080] The data processing method provided by the embodiment of the present invention comprises the following steps:

[0081] S210, obtaining target JSON data within a preset time period from a preset data source.

[0082] In this step, the execution subject of the present invention may be a data processing system. The data source may be a business data source, such as a real-time JSON data source. There is no limitation on the time selection of the preset time period, which may be set according to actual business needs, such as 5 minutes.

[0083] S220, accessing the target JSON data into a preset message queue.

[0084] In this step, you can select a suitable message queue, such as Kafka, RabbitMQ, etc., according to actual business needs. Then connect the above data source to the message queue to ensure that the acquired target JSON data can be sent to the message queue in real time. The message queue can be tested in advance to ensure that the connection between the data source and the message queue is normal and the data can be sent and received correctly by sending test data.

[0085] S230, obtaining a preset target configuration file and alarm rules.

[0086] In this step, the specific implementation process of the construction process of the above-mentioned target configuration file and alarm rules will be further described in detail in the subsequent specific embodiments of the present invention, and will not be elaborated on here.

[0087] S240, based on a preset target processing engine, filter out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue.

[0088] In this step, the specific implementation process of filtering out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue based on the preset target processing engine will be further described in detail in the subsequent specific embodiments of the present invention and will not be elaborated on here.

[0089] S250: Perform an alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data.

[0090] In this step, the specific implementation process of performing alarm judgment on the abnormal object data based on the alarm rule to obtain the alarm judgment result corresponding to the abnormal object data will be further described in detail in the subsequent specific embodiments of the present invention, and will not be elaborated here.

[0091] S260: If the alarm determination result is that the abnormal object data triggers an alarm, then an alarm process corresponding to the abnormal object data is executed.

[0092] In this step, the content of the alarm judgment result may include: the alarm judgment result is that the abnormal object data triggers an alarm, or the alarm judgment result is that the abnormal object data does not trigger an alarm. The specific implementation process of the above-mentioned execution of the alarm processing corresponding to the abnormal object data will be further described in detail in the subsequent specific embodiments of the present invention, and will not be elaborated on here.

[0093] In an embodiment of the present invention, first, target JSON data within a preset time period is obtained from a preset data source; then the target JSON data is connected to a preset message queue; then a preset target configuration file and an alarm rule are obtained; subsequently, based on a preset target processing engine, abnormal object data that does not conform to the target configuration file is screened out from the target JSON data in the message queue; finally, an alarm judgment is performed on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data; if the alarm judgment result is that the abnormal object data triggers an alarm, the alarm processing corresponding to the abnormal object data is executed. The present invention obtains the target JSON data within a preset time period from a preset data source, and connects the target JSON data to a preset message queue, then obtains a preset target configuration file and an alarm rule, then based on the use of the target processing engine, the abnormal object data that does not conform to the target configuration file is screened out from the target JSON data in the message queue, and then an alarm judgment is performed on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data, and when it is detected that the alarm judgment result is that the abnormal object data triggers an alarm, the alarm processing corresponding to the abnormal object data is executed. The present invention connects JSON data to a real-time message queue and utilizes the real-time processing capability of a target processing engine to efficiently and intelligently implement quasi-real-time data quality monitoring and alarming for JSON data, thereby effectively improving the processing efficiency and accuracy of data quality monitoring and alarming for JSON data.

[0094] Optionally, the preset target processing engine based on which the abnormal object data not conforming to the target configuration file is screened out from the target JSON data in the message queue includes:

[0095] The target processing engine is invoked.

[0096] In this step, the target processing engine may specifically use the Flink real-time computing engine. Flink is an open source framework and distributed processing engine for stateful computing of unbounded and bounded data streams. Flink is designed to run in all common cluster environments and perform computing at memory speed and at any scale.

[0097] Based on the target processing engine, the target JSON data is consumed from the message queue.

[0098] In this step, the target JSON data in the message queue can be consumed in real time by using the target processing engine.

[0099] A corresponding target template is generated based on the target configuration file.

[0100] In this step, the target processing engine can generate a matching template, namely the above-mentioned target template, according to the pre-configured target configuration file, namely the JSON Schema description file. Among them, a JSON Schema description file is configured in advance according to the needs of the business data source. This file defines the structure of the JSON data, including field names, data types, whether the field is required, etc. In addition, the FlatMapFunction interface of Flink can be implemented in JAVA language, and then the corresponding target template can be generated according to the configured JSON Schema description file.

[0101] The target JSON data is structurally compared with the target template to identify whether the target JSON data conforms to the target configuration file, and a corresponding identification result is obtained.

[0102] In this step, the target JSON data and the target template are compared by comparing their structures, that is, comparing the target JSON data and the target template, to identify whether the target JSON data conforms to the above target configuration file, and generate a corresponding recognition result. The content of the recognition result includes: conforming to the target configuration file, or not conforming to the target configuration file.

[0103] Filter out the designated JSON data whose identification result is that it does not conform to the target configuration file from the target JSON data.

[0104] In this step, the recognition result can be analyzed to filter out the designated JSON data whose recognition result is that it does not conform to the target configuration file from the target JSON data.

[0105] The specified JSON data is used as the abnormal object data.

[0106] In this embodiment, by calling the target processing engine; then based on the target processing engine, consuming the target JSON data from the message queue; then generating a corresponding target template based on the target configuration file; subsequently comparing the target JSON data with the target template to identify whether the target JSON data conforms to the target configuration file, and obtaining a corresponding recognition result; further filtering out the specified JSON data whose recognition result is not conforming to the target configuration file from the target JSON data, and using the specified JSON data as the abnormal object data. The present invention consumes the target JSON data from the message queue based on the use of the target processing engine, and generates a corresponding target template based on the target configuration file, and then compares the target JSON data with the target template to identify whether the target JSON data conforms to the target configuration file, and obtains a corresponding recognition result, and then according to the obtained recognition result, filtering out the specified JSON data whose recognition result is not conforming to the target configuration file from the target JSON data and using it as the final abnormal object data, so that based on the use of the target processing engine, it can be realized that the abnormal object data that does not conform to the target configuration file within a preset time period can be quickly and accurately filtered out from the target JSON data in the message queue, effectively improving the screening efficiency of the abnormal object data, and ensuring the accuracy of the obtained abnormal object data.

[0107] Optionally, performing alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data includes:

[0108] The abnormal object data are grouped and counted to obtain the number of the abnormal object data within the preset time period.

[0109] In this step, the target processing engine may be used to group and count the abnormal object data to calculate the number of abnormal object data that do not conform to the target configuration file within the preset time period.

[0110] Obtain the alarm quantity threshold corresponding to the alarm rule.

[0111] In this step, the alarm quantity threshold value pre-configured in the above alarm rule can be obtained by extracting information about the alarm threshold value from the above alarm rule. The alarm quantity threshold value of the number of JSON data that does not conform to the target configuration file, i.e., the JSON Schema description file, can be determined according to business needs. For example, the alarm quantity threshold value is set to 5. If more than 5 pieces of data are continuously exceeded, for example, 6 pieces of data do not conform to the JSON Schema description file, an alarm is triggered.

[0112] Determine whether the data quantity is greater than the alarm quantity threshold.

[0113] In this step, the data quantity of the abnormal object data can be numerically compared with the alarm quantity threshold to obtain a corresponding numerical comparison result, wherein the content of the numerical comparison result may include: the data quantity is greater than the alarm quantity threshold, or the data quantity is not greater than the alarm quantity threshold.

[0114] If so, a first alarm judgment result of the abnormal object data triggering an alarm is generated.

[0115] In this step, if it is detected that the data quantity is greater than the alarm quantity threshold, it is determined that the abnormal object data triggers a pre-configured alarm rule, and then a first alarm judgment result that the abnormal object data triggers an alarm is generated.

[0116] If not, a second alarm judgment result indicating that the abnormal object data does not trigger an alarm is generated.

[0117] In this step, if it is detected that the data quantity is not greater than the alarm quantity threshold, it is determined that the abnormal object data does not trigger the pre-configured alarm rule, and then a second alarm judgment result that the abnormal object data does not trigger an alarm is generated.

[0118] In this embodiment, the number of data of the abnormal object data within the preset time period is counted by grouping and counting the abnormal object data; then the alarm number threshold value corresponding to the alarm rule is obtained; subsequently, it is determined whether the number of data is greater than the alarm number threshold value; if so, a first alarm judgment result indicating that the abnormal object data triggers an alarm is generated; if not, a second alarm judgment result indicating that the abnormal object data does not trigger an alarm is generated. The present invention counts the number of data of the abnormal object data within the preset time period by grouping and counting the abnormal object data, and obtains the alarm number threshold value corresponding to the alarm rule, and then numerically compares the number of data of the abnormal object data with the alarm number threshold value, so that the alarm judgment processing for the abnormal object data can be completed efficiently and accurately based on the obtained numerical comparison result, thereby ensuring the data accuracy of the alarm judgment result corresponding to the abnormal object data.

[0119] Optionally, the executing alarm processing corresponding to the abnormal object data includes:

[0120] Generate corresponding warning information based on the abnormal object data.

[0121] In this step, the specific implementation process of generating corresponding alarm information based on the abnormal object data will be further described in detail in the subsequent specific embodiments of the present invention, and will not be elaborated on here.

[0122] Get the preset target push method.

[0123] In this step, the information about the alarm method can be extracted from the alarm rules to obtain the push method corresponding to the alarm information, that is, the target push method. The target push method can be selected according to actual business needs, such as email, SMS or instant messaging tools.

[0124] Identify designated personnel associated with alert push.

[0125] In this step, information about the alarm recipient can be extracted from the above alarm rules to obtain the pre-specified R&D personnel or team that receives the alarm information, that is, the above-specified personnel.

[0126] Based on the target push mode, the alarm information is sent to the designated person.

[0127] In this step, the above-mentioned target push method can be used to complete the alarm push processing of sending the alarm information to the designated personnel. In addition, after receiving the alarm information, the designated personnel can promptly discover and repair the structural change problem in the data source, thereby ensuring data quality.

[0128] In this embodiment, the corresponding alarm information is generated based on the abnormal object data; then a preset target push method is obtained; then a designated person related to the alarm push is determined; and subsequently, the alarm information is sent to the designated person based on the target push method. The present invention generates corresponding alarm information based on the abnormal object data, obtains a preset target push method, and determines a designated person related to the alarm push, and then sends the alarm information to the designated person based on the target push method, so that in the scenario where the alarm judgment result is that the abnormal object data triggers an alarm, it is possible to automatically and intelligently complete the execution of the alarm processing corresponding to the abnormal object data, thereby improving the processing standardization and processing intelligence of the alarm processing.

[0129] Optionally, the generating corresponding alarm information based on the abnormal object data includes:

[0130] Get the preset alarm template.

[0131] In this step, a template with alarm information, i.e., the above-mentioned alarm template, can be pre-written according to actual business needs. Exemplarily, the alarm template may include multiple information areas such as an alarm type area, an occurrence time area, an abnormal object data area, and an abnormal object data quantity area.

[0132] Based on the alarm template, associated information related to the abnormal object data is obtained.

[0133] In this step, the above-mentioned associated information may include the target alarm type related to the abnormal object data, the target occurrence time, and the data quantity of the abnormal object data.

[0134] The abnormal object data and the associated information are filled into corresponding positions in the alarm template to obtain a filled alarm template.

[0135] In this step, the above-mentioned abnormal object data can be filled into the abnormal object data area in the alarm template, and the target alarm type, target occurrence time, and data quantity of the abnormal object data contained in the above-mentioned related information can be filled into the alarm type area, occurrence time area and abnormal object data quantity area in the alarm template respectively, so as to obtain the filled alarm template.

[0136] The filled alarm template is used as the alarm information.

[0137] In this embodiment, a preset alarm template is obtained; then, based on the alarm template, associated information related to the abnormal object data is obtained; then, the abnormal object data and the associated information are filled into corresponding positions in the alarm template to obtain a filled alarm template; and subsequently, the filled alarm template is used as the alarm information. The present invention obtains a preset alarm template, and based on the use of the alarm template, associated information related to the abnormal object data is obtained, and then the abnormal object data and the associated information are filled into corresponding positions in the alarm template, so that the alarm information corresponding to the abnormal object data can be automatically and accurately generated, thereby improving the standardization and accuracy of the generation of alarm information.

[0138] Optionally, before obtaining the preset target configuration file and the alarm rule, the method further includes:

[0139] Determine whether a file configuration request corresponding to the JSON data triggered by the user is received.

[0140] In this step, the file configuration request is a request triggered by a user for configuring a target configuration file required for alarm processing of JSON data.

[0141] If so, display the default file configuration interface.

[0142] In this step, the file configuration interface is a user-friendly interface developed in advance to assist users in performing file configuration.

[0143] An initial configuration file input by the user in the file configuration interface is received.

[0144] In this step, the user can enter the corresponding initial configuration file in the file configuration interface according to actual personal needs or business needs. Specifically, the configuration process of the initial configuration file includes: first, according to the needs of the business data source, determine the field names in the JSON data, such as user ID, name, age, etc. Then define the data type for each field, such as string, integer, floating point number, etc. Then set the field requirements to specify which fields are required and which fields are optional. Subsequently, write the JSON Schema file and use the standard format of JSON Schema to write the description file to ensure that the file complies with the specifications of JSONSchema. Exemplarily, the configuration content of the initial configuration file is as follows:

[0145]

[0146]

[0147] The JSON data schema complies with the draft 04 version specification. The type is object and contains fields such as name, gender, age, and mobile phone number. The name and gender data types are strings, the age data type is integer, and the minimum value is 0. The minimum value of 0 is not excluded. The mobile phone number data type is a string. To meet the above regular expression, the required fields are name and gender, and other fields can be included.

[0148] In addition, the configuration content of the initial configuration file may also include the judgment of verifying JSON Schema, as follows:

[0149]

[0150] if (errors.isEmpty()) {

[0151] right=true;

[0152] }

[0153] return right;

[0154] }catch(IOException e){

[0155] return false;

[0156] }

[0157] }

[0158] The initial configuration file is verified.

[0159] In this step, the content verification may include performing a syntax check on the initial configuration file. The initial configuration file may be subjected to a syntax check by using a common syntax check tool to obtain a corresponding syntax check result. The syntax check result includes: the initial configuration file passes the syntax check or the initial configuration file fails the syntax check. In addition, if it is detected that the initial configuration file passes the syntax check, it is determined that the initial configuration file passes the content check, otherwise it is determined that the initial configuration file fails the content check.

[0160] If the initial configuration file passes the content verification, the initial configuration file is used as the target configuration file, and the target configuration file is stored.

[0161] In this step, there is no specific limitation on the storage method of the target configuration file, which can be determined according to the actual business usage requirements. Exemplarily, the storage method of the target configuration file can be any one of local database storage, local disk storage, network disk storage, blockchain storage, etc. Among them, blockchain storage is preferably used as the storage method of the target configuration file. By using blockchain storage to store and manage the target configuration file, the security and non-tamperability of the target configuration file can be effectively guaranteed.

[0162] In this embodiment, it is determined whether a file configuration request corresponding to JSON data triggered by a user is received; if so, a preset file configuration interface is displayed; then the initial configuration file input by the user in the file configuration interface is received; the initial configuration file is subsequently verified for content; if the initial configuration file passes the content verification, the initial configuration file is used as the target configuration file, and the target configuration file is stored. After receiving a file configuration request corresponding to JSON data triggered by a user, the present invention displays a preset file configuration interface, and based on the use of the file configuration interface, the user can input the initial configuration file according to personal needs, and then the initial configuration file is intelligently verified for content, and after the initial configuration file passes the content verification, the initial configuration file is used as the target configuration file, and the target configuration file is stored, thereby realizing the rapid and intelligent generation of a target configuration file that meets user needs, effectively improving the accuracy and usability of the generated target configuration file, improving the user experience, and ensuring the security of the generated target configuration file.

[0163] Optionally, before obtaining the preset target configuration file and the alarm rule, the method further includes:

[0164] Determine whether an alarm rule configuration request corresponding to the JSON data triggered by the user is received.

[0165] In this step, the alarm rule configuration request is a request triggered by a user for configuring alarm rules required for alarm processing of JSON data.

[0166] If yes, the preset rule configuration interface is displayed.

[0167] In this step, the file configuration interface is a pre-developed user-friendly interface for assisting users in configuring alarm rules.

[0168] Receive the initial alarm rule input by the user in the rule configuration interface.

[0169] In this step, the user can enter the corresponding initial alarm rules in the rule configuration interface according to actual personal needs or business needs. Specifically, the configuration process of the initial alarm rule includes: 1) Determine the alarm quantity threshold: According to business needs, determine the threshold of the number of JSON data that does not conform to the JSON Schema description file. 2) Configure the alarm method: Determine the push method of the alarm information, such as email, SMS or instant messaging tools. 3) Set the alarm recipient: Specify the R&D personnel or team who receives the alarm information. Exemplary, the configuration content of the initial alarm rule is as follows:

[0170] public void process(String s,Context context,

[0171] java.lang.Iterable <Tuple2<String,Integer> >values,Collector <string>collector)

[0172] long sum = 0L;

[0173] boolean sendAlarm = true;

[0174] for(Tuple2<String,Integer> value:values){

[0175] if(value.f0.equalsIgnoreCase("false")){

[0176] sum+=value.f1;

[0177] if(sendAlarm&&sum>=ExceptionThreshold){

[0178] FeiShuAlarm.sendFeiShu("xxx","xxx schema is wrong");

[0179] sendAlarm = false;

[0180]

[0181] A rule check is performed on the initial alarm rule.

[0182] In this step, the above rule verification includes syntax verification, logic verification and conflict verification. Among them, syntax verification includes verifying whether the syntax of the initial alarm rule conforms to the syntax rules. Logic verification includes verifying whether the logical relationship of the initial alarm rule is reasonable. Conflict verification includes verifying whether there is a potential conflict between the initial alarm rule and other rules. Specifically, only when it is detected that the initial alarm rule has passed the syntax verification, logic verification and conflict verification at the same time, will it be determined that the initial alarm rule has passed the rule verification, otherwise it is determined that the initial alarm rule has not passed the rule verification.

[0183] If the initial alarm rule passes the rule verification, the initial alarm rule is used as the alarm rule, and the alarm rule is stored.

[0184] In this step, there is no specific limitation on the storage method of the target configuration file, which can be determined according to actual business usage requirements. The specific storage process can refer to the above-mentioned storage process of the target configuration file, which will not be described in detail here.

[0185] In this embodiment, it is determined whether an alarm rule configuration request corresponding to JSON data triggered by a user is received; if so, a preset rule configuration interface is displayed; then the initial alarm rule input by the user in the rule configuration interface is received; and the initial alarm rule is subsequently subjected to rule verification. If the initial alarm rule passes the rule verification, the initial alarm rule is used as the alarm rule, and the alarm rule is stored and processed. After receiving the alarm rule configuration request corresponding to JSON data triggered by a user, the present invention displays a preset rule configuration interface, and based on the use of the rule configuration interface, the user can input the initial alarm rule according to personal needs, and then the initial alarm rule is intelligently verified, and after the initial alarm rule passes the rule verification, the initial alarm rule is used as the alarm rule, and the alarm rule is stored and processed, thereby realizing the rapid and intelligent generation of alarm rules that meet user needs, effectively improving the accuracy and availability of the generated alarm rules, improving the user experience, and ensuring the security of the generated alarm rules.

[0186] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data processing device 300, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0187] An embodiment of the present invention provides a data processing device 300, the data processing device 300 comprising:

[0188] The first acquisition module 310 is used to acquire target JSON data within a preset time period from a preset data source;

[0189] The first processing module 320 is used to connect the target JSON data to a preset message queue;

[0190] The second acquisition module 330 is used to acquire a preset target configuration file and an alarm rule;

[0191] A screening module 340 is used to screen out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue based on a preset target processing engine;

[0192] A second processing module 350 is used to perform an alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data;

[0193] The third processing module 360 ​​is configured to execute an alarm process corresponding to the abnormal object data if the alarm determination result is that the abnormal object data triggers an alarm.

[0194] Optionally, the screening module 340 includes:

[0195] A calling submodule, used for calling the target processing engine;

[0196] A consumption submodule, used for consuming the target JSON data from the message queue based on the target processing engine;

[0197] A first generating submodule, configured to generate a corresponding target template based on the target configuration file;

[0198] An identification submodule, used to compare the target JSON data with the target template to identify whether the target JSON data conforms to the target configuration file and obtain a corresponding identification result;

[0199] A screening submodule, used for screening out, from the target JSON data, designated JSON data whose identification result is that it does not conform to the target configuration file;

[0200] The first determination submodule is used to use the specified JSON data as the abnormal object data.

[0201] Optionally, the second processing module 350 includes:

[0202] A statistics submodule, used for grouping and counting the abnormal object data, and counting the number of the abnormal object data within the preset time period;

[0203] A first acquisition submodule is used to obtain an alarm quantity threshold corresponding to the alarm rule;

[0204] A judgment submodule, used to judge whether the data quantity is greater than the alarm quantity threshold;

[0205] A second generating submodule is used to generate a first alarm judgment result of the abnormal object data triggering an alarm if yes;

[0206] The third generating submodule is used to generate a second alarm judgment result indicating that the abnormal object data does not trigger an alarm.

[0207] Optionally, the third processing module 360 ​​includes:

[0208] A fourth generating submodule, configured to generate corresponding warning information based on the abnormal object data;

[0209] The second acquisition submodule is used to acquire a preset target push mode;

[0210] The second determination submodule is used to determine the designated personnel related to the alarm push;

[0211] The sending submodule is used to send the alarm information to the designated person based on the target push mode.

[0212] Optionally, the fourth generation submodule includes:

[0213] A first acquisition unit, used to acquire a preset alarm template;

[0214] A second acquisition unit, configured to acquire associated information related to the abnormal object data based on the alarm template;

[0215] A filling unit, used to fill the abnormal object data and the associated information into corresponding positions in the alarm template to obtain a filled alarm template;

[0216] A determination unit is used to use the filled alarm template as the alarm information.

[0217] Optionally, the data processing device 300 further includes:

[0218] A first judgment module is used to judge whether a file configuration request corresponding to the JSON data triggered by the user is received;

[0219] A first display module is used to display a preset file configuration interface if yes;

[0220] A first receiving module, used to receive an initial configuration file input by the user in the file configuration interface;

[0221] A first verification module, used to perform content verification on the initial configuration file;

[0222] The first storage module is used to use the initial configuration file as the target configuration file if the initial configuration file passes the content verification, and store the target configuration file.

[0223] Optionally, the data processing device 300 further includes:

[0224] The second judgment module is used to judge whether an alarm rule configuration request corresponding to the JSON data triggered by the user is received;

[0225] The second display module is used to display the preset rule configuration interface;

[0226] A second receiving module, used to receive the initial alarm rule input by the user in the rule configuration interface;

[0227] A second verification module, used to perform rule verification on the initial alarm rule;

[0228] The second storage module is used to use the initial alarm rule as the alarm rule if the initial alarm rule passes the rule verification, and store the alarm rule.

[0229] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0230] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0231] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0232] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the data processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0233] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the data processing method.

[0234] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0235] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores the application crash processing program, and the application crash processing program can be executed by at least one processor to enable the at least one processor to perform the steps of the data processing method as described above.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0237] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0238] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.< / string>

Claims

1. A data processing method, characterized in that: include: Get the target JSON data within the preset time period from the preset data source; Connect the target JSON data to a preset message queue; Get the preset target configuration files and alarm rules; Based on a preset target processing engine, filter out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue; Performing an alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data; If the alarm determination result is that the abnormal object data triggers an alarm, an alarm process corresponding to the abnormal object data is executed.

2. The method according to claim 1, characterized in that The preset target processing engine filters out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue, including: Invoking the target processing engine; Based on the target processing engine, consume the target JSON data from the message queue; Generate a corresponding target template based on the target configuration file; Compare the target JSON data with the target template to identify whether the target JSON data conforms to the target configuration file and obtain a corresponding identification result; Filter out, from the target JSON data, designated JSON data whose identification result is that it does not conform to the target configuration file; The specified JSON data is used as the abnormal object data.

3. The method according to claim 1, characterized in that The performing alarm judgment on the abnormal object data based on the alarm rule to obtain an alarm judgment result corresponding to the abnormal object data includes: Counting the abnormal object data in groups, and calculating the number of the abnormal object data within the preset time period; Obtaining an alarm quantity threshold corresponding to the alarm rule; Determine whether the data quantity is greater than the alarm quantity threshold; If so, generating a first alarm judgment result of the abnormal object data triggering an alarm; If not, a second alarm judgment result indicating that the abnormal object data does not trigger an alarm is generated.

4. The method according to claim 1, characterized in that The performing of alarm processing corresponding to the abnormal object data includes: Generate corresponding warning information based on the abnormal object data; Get the preset target push method; Identify designated personnel associated with alert push; Based on the target push mode, the alarm information is sent to the designated person.

5. The method according to claim 4, characterized in that The generating corresponding alarm information based on the abnormal object data includes: Get the preset alarm template; Based on the alarm template, obtaining associated information related to the abnormal object data; Filling the abnormal object data and the associated information into corresponding positions in the alarm template to obtain a filled alarm template; The filled alarm template is used as the alarm information.

6. The method according to claim 1, characterized in that Before obtaining the preset target configuration file and the alarm rule, the method further includes: Determine whether a file configuration request corresponding to the JSON data triggered by the user is received; If yes, display the preset file configuration interface; Receiving an initial configuration file input by the user in the file configuration interface; Performing content verification on the initial configuration file; If the initial configuration file passes the content verification, the initial configuration file is used as the target configuration file, and the target configuration file is stored.

7. The method according to claim 1, characterized in that Before obtaining the preset target configuration file and the alarm rule, the method further includes: Determine whether an alarm rule configuration request corresponding to the JSON data triggered by the user is received; If yes, display the preset rule configuration interface; Receiving the initial alarm rule input by the user in the rule configuration interface; Performing rule verification on the initial alarm rule; If the initial alarm rule passes the rule verification, the initial alarm rule is used as the alarm rule, and the alarm rule is stored.

8. A data processing device, characterized in that: include: The first acquisition module is used to acquire target JSON data within a preset time period from a preset data source; A first processing module, used for connecting the target JSON data to a preset message queue; The second acquisition module is used to obtain a preset target configuration file and an alarm rule; A screening module, used for screening out abnormal object data that does not conform to the target configuration file from the target JSON data in the message queue based on a preset target processing engine; A second processing module, configured to perform an alarm judgment on the abnormal object data based on the alarm rule, and obtain an alarm judgment result corresponding to the abnormal object data; The third processing module is used to execute alarm processing corresponding to the abnormal object data if the alarm judgment result is that the abnormal object data triggers an alarm.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images