Log dynamic analysis and verification method and system based on Flink and JS engine

Abstract

The invention belongs to the technical field of Internet, and provides a log dynamic analysis and verification method and system based on an Flink and a JS engine, and the technical scheme is as follows: configuring a configuration file and environment variable dependency information required by an Flink environment and a Pulsar environment; configuring a dynamic log data access table of to-be-analyzed business data in the Pulsar environment according to business requirements; reading data in the dynamic log data access table in the Flink application, judging whether the read log data conforms to a set format or not, if so, calling an analysis and verification rule table, traversing analysis and verification rules in combination with a JS engine, filtering data which does not conform to the analysis and verification rules into an early warning information table, and sending the data into the early warning information table; and writing the successfully analyzed data into the corresponding theme to obtain an analysis result. Code updating and application starting and stopping do not need to be carried out, and the secondary development cost of research and development personnel is effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the field of Internet technologies, and in particular, relates to a method and system for dynamically parsing and verifying logs based on Flink and a JS engine. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] The dynamic parsing of log data access is extremely important for R & D personnel, operation personnel, and operation and maintenance personnel. The technology of dynamically parsing log data access based on the Flink framework is even more important in big data scenarios;

[0004] Currently, for the data parsing solution of the Flink engine, such as the solution with the publication number CN 110737631A and the invention name of a data parsing method and device based on the Flink engine, it overcomes the problem that existing computing engines cannot meet multiple data formats. It can generate a parser according to the parsing logic configured by the user and the data source information, and parse the data to be parsed in different formats to meet multiple business scenarios;

[0005] Another example is the solution with the publication number CN116126906A and the invention name of a method and system for monitoring the quality of streaming data based on Flink. It can monitor streaming data through the combined components and designed operators of Flink, and ensure and monitor the quality of streaming data to a certain extent.

[0006] However, in the field of log reception and processing, there are various situations such as multiple data formats, chaotic formats, and inconsistent parameter types at the log source end, which are likely to cause redundant work for log parsing manufacturers and developers, increase development costs, and affect the development cycle, seriously affecting data parsing and use. Conventional parsing methods and practices can meet the log parsing and verification at that time. With the update of log types, the addition and expansion of collection devices, etc., conventional parsing methods and practices are difficult to expand, and the support for parsing and verifying new types and new formats of data is low. Summary of the Invention

[0007] In order to solve at least one of the technical problems in the above background art, the present invention provides a method and system for dynamically parsing and verifying logs based on Flink and a JS engine. Based on the real-time data stream processing framework of Flink, it is widely applied to big data processing services. The incorporated JS engine, combined with the set parsing and verification rule traversal rules, makes the parsing and verification of data simpler, easier to use, and more convenient, enhances the compatibility of the entire data processing, reduces the operation difficulty of data verification, and reduces personnel costs.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A first aspect of the present invention provides a method for dynamically parsing and verifying logs based on Flink and a JS engine, including the following steps:

[0010] Configure the configuration files and environment variable dependency information required for the Flink environment and the Pulsar environment;

[0011] According to the requirements of the business, configure the topic for dynamically logging access data of the business data to be parsed in the Pulsar environment;

[0012] Configure a parsing and verification rule table for dynamic log data in the database;

[0013] Read the dynamic log access data in the Flink environment according to the topic, and determine whether the read log data conforms to the set format. If it conforms, call the parsing and verification rule table, combine with the JS engine, traverse the parsing and verification rules, and filter the data that does not conform to the parsing and verification rules into the warning information table. If it does not conform, send the data to the warning information table;

[0014] Write the successfully parsed data into the corresponding topic to obtain the parsing result.

[0015] Furthermore, the fields and data of the configured parsing and verification rule table, including data type data_type, data type code data_code, field name col_name, field default value default_value, field type col_type, field mapping name col_parse, and verification rule col_validator, are js statements or js code that meet the syntax definition of the js engine.

[0016] Furthermore, the process of traversing the parsing and verification rules includes:

[0017] Perform JSON formatting on the accessed data, obtain the data type acq_type in the formatted data, and match the corresponding data type in the parsing and verification list;

[0018] According to the obtained data type data_type, parse the JSON through a third-party open-source component, and convert and obtain all keys in the JSON;

[0019] Then, according to the key, match the field name col_name in the parsing rule, replace all key values in the JSON with the new column name data_parser information, and match the corresponding verification rule col_validator information;

[0020] Data is verified according to the verification rule col_validator. If it meets the JavaScript statements or JavaScript code defined by the JavaScript engine syntax, the output result is encapsulated into a Map collection for output and parsing to complete the parsing and output of JSON data. Otherwise, it is excluded as abnormal data or formatted.

[0021] Further, the content of the acq_type field in the data stream of the message middleware is matched and associated with the data type of the mapping table through the constructed asynchronous operator AsyncFunction.

[0022] Further, dynamic log data is read in the Flink environment, including: reading corresponding dynamic log data based on the predefined message middleware according to the dynamic log access data topic. The predefined message middleware is defined through a third-party jar package or self-written.

[0023] Further, the dynamic data of the data to be parsed includes the field name, field type, mapping name, default value information, data type code, and data type name corresponding to the business data.

[0024] The second aspect of the present invention provides a log dynamic parsing and verification system based on Flink and JavaScript engine, including:

[0025] An environment information configuration module, which is used to configure the configuration files and environment variable dependency information required for the Flink environment and Pulsar environment;

[0026] A data access configuration module, which is used to configure the topic of the dynamic log access data of the business data to be parsed in the Pulsar environment according to the business requirements;

[0027] A rule verification configuration module, which is used to configure the parsing and verification rule table of the dynamic log data in the database;

[0028] A parsing module, which is used to read the dynamic log access data in the Flink environment according to the topic, determine whether the read log data meets the set format. If it meets, it calls the parsing and verification rule table, combines with the JavaScript engine, traverses the parsing and verification rules, filters the data that does not meet the parsing and verification rules into the warning information table. If it does not meet, it sends the data to the warning information table; writes the successfully parsed data into the corresponding topic to obtain the parsing result.

[0029] The third aspect of the present invention provides a computer-readable storage medium.

[0030] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in a log dynamic parsing and verification method based on Flink and JavaScript engine as described above.

[0031] The fourth aspect of the present invention provides a computer device.

[0032] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps in a log dynamic parsing and verification method based on Flink and a JS engine as described above are implemented.

[0033] The fifth aspect of the present invention provides a program product.

[0034] A program product, which is a computer program product, includes a computer program. When the computer program is executed by a processor, the steps in a log dynamic parsing and verification method based on Flink and a JS engine as described above are implemented.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] 1. The present invention has stronger compatibility and simpler operation for log data parsing and extension. The configuration of verification rules is more convenient, and there is no need to update the code and start and stop the application. It effectively reduces the cost of secondary development for R & D personnel. After the application of the present invention, the following advantages can be achieved: 1) New types of log data can be added, and the configuration types and configuration items can be added in a timely manner to complete the parsing and verification of new type data; 2) The verification rules of old type log data can be changed in real time, and the verification of new rules can be carried out in real time without starting and stopping the application, without affecting the real-time nature of the data; 3) Effective interception of impurity data, and some impurity data do not need to be configured and can be directly filtered into the warning data.

[0037] 2. The present invention can complete the conversion of different data types and different data through the configured parsing rules and verification rules, and is compatible with integer type, long integer type, string, JSON, JSONArray, date type, date and time type, and BigDecimal, meeting the governance of standard data.

[0038] 3. The present invention has strong compatibility and high real-time nature. R & D personnel or operation personnel can adjust the access rules and verification rules in real time and the parsing and verification of log data will take effect immediately, quickly and accurately completing problems such as data parsing and warehousing, improving the development and operation and maintenance efficiency.

[0039] 4. The present invention innovatively introduces a JS engine for data verification, effectively solving the randomness and uncertainty of data verification in dynamic data access. Through a configurable method, the verification rules can be updated in real time and the verification of complex data can be effectively solved. It effectively avoids mixing the verification rules in the code and can modify the rules immediately according to the scenario requirements.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. Brief Description of the Drawings

[0041] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0042] Figure 1 is the existing overall Flink process;

[0043] Figure 2 is the source - end data to the message queue;

[0044] Figure 3 is a flowchart of a method for dynamically parsing and verifying logs based on Flink and a JS engine provided by an embodiment of the present invention;

[0045] Figure 4 is a flowchart of a method for traversing and parsing verification rules provided by an embodiment of the present invention. Detailed Embodiments

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0049] The existing development process for dynamic parsing based on the protobuf protocol is suitable for data transmission scenarios with high performance and requirements for response speed. Since protobuf is a binary data format, encoding and decoding are required. The data itself is not readable, so only after deserialization can the truly readable data be obtained. Currently, the more extensive application development of protobuf is based on C++. It is not very friendly to the Java development language in Flink applications. Only with the help of third - party software or components can the generation of protobuf definition files be achieved, increasing the development steps and resources.

[0050] After the project or application starts, it is impossible to make changes to protobuf. All dynamic parsing is based on the content of the already generated protobuf files. For some data types that may be temporary or newly added, the original application needs to adjust the definition content of the protobuf file and restart the application for the parsing and reading of the new data types to take effect; the definition method is too complex and close to the programming language level, with a high learning cost. In actual project experience, it is very difficult for operation and maintenance personnel to learn and improve the rules entirely by themselves.

[0051] The Flink framework currently supports more modular extensions in the ecosystem. First, application personnel only need to add or delete modules according to module requirements, and more conveniently reference the connectors in the Flink ecosystem, providing access to various data sources and more ways to write data to different data sources; second, to ensure the independence and non-conflict characteristics when modules are dynamically loaded, Flink uses class loader isolation technology; third, to define connectors, it is necessary to comply with the Connectors API interfaces defined by Flink for data sources and data sinks to ensure the compatibility of the functions of new modules with the existing architecture. Finally, it is necessary to reasonably configure the dependency management of the project to solve various possible library conflict problems. Developers must encapsulate the underlying dependency frameworks referenced by Connectors themselves, see Figure 1 。

[0052] The dynamic parsing of log data access is extremely important for R & D personnel, operation personnel, and operation and maintenance personnel. The application of the dynamic parsing technology of log data access based on the Flink framework is even more important in the big data scenario;

[0053] In the early stage, business personnel conduct a comprehensive investigation and analysis of data types, data formats, etc. based on the data within the business scenario, and compatibly conduct preliminary design, definition, and scalability planning for the data entry within the business to form a data standard that can meet all data access. The source data is configured into a standard format that conforms to the data standard through methods such as api, logstash, nifi, and flume, and output to a unified message queue. Use the Flink framework to define the application and configure the parser to complete the data association between the message middleware data and the dynamic parsing table, and perform operations such as extraction, parsing, splitting, and conversion on the data in the message queue according to the matching parsing rule list, see Figure 2 。

[0054] However, in the field of log reception and processing, there are various situations such as multiple and chaotic data formats at the log source side, and inconsistent parameter types, which can easily cause redundant work for log parsing manufacturers and developers, increase development costs, and affect the development cycle, seriously affecting data parsing and usage. Conventional parsing methods and practices can meet the log parsing and verification at that time. With the update of log types, the addition and expansion of collection devices, etc., conventional parsing methods and practices are difficult to expand, and have low support for parsing and verifying new types and new format data. There are various template type data in the log processing field, and new types of template data are added with the expansion of business. The present invention can complete the parsing, verification of new type data, and output the parsed data by configuring new data sources, data source field mapping, and field verification rules.

[0055] Embodiment 1

[0056] As Figure 3 shown, this embodiment provides a method for dynamically parsing and verifying logs based on Flink and JS engine, including the following steps:

[0057] Step 1: Build a Flink environment and a Pulsar environment, and configure the configuration files and environment variable dependency information required for the Flink environment and the Pulsar environment;

[0058] In this embodiment, install Pulsar, including building Pulsar nodes on the server side and enabling the authentication mode; install Flink nodes; in application development, it is necessary to build a Java application based on idea and apply the relevant dependent jar packages of Pulsar and Flink to complete a preliminary environment construction.

[0059] Step 2: According to the business requirements, configure the topic for dynamically accessing log data of the business data to be parsed in the Pulsar environment;

[0060] In this embodiment, the dynamically accessed data of the data to be parsed includes information such as the field name, field type, mapping name, default value, etc. corresponding to the business data, data type encoding, and data type name;

[0061] Step 3: Configure a parsing and verification rule table for dynamic log data in the database;

[0062] The data type data_type, data type encoding data_code, field name col_name, field default value default_value, field type col_type, field mapping name col_parse, and verification rule col_validator are js statements or js code that meet the js engine syntax definition.

[0063] Taking a certain business as an example, the parsing and verification rules for its corresponding dynamic log data are shown in Table 1:

[0064] Table 1 Parsing and Verification Rule Table

[0065]

[0066] The use of parsing rules enhances the robustness of the data parsing process in the message queue. At the same time, it identifies the data_type sources of illegal types and directly filters the data types that do not conform to the parsing rules into the warning table, avoiding data loss and the intrusion of invalid data into correct data. The purpose of configuring parsing rules is to dynamically configure data types, field formats, and field verification rules. The parsing rule configuration can adapt to adjustments such as the addition, deletion, and update of fields in the original data, enhancing the robustness of the application's data processing and avoiding abnormal situations such as parsing failures caused by field updates in the original data.

[0067] Step 4: Read the dynamic log data in the constructed Flink environment according to the topic, and determine whether the read log data conforms to the set format. If it conforms, call the parsing and verification rule table, traverse the parsing and verification rules, and filter the data that does not conform to the parsing and verification rules into the warning information table. If it does not conform, send the data to the warning information table;

[0068] Specifically, it includes the following steps:

[0069] Step 401: Read and access the dynamic log data in the Flink environment;

[0070] In this embodiment, the dynamic log data in the access table is read based on a predefined message middleware, and the predefined message middleware includes information such as address information, port information, username and password. It has been defined in the prefabricated code or configuration file in the application.

[0071] Specifically, it can be defined through a third-party jar package or can also be written by oneself. For example, in the way of using a custom Pulsar data source, by defining the PulsarSourceCustom type and inheriting the RichParallelSourceFunction type, the Topic data of Pulsar can be read through this class;

[0072] In this embodiment, it also includes building the StreamEnvironment context environment of Flink, configuring checkpoint information, configuring relevant semantic information, and the restart strategy of the application, etc., which helps the application to restart and recover data in case of failure or error.

[0073] Step 402: Determine whether the read log data conforms to the set format;

[0074] In this embodiment, the set format is the JSON data format. The specific format of the preset JSON includes core data fields, extended fields, and auxiliary data fields, enhancing the necessity of data retention and the traceability of data. In this embodiment, the default specified field acq_content carries the transmitted data information and it is set that the content of this field also conforms to the JSON data format. Among them, acq_content carries the entire content of type-a data or the entire content of type-b data;

[0075] When obtaining the content of the acq_content field, through the map function, the content of the acq_content field is obtained, and the JSON key values in acq_content are mapped to the mapped fields in the parsing list; according to the field name, the name after being mapped and converted is obtained, and during the conversion process, the type is converted again according to the field type;

[0076] Step 403: If the set format is satisfied, an asynchronous request is sent to the database based on the asynchronous operator of the distribution request, the parsing and verification rule table is called back, the parsing and verification rules are traversed, and the successfully parsed data is written into the corresponding topic to obtain the parsed data.

[0077] In this embodiment, as Figure 4 shown, the specific process of traversing the parsing and verification rules includes:

[0078] Step 4031: Perform JSON formatting on the accessed data, obtain the data type acq_type in the formatted data, and match the data type in the corresponding parsing and verification list;

[0079] In this embodiment, an asynchronous operator AsyncFunction is constructed. The asynchronous I / O operator provides a complete exactly-once fault tolerance guarantee. It saves the records of the in-flight asynchronous requests in the checkpoint and re-triggers the requests during fault recovery. This asynchronous operator matches and associates the content of the acq_type field in the data stream of the message middleware with the data types in the mapping table and returns the matching parsing list;

[0080] acq_type is the code name of the data type carried by acq_content, representing a data type, such as type-a data or type-b data. Type-a data includes, for example, time data, address data, uplink and downlink traffic, etc.; type-b data can include the user's mobile phone number, base station information of the location, etc.;

[0081] Step 4032: According to the obtained data type data_type, use a third-party open-source component to parse the JSON and convert all the keys in the JSON;

[0082] In this embodiment, third-party open-source components such as fastjson or gson are adopted;

[0083] The conversion types include: types such as Integer / Long / Date / BigDecimal / Timestamp / String / JsonB, etc. By encapsulating data conversion methods, the corresponding conversion types can be matched to convert the content of the JSON field to be converted into the specified data type. For the diverse JsonB data types, the key value in the JSON array or object to be parsed can be specified according to the specific data.

[0084] Step 4033: Then, according to the field name col_name in the parsing rule matched by the key, all the key values in the JSON are replaced with the new column name data_parser information, and the corresponding verification rule col_validator information is matched;

[0085] Step 4034: Verify the data according to the verification rule col_validator. If it meets the js statements or js code defined by the js engine syntax, the output result is encapsulated into a Map collection for output and parsing to complete the parsing and output of the JSON data. Otherwise, it is excluded as abnormal data or formatted.

[0086] The conversion method internally integrates dynamic verification rules. The dynamic verification rules rely on the JS engine to implement the rule verification for specific fields, and can return success or failure or specific content information. In the application, the data with verification failure can be encapsulated and output to the warning table in the form of warning data.

[0087] In this embodiment, the verification rules are divided into two verification methods: global verification and custom verification:

[0088] Global verification has extensive and general verification capabilities. For example, it can implement domain name verification, mobile phone number verification, parameter information verification, numerical value verification, email verification, etc. Such verification can meet the basic and general verification requirements; custom verification is for the diverse configuration of fields in the business to meet the business needs. For example, address information verification and mac address verification. Such verification information is configured in the col_validator data column, and the data is judged and verified through js code.

[0089] In the above manner, rule verification is performed on specific fields of the table, and the data that complies with the rules is written into the standard table, enabling the conversion of the data of the original field types, from the original string type or JSON type, etc. to the standard numerical type, list, etc. In addition, data verification can be enriched through col_validator, and the data that does not meet the requirements is excluded or formatted; effectively preventing the output of incorrect or invalid data to the standard data table.

[0090] Embodiment Two

[0091] This embodiment provides a log dynamic parsing and verification system based on Flink and JS engines, including:

[0092] An environment information configuration module, which is used to configure the configuration files and environment variable dependency information required for the Flink environment and the Pulsar environment;

[0093] A data access configuration module, which is used to configure the topic for dynamically logging and accessing the service data to be parsed in the Pulsar environment according to the requirements of the service;

[0094] A rule verification configuration module, which is used to configure a parsing and verification rule table for dynamic log data in the database;

[0095] A parsing module, which is used to read the dynamically logged access data in the Flink environment according to the topic, determine whether the logged data read conforms to the set format. If it conforms, it calls the parsing and verification rule table, combines with the JS engine, traverses the parsing and verification rules, filters the data that does not conform to the parsing and verification rules into the warning information table. If it does not conform, it sends the data to the warning information table; writes the successfully parsed data into the corresponding topic to obtain the parsing result.

[0096] Embodiment Three

[0097] This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in a log dynamic parsing and verification method based on Flink and JS engines as described above.

[0098] Embodiment Four

[0099] This embodiment provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a log dynamic parsing and verification method based on Flink and JS engines as described above.

[0100] Embodiment Five

[0101] This embodiment provides a program product, which is a computer program product including a computer program. When the computer program is executed by a processor, it implements the steps in a method for dynamically parsing and verifying logs based on Flink and a JS engine as described above.

[0102] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0106] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0107] The foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A log dynamic parsing and verification method based on Flink and JS engine, characterized in that: The steps include: Configuration files and environment variable dependency information required to configure the Flink environment and Pulsar environment; According to business needs, configure the topic of dynamic log access data of the business data to be parsed in the Pulsar environment; Configure the parsing and verification rule table of dynamic log data in the database; Read dynamic log access data in the Flink environment according to the topic, and determine whether the read log data conforms to the set format. If it conforms, call the parsing and verification rule table, combine with the JS engine, traverse the parsing and verification rules, and filter the data that does not conform to the parsing and verification rules into the warning information table. If it does not conform, send the data to the warning information table; Write the successfully parsed data into the corresponding topic to obtain the parsing result.

2. A log dynamic parsing and verification method based on Flink and JS engine as described in claim 1, characterized in that: The configured parsing validation rule table includes data type data_type, data type code data_code, field name col_name, field default value default_value, field type col_type, field mapping name col_parse, and validation rule col_validator, which is a js statement or js code that meets the js engine syntax definition.

3. A log dynamic parsing and verification method based on Flink and JS engine as described in claim 1, characterized in that: The parsing and verification rule traversal process includes: Format the received data in JSON, obtain the data type acq_type in the formatted data and match it with the data type in the corresponding parsing and verification list; According to the matched data type data_type, parse JSON through a third-party open source component, convert and obtain all keys in JSON; Then, according to the key matching parsing rule field name col_name, all key values ​​in JSON are replaced with the new column name data_parser information, and the corresponding validation rule col_validator information is matched; The data is validated according to the validation rule col_validator. If the data meets the js statement or js code defined by the js engine syntax, the output result is encapsulated into a Map collection and output and parsed to complete the parsing and output of the JSON data. Otherwise, it is removed as abnormal data or formatted.

4. A log dynamic parsing and verification method based on Flink and JS engine as described in claim 3, characterized in that: The constructed asynchronous operator AsyncFunction matches and associates the acq_type field content in the message middleware data stream with the data type of the mapping table.

5. A log dynamic parsing and verification method based on Flink and JS engine as claimed in claim 1, characterized in that: Reading dynamic log data in the Flink environment includes: reading the corresponding dynamic log data according to the dynamic log access data topic based on the predefined message middleware. The predefined message middleware is defined through a third-party jar package or self-written.

6. A log dynamic parsing and verification method based on Flink and JS engine as claimed in claim 1, characterized in that: The dynamic data of the data to be parsed includes the field name, field type, mapping name, default value information, data type code and data type name corresponding to the business data.

7. A log dynamic parsing and verification system based on Flink and JS engine, characterized in that: include: The environment information configuration module is used to configure the configuration files and environment variable dependency information required for the Flink environment and the Pulsar environment; The data access configuration module is used to configure the topic of dynamic log access data of the business data to be parsed in the Pulsar environment according to business needs; A rule verification configuration module is used to configure a parsing verification rule table for dynamic log data in a database; The parsing module is used to read dynamic log access data in the Flink environment according to the topic, and determine whether the read log data conforms to the set format. If it conforms, the parsing verification rule table is called, combined with the JS engine, the parsing verification rules are traversed, and the data that does not conform to the parsing verification rules is filtered into the warning information table. If it does not conform, the data is sent to the warning information table; Write the successfully parsed data into the corresponding topic to obtain the parsing result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a log dynamic parsing and verification method based on Flink and JS engine as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the log dynamic parsing and verification method based on Flink and JS engine as described in any one of claims 1 to 6 are implemented.

10. A program product, the program product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, the steps in the log dynamic parsing and verification method based on Flink and JS engine as described in any one of claims 1 to 6 are implemented.

Images