A log data analysis method and device, electronic equipment and storage medium

Abstract

The application discloses a log data analysis method and device, electronic equipment and storage medium, and is applied to the field of big data. The method comprises the following steps: for any data stream of data in a data warehouse, a corresponding broadcast stream is created, and the data stream is connected with the broadcast stream. When a log data enters the data stream, it is judged whether the target field structure of the log data exists in each field structure broadcasted by the broadcast stream. If the target field structure exists, the log data is analyzed; if the target field structure does not exist, the log data is temporarily stored, and when the log data entering the data stream is successfully analyzed, all the temporarily stored log data is added to the data stream for reanalysis. Through the application of the broadcast stream, the changed field structure can be automatically acquired in the job process of data consumption, and the problem of delayed data consumption is solved.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a log data parsing method, apparatus, electronic device, and storage medium. Background Technology

[0002] ETL (Extract-Transform-Load) is the process of extracting, transforming, and loading data from a source to a destination. In existing solutions that include real-time ETL, Flink (an engine for distributed data stream processing) is typically used to consume data from the data warehouse in real time. During this process, the data is parsed, transformed into the log data format required by downstream systems, and then written to the downstream systems in real time. This ensures that the data is visible to the downstream systems in real time; that is, every piece of data that arrives can be immediately processed by Flink and written to the downstream system.

[0003] However, in practical applications, the fields corresponding to the data often change according to requirements, and once the data fields change, the data cannot be parsed correctly. Related technologies require manually stopping the real-time ELT job, manually modifying the parsing code logic to handle the changed data, and then restarting the real-time ELT job to consume the data from the modified day. However, because data fields change frequently, modifying the code and restarting the job not only wastes manpower on code modifications but also causes significant data delays, making it difficult to meet the real-time requirements of data consumption. Summary of the Invention

[0004] This application provides a log data parsing method, apparatus, electronic device, and storage medium. By applying broadcast streaming technology, the changed field structure can be automatically obtained during the data consumption process, thereby solving the problems of wasted manpower and significant data delays in the prior art.

[0005] The first aspect of this application provides a log data parsing method, the method comprising:

[0006] For any data stream that performs stream computation on data in the data warehouse, a corresponding broadcast stream is created; the broadcast stream is used to periodically retrieve at least one field structure from a pre-defined configuration table and broadcast each field structure.

[0007] Connect the data stream to the broadcast stream so that the broadcast stream broadcasts the field structure to the data stream;

[0008] Whenever a log data enters the data stream, it is determined whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream;

[0009] If it exists, then parse the log data;

[0010] If the log data does not exist, it will be temporarily stored until the log data entering the data stream is successfully parsed. Then, all the temporarily stored log data will be added back to the data stream for re-parsing.

[0011] A second aspect of this application provides a log data parsing apparatus, the apparatus comprising:

[0012] The data broadcasting unit is used to create a corresponding broadcast stream for any data stream that performs stream computing on data in the data warehouse; the broadcast stream is used to periodically obtain at least one field structure from a preset configuration table and broadcast each field structure.

[0013] The data connection unit connects the data stream and the broadcast stream, enabling the broadcast stream to broadcast the field structures to the data stream.

[0014] The judgment and processing unit is used to determine whether the target field structure of the log data exists in the field structure broadcast by the broadcast stream whenever a log data enters the data stream;

[0015] If it exists, the data parsing unit is invoked, which is used to parse the log data;

[0016] If the log data does not exist, it will be temporarily stored until the log data entering the data stream is successfully parsed. Then, all the temporarily stored log data will be added back to the data stream for re-parsing.

[0017] A third aspect of this application also provides an electronic device, including: a memory and a processor, wherein the memory and the processor are coupled. The memory is used to store one or more computer instructions; the processor is used to execute one or more computer instructions to implement the above-described log data parsing method.

[0018] Furthermore, this application also provides a computer-readable storage medium storing one or more computer instructions that are executed by a processor to implement the above-described log data parsing method.

[0019] In the embodiments of this application described above, a corresponding broadcast stream is created for any data stream undergoing stream computing. The broadcast stream periodically retrieves at least one field structure from a preset configuration table and broadcasts each field structure. The data stream is connected to the broadcast stream so that the broadcast stream broadcasts each field structure to the data stream. Whenever log data enters the data stream, it is determined whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream. If it exists, the log data is parsed; if it does not exist, the log data is temporarily stored until the log data entering the data stream is successfully parsed. Then, all the temporarily stored log data is added back to the data stream for re-parsing. This method utilizes broadcast stream technology to broadcast the field structure of log data in the form of a stream, ensuring that any changes in the field structure are always known to the data stream and automatically retrieved to complete the data parsing process. This allows the process of retrieving the changed field structure to be completed automatically without manual code modification, thus avoiding the step of stopping and restarting the real-time ETL job to modify the code, eliminating the defects of delayed data consumption, and meeting the real-time requirements of data consumption. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the log data parsing method provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating the process of preparing to create a broadcast stream, as described in an embodiment of this application.

[0022] Figure 3 This is a flowchart illustrating the method for creating a broadcast stream according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the log data parsing device provided in the embodiments of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0026] The technical solutions provided in this application are mainly applied in the field of big data. In this field, ETL is often used to integrate scattered, disorganized, and inconsistent data within an enterprise, providing analytical support for enterprise decision-making.

[0027] The rapid pace of data updates today has made real-time ETL a new requirement for enterprises. In practice, ETL typically relies on data consumption. Data consumption refers to the process by which consumers consume data provided by producers within the data processing dimension. In this concept, producers are the data providers, consumers are the data users who need the data provided by producers, and consumption refers to the entire process of acquiring, processing, and transmitting the data provided by producers. The data provided by producers is stored in a data warehouse for consumers to consume. Consumption is considered successful only when the consumer completes the entire process of data acquisition, processing, and transmission; any error at any stage leading to data transmission failure constitutes a consumption failure.

[0028] Existing solutions involving real-time ETL typically use Flink to consume data from a data warehouse. This process parses the data, transforms it into the log data format required by downstream systems, and then loads the data into those systems. It's important to note that in this embodiment, the data warehouse refers to a strategic collection that can store all data. In practical applications, real-time data warehouses such as Kafka are often used for real-time data consumption, and Flink is typically used to consume data from Kafka to complete the real-time ETL process.

[0029] However, during this process, the fields corresponding to the data frequently change according to requirements, and once the data fields change, the data cannot be parsed correctly. At this point, the only solution is to stop the real-time ELT job, manually modify the parsing code logic to handle the changed data, restart the real-time ELT job, and re-consume the data for the current day. However, this process of stopping and restarting, and re-consuming the code for the current day, not only wastes manpower but also causes significant data delays, making it difficult to meet the real-time requirements of data consumption.

[0030] In this embodiment, a broadcast stream is created corresponding to the consumed data stream, broadcasting the field structure of the log data to the data stream. During this process, the field structure in the broadcast stream is always visible to the data stream and can be automatically retrieved by the data stream. Therefore, as long as the broadcast stream can obtain the latest field structure in a timely manner, the data stream can obtain the latest field structure promptly and parse the log data based on the latest field structure. The timely retrieval and use of the updated field structure for data parsing solves the aforementioned problems of wasted manpower and delayed data consumption.

[0031] Figure 1 This is a flowchart illustrating the log data parsing method provided in an embodiment of this application. It should be noted that the steps shown in this flowchart can be executed in a computer system, such as a set of computer-executable instructions. Furthermore, in some cases, the steps shown may be executed in a different logical order than that shown in the flowchart.

[0032] like Figure 1 As shown, the method includes the following steps S101-S105:

[0033] S101. For any data stream that performs stream computation on data in the data warehouse, create a corresponding broadcast stream; the broadcast stream is used to periodically obtain at least one field structure from a preset configuration table and broadcast each field structure.

[0034] First, let's introduce data streams and broadcast streams.

[0035] A data stream refers to an infinitely large and dynamic collection of data in terms of time distribution and quantity, within which data continuously progresses over time. The data processing method for data streams is streaming processing, meaning that data is processed in real-time according to a time-bound process. The environment in which data streams exist is a streaming computing framework; that is, the definition of a data stream and the real-time computing process can only be completed within a streaming computing framework. Common streaming computing frameworks include Storm, Spark Streaming, and Flink. This embodiment primarily uses Flink as an exemplary application scenario for the log data parsing method provided in this embodiment to specifically illustrate the method, but it does not limit the application of the method in this embodiment to other applicable streaming computing frameworks.

[0036] Broadcast streams are a special type of data stream, typically used to broadcast data processing rules, and the amount of data in a broadcast stream is much smaller than that in a regular data stream. In a streaming computing framework, a task to process a data stream is implemented through operators. Different operators represent different processing logics and achieve different computational purposes. Each operator can be further divided into multiple parallel operators based on its degree of parallelism, allowing for parallel computation of the data. This means dividing the computational task of an operator into multiple parallel computational subtasks (or operator subtasks, parallel operator tasks) to improve computational efficiency. If the degree of parallelism of an operator is N, then that operator can have N parallel operators. Data in a regular data stream can be split into multiple data fragments and processed in parallel by multiple parallel operators, while data in a broadcast stream can be received by each parallel operator and applied in the data processing process. Broadcast stream data is broadcast to all parallel operators of all operators. However, in this embodiment, only after all parallel operators of the parsing operator receive the broadcast data can they store it as their own state, serving as rule data needed for their own data parsing.

[0037] In the embodiments of this application, a pair of corresponding data streams and broadcast streams are created for the same data processing task, that is, the same Flink task. A Flink task can be used to consume log data of a topic in Kafka.

[0038] Before creating a broadcast stream, preparations should be made for its creation. Figure 2 The following is a flowchart illustrating the preparation process for creating a broadcast stream according to an embodiment of this application:

[0039] 201. Create a configuration table; this configuration table is used to store the field structure corresponding to each type of business data in the data warehouse.

[0040] First, a configuration table can be created in MySQL (a relational database management engine). This table stores the field structure corresponding to each type of business data in the data warehouse. The MySQL engine can be used to provide table lookup services. In this embodiment, the Flink engine can obtain the configuration table containing the field structure from the MySQL engine. In a specific implementation, taking Flink, a streaming computing framework, as an example, the data warehouse can be Kafka. The purpose of creating the configuration table is to store the field structure corresponding to each type of business data in the data warehouse in advance, thereby providing a data source for the creation of broadcast streams.

[0041] The Kafka data warehouse contains data from multiple topics. Each topic includes various business functions, and each business function's data corresponds to a log ID. Different log IDs correspond to different business functions' data, and business data for the same business function has the same field structure. In this embodiment, the field structure corresponding to each log ID in each topic can be stored in a configuration table.

[0042] In this embodiment, the configuration table refers to a collection of tables that store the field structure corresponding to each type of business data in the data warehouse. In specific implementation, multiple configuration sub-tables can be set up. Various business data in the data warehouse are divided into business data of at least one theme. The configuration table includes at least one configuration sub-table. Different configuration sub-tables are used to store the field structure corresponding to each type of business data in different themes in the data warehouse.

[0043] For example, there are N configuration sub-tables and M topics in the data warehouse, where N and M are both positive integers greater than or equal to 1. Preferably, one configuration sub-table corresponds to all log field structures in one topic, i.e., N=M; however, it is not excluded that one configuration sub-table corresponds to the log field structures in multiple topics, i.e., N<M; and there are also cases where multiple configuration sub-tables correspond to the log field structures in one topic, i.e., N>M.

[0044] The establishment of the configuration table provides the data source conditions for the broadcast stream to broadcast the field structure. It also requires specifying the type of the broadcast field structure and providing the basic methods for data acquisition. Specifying the type of the broadcast field structure is achieved through step 202 below, while data acquisition is achieved through step 203 below.

[0045] 202. Create a target state descriptor subclass; this target state descriptor subclass inherits from the preset state descriptor abstract class and is used to define the target type of state required for parallel operator tasks in performing stream computation on the data stream.

[0046] This step is used to create a target state descriptor subclass that inherits from the preset abstract state descriptor class StateDescriptor and is used to define the target type of state required for parallel operator tasks in the data stream computation, thereby defining the data type of the broadcast field structure.

[0047] Abstract classes cannot be directly instantiated; instead, a subclass inheriting from the abstract class must be created and then instantiated to implement the methods defined in the abstract class. The abstract class `StateDescriptor` defines a method for defining state.

[0048] State refers to the information a stream processing application stores within a certain timeframe of received events or intermediate results, which can then be accessed and processed at a later point in time. In other words, state is the data required by the parallel operators of each stream processing unit to execute their tasks, such as historical computation results and the field structures needed to parse data.

[0049] In Flink, a data stream and a broadcast stream belong to a Flink task. The execution of a Flink task relies on multiple operators, each corresponding to different data processing logic. Due to the large amount of computation, and because the same operator is often used by multiple parallel operators for parallel computation, these parallel operators use the same processing logic. During data processing, parallel operators first store the data needed for computation in their own maintained state. This allows them to perform computations based on the data in the state.

[0050] By creating a target state descriptor subclass, the predefined state descriptor abstract class is concretized. This allows the state definition to be completed through the target state descriptor subclass, specifying the storage type of the broadcast stream data. This defines the target type of state required for parallel operator tasks in the data stream execution, thus providing the conditions for the implementation of the broadcast stream.

[0051] Additionally, the target type's state can be either a value state (ValueState) or a list state (ListState). Correspondingly, when the target type's state is a value state, the target state descriptor subclass is ValueStateDescriptor; when the target type's state is a list state, the target state descriptor subclass is ListStateDescriptor. A value state refers to a state that is a single value. If multiple field structures need to be broadcast, these structures can be arranged sequentially into a string, with fields separated by a preset delimiter. This string can then be considered a single value. When parsing multiple broadcast field structures, each field structure can be extracted based on the delimiter between them. A list state, on the other hand, temporarily stores data in the form of a list, where each list item is a field structure.

[0052] For example, when applying this step in Flink, a target state descriptor subclass, ValueStateDescriptor or ListStateDescriptor, is created. This subclass inherits from the predefined abstract state descriptor class, StateDescriptor. Through this target state descriptor subclass, the specific type of the state can be defined. ValueStateDescriptor defines the state type as ValueState under the keyedState type, and ListStateDescriptor defines the state type as ListState under the keyedState type. For ease of explanation, the following will use ValueStateDescriptor as the target state descriptor subclass as an example.

[0053] 203. Create a target rich function subclass; this target rich function subclass inherits from the preset rich function abstract class and is used to periodically retrieve the structure of at least one field in the configuration table.

[0054] Similar to the previous step, the methods defined in the rich function abstract class RichSourceFunction need to be implemented by creating a target rich function subclass MysqlSourceFunction that inherits from the rich function abstract class, and then instantiating the target rich function subclass MysqlSourceFunction.

[0055] The target rich function subclass MysqlSourceFunction can be used to periodically read at least one field structure from a configuration table from the MySQL engine, for example, it can be set to pull at least one field structure from the MySQL engine every 30 seconds or every 10 seconds.

[0056] It should be noted that steps 201-203 are preliminary steps before step S101, and there is no restriction on the order of steps 201-203.

[0057] Once the configuration table, target state descriptor subclass, and target rich function subclass are created, the broadcast stream can be created based on these three elements, such as... Figure 3 As shown, step S101 specifically includes the following steps 301-304:

[0058] 301. Instantiate the target rich function subclass to obtain the target rich function of the target rich function subclass.

[0059] 302. Instantiate the target state descriptor subclass to obtain the target state descriptor of the target state descriptor subclass.

[0060] First, it is necessary to instantiate the target rich function subclass and the target rich function subclass to obtain an object of the target rich function subclass, i.e., the target rich function, and an object of the target state descriptor subclass, i.e., the target state descriptor.

[0061] The broadcast stream establishes a connection with the configuration table and retrieves the field structures stored in the configuration table by calling the target rich function. It's important to understand that during the instantiation of the target rich function subclass, since this subclass is configured to periodically retrieve at least one field structure from the configuration table, when the field structure in the configuration table is updated, the target rich function can promptly add the updated field structure to the broadcast stream in the next cycle.

[0062] The target state descriptor defines the type of state, and the broadcast stream uses this target state descriptor to define the data type of the broadcast.

[0063] 303. Call the source operator used to read data and pass the target rich function parameters to the source operator so that the field structure obtained by the target rich function can be used as the data source of the broadcast stream.

[0064] In this step, the data source for the broadcast stream is established by calling the source operator (i.e., the addSource operator) used to read data, passing parameters of the target rich function to the source operator, and thus using the field structure obtained by the target rich function as the data source for the broadcast stream.

[0065] In this step, the target rich function is used as the data source parameter of the source operator. The target rich function periodically retrieves the field structure from the configuration table. Therefore, retrieving data from the data source through the source operator is the process of periodically retrieving the field structure stored in the configuration table.

[0066] Furthermore, when calling the target rich function through the source operator to obtain the field structure, it's necessary to know which field structures need to be retrieved. In one implementation, where a configuration sub-table corresponds to all field structures in a topic, the target rich function can be specifically used to periodically retrieve the field structures of a particular configuration sub-table. Optionally, the field structures of a configuration sub-table can be used as the data source for a broadcast stream by passing parameters or not passing parameters.

[0067] For the parameter passing method, step 303 specifically includes: calling the source operator used to read data, and passing the identifier of the target configuration sub-table corresponding to the data stream and the target rich function to the source operator, so as to use the field structure of each field obtained by the target rich function from the target configuration sub-table as the data source of the broadcast stream.

[0068] The method for passing parameters to acquire data involves including the identifier of the target configuration sub-table corresponding to the data stream as one of the parameters, passed together with the target rich function to the source operator. Specifically, the source operator for reading data is called, and the identifier of the target configuration sub-table and the target rich function are passed to the source operator. The field structure obtained by the target rich function from the target configuration sub-table serves as the data source for the broadcast stream. The identifier of the target configuration sub-table is data information that uniquely identifies its storage location, such as the storage path and name of the configuration sub-table.

[0069] For the parameterless method, the target rich function subclass is specifically used to periodically retrieve the field structure from the target configuration subtable. Accordingly, step 303 specifically includes: calling the source operator for reading data and passing the target rich function parameters to the source operator, so as to use the field structure obtained by the target rich function from the target configuration subtable as the data source of the broadcast stream.

[0070] In the parameterless approach, multiple target rich function subclasses need to be created for each configuration subtable, and the identifier of each configuration subtable is written into the corresponding target rich function subclass. Thus, without passing the specific configuration subtable identifier, the field structure of the configuration subtable can be obtained based on the pre-set configuration subtable identifier.

[0071] This step completes the process of creating a data source for the broadcast stream and retrieving the field structure from the configuration table. After obtaining the field structure, it's necessary to distribute this data to each parallel operator—a process known as data broadcasting—while ensuring the data is stored using a specified data type. This will be detailed below.

[0072] 304. Invoke the broadcast operator for broadcasting data and pass the target state descriptor to the source operator to broadcast the data source of the broadcast stream to the parallel operators that perform stream computation on the data stream in the target type indicated by the target state descriptor.

[0073] This step is used to broadcast the data based on step 303.

[0074] Specifically, the broadcast operator has a broadcasting function, which means broadcasting the data of the broadcast stream, i.e., the field structure obtained from the data source in step 303, to the parallel operators of each operator. In this process, the target state descriptor is passed as a parameter to the broadcast operator. When the broadcast operator is called, the data type is specified for the broadcast data, and this data type is temporarily stored as a state by the parsing parallel operator of the parsing operator. Therefore, the data type is consistent with the state type. That is, in this embodiment, the broadcast stream broadcasts data to the parallel operators of all operators in the computation data stream, but only the parallel operators of the parsing operator store the received broadcast data (i.e., the field structure) into their own maintained state.

[0075] Therefore, step 304 specifically includes: for each parsing parallel operator among the parallel operators performing stream computation on the data stream, receiving the field structures broadcast by the broadcast stream; and saving the field structures broadcast by the broadcast stream as the state of the parsing parallel operator according to the target type indicated by the target state descriptor.

[0076] Corresponding to the Flink application scenario, after instantiating the target rich function subclass and the target state descriptor subclass, we obtain the target rich function `MysqlSourceFunction` and the target state descriptor `ValueStateDescriptor`. Here, `MysqlSourceFunction` and `ValueStateDescriptor` are the results obtained after instantiation. Then, we pass `MysqlSourceFunction` as a parameter to the source operator `addSource` that reads data, and pass `ValueStateDescriptor` as a parameter to the broadcast operator `broadcast`, thus obtaining the broadcast stream that Flink periodically retrieves the field structure from the configuration table and broadcasts it: `addSource(MysqlSourceFunction).broadcast(ValueStateDescriptor)`. Through this broadcast stream, the data from `MysqlSourceFunction` is broadcast as `ValueState` type to each parallel operator performing stream computation on the data stream.

[0077] This step completes the creation of the broadcast stream. After the broadcast stream is created, its field structure will be periodically updated because it continuously retrieves data from the configuration table, ensuring that the parsing operator can obtain the latest field structure in a timely manner.

[0078] S102. Connect the data stream to the broadcast stream so that the broadcast stream broadcasts the field structures to the data stream;

[0079] After creating the broadcast stream, you need to connect the data stream to the broadcast stream so that the broadcast stream can broadcast the field structure to the data stream.

[0080] Since both broadcast streams and data streams are essentially data streams, although they are created for the same data processing task, they are not interconnected. Therefore, in order for data in the broadcast stream to be used for computation in the data stream, the data stream and the broadcast stream must be connected. In Flink's application environment, the connect operator is used to connect the data stream and the broadcast stream.

[0081] S103. Whenever a log data enters the data stream, determine whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream.

[0082] After connecting the broadcast stream and the data stream, the field structure can be obtained from the broadcast stream. At this point, whenever a log data enters the data stream, in order to ensure successful data parsing, it is necessary to first determine whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream.

[0083] This step determines whether the target field structure of the log data exists in the field structure broadcast by the broadcast stream. It is a preliminary step for the parsing operator. It is important to understand that when log data enters the data stream, it is not necessarily parsed directly; it may first undergo other processing steps, such as clustering, mapping, and operations.

[0084] Once log data enters the data stream, it undergoes a series of processing steps before being parsed. Only then is the determination made regarding whether the target field structure of the log data exists within the field structure broadcast by the broadcast stream.

[0085] Specifically, since the broadcast stream has already broadcast the field structure to each parsing parallel operator, the state of each parsing parallel operator already contains the field structure broadcast by the broadcast stream. Therefore, it is only necessary to read the current state maintained by the parsing parallel operator and determine whether the target field structure of the log data exists in the field structures read from the current state maintained by the parsing parallel operator. This will give the result of whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream.

[0086] Therefore, this judgment process is actually implemented through the parsing parallel operator. That is, whenever a log data enters the data stream and the log data is processed as data to be parsed, the current state maintained by the parsing parallel operator is read. The current state includes the field structure broadcast by the broadcast stream. It is then determined whether the target field structure of the log data exists in the field structure read from the current state maintained by the parsing parallel operator.

[0087] Once the judgment result is obtained, the log data is processed differently according to different judgment results, that is, step S104 or step S105 is executed according to different results.

[0088] S104. If it exists, parse the log data.

[0089] If the target field structure of the log data exists in the field structures read from the current state maintained by the parsing parallel operator, it indicates that the parsing parallel operator that parses the log data has maintained the latest field structure. Therefore, the log data can be parsed directly based on the target field structure.

[0090] After parsing, the parsed log data is loaded into downstream systems, such as HDFS, HBase, ES, and Kudu storage systems. Furthermore, after parsing the log data, the parallel parsing operator needs to update its current state based on the parsing results to ensure its real-time status and thus guarantee the normal parsing of subsequent data.

[0091] In summary, after receiving the data to be parsed, the parallel parsing operator first reads its current state, which includes the field structure obtained from the configuration table. If the target field structure of the log data exists in the current state, the log data can be parsed. Once the data is parsed, the parsed data is loaded into the downstream system, and the operator simultaneously updates and maintains its own state based on the parsing results.

[0092] S105. If it does not exist, the log data will be temporarily stored until the log data entering the data stream is successfully parsed. Then, all the temporarily stored log data will be added back to the data stream for re-parsing.

[0093] If the target field structure of the log data does not exist in the field structures read from the current state maintained by the parsing parallel operator, it indicates that the parsing parallel operator has not yet obtained the corresponding field structure when the log data enters the parsing phase, and the log data is temporarily stored. After temporary storage, when the log data entering the data stream is successfully parsed, it means that the configuration table has been updated, the state of the parsing parallel operator has been updated accordingly, and the field structure corresponding to the temporarily stored log data is now available. At this point, all the temporarily stored log data is added back to the data stream for re-parsing.

[0094] Specifically, because broadcast streams periodically retrieve the field structure of the configuration table, during the data updates of both the data stream and the broadcast stream over time, when the field structure changes, due to the periodic nature of the broadcast stream's retrieval of the field structure, there may be instances where the broadcast field structure is not updated in time before the next cycle arrives. That is, data enters the data stream and has already been processed into data awaiting parsing, but the target rich function in the broadcast stream has not yet reached the time to retrieve the target field structure, thus failing to complete the parsing process. Furthermore, since the data stream's data processing is time-driven and sequential, this situation where log data has been updated but the corresponding field structure has not will inevitably continue until the next cycle when the broadcast stream retrieves the field structure. Therefore, in this case, it is only necessary to re-parse the data that could not be parsed successfully within this cycle.

[0095] In other words, once the broadcast stream retrieves the latest field structure from the configuration table in the next cycle and broadcasts this latest field structure to the parsing parallel operator, causing the parsing parallel operator to complete its state update, the parsing parallel operator can then re-parse the log data that was not successfully parsed in the current cycle. Specifically, after temporarily storing the unparseable log data, when a new log data is successfully parsed, it indicates that the parallel operator has completed its state update.

[0096] Therefore, when executing step S105, the temporarily stored log data can be re-parsed when a new log data parsing success event occurs. Specifically, it is determined whether the target field structure of the log data exists in the field structures broadcast by the broadcast stream. If it does not exist, the log data is temporarily stored until the broadcast stream obtains at least one field structure after the configuration table is updated, and successfully parses the log data entering the data stream based on at least one field structure after the configuration table is updated. At that time, all the temporarily stored log data is added back to the data stream for re-parsing.

[0097] Furthermore, the successful parsing of new data logs can be considered for a single parsing parallel operator or for all parsing parallel operators. In other words, the time point at which temporarily stored, unparsed data is put back into the data stream for re-parsing can be configured for a single parsing parallel operator, allowing that operator to re-parse its temporarily stored log data as soon as it finishes parsing the next new data. Alternatively, it can be configured for the entire parsing process. That is, for each parsing parallel operator, if a new data set cannot be parsed by one of them, as long as one of the parsing parallel operators completes the parsing of a new set of data, all temporarily stored data can be put back into the data stream for re-parsing by the individual parsing parallel operators.

[0098] For log data that failed to be parsed previously, it can be pushed back to the Kafka topic being consumed, so that Flink can re-parse the log data that failed to be parsed due to field structure updates.

[0099] When the temporarily stored log data is added back to the data stream, the parsing parallel operator will have already stored the latest field structure in the state, so the data parsing can be completed at this point, satisfying condition S104 and completing the parsing process. Thus, even if the field structure is not updated in time due to the data acquisition cycle, the parsing process for all data can still be completed, ensuring that the field structure of the data is automatically acquired by the data stream's parsing parallel operator in a timely manner, thereby guaranteeing successful data parsing.

[0100] The log data parsing method provided in the embodiments of this application has been described in detail above. The following is a detailed description in conjunction with... Figure 4 This section introduces the log data parsing device, including... Figure 4 A schematic diagram of the log data parsing device provided by the present invention is shown below. Figure 4 As shown, the device includes:

[0101] The data broadcasting unit 401 is used to create a corresponding broadcast stream for any data stream that performs stream computing on the data in the data warehouse; the broadcast stream is used to periodically obtain at least one field structure from a preset configuration table and broadcast each field structure.

[0102] Data connection unit 403 connects the data stream to the broadcast stream so that the broadcast stream broadcasts each field structure to the data stream;

[0103] The judgment and processing unit 405 is used to determine whether the target field structure of the log data exists in the field structure broadcast by the broadcast stream whenever a log data enters the data stream.

[0104] The data parsing unit 407 is used to parse the log data if the target field structure of the log data exists in the field structure broadcast by the broadcast stream.

[0105] The re-parsing unit 409 is used to temporarily store the log data if the target field structure of the log data does not exist in the field structure broadcast by the broadcast stream, until the log data entering the data stream is successfully parsed, and then add all the temporarily stored log data back to the data stream for re-parsing.

[0106] In an optional implementation, the re-parsing unit 409 is further configured to temporarily store the log data if the target field structure of the log data does not exist in the field structures broadcast by the broadcast stream, until the broadcast stream obtains at least one field structure after the configuration table is updated, and successfully parses the log data entering the data stream according to at least one field structure after the configuration table is updated, then add all the temporarily stored log data back to the data stream for re-parsing.

[0107] In an optional implementation, the data broadcasting unit 401 is also used to create a data configuration table; the configuration table is used to store the field structure corresponding to each type of business data in the data warehouse;

[0108] Create a target state descriptor subclass; the target state descriptor subclass inherits from the preset state descriptor abstract class and is used to define the target type of state required for parallel operator tasks in performing stream computation on the data stream.

[0109] Create a target rich function subclass; this target rich function subclass inherits from the preset rich function abstract class and is used to periodically retrieve the structure of at least one field in the configuration table.

[0110] In an optional implementation, the data broadcasting unit 401 is further configured to instantiate the target rich function subclass to obtain the target rich function of the target rich function subclass.

[0111] Instantiate the target state descriptor subclass to obtain the target state descriptor of the target state descriptor subclass;

[0112] Call the source operator used to read data and pass the target rich function parameters to the source operator so that the field structure obtained by the target rich function can be used as the data source of the broadcast stream;

[0113] Invoke the broadcast operator for broadcasting data and pass the target state descriptor to the source operator to broadcast the data source of the broadcast stream to the various parallel operators that perform stream computation on the data stream, with the target type indicated by the target state descriptor.

[0114] In an optional implementation, the data broadcasting unit 401 is further configured to receive, for each of the parsing parallel operators in the parallel operators performing stream computation on the data stream, the field structures broadcast by the broadcast stream.

[0115] According to the target type indicated by the target state descriptor, the structure of each field broadcast by the broadcast stream is saved as the state of the parsing parallel operator.

[0116] In an optional implementation, the determination processing unit 405 is further configured to read the current state maintained by the parsing parallel operator whenever a log data enters the data stream and the log data is processed into data to be parsed; the current state includes the structure of each field broadcast by the broadcast stream.

[0117] Determine whether the target field structure of the log data exists in the field structures read from the current state maintained by the parsing parallel operator.

[0118] In an optional implementation, the data parsing unit 407 is further configured to update the current state maintained by the parsing parallel operator based on the parsing result of the log data after the log data is parsed.

[0119] In one optional implementation, the various business data in the data warehouse are divided into business data of at least one theme, and the configuration table includes at least one configuration sub-table. Different configuration sub-tables are used to store the field structure corresponding to each business data in different themes in the data warehouse.

[0120] In an optional implementation, the data broadcasting unit 401 is further configured to call the source operator for reading data, and pass the identifier of the target configuration sub-table corresponding to the data stream and the target rich function as parameters to the source operator, so as to use the field structure obtained by the target rich function from the target configuration sub-table as the data source of the broadcast stream.

[0121] In an optional implementation, the target rich function subclass is specifically used to periodically retrieve the field structure from the target configuration subtable;

[0122] Accordingly, the data broadcasting unit 401 is also used to call the source operator for reading data and pass the target rich function to the source operator so as to use the field structure of each field obtained by the target rich function from the target configuration sub-table as the data source of the broadcast stream.

[0123] In one alternative implementation, the state of the target type is either a value state or a list state.

[0124] It should be noted that the information interaction and execution process between the modules / units in the processing device are different from those in this application. Figures 1 to 3The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0125] Additionally, embodiments of this application provide an electronic device comprising a memory and a processor, wherein the memory and the processor are coupled together.

[0126] The memory is used to store one or more computer instructions.

[0127] The processor is used to execute one or more computer instructions to implement the log data parsing method described in the first embodiment above.

[0128] This application also provides a computer-readable storage medium, which includes computer instructions. When executed by a processor, the computer instructions are used to implement any of the log data parsing techniques described in this application.

[0129] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims of the present invention.

Claims

1. A log data parsing method, characterized in that, include: For any data stream that performs stream computation on data in the data warehouse, create a corresponding broadcast stream; The broadcast stream is used to periodically obtain at least one field structure from a preset configuration table and broadcast each of the field structures. Connect the data stream to the broadcast stream so that the broadcast stream broadcasts each of the field structures to the data stream; Whenever a log data enters the data stream, it is determined whether the target field structure of the log data exists in each of the field structures broadcast by the broadcast stream; If it exists, then parse the log data; If it does not exist, the log data will be temporarily stored until the log data entering the data stream is successfully parsed. Then, all the temporarily stored log data will be added back to the data stream for re-parsing, including: If the log data does not exist, it will be temporarily stored until the broadcast stream obtains at least one field structure of the updated configuration table and successfully parses the log data entering the data stream according to the updated configuration table. Then, all the temporarily stored log data will be added back to the data stream for re-parsing.

2. The method according to claim 1, characterized in that, Before creating the corresponding broadcast stream for any data stream performing stream computation on data in the data warehouse, the process also includes: Create a configuration table; the configuration table is used to store the field structure corresponding to each type of business data in the data warehouse; Create a target state descriptor subclass; the target state descriptor subclass inherits from the preset state descriptor abstract class and is used to define the target type of state required for parallel operator tasks in stream computation of data stream; Create a target rich function subclass; the target rich function subclass inherits from a preset rich function abstract class and is used to periodically obtain the structure of at least one field in the configuration table.

3. The method according to claim 2, characterized in that, The step of creating a corresponding broadcast stream for any data stream performing stream computation on data in the data warehouse includes: Instantiate the target rich function subclass to obtain the target rich function of the target rich function subclass; Instantiate the target state descriptor subclass to obtain the target state descriptor of the target state descriptor subclass; Invoke the source operator for reading data and pass the target rich function as parameters to the source operator so that the field structures obtained by the target rich function are used as the data source of the broadcast stream; The broadcast operator for broadcasting data is invoked, and the target state descriptor is passed as a parameter to the source operator to broadcast the data source of the broadcast stream to the various parallel operators performing stream computation on the data stream in the target type indicated by the target state descriptor.

4. The method according to claim 3, characterized in that, The step of invoking the broadcast operator for broadcasting data and passing the target state descriptor as a parameter to the source operator to broadcast the data source of the broadcast stream to the various parallel operators performing stream computation on the data stream according to the target type indicated by the target state descriptor further includes: For each parsing parallel operator in the parallel operators that perform stream computation on the data stream, the field structures broadcast by the broadcast stream are received; According to the target type indicated by the target state descriptor, the field structures broadcast by the broadcast stream are saved as the state of the parsing parallel operator.

5. The method according to claim 4, characterized in that, The step of determining whether the target field structure of the log data exists in each of the field structures broadcast by the broadcast stream whenever a log data enters the data stream includes: Whenever a log data enters the data stream and is processed into data to be parsed, the current state maintained by the parsing parallel operator is read; the current state includes the structure of each field broadcast by the broadcast stream. Determine whether the target field structure of the log data exists in each of the field structures read from the current state maintained by the parsing parallel operator.

6. The method according to claim 4, characterized in that, After parsing the log data, the process also includes: The current state maintained by the parsing parallel operator is updated based on the parsing results of the log data.

7. The method according to claim 3, characterized in that, The various business data in the data warehouse are divided into business data of at least one theme, and the configuration table includes at least one configuration sub-table. Different configuration sub-tables are used to store the field structure corresponding to each business data in different themes in the data warehouse.

8. The method according to claim 7, characterized in that, The invocation of the source operator for reading data, and the passing of the target rich function as parameters to the source operator, to use the field structures obtained by the target rich function as the data source of the broadcast stream, includes: The source operator for reading data is invoked, and the identifier of the target configuration sub-table corresponding to the data stream and the target rich function are passed as parameters to the source operator, so that the field structures obtained by the target rich function from the target configuration sub-table are used as the data source of the broadcast stream.

9. The method according to claim 7, characterized in that, The target rich function subclass is specifically used to periodically obtain the field structure of the target configuration subtable; Accordingly, the invocation of the source operator for reading data and the passing of the target rich function as parameters to the source operator, so as to use the field structures obtained by the target rich function as the data source of the broadcast stream, includes: The source operator for reading data is invoked, and the target rich function is passed as parameters to the source operator so that the field structures obtained by the target rich function from the target configuration sub-table are used as the data source of the broadcast stream.

10. The method according to claim 2, characterized in that, The state of the target type is either a value state or a list state.

11. A log data parsing device, characterized in that, include: The data broadcasting unit is used to create a corresponding broadcast stream for any data stream that performs stream computing on data in the data warehouse; The broadcast stream is used to periodically obtain at least one field structure from a preset configuration table and broadcast each of the field structures. A data connection unit connects the data stream to the broadcast stream, so that the broadcast stream broadcasts each of the field structures to the data stream; The judgment and processing unit is used to determine whether the target field structure of the log data exists in each of the field structures broadcast by the broadcast stream whenever a log data enters the data stream; If it exists, the data parsing unit is invoked, which is used to parse the log data; If it does not exist, the re-parsing unit is invoked. This re-parsing unit temporarily stores the log data until the log data entering the data stream is successfully parsed. Then, all temporarily stored log data is added back to the data stream for re-parsing, including: If the log data does not exist, it will be temporarily stored until the broadcast stream obtains at least one field structure of the updated configuration table and successfully parses the log data entering the data stream according to the updated configuration table. Then, all the temporarily stored log data will be added back to the data stream for re-parsing.

12. An electronic device, characterized in that, include: The memory and the processor are coupled; The memory is used to store one or more computer instructions; The processor is used to execute one or more computer instructions to implement the log data parsing method as described in any one of claims 1-10.

13. A computer-readable storage medium storing one or more computer instructions thereon, characterized in that, The instruction is executed by the processor to implement the log data parsing method as described in any one of claims 1-10.

Images