Data processing method and device

Abstract

The embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: obtaining data to be processed, and inputting the data to be processed into a data processing link, wherein the data processing link comprises at least two data processing nodes, and performing preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes, and determining whether the data to be loaded meets an output condition, and if so, performing interrupt processing on the target data processing node, and outputting and loading the data to be loaded to a target loading end.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a data processing method. Background Art

[0002] ETL (Extraction-Transformation-Loading) is the process of extracting, transforming and loading data. It is an important part of building a data warehouse. Users extract the required data from the database, clean the data, and finally load the data into the data warehouse according to the pre-defined data warehouse model.

[0003] Traditional ETL tools and software solve the problem of importing data from a data source to a target loading end in a data warehouse, for example, from a table in database A to another table in database B. However, in the traditional data import process, the data processing function will split the event (input one event and output multiple events). In this case, the memory occupied by the function cache may be much larger than the original data, and each original data needs to be fully calculated and cached, which will increase unnecessary resource overhead and reduce resource utilization. Therefore, an effective processing method is urgently needed to solve this problem. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0006] Acquire data to be processed, and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes;

[0007] Performing preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes;

[0008] Determine whether the data to be loaded meets the output condition;

[0009] If yes, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to the target loading end.

[0010] Optionally, the data processing method further includes:

[0011] If the result of the execution of the determination of whether the data to be loaded meets the output condition is no, the data to be loaded is transmitted to a data processing node next to the target data processing node for processing, wherein the next data processing node is one of the at least two data processing nodes; or

[0012] If the result of the execution of determining whether the data to be loaded meets the output condition is no, then in the case where it is determined that the data to be loaded meets the deletion condition, the data to be loaded is deleted.

[0013] Optionally, the transmitting the data to be loaded to a data processing node next to the target data processing node for processing includes:

[0014] The data to be loaded is transmitted to a data processing node next to the target data processing node, and the data to be loaded is processed by a data processing function corresponding to the next data processing node to generate a corresponding processing result.

[0015] Optionally, the data processing method further includes:

[0016] Determining whether the processing result meets the output condition;

[0017] If so, interrupt processing is performed on the next data processing node, and the processing result is output and loaded to the target loading end.

[0018] Optionally, obtaining the data to be processed and inputting the data to be processed into a data processing link includes:

[0019] Get multiple log data to be processed;

[0020] Inputting first log data among the plurality of log data to be processed into a data processing link, wherein the first log data is one of the plurality of log data to be processed.

[0021] Optionally, after determining whether the data to be loaded meets the output condition, the method further includes:

[0022] If the result of determining whether the data to be loaded meets the output condition is yes, the second log data among the multiple log data to be processed is input into the target data processing node, wherein the second log data is one of the multiple log data to be processed.

[0023] Optionally, performing preset processing on the data to be processed by a target data processing node in the data processing link to generate data to be loaded includes:

[0024] A preset data processing function corresponding to a target data processing node in the data processing link is determined, and the data to be processed is processed by the preset data processing function to generate data to be loaded.

[0025] Optionally, the step of outputting the data to be loaded and loading it to a target loading end includes:

[0026] Outputting the data to be loaded from the data processing link, and storing the output data to be loaded in a data output module, wherein the data output module divides the data to be loaded according to data attributes of the data to be loaded, and generates corresponding data blocks to be loaded;

[0027] When it is determined that the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data block to be loaded is loaded to the corresponding target loading end.

[0028] Optionally, the data to be loaded is a key-value pair;

[0029] Accordingly, judging whether the data to be loaded meets the output condition includes:

[0030] Determine the data type of the data to be loaded according to the keyword in the key-value pair, and determine the target output condition corresponding to the data type;

[0031] It is determined whether the data corresponding to the target field of the key value in the key-value pair meets the target output condition.

[0032] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0033] An acquisition module is configured to acquire data to be processed and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes;

[0034] a processing module configured to perform preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes;

[0035] A judging module, configured to judge whether the data to be loaded meets the output condition;

[0036] If the running result of the judgment module is yes, then the loading module is run;

[0037] The loading module is configured to perform interrupt processing on the target data processing node, output the data to be loaded, and load it to the target loading end.

[0038] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0039] Memory and processor;

[0040] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0041] Acquire data to be processed, and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes;

[0042] Performing preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes;

[0043] Determine whether the data to be loaded meets the output condition;

[0044] If yes, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to the target loading end.

[0045] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the data processing methods are implemented.

[0046] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0047] An embodiment of the present specification obtains data to be processed and inputs the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes, performs preset processing on the data to be processed through a target data processing node in the data processing link, and generates data to be loaded, wherein the target data processing node is one of the at least two data processing nodes, and determines whether the data to be loaded meets an output condition. If so, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to a target loading end.

[0048] In the embodiments of the present specification, for the data to be processed, each time any data processing node in the data processing link is used to process the data and obtain the corresponding processing result, it can be further determined whether the processing result needs to be output. If it needs to be output, the processing operation on the processing result can be interrupted, that is, there is no need to use other data processing nodes after any of the aforementioned data processing nodes in the data processing link to further process the processing result, and the processing result can be directly output and loaded to the target loading end. In this case, the data to be processed does not need to be fully calculated and cached, which is beneficial to reducing the resource overhead required for the data processing process, thereby helping to improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification;

[0050] Figure 2 It is a schematic diagram of a single event stream processing process provided by an embodiment of this specification.

[0051] Figure 3 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0052] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0053] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0054] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0055] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0056] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0057] First, the terms involved in one or more embodiments of this specification are explained.

[0058] Extract-Transform-Load (ETL for short): The process of reading, transforming, and outputting data from the source to the destination.

[0059] Out Of Memory (OOM for short): Memory overflow refers to the situation where there is unreclaimable memory or too much memory is used in the application system, which eventually causes the memory required for the program to run to be greater than the maximum memory that can be provided.

[0060] Interrupt mechanism: One of the basic mechanisms in modern computer operating systems. During the operation of the operating system, when certain interrupt signals (such as mouse clicks) are received, the currently running program can be interrupted and the received signal can be processed. After the processing is completed, the task executed before the interruption can be returned to continue execution.

[0061] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0062] Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0063] Step 102: Acquire data to be processed, and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes.

[0064] The data processing method provided in the embodiments of this specification is used to load data from a data source into a data warehouse, that is, the data in the data warehouse can be obtained by processing the data in the original scattered data source. The data in the data source can be log data, and the log data can be used as the data to be processed, and the data to be processed can be processed through the data processing link, so that the data that meets the loading conditions in the obtained processing results can be loaded into the data warehouse.

[0065] ETL (Extraction-Transformation-Loading) is the process of extracting, transforming and loading data. It is an important part of building a data warehouse. Users extract the required data from the database, clean the data, and finally load the data into the data warehouse according to the pre-defined data warehouse model.

[0066] Among them, data can be extracted from one or at least two data sources, that is, the data to be processed can be obtained, and the method of extracting data can be set according to actual needs. The data source can be a database, a business system, a system module, an application program, etc., and the data extracted from the database can be log data.

[0067] Since various log data are mixed together (need to filter out what you need), the data formats are diverse (the formats may not be uniform), and the data volume is relatively large. For this type of data, the usual method is filtering and extraction. Therefore, after the data to be processed is extracted from the data source, it can be input into the data processing link to perform preset processing on the data to be processed and obtain the data to be loaded to the loading end.

[0068] In a specific implementation, when the data to be processed is log data, the data to be processed is obtained, and the data to be processed is input into a data processing link, which can be specifically implemented in the following manner:

[0069] Get multiple log data to be processed;

[0070] Inputting first log data among the plurality of log data to be processed into a data processing link, wherein the first log data is one of the plurality of log data to be processed.

[0071] Specifically, since the log data is characterized by irregularity, it needs to be processed before it can be conveniently analyzed and applied. Therefore, when the log data needs to be loaded into the data warehouse, the log data can be used as the data to be processed, and when there are multiple log data to be processed, the log data to be processed can be input into the data processing link one by one for processing, specifically, the first log data among the multiple log data to be processed is input into the data processing link, and the first log data is any one of the multiple log data to be processed. In this case, the data processing link only processes a single log data at a time, which is conducive to finely controlling the use of memory, thereby avoiding the problem of memory overflow that may be caused by the data processing process.

[0072] Step 104: Perform preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes.

[0073] Specifically, the target data processing node may be the first data processing node in the data processing link.

[0074] Since the data processing link is composed of multiple data processing nodes, after each piece of data to be processed is input into the data processing node, it can be first processed by the first data processing node in the data processing link. In addition, since different data processing nodes correspond to different data processing functions, that is, different data processing nodes may have different processing methods for processing data to be processed, the data to be processed is processed by the target data processing node in the data processing link, and the data to be loaded is generated. Specifically, the preset data processing function corresponding to the target data processing node in the data processing link is determined, and the data to be processed is processed by the preset data processing function to generate the data to be loaded.

[0075] Specifically, different data processing nodes correspond to different data processing functions. Therefore, after the data to be processed is input into the data processing link, the data to be processed is first processed by the preset data processing function corresponding to the first data processing node in the data processing link, that is, the target data processing node. The preset data processing function can be used to split the data to be processed, extract key information from the data to be processed, etc., but the specific data processing function can be determined according to actual needs and is not limited here.

[0076] In actual applications, if the log data processing process is applied to the ETL processing flow, the preset data processing function here can be used to clean and convert the data to be processed. The task of data cleaning is to filter out data that does not meet the requirements, such as incomplete, erroneous or duplicate data; the task of data conversion is mainly to integrate the same type of data from different sources, aggregate data from different sources according to the granularity of the data warehouse, etc. The specific details can be determined according to actual needs and are not limited here.

[0077] Step 106: determine whether the data to be loaded meets the output condition.

[0078] Specifically, if the execution result of step 106 is yes, step 108 is executed; if the execution result is no, the data to be loaded may be transmitted to a data processing node next to the target data processing node for processing, wherein the next data processing node is one of the at least two data processing nodes; or,

[0079] If the result of the execution of determining whether the data to be loaded meets the output condition is no, then in the case where it is determined that the data to be loaded meets the deletion condition, the data to be loaded is deleted.

[0080] Furthermore, the data to be loaded is transmitted to the next data processing node of the target data processing node for processing. Specifically, the data to be loaded can be transmitted to the next data processing node of the target data processing node, and the data to be loaded is processed by the data processing function corresponding to the next data processing node to generate a corresponding processing result.

[0081] Furthermore, the data to be loaded is processed by the data processing function corresponding to the next data processing node, and after the corresponding processing result is generated, it can also be determined whether the processing result meets the output condition;

[0082] If so, interrupt processing is performed on the next data processing node, and the processing result is output and loaded to the target loading end.

[0083] Specifically, after the target data processing node is used to extract, convert, and process the data to be processed to generate the corresponding data to be loaded, it can be determined whether the data to be loaded meets the output condition. If it is determined that the data to be loaded does not meet the output condition, the data to be loaded can be input to the next data processing node of the target data processing node, that is, the second data processing node in the data processing chain for processing.

[0084] Alternatively, when it is determined that the data to be loaded does not satisfy the output condition, it is possible to first determine whether the data to be loaded satisfies the deletion condition (discarding condition). When it is determined that it satisfies the deletion condition, the data to be loaded can be deleted (discarded); and when it is determined that it does not satisfy the deletion condition, the data to be loaded can be input into the next data processing node of the target data processing node, that is, the second data processing node in the data processing link for processing.

[0085] Similarly, after the data to be loaded is transmitted to the second data processing node, the second data processing node also processes the data to be loaded through the data processing function corresponding to the data processing node to generate a corresponding processing result.

[0086] Furthermore, the processing method for the processing result is consistent with the processing method for the data to be loaded, that is, to determine whether it meets the output conditions. If so, it is output; if not, it can be further determined whether it meets the deletion conditions. If so, it is deleted; if not, it is transmitted to the third data processing node in the data processing link, and the processing result is processed by the preset data processing function corresponding to the third data processing node.

[0087] In specific implementation, the data to be loaded may be in the form of a key-value pair, so judging whether the data to be loaded meets the output condition may be implemented in the following manner:

[0088] Determine the data type of the data to be loaded according to the keyword in the key-value pair, and determine the target output condition corresponding to the data type;

[0089] It is determined whether the data corresponding to the target field of the key value in the key-value pair meets the target output condition.

[0090] Specifically, the data to be loaded generated by extracting, converting, etc. the data to be processed in the embodiments of this specification may be in the form of a key-value pair (KV), that is, composed of a keyword (key) and a key value (value). In this case, to determine whether the data to be loaded meets the output condition, the data type of the data to be loaded can be determined based on the keyword in the key-value pair, and the target output condition corresponding to the data type can be determined, and then it is determined whether the data corresponding to the target field of the key value in the key-value pair meets the target output condition.

[0091] For example, if the target output condition of the data to be loaded is a type a, the b field of the data to be loaded (the b field of the key value in the key-value pair) is equal to x, then it is output. Therefore, if the data to be loaded is determined to be type a according to the keyword in the key-value pair, it can be determined whether the b field of the data to be loaded (the b field in the key value) is equal to x. If so, it is determined that it meets the target output condition; if not, it is determined that it does not meet the target output condition.

[0092] In the embodiments of the present specification, for the data to be processed, each time any data processing node in the data processing link is used to process it and obtain the corresponding processing result, it can be further determined whether the processing result needs to be output or deleted. If it needs to be output or deleted, the processing operation on the processing result can be interrupted, that is, there is no need to use other data processing nodes after any of the aforementioned data processing nodes in the data processing link to further process the processing result, and the processing result can be directly output and loaded to the target loading end of the data warehouse, or deleted. In this case, the data to be processed does not need to be fully calculated and cached, which is beneficial to reduce the resource overhead required for the data processing process, thereby helping to improve resource utilization.

[0093] Step 108: interrupt the target data processing node, output the data to be loaded, and load it to the target loading end.

[0094] Specifically, the target loading end may be a data storage unit in a data warehouse, for example, a folder or a data table in the data warehouse.

[0095] The embodiments of this specification add an interrupt mechanism to the ETL processing flow. When it is determined that the data to be loaded meets the output conditions, the interrupt mechanism can be enabled to interrupt the processing of the data to be loaded by other data processing nodes in the data processing link, output it, and load it to the target loading end of the data warehouse.

[0096] The target loading end can be determined based on the mapping relationship between the pre-set data attributes and the loading end, that is, the data attributes of the data to be loaded are first determined, and the target loading end in the mapping relationship with the data attributes is searched according to the data attributes. Alternatively, the target loading end can be obtained by performing a calculation on the data to be loaded, for example, a hash calculation is performed on the data to be loaded to obtain the number of the target loading end, and the loading end corresponding to the number is used as the target loading end.

[0097] In a specific implementation, the data to be loaded is output and loaded to the target loading end, which can also be achieved by the following methods:

[0098] Outputting the data to be loaded from the data processing link, and storing the output data to be loaded in a data output module, wherein the data output module divides the data to be loaded according to data attributes of the data to be loaded, and generates corresponding data blocks to be loaded;

[0099] When it is determined that the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data block to be loaded is loaded to the corresponding target loading end.

[0100] Specifically, when it is determined that the data to be loaded meets the output conditions, it can be output and temporarily stored in the data output module. After the data output module receives the data to be loaded, if its memory has reached the limit (this limit is specified in advance and is safe and controllable), the data to be loaded in its cache is written to the corresponding target loading end, and the cache is cleared, and then the data to be loaded is placed in the cache; or, after the data output module receives the data to be loaded, if its memory does not reach the limit, the data to be loaded can be directly placed in a controllable cache, and the data to be loaded can be integrated, and when the integration result meets certain conditions, it is output in a unified manner.

[0101] In addition, the data output module can divide the data to be loaded according to the data attributes of the data to be loaded in the controllable cache, generate corresponding data blocks to be loaded, and when its memory reaches the limit, that is, the data volume corresponding to the data to be loaded stored in the data output module is greater than the preset data volume threshold, the data blocks to be loaded in the cache are loaded to the corresponding target loading end in the data warehouse.

[0102] In an embodiment of the present specification, when there is a mapping relationship between the data attributes of the data to be loaded and the loading end, the data output module can divide the data to be loaded according to the data attributes of the data to be loaded, and generate corresponding data blocks to be loaded. Then, when the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data blocks to be loaded can be loaded to the target loading end according to the mapping relationship between the data attributes of the data to be loaded and the loading end in the division result.

[0103] Among them, the data attributes of the data to be loaded may include time information, data source information of the data to be loaded, and one or at least two of the data types. Specifically, it can be a setting content directly extracted from the data to be loaded, or it can be obtained by performing statistical analysis on the setting content of the data to be loaded or by using a certain preset algorithm for calculation. The specific information can be determined according to actual needs and is not limited here.

[0104] In addition, after determining whether the data to be loaded meets the output condition, if the execution result of determining whether the data to be loaded meets the output condition is yes, the second log data among the multiple log data to be processed is input into the target data processing node, wherein the second log data is one of the multiple log data to be processed.

[0105] Specifically, when it is determined that the data to be loaded meets the output conditions, the data to be loaded will be output, and any other data to be processed among the multiple data to be processed can be input into the data processing link for processing. The processing process is similar to the processing process of the aforementioned data to be processed, and will not be repeated here.

[0106] The embodiments of this specification implement an efficient execution framework for processing a single log event stream. Based on the interrupt mechanism, it can finely control memory overhead and prevent memory overflow in complex scenarios (such as event splitting and one-to-many memory expansion). On the other hand, during the processing of the log event stream, it can determine whether its processing logic is terminated at the location of any data processing node, eliminating unnecessary overhead.

[0107] A schematic diagram of a single event stream processing process provided in the embodiment of this specification is as follows Figure 2 As shown. First, the data to be processed is obtained from the data source, and the data to be processed is input into the data processing link. The data to be processed is processed by function 1 in the data processing link to generate the data to be loaded, and it is determined whether the data to be loaded needs to be discarded. If it does not need to be discarded, it is determined whether it meets the output condition. If it meets the output condition, it is output and loaded into the data warehouse; if it does not meet the output condition, it is transmitted to function 2 in the data processing link, and is processed by function 2; in addition, if the data to be loaded needs to be discarded, it can be directly discarded.

[0108] Similarly, function 2 processes the data to be loaded, generates a processing result, and determines whether the processing result needs to be discarded. If it does not need to be discarded, it determines whether it meets the output condition. If it meets the output condition, it is output and loaded into the data warehouse; if it does not meet the output condition, it is transmitted to function 3 in the data processing link, and function 3 processes it; in addition, if the processing result needs to be discarded, it can be discarded directly.

[0109] The processing process of function 3 on the processing result is similar to the processing process of the aforementioned function 1 and function 2, which will not be repeated here.

[0110] Figure 2 Taking a piece of data to be processed as an example, after obtaining multiple pieces of data to be processed from the data source, the data to be processed can be input into the data processing link one by one for processing. Specifically, one of the data to be processed can be input into the data processing link for processing first. After the data to be loaded in the piece of data to be processed is deleted or output, or the processing result of the data to be loaded (except function 1, the output results of function 2, function 3 to function n can all be understood as the processing results, and n is a positive integer greater than 3) is deleted or output, the next piece of data to be processed can be input.

[0111] In the embodiments of the present specification, each event (original event or intermediate result event) can independently complete the entire processing flow, eliminating uncontrollable intermediate cache dependencies, so that memory usage can be finely controlled to solve the problem of memory overflow; in addition, each event can be judged at any location whether it needs to be discarded or output to a specific target, rather than making a judgment at the end, thereby reducing unnecessary overhead.

[0112] An embodiment of the present specification obtains data to be processed and inputs the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes, performs preset processing on the data to be processed through a target data processing node in the data processing link, and generates data to be loaded, wherein the target data processing node is one of the at least two data processing nodes, and determines whether the data to be loaded meets an output condition. If so, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to a target loading end.

[0113] In the embodiments of the present specification, for the data to be processed, each time any data processing node in the data processing link is used to process it and obtain the corresponding processing result, it can be further determined whether the processing result needs to be output. If it needs to be output, the processing operation on the processing result can be interrupted, that is, there is no need to use other data processing nodes after any of the aforementioned data processing nodes in the data processing link to further process the processing result, and the processing result can be directly output and loaded to the target loading end of the data warehouse. In this case, the data to be processed does not need to be full-link calculated and cached, which is beneficial to reduce the resource overhead required for the data processing process, thereby helping to improve resource utilization.

[0114] The following combination Figure 3 , taking the application of the data processing method provided in this specification in a practical scenario as an example, the data processing method is further explained. Figure 3 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0115] Step 302: Acquire multiple log data to be processed.

[0116] Step 304: input the first log data among the plurality of log data to be processed into a data processing link.

[0117] The first log data is any one of multiple log data to be processed, and the data processing link includes at least two data processing nodes.

[0118] Step 306: Perform preset processing on the first log data through a preset data processing function corresponding to the target data processing node in the data processing link to generate data to be loaded.

[0119] The target data processing node is one of the at least two data processing nodes.

[0120] Step 308: Determine whether the data to be loaded meets the output condition.

[0121] If the execution result of step 308 is yes, then step 310 is executed; if the execution result of step 308 is no, then step 316 is executed.

[0122] Step 310: interrupt the target data processing node, output the data to be loaded from the data processing link, and store the output data to be loaded into a data output module.

[0123] The data output module divides the data to be loaded according to data attributes of the data to be loaded, and generates corresponding data blocks to be loaded.

[0124] Step 312: when it is determined that the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data block to be loaded is loaded to the corresponding target loading end in the data warehouse.

[0125] Step 314: Use the second log data in the plurality of log data to be processed as the first log data.

[0126] Return to step 304. The second log data is one of the plurality of log data to be processed.

[0127] Step 316, determining whether the data to be loaded meets the deletion condition.

[0128] If the execution result of step 316 is yes, then step 318 is executed; if the execution result of step 316 is no, then step 320 is executed.

[0129] Step 318: delete the data to be loaded.

[0130] Return to step 314.

[0131] Step 320: transmitting the data to be loaded to the next data processing node of the target data processing node, and processing the data to be loaded by a data processing function corresponding to the next data processing node to generate the data to be loaded.

[0132] Return to step 308.

[0133] After the multiple log data to be processed are processed through the above steps, the data processing process ends.

[0134] An embodiment of the present specification obtains log data to be processed and inputs the log data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes, performs preset processing on the log data to be processed through a target data processing node in the data processing link, and generates data to be loaded, wherein the target data processing node is one of the at least two data processing nodes, and determines whether the data to be loaded meets the output condition. If so, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to the target loading end.

[0135] In the embodiments of the present specification, for any piece of log data to be processed, each time any data processing node in the data processing link is used to process it and obtain the corresponding processing result, it can be further determined whether the processing result needs to be output. If it needs to be output, the processing operation on the processing result can be interrupted, that is, there is no need to use other data processing nodes after any of the aforementioned data processing nodes in the data processing link to further process the processing result, and the processing result can be directly output and loaded to the target loading end of the data warehouse. In this case, the data to be processed does not need to be fully calculated and cached, which is beneficial to reducing the resource overhead required for the data processing process, thereby helping to improve resource utilization.

[0136] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0137] An acquisition module 402 is configured to acquire data to be processed and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes;

[0138] The processing module 404 is configured to perform preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes;

[0139] The judging module 406 is configured to judge whether the data to be loaded meets the output condition;

[0140] If the result of the operation of the judgment module 406 is yes, the loading module 408 is executed;

[0141] The loading module 408 is configured to perform interrupt processing on the target data processing node, output the data to be loaded, and load it to the target loading end.

[0142] Optionally, the data processing device further includes a transmission module configured to:

[0143] If the result of determining whether the data to be loaded meets the output condition is no, the data to be loaded is transmitted to a next data processing node of the target data processing node for processing, wherein the next data processing node is one of the at least two data processing nodes.

[0144] Optionally, the data processing device further includes a deletion module configured to:

[0145] If the result of the execution of determining whether the data to be loaded meets the output condition is no, then in the case where it is determined that the data to be loaded meets the deletion condition, the data to be loaded is deleted.

[0146] Optionally, the transmission module is further configured to:

[0147] The data to be loaded is transmitted to a data processing node next to the target data processing node, and the data to be loaded is processed by a data processing function corresponding to the next data processing node to generate a corresponding processing result.

[0148] Optionally, the data processing device further includes a condition judgment module configured to:

[0149] Determining whether the processing result meets the output condition;

[0150] If so, interrupt processing is performed on the next data processing node, and the processing result is output and loaded to the target loading end.

[0151] Optionally, the acquisition module 402 is further configured to:

[0152] Get multiple log data to be processed;

[0153] Inputting first log data among the plurality of log data to be processed into a data processing link, wherein the first log data is one of the plurality of log data to be processed.

[0154] Optionally, the data processing device further includes an input module configured to:

[0155] If the result of determining whether the data to be loaded meets the output condition is yes, the second log data among the multiple log data to be processed is input into the target data processing node, wherein the second log data is one of the multiple log data to be processed.

[0156] Optionally, the processing module 404 is further configured to:

[0157] A preset data processing function corresponding to a target data processing node in the data processing link is determined, and the data to be processed is processed by the preset data processing function to generate data to be loaded.

[0158] Optionally, the loading module 408 is further configured to:

[0159] Outputting the data to be loaded from the data processing link, and storing the output data to be loaded in a data output module, wherein the data output module divides the data to be loaded according to data attributes of the data to be loaded, and generates corresponding data blocks to be loaded;

[0160] When it is determined that the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data block to be loaded is loaded to the corresponding target loading end.

[0161] Optionally, the data to be loaded is a key-value pair;

[0162] Accordingly, the determination module 406 is further configured to:

[0163] Determine the data type of the data to be loaded according to the keyword in the key-value pair, and determine the target output condition corresponding to the data type;

[0164] It is determined whether the data corresponding to the target field of the key value in the key-value pair meets the target output condition.

[0165] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0166] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0167] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0168] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0169] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0170] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0171] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0172] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0173] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0174] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0175] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0176] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0178] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0179] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0180] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Acquire data to be processed, and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes; Performing preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes; Determine whether the data to be loaded meets the output condition; If so, interrupt processing is performed on the target data processing node, and the data to be loaded is output and loaded to the target loading end, wherein the outputting and loading of the data to be loaded to the target loading end includes: temporarily storing the data to be loaded in a data output module, and after the data output module receives the data to be loaded, if the memory of the data output module reaches the limit, writing the data to be loaded in the cache of the data output module to the corresponding target loading end.

2. The data processing method according to claim 1, further comprising: If the result of the execution of the determination of whether the data to be loaded meets the output condition is no, the data to be loaded is transmitted to a data processing node next to the target data processing node for processing, wherein the next data processing node is one of the at least two data processing nodes; or If the result of the execution of determining whether the data to be loaded meets the output condition is no, then in the case where it is determined that the data to be loaded meets the deletion condition, the data to be loaded is deleted.

3. The data processing method according to claim 2, wherein the step of transmitting the data to be loaded to a data processing node next to the target data processing node for processing comprises: The data to be loaded is transmitted to a data processing node next to the target data processing node, and the data to be loaded is processed by a data processing function corresponding to the next data processing node to generate a corresponding processing result.

4. The data processing method according to claim 3, further comprising: Determining whether the processing result meets the output condition; If so, interrupt processing is performed on the next data processing node, and the processing result is output and loaded to the target loading end.

5. The data processing method according to claim 1, wherein the step of obtaining the data to be processed and inputting the data to be processed into a data processing link comprises: Get multiple log data to be processed; Inputting first log data among the plurality of log data to be processed into a data processing link, wherein the first log data is one of the plurality of log data to be processed.

6. The data processing method according to claim 5, after determining whether the data to be loaded meets the output condition, further comprising: If the result of determining whether the data to be loaded meets the output condition is yes, the second log data among the multiple log data to be processed is input into the target data processing node, wherein the second log data is one of the multiple log data to be processed.

7. The data processing method according to claim 1, wherein the target data processing node in the data processing link performs preset processing on the data to be processed to generate data to be loaded, comprising: A preset data processing function corresponding to a target data processing node in the data processing link is determined, and the data to be processed is processed by the preset data processing function to generate data to be loaded.

8. The data processing method according to claim 1, wherein the step of outputting the data to be loaded and loading the data to a target loading end comprises: Outputting the data to be loaded from the data processing link, and storing the output data to be loaded in a data output module, wherein the data output module divides the data to be loaded according to data attributes of the data to be loaded, and generates corresponding data blocks to be loaded; When it is determined that the data volume corresponding to the data to be loaded stored in the data output module is greater than a preset data volume threshold, the data block to be loaded is loaded to the corresponding target loading end.

9. The data processing method according to claim 1, wherein the data to be loaded is a key-value pair; Accordingly, judging whether the data to be loaded meets the output condition includes: Determine the data type of the data to be loaded according to the keyword in the key-value pair, and determine the target output condition corresponding to the data type; It is determined whether the data corresponding to the target field of the key value in the key-value pair meets the target output condition.

10. A data processing device, comprising: An acquisition module is configured to acquire data to be processed and input the data to be processed into a data processing link, wherein the data processing link includes at least two data processing nodes; a processing module configured to perform preset processing on the data to be processed through a target data processing node in the data processing link to generate data to be loaded, wherein the target data processing node is one of the at least two data processing nodes; A judging module, configured to judge whether the data to be loaded meets the output condition; If the running result of the judgment module is yes, then the loading module is run; The loading module is configured to perform interrupt processing on the target data processing node, output the data to be loaded and load it to the target loading end, wherein the outputting and loading the data to be loaded to the target loading end includes: temporarily storing the data to be loaded in the data output module, and after the data output module receives the data to be loaded, if the memory of the data output module reaches the limit, writing the data to be loaded in the cache of the data output module to the corresponding target loading end.

11. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images