Solution processing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of big data processing technology, and specifically to a solution processing method, device, computer equipment and storage medium. The method includes: the execution engine monitors the solution processing message of the message queue through the real-time stream task, and the solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the processing of the target solution, indicating the message queue; according to the solution processing message, the solution configuration information of the target solution configured by the user in the configuration engine is obtained; according to the solution configuration information, the data to be processed is extracted from the target data source, and the data to be processed is subjected to ETL processing. In the embodiment of the present invention, the user can implement complex logic by configuring simple SQL statements, without the need for the user to master the SQL syntax of different data sources, strengthening configuration, and supporting multiple data sources. The execution engine is universal, so there is no need for developers to repeatedly write logic for different scenarios.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a solution processing method, apparatus, computer equipment and storage medium. Background Art

[0002] The following statements merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In the new internet retail industry, when faced with complex system interactions and interactive processing of multiple data sources, it is necessary to frequently add interaction logic with the database, and at the same time, add ETL processing logic and data exchange tasks. The workload is repetitive, and there are also major bottlenecks in performance and versatility. Summary of the Invention

[0004] In response to the above-mentioned deficiencies or shortcomings, the present invention provides a solution processing method, apparatus, computer device and storage medium. The embodiments of the present invention do not require users to master the SQL syntax of different data sources, enhance configuration, support multiple data sources, and the execution engine is universal, so developers do not need to repeatedly write logic for different scenarios.

[0005] According to a first aspect, the present invention provides a solution processing method. In one embodiment, the method includes:

[0006] The execution engine monitors the solution processing messages from the message queue through the real-time stream task. The solution processing messages are generated by the configuration engine in response to the solution processing instructions issued by the user to instruct the message queue to process the target solution. The execution engine and the configuration engine are deployed on different servers.

[0007] Obtain the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message;

[0008] Extract the data to be processed from the target data source according to the solution configuration information, and perform ETL processing on the data to be processed.

[0009] In one embodiment, the execution engine is deployed on a Hadoop cluster based on a JVM environment, and the configuration engine is deployed on a WEB server based on a JVM environment.

[0010] In one embodiment, the step of obtaining solution configuration information of a target solution configured by a user in a configuration engine according to a solution processing message includes:

[0011] Parse the solution processing message and obtain the unique solution code obtained by parsing;

[0012] The solution configuration information of the target solution configured by the user in the configuration engine corresponding to the solution unique code is queried from the configuration engine.

[0013] In one embodiment, the solution configuration information includes data source configuration information, data output configuration information, table association configuration information, filter condition configuration information, and solution execution script;

[0014] The steps of extracting the data to be processed from the target data source according to the solution configuration information and performing ETL processing on the data to be processed include:

[0015] Establish a connection with the target data source based on the data source configuration information, and extract the data to be processed from it into the Hive table corresponding to the data source configuration information;

[0016] Perform ETL processing on the data to be processed based on table association configuration information, filter condition configuration information and solution execution script.

[0017] In one embodiment, the steps of performing ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information, and the solution execution script include:

[0018] The execution script of the execution plan processes the data to be processed and obtains the preliminary processing results;

[0019] According to the table association configuration information and filter condition configuration information, the preliminary processing results are deduplicated and summarized to obtain the ETL processing results.

[0020] In one embodiment, the solution configuration information also includes data output configuration information; the steps of performing data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information to obtain the ETL processing results include:

[0021] Perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information; or

[0022] The preliminary processing results are deduplicated and summarized according to the table association configuration information and the filter condition configuration information. The processed data is stored in the Hive intermediate table corresponding to the data output configuration information. The data manually imported by the user into the preset import table is obtained, and the data in the Hive intermediate table and the data in the preset import table are merged. The merged result is stored as the ETL processing result in the Hive table corresponding to the data output configuration information.

[0023] In one embodiment, the method further includes: when performing ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information and the scheme execution script, generating a data processing log through a log engine, and recording the generated data processing log into a table for storing log data.

[0024] According to a second aspect, the present invention provides a solution processing device. In one embodiment, the device includes:

[0025] The monitoring module is used by the execution engine to monitor the solution processing messages in the message queue through the real-time streaming task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the message queue to process the target solution. The execution engine and the configuration engine are deployed on different servers respectively.

[0026] The configuration information acquisition module is used to obtain the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message;

[0027] The processing module is used to extract the data to be processed from the target data source according to the solution configuration information and perform ETL processing on the data to be processed.

[0028] According to a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0029] According to a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above method embodiments when the computer program is executed by a processor.

[0030] In an embodiment of the present invention, the execution engine monitors the solution processing message of the message queue through the real-time stream task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the message queue to process the target solution; the execution engine and the configuration engine are deployed on different servers respectively; the solution configuration information of the target solution configured by the user in the configuration engine is obtained according to the solution processing message; the data to be processed is extracted from the target data source according to the solution configuration information, and the ETL processing is performed on the data to be processed. By separating the business of the configuration engine and the execution engine, the user can implement complex logic by configuring simple SQL statements. The user does not need to master the SQL syntax of different data sources, strengthens configuration, and supports multiple data sources. The execution engine is universal, so there is no need for developers to repeatedly write logic for different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1( a ) is a diagram illustrating an application environment of a solution processing method in one embodiment;

[0032] FIG1( b ) is an application environment diagram of a solution processing method in another embodiment;

[0033] Figure 2 A schematic flow chart of a method for processing a solution in one embodiment;

[0034] Figure 3 A schematic diagram of a solution configuration page in an embodiment;

[0035] Figure 4 A schematic diagram of a user-designed model in one embodiment;

[0036] Figure 5 A schematic diagram of a task configuration page in one embodiment;

[0037] Figure 6 A schematic diagram of a process in which an execution engine processes data to be processed in one embodiment;

[0038] Figure 7 A schematic diagram of a log generated in one embodiment;

[0039] Figure 8 It is a structural block diagram of a solution processing device in one embodiment;

[0040] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] The present invention provides a solution processing method, which can be applied on a big data processing platform. The big data processing platform may include a WEB server for performing business interactions with users, a Hadoop cluster for executing user-configured solutions, and a series of databases, such as a MySQL database, HDFS distributed storage, ES (i.e., elasticsearch), etc. In one embodiment, the solution processing solution of this embodiment can be applied to an application environment as shown in Figure 1(a). In a possible embodiment, the solution processing solution of this embodiment can be applied to an application environment as shown in Figure 1(b). Among them, the configuration engine and the execution engine are respectively deployed on different servers (which can be independent servers or server clusters composed of multiple servers). For example, the execution engine can be deployed on a Hadoop cluster based on a JVM environment, and the configuration engine can be deployed on a WEB server based on a JVM environment. This makes packaging simple and allows for rapid deployment. Users can use a user client (such as a smart phone, tablet computer, laptop computer, or other computing device) on the configuration engine through the solution configuration page provided by the configuration engine (for example, it can be Figure 3 The configuration page shown in the figure is used to configure the corresponding configuration information for the solution to be processed. After the configuration information of the solution is configured, a solution processing instruction is issued to the configuration engine to instruct the configuration engine to process the solution. The configuration engine will generate a solution processing message in response to the solution processing instruction (which can be implemented using the Kafka message middleware). The execution engine can then listen to the solution processing message through a real-time stream task, and then obtain the corresponding solution configuration information based on the solution processing message. It can also extract the data to be processed, i.e., the data to be processed, from the target data source based on the solution configuration information, and then perform corresponding processing on the data to be processed.

[0043] The solution approach includes: Figure 2 The steps shown are as follows. The method is described below by taking the method applied to the execution engine in FIG1 as an example.

[0044] S110: The execution engine monitors the solution processing message of the message queue through the real-time stream task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user for instructing the message queue to process the target solution.

[0045] In this embodiment, the execution engine uses a real-time streaming task to monitor the message queue for messages to be processed. This real-time streaming task can be implemented using stream processing components such as Spark Streaming and Flink. For example, if the message queue is Kafka, the execution engine can use a Spark Streaming task to monitor Kafka topic messages.

[0046] S120: Obtaining solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message.

[0047] The solution processing message can be a string of assembled JSON strings, which contains information of several fields. In some scenarios, based on the needs of data transmission, the solution processing message can be a string of assembled JSON strings. After receiving the solution processing message, it can be parsed to obtain the information of each field. Specifically, in the scenario where the solution processing message needs to be parsed, the step of obtaining the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message includes: parsing the solution processing message and obtaining the solution unique code obtained by the parsing; querying from the configuration engine to obtain the solution configuration information of the target solution configured by the user in the configuration engine corresponding to the solution unique code. Each solution configured by the user will be assigned a unique identifier, that is, the solution unique code, for querying the corresponding information.

[0048] Among them, in addition to the unique code of the solution, the information of which fields can be included in the solution processing message can be adjusted according to the needs of the actual scenario. For example, taking the target solution as an example of obtaining budget expenses, the solution processing message can be as follows:

[0049] map:{"apportion"="exe#PWPCE00002#2021001"}

[0050] After parsing it, you can get information of the following three fields.

[0051] The "exe" field indicates the task type of the target solution, such as an ETL execution task.

[0052] The field information "PWPCE00002" is used to indicate the unique code of the target solution;

[0053] The field information "2021001" is used to indicate which billing period the target plan is to be executed.

[0054] S130: extracting the data to be processed from the target data source according to the solution configuration information, and performing ETL processing on the data to be processed.

[0055] In this embodiment, the solution configuration information includes data source configuration information, data output configuration information, table association configuration information, filter condition configuration information, and solution execution script information, which are pre-configured by the user on the configuration engine through the user client. Data source configuration information is used to determine the source of data to be processed during solution processing, such as a table name. Data output configuration information is used to specify where the processed data is to be output, such as a table name. Filter condition configuration information is generated by users by performing calculation formulas and filter configurations on the filter condition configuration page. For example, a user-configured page is provided to configure filter condition assembly configurations. If the user-configured filter condition is "date greater than January 1, 2020", the configuration engine will generate the filter condition based on this: where date > '20200101'. The solution execution script is an SQL script. Users can manually enter the SQL script or import or upload SQL files from OSS, FTP, or SVN sources to the configuration engine for increased flexibility. In addition, users can also operate on the script configuration page provided by the configuration engine. The configuration engine can generate SQL scripts based on user operations. This allows the execution engine to connect to multiple data sources, such as HIVE, MySQL, Oracle, and others. Users do not need to master the SQL syntax of different data sources when configuring SQL scripts, thereby reducing the difficulty of solution configuration. In this step, the ETL processing of the data to be processed can be performed through the Spark engine.

[0056] This embodiment is based on JVM environment deployment, combined with Hadoop distributed system architecture, and uses Spark real-time computing engine. It freely configures simple SQL form step-by-step layered logic, providing a large-scale data processing logic implementation solution under cross-platform interaction. Specifically, by separating the configuration engine from the execution engine, users can implement complex logic by configuring simple SQL statements. Users do not need to master the SQL syntax of different data sources. The configuration is strengthened and multiple data sources are supported. In addition, the execution engine is universal, and users do not need to customize different solutions. Therefore, developers do not need to repeatedly write logic for different scenarios.

[0057] Furthermore, the method further includes generating a data processing log using a logging engine during ETL processing of the data to be processed based on the table association configuration information, the filter condition configuration information, and the solution execution script, and recording the generated data processing log in a table for storing log data. The logging engine continues to record logs throughout the ETL processing process, allowing users to track the progress of solution execution through the logs.

[0058] In one embodiment, the steps of extracting the data to be processed from the target data source according to the scheme configuration information and performing ETL processing on the data to be processed in the above embodiment include: establishing a connection with the target data source according to the data source configuration information, and extracting the data to be processed from it into the Hive table corresponding to the data source configuration information; performing ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information and the scheme execution script.

[0059] In terms of data processing, compared with the previous solution processing in the SAP-BW business system, this embodiment has the problem that many SQLs occupy server resources and are difficult to trace. The configuration engine and the execution engine are deployed separately through the big data processing platform to separate the business and configuration, which can achieve higher execution efficiency (for example, some SQL statements that used to take minutes or hours to process can now be processed in seconds) and is more convenient to trace. In addition, based on the support of the execution engine, the execution steps can be quickly responded to through log records after web management. Moreover, based on the powerful support of the configuration engine and the execution engine, HDFS storage and relational databases (such as MySQL databases) can be connected to achieve real-time intercommunication, making data storage and message transmission simple and easy to use, while covering offline computing and real-time computing scenarios, achieving wide applicability of scenarios. It can be used for cost calculation in scenarios such as salary, marketing, offline convenience stores, logistics, etc., and can also be applied to calculations in various other SQL scenarios.

[0060] The steps of performing ETL processing on the data to be processed based on the table association configuration information, the filter condition configuration information, and the solution execution script include:

[0061] The execution plan executes the script to process the data to be processed to obtain preliminary processing results; the preliminary processing results are deduplicated and summarized according to the table association configuration information and filter condition configuration information to obtain the ETL processing results.

[0062] Furthermore, in one embodiment, the solution configuration information also includes data output configuration information; accordingly, the steps of performing data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information to obtain the ETL processing results include:

[0063] Perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information; or

[0064] The preliminary processing results are deduplicated and summarized according to the table association configuration information and the filter condition configuration information. The processed data is stored in the Hive intermediate table corresponding to the data output configuration information. The data manually imported by the user into the preset import table is obtained, and the data in the Hive intermediate table and the data in the preset import table are merged. The merged result is stored as the ETL processing result in the Hive table corresponding to the data output configuration information.

[0065] The processed results (i.e., the processed data) obtained by deduplicating and aggregating the preliminary processing results based on the table association configuration information and the filter condition configuration information may contain issues such as inaccuracies or missing information in certain fields. In this case, the user must manually import data to adjust or supplement the results. The processed results obtained by deduplicating and aggregating the preliminary processing results based on the table association configuration information and the filter condition configuration information must be merged with the manually imported data and stored in the Hive table corresponding to the data output configuration information.

[0066] In one embodiment, each plan may consist of only one task or multiple tasks. If the plan consists of multiple tasks, the user needs to configure corresponding configuration information (or task configuration information) for each subtask in the plan when configuring the plan.

[0067] The following uses the task configuration for obtaining budget costs as an example to introduce the task configuration process.

[0068] 1. The user needs to design the model first. For example, the model designed by the user can be Figure 4 The tables shown are the source table, dimension table, intermediate tables T1 / T3 / T6, and import table.

[0069] 2. After the user completes the model design, he can Figure 5 The page shown is used to configure task configuration information to determine which data the task processes, how to perform ETL and processing on the data, etc. It can include the following configurations:

[0070] a) Data source configuration: configure the source information of the data to be extracted in the task.

[0071] b) Data output configuration: Configure the result table (or target table) to which the processed data will be output, such as a Hive table or a relational database table (such as a MySQL table).

[0072] c) Table association configuration: You can select the corresponding master table, detail table, conditional association, etc. on the relevant page.

[0073] d) Filter condition settings: You can configure filter conditions by entering calculation formulas, text, etc. on the relevant page.

[0074] e) SQL script: Generate SQL script through relevant page configuration, enter SQL script manually, or import and upload SQL script.

[0075] 3. After configuring the tasks required to be performed by the plan, you can configure the plan, such as configuring the processing time of the plan, executing tasks, or performing data rollback.

[0076] The following is an example to illustrate the process of executing the engine processing solution. Figure 6 shown.

[0077] a) The execution engine listens to the Kafka topic message in the Spark Streaming task, parses the topic message, and obtains the unique solution code.

[0078] b) Put the obtained solution unique code into the thread pool and start a WorkTask task to process the solution corresponding to the solution unique code.

[0079] c) The log records the process ID: programId and marks the start of the task.

[0080] The process of the above WorkTask task processing the solution corresponding to the unique code of the solution is as follows:

[0081] 1) Connection initialization: Get the connection of MySQL database and connect to it.

[0082] 2) Log engine startup: The generated logs are recorded in the table used to store log data. The operation of generating logs by the log engine will run through the entire processing process of the WorkTask task.

[0083] 3) Basic data initialization: According to the unique code of the solution, the corresponding solution configuration information is queried from the configuration engine, such as source table information, target table information, source table fields, target table fields, and SQL logical processing statements (i.e., SQL scripts).

[0084] 4) Task processing: Synchronize the data in the MySQL table corresponding to the source table (i.e., the source table) to the Hive table corresponding to the source table; create a Hive intermediate table with the same structure as the Hive table corresponding to the target table (i.e., the target table); set its name to have the suffix _tmp; obtain and execute the SQL script, deduplicate and summarize the preliminary processing results obtained by executing the SQL script, and store the processed data in the Hive intermediate table;

[0085] 5) Result Output: If the user did not manually import data, the data in the Hive intermediate table is further synchronized with the Hive table corresponding to the target table. If the user manually imported data, the schema information of the imported table is first obtained. Based on this schema information, the imported data is obtained and merged with the data in the Hive intermediate table. The merged data is then stored in the Hive table corresponding to the target table. The schema information is obtained to determine the structure of the imported table for correct data import (for example, the imported data may only expose 10 fields, but the target table has 20 fields. In this case, the remaining 10 fields are obtained through default values or associations to complete the fields).

[0086] 6) Finally, the task processing flow ends. The process status can be set to success or failure according to the task processing result, and the task end log is recorded and the related database connection is closed. The content of the log after the task processing can be as follows: Figure 7 shown.

[0087] Figure 2 FIG. 1 is a flow chart of a solution processing method in an embodiment. It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0088] Based on the same inventive concept, the present invention also provides a solution processing device. In one embodiment, Figure 8 As shown, the solution processing device includes the following modules:

[0089] Monitoring module 110, configured for the execution engine to monitor solution processing messages in the message queue through real-time streaming tasks. Solution processing messages are generated by the configuration engine in response to a solution processing instruction issued by a user to instruct the message queue to process the target solution. The execution engine and the configuration engine are deployed on different servers.

[0090] The configuration information acquisition module 120 is used to obtain the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message;

[0091] The processing module 130 is used to extract the data to be processed from the target data source according to the solution configuration information, and perform ETL processing on the data to be processed.

[0092] In one embodiment, the configuration information acquisition module includes:

[0093] The parsing submodule is used to parse the solution processing message and obtain the unique solution code obtained by parsing;

[0094] The configuration information acquisition submodule is used to query the configuration engine to obtain the solution configuration information of the target solution configured by the user in the configuration engine corresponding to the solution unique code.

[0095] In one embodiment, the solution configuration information includes data source configuration information, data output configuration information, table association configuration information, filter condition configuration information, and solution execution script; the processing module includes:

[0096] The data extraction submodule is used to establish a connection with the target data source according to the data source configuration information, and extract the data to be processed from it into the Hive table corresponding to the data source configuration information;

[0097] The processing submodule is used to perform ETL processing on the data to be processed based on the table association configuration information, filter condition configuration information and solution execution script.

[0098] In one embodiment, the processing submodule includes:

[0099] A script execution unit is used to execute the program execution script to process the processing data and obtain preliminary processing results;

[0100] The processing unit is used to deduplicate and summarize the preliminary processing results according to the table association configuration information and the filter condition configuration information to obtain the ETL processing results.

[0101] In one embodiment, the processing unit is specifically used to perform data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information; or is specifically used to perform data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information, store the processed data in the Hive intermediate table corresponding to the data output configuration information, obtain the data manually imported by the user into the preset import table, merge the data in the Hive intermediate table with the data in the preset import table, and store the merged result as the ETL processing result in the Hive table corresponding to the data output configuration information.

[0102] In one embodiment, the device also includes a log generation module, which is used to generate a data processing log through a log engine when ETL processing is performed on the data to be processed according to table association configuration information, filter condition configuration information and scheme execution script, and record the generated data processing log in a table for storing log data.

[0103] For the specific definition of the solution processing device, please refer to the definition of the solution processing method above and will not be repeated here. Each module in the above-mentioned solution processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0104] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store intermediate data, final data and other data generated during processing. The specific stored data can also be found in the definitions in the above method embodiments. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a solution processing method is implemented.

[0105] Those skilled in the art will understand that Figure 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0107] The execution engine monitors the solution processing message of the message queue through the real-time stream task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the message queue to process the target solution. The execution engine and the configuration engine are deployed on different servers respectively. The solution configuration information of the target solution configured by the user in the configuration engine is obtained according to the solution processing message. The data to be processed is extracted from the target data source according to the solution configuration information, and the data to be processed is processed by ETL.

[0108] In one embodiment, the processor executes a computer program to implement the steps of obtaining solution configuration information of a target solution configured by a user in a configuration engine according to a solution processing message, including:

[0109] Parse the solution processing message and obtain the unique solution code obtained by parsing;

[0110] The solution configuration information of the target solution configured by the user in the configuration engine corresponding to the solution unique code is queried from the configuration engine.

[0111] In one embodiment, the solution configuration information includes data source configuration information, data output configuration information, table association configuration information, filter condition configuration information, and a solution execution script; the processor executes a computer program to implement the steps of extracting data to be processed from a target data source according to the solution configuration information and performing ETL processing on the data to be processed, including:

[0112] Establish a connection with the target data source based on the data source configuration information, and extract the data to be processed into the Hive table corresponding to the data source configuration information; perform ETL processing on the data to be processed based on the table association configuration information, filter condition configuration information and solution execution script.

[0113] In one embodiment, a processor executes a computer program to implement steps of performing ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information, and the solution execution script, including:

[0114] The execution plan executes the script to process the data to be processed to obtain preliminary processing results; the preliminary processing results are deduplicated and summarized according to the table association configuration information and filter condition configuration information to obtain the ETL processing results.

[0115] In one embodiment, the solution configuration information further includes data output configuration information; the processor executes a computer program to implement data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information to obtain the ETL processing results, including the following steps:

[0116] Perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information; or perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, store the processed data in the Hive intermediate table corresponding to the data output configuration information, obtain the data manually imported by the user into the preset import table, merge the data in the Hive intermediate table with the data in the preset import table, and store the merged result as the ETL processing result in the Hive table corresponding to the data output configuration information.

[0117] In one embodiment, when the processor executes the computer program, it further implements the following steps:

[0118] When ETL processing is performed on the data to be processed according to the table association configuration information, the filter condition configuration information and the scheme execution script, a data processing log is generated by the log engine and the generated data processing log is recorded in a table for storing log data.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0120] The execution engine monitors the solution processing message of the message queue through the real-time stream task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the message queue to process the target solution. The execution engine and the configuration engine are deployed on different servers respectively. The solution configuration information of the target solution configured by the user in the configuration engine is obtained according to the solution processing message. The data to be processed is extracted from the target data source according to the solution configuration information, and the data to be processed is processed by ETL.

[0121] In one embodiment, the computer program is executed by a processor to obtain solution configuration information of a target solution configured by a user in a configuration engine according to a solution processing message, including:

[0122] Parse the solution processing message and obtain the unique solution code obtained by parsing;

[0123] The solution configuration information of the target solution configured by the user in the configuration engine corresponding to the solution unique code is queried from the configuration engine.

[0124] In one embodiment, the solution configuration information includes data source configuration information, data output configuration information, table association configuration information, filter condition configuration information, and a solution execution script; the computer program is executed by a processor to extract data to be processed from a target data source according to the solution configuration information, and perform ETL processing on the data to be processed, including the following steps:

[0125] Establish a connection with the target data source based on the data source configuration information, and extract the data to be processed into the Hive table corresponding to the data source configuration information; perform ETL processing on the data to be processed based on the table association configuration information, filter condition configuration information and solution execution script.

[0126] In one embodiment, the computer program is executed by a processor to perform ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information, and the solution execution script, including the following steps:

[0127] The execution plan executes the script to process the data to be processed to obtain preliminary processing results; the preliminary processing results are deduplicated and summarized according to the table association configuration information and filter condition configuration information to obtain the ETL processing results.

[0128] In one embodiment, the solution configuration information further includes data output configuration information; the computer program is executed by a processor to perform data deduplication and aggregation processing on the preliminary processing results according to the table association configuration information and the filter condition configuration information to obtain the ETL processing results, including the following steps:

[0129] Perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information; or perform data deduplication and aggregation on the preliminary processing results according to the table association configuration information and the filter condition configuration information, store the processed data in the Hive intermediate table corresponding to the data output configuration information, obtain the data manually imported by the user into the preset import table, merge the data in the Hive intermediate table with the data in the preset import table, and store the merged result as the ETL processing result in the Hive table corresponding to the data output configuration information.

[0130] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0131] When ETL processing is performed on the data to be processed according to the table association configuration information, the filter condition configuration information and the scheme execution script, a data processing log is generated by the log engine and the generated data processing log is recorded in a table for storing log data.

[0132] Those skilled in the art will appreciate that all or part of the processes in the above method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A solution processing method, characterized in that: The method comprises: The execution engine monitors the solution processing message of the message queue through the real-time stream task. The solution processing message is generated by the configuration engine in response to the solution processing instruction issued by the user to instruct the message queue to process the target solution. The execution engine and the configuration engine are respectively deployed on different servers. Obtaining, according to the solution processing message, solution configuration information of the target solution configured by the user in the configuration engine; the solution configuration information includes data output configuration information, data source configuration information, table association configuration information, filter condition configuration information and solution execution script; Establish a connection with the target data source according to the data source configuration information, and extract the data to be processed from it into the Hive table corresponding to the data source configuration information; execute the scheme execution script to process the data to be processed to obtain a preliminary processing result; deduplicate and summarize the preliminary processing result according to the table association configuration information and the filter condition configuration information, and store the processed data as the ETL processing result in the Hive table corresponding to the data output configuration information, or deduplicate and summarize the preliminary processing result according to the table association configuration information and the filter condition configuration information, and store the processed data in the Hive intermediate table corresponding to the data output configuration information, obtain the data manually imported by the user into the preset import table, merge the data in the Hive intermediate table and the data in the preset import table, and store the merged result as the ETL processing result in the Hive table corresponding to the data output configuration information.

2. The method according to claim 1, wherein The execution engine is deployed on the Hadoop cluster based on the JVM environment, and the configuration engine is deployed on the WEB server based on the JVM environment.

3. The method according to claim 1, wherein The step of obtaining the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message includes: Parsing the solution processing message and obtaining the solution unique code obtained by parsing; The solution configuration information of the target solution configured by the user in the configuration engine and corresponding to the unique code of the solution is obtained from the configuration engine.

4. The method according to claim 1, wherein The method further includes: when performing ETL processing on the data to be processed according to the table association configuration information, the filter condition configuration information and the scheme execution script, generating a data processing log through a log engine, and recording the generated data processing log in a table for storing log data.

5. A solution processing device, characterized in that: The device comprises: A monitoring module is configured to monitor a solution processing message from a message queue received by the execution engine through a real-time streaming task. The solution processing message is generated by the configuration engine in response to a solution processing instruction issued by a user to instruct the message queue to process a target solution. The execution engine and the configuration engine are deployed on different servers. A configuration information acquisition module is configured to obtain the solution configuration information of the target solution configured by the user in the configuration engine according to the solution processing message; the solution configuration information includes data output configuration information, data source configuration information, table association configuration information, filter condition configuration information and solution execution script; A processing module is used to establish a connection with a target data source according to the data source configuration information, and extract the data to be processed from it into a Hive table corresponding to the data source configuration information; execute the scheme execution script to process the data to be processed to obtain a preliminary processing result; perform data deduplication and aggregation processing on the preliminary processing result according to the table association configuration information and the filter condition configuration information, and store the processed data as an ETL processing result in the Hive table corresponding to the data output configuration information, or perform data deduplication and aggregation processing on the preliminary processing result according to the table association configuration information and the filter condition configuration information, and store the processed data in the Hive intermediate table corresponding to the data output configuration information, obtain the data manually imported by the user into the preset import table, merge the data in the Hive intermediate table with the data in the preset import table, and store the merged result as the ETL processing result in the Hive table corresponding to the data output configuration information.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images