Data warehouse automated processing method, device, equipment and storage medium

Abstract

The present invention relates to the field of big data, and discloses a method, apparatus, device, and storage medium for automated processing of data warehouses, which are used to improve the efficiency of developing data warehouses. The method for automated processing of data warehouses includes: obtaining information to be processed, classifying the information to be processed to obtain scheduling information, creating classified information of data table information and / or developing script information; creating a target subject of a processing scheme for data processing the classified information, and calling a processing interface corresponding to the target subject; preprocessing the classified information through the processing interface to obtain processed information; identifying the information type of the processed information to obtain first target information and second target information; connecting a download interface and a modification interface corresponding to the first target information, and rendering the second target information on a preset display page. In addition, the present invention also relates to blockchain technology, and the processed information can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of ETL architecture tools, and in particular to a method, apparatus, device and storage medium for automated processing of data warehouses. Background Art

[0002] Data warehouses, as structured data environments serving as data sources for decision support systems and online analytical applications, are often used in various business domains. Business project development is enabled through data extraction, transformation, and loading (ETL) operations within data warehouses.

[0003] In the existing technology, when developing a data warehouse, developers are required to manually fill in scheduling documents and corresponding table contents when deploying versions. When building tables, they need to synchronize table information from the database management system and manually edit table building statements. The review of development scripts is manual review, which takes a long time, thus resulting in low efficiency in data warehouse development. Summary of the Invention

[0004] The main purpose of the present invention is to solve the problem of low efficiency in developing a data warehouse.

[0005] A first aspect of the present invention provides a method for automated data warehouse processing, comprising:

[0006] Acquiring information to be processed, and classifying the information to be processed to obtain classified information, wherein the classified information includes scheduling information, creation data table information, and / or development script information;

[0007] Creating a target topic for the classified information, and calling a processing interface corresponding to the target topic;

[0008] Performing information preprocessing on the classified information through the processing interface to obtain processed information;

[0009] Identifying the information type of the processing information to obtain first target information and second target information, wherein the first target information includes scheduling file information and table creation template information, and the second target information includes audit information and exception information, wherein the audit information is information obtained by performing script performance audit processing, and the exception information is information obtained by performing exception monitoring processing;

[0010] Connecting the download interface and the modification interface corresponding to the first target information, and rendering the second target information on a preset display page.

[0011] Optionally, in a first implementation of the first aspect of the present invention, preprocessing the classified information through the processing interface to obtain processed information includes:

[0012] Through the processing interface, the classification information is processed by scheduling file generation processing, table creation template generation processing and script performance audit processing, and the scheduling file, table creation template and audit information are obtained accordingly;

[0013] Obtaining operation log information of the scheduling file, the table creation template, and the audit information;

[0014] Performing abnormal monitoring on the operation log information to obtain abnormal information;

[0015] The scheduling file, the table creation template, the audit information and the exception information are determined as processing information.

[0016] Optionally, in a second implementation of the first aspect of the present invention, the processing interface performs scheduling file generation processing, table creation template generation processing, and script performance audit processing on the classification information, respectively, to obtain a scheduling file, a table creation template, and audit information, including:

[0017] When the classification information is scheduling information, obtaining a target scheduling table structure and a first processing interface corresponding to the scheduling information;

[0018] Filling the scheduling information into the target scheduling table structure through the first processing interface to obtain a scheduling file;

[0019] When the classification information is data table creation information, obtaining a target data table structure and a second processing interface corresponding to the data table creation information;

[0020] Filling the target data table structure with the data table creation information through the second processing interface to obtain a table creation template;

[0021] When the classification information is development script information, obtaining script performance audit rules from a preset rule library, and obtaining a third processing interface corresponding to the development script information;

[0022] The performance audit is performed on the development script information through the third processing interface and the script performance audit rule to obtain audit information.

[0023] Optionally, in a third implementation of the first aspect of the present invention, after performing a performance audit on the development script information using the third processing interface and the script performance audit rule and obtaining the audit information, the method further includes:

[0024] Acquire historical development script information corresponding to the audit information from a preset script library;

[0025] Calculating the similarity between the historical development script information and the development script information;

[0026] Sorting the historical development script information in descending order of similarity values ​​to obtain target historical development script information ranked first;

[0027] A script optimization solution corresponding to the target historical development script information is obtained, and the script optimization solution is added to the audit information to obtain final audit information.

[0028] Optionally, in a fourth implementation of the first aspect of the present invention, performing a performance audit on the development script information using the third processing interface and the script performance audit rule to obtain audit information includes:

[0029] Obtaining the preset performance statement and preset degree table in the script performance audit rule;

[0030] Detecting the development script information through the third processing interface and the preset performance statement to obtain target information, where the target information includes a target script statement corresponding to the preset performance statement and a target number of the target script statements;

[0031] Traversing the preset degree table according to the target information to obtain a target score corresponding to the target information;

[0032] The target script statement and the target score are determined as review information of the development script information.

[0033] Optionally, in a fifth implementation of the first aspect of the present invention, detecting the development script information through the third processing interface and the preset performance statement to obtain target information includes:

[0034] Creating, through the third processing interface, a hash value of the preset performance statement, a hash table of the development script information, and multiple processes respectively;

[0035] The hash table is searched according to a preset dictionary, the hash value and the multiple processes to obtain target information.

[0036] Optionally, in a sixth implementation of the first aspect of the present invention, before obtaining the information to be processed and classifying the information to be processed to obtain classified information, the steps include:

[0037] Constructing a user page element framework, connecting the user page element framework with a preset data warehouse, and generating a rule base;

[0038] The preset script performance audit rules are encapsulated in the rule library through a preset open source web application framework, and the script performance audit rules are used to indicate rules for auditing whether the code performance of the script meets preset requirements.

[0039] A second aspect of the present invention provides a device for automated data warehouse processing, comprising:

[0040] A first classification module is used to obtain information to be processed, classify the information to be processed to obtain classified information, and the classified information includes scheduling information, creation data table information and / or development script information;

[0041] A creation module, configured to create a target topic of the classified information and call a processing interface corresponding to the target topic;

[0042] a processing module, configured to perform information preprocessing on the classified information through the processing interface to obtain processed information;

[0043] a second classification module for identifying an information type of the processing information to obtain first target information and second target information, wherein the first target information includes scheduling file information and table creation template information, and the second target information includes audit information and exception information, wherein the audit information is information obtained by performing script performance audit processing, and the exception information is information obtained by performing exception monitoring processing;

[0044] A connection module is used to connect the download interface and the modification interface corresponding to the first target information, and render the second target information on a preset display page.

[0045] Optionally, in a first implementation of the second aspect of the present invention, the processing module includes:

[0046] A processing submodule is used to perform scheduling file generation processing, table creation template generation processing and script performance audit processing on the classification information through the processing interface, and obtain a scheduling file, a table creation template and audit information accordingly;

[0047] A first acquisition submodule is used to obtain the operation log information of the scheduling file, the table creation template and the audit information;

[0048] The exception monitoring submodule is used to perform exception monitoring on the operation log information to obtain exception information;

[0049] A determination submodule is used to determine the scheduling file, the table creation template, the audit information and the exception information as processing information.

[0050] Optionally, in a second implementation of the second aspect of the present invention, the processing submodule includes:

[0051] a first acquiring unit, configured to acquire a target scheduling table structure and a first processing interface corresponding to the scheduling information when the classification information is scheduling information;

[0052] A first filling unit, configured to fill the scheduling information into the target scheduling table structure through the first processing interface to obtain a scheduling file;

[0053] A second acquiring unit is configured to acquire a target data table structure and a second processing interface corresponding to the data table creation information when the classification information is data table creation information;

[0054] A second filling unit is configured to fill the data table creation information into the target data table structure through the second processing interface to obtain a table creation template;

[0055] a third acquiring unit, configured to acquire, when the classification information is development script information, a script performance audit rule from a preset rule library, and acquire a third processing interface corresponding to the development script information;

[0056] The performance audit unit is used to perform a performance audit on the development script information through the third processing interface and the script performance audit rule to obtain audit information.

[0057] Optionally, in a third implementation of the second aspect of the present invention, the processing submodule further includes:

[0058] A first acquiring unit is configured to acquire historical development script information corresponding to the audit information from a preset script library;

[0059] a calculation unit, configured to calculate a similarity between the historical development script information and the development script information;

[0060] a sorting unit, configured to sort the historical development script information in descending order of similarity values ​​to obtain target historical development script information ranked first;

[0061] The adding unit is used to obtain a script optimization solution corresponding to the target historical development script information, and add the script optimization solution to the audit information to obtain final audit information.

[0062] Optionally, in a fourth implementation of the second aspect of the present invention, the performance audit unit includes:

[0063] An acquisition subunit, configured to acquire a preset performance statement and a preset degree table in the script performance audit rule;

[0064] a detection subunit, configured to detect the development script information through the third processing interface and the preset performance statement to obtain target information, wherein the target information includes a target script statement corresponding to the preset performance statement and a target number of the target script statements;

[0065] a traversal subunit, configured to traverse the preset degree table according to the target information to obtain a target score corresponding to the target information;

[0066] The determination subunit is configured to determine the target script statement and the target score as the review information of the development script information.

[0067] Optionally, in a fifth implementation of the second aspect of the present invention, the detection subunit is specifically configured to:

[0068] Creating, through the third processing interface, a hash value of the preset performance statement, a hash table of the development script information, and multiple processes respectively;

[0069] The hash table is searched according to a preset dictionary, the hash value and the multiple processes to obtain target information.

[0070] Optionally, in a sixth implementation of the second aspect of the present invention, the apparatus for automated data warehouse processing further includes:

[0071] A construction module is used to construct a user page element framework, connect the user page element framework with a preset data warehouse, and generate a rule base;

[0072] The encapsulation module is used to encapsulate preset script performance audit rules in the rule library through a preset open source web application framework, wherein the script performance audit rules are used to indicate rules for auditing whether the script code performance meets preset requirements.

[0073] The third aspect of the present invention provides a device for automated processing of data warehouses, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the device for automated processing of data warehouses executes the above-mentioned method for automated processing of data warehouses.

[0074] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned method for automated data warehouse processing.

[0075] In the technical solution provided by the present invention, the information to be processed is classified to obtain scheduling configuration information, data table creation configuration information and / or classification information of development script information; a target subject of a processing scheme for data processing of the classified information is created, and a processing interface corresponding to the target subject is called; the classified information is processed through the processing interface to generate a scheduling file, generate a table template, perform a script performance audit, and perform an exception monitoring process to obtain processing information; the information type of the processed information is identified to obtain first target information and second target information, the first target information includes scheduling file information and table template information, the second target information includes audit information and exception information, the audit information is audit information obtained by performing a script performance audit process, and the exception information is exception information obtained by performing an exception monitoring process; the download interface and the modification interface corresponding to the first target information are connected, and the second target information is rendered on a preset display page. In the present invention, by realizing automatic configuration of data warehouse to generate scheduling files and table creation templates, and automatically performing script performance audits, manual operations in the development process are reduced, standardization and templateization of data table creation are achieved, and excessive operations of querying source table information are reduced. Script tasks that affect script performance are predicted in advance and script codes are standardized, which reduces the workload of manual audits and thereby improves the efficiency of data warehouse development. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of an embodiment of a method for automated data warehouse processing according to an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of an embodiment of a device for automated data warehouse processing according to an embodiment of the present invention;

[0078] Figure 3 This is a schematic diagram of an embodiment of a device for automated data warehouse processing in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The embodiments of the present invention provide a method, apparatus, device and storage medium for automated processing of a data warehouse, thereby improving the efficiency of developing a data warehouse.

[0080] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0081] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of the method for automated data warehouse processing in the embodiment of the present invention includes:

[0082] 101. Obtain information to be processed, classify the information to be processed to obtain classified information, where the classified information includes scheduling information, data table creation information, and / or development script information;

[0083] It is understandable that the execution subject of the present invention can be a device for data warehouse automation processing, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0084] The server may receive at least one of the following information to be processed: the configuration information for scheduling, the configuration information for creating a data table, and the script input on a preset configuration page and / or input page. For example, the input information to be processed may include the source database information of the data table, the data table information (including the table name, field name, type and length of each field, and other definition information), the owner information of the data table, and the development script information; or the input information to be processed may only include the development script information. By classifying the information to be processed, it is possible to facilitate the subsequent efficient calling and processing of the information to be processed.

[0085] It should be noted that the scheduling information is the scheduling configuration information, which may include the scheduling module of the data warehouse, the scheduling time, the dependency of the scheduling task and the scheduling task type. The data table creation information is the configuration information for creating the data table, which may include the source library information of the data table, the data table information (including the table name, field name, the type and length of each field and other definition information) and the owner information of the data table.

[0086] Specifically, the server obtains the information to be processed, and before classifying the information to be processed to obtain the classified information, it constructs a user page element framework, connects the user page element framework with a preset data warehouse, and generates a rule library; through a preset open source web application framework, the preset script performance audit rules are encapsulated in the rule library, and the script performance audit rules are used to indicate the rules for auditing whether the code performance of the script meets the preset requirements.

[0087] The server builds the user page element framework Vue+Elementui to achieve the acquisition and transmission of pending information on the front-end interface, as well as the display of secondary target information. The server generates data processing classes using the pre-installed open-source web application framework Django to encapsulate pre-set script performance audit rules in a rule library, implementing back-end logical management of script performance audits within the rule library. Script performance audit rules can include multiple audit rules, customizable by users based on business needs. For example, rules include whether the script code contains Cartesian products, whether it contains select* statements, and whether the number of joins exceeds a preset threshold. This rule library automates data warehouse development and improves data warehouse development efficiency.

[0088] 102. Create a target topic for the classified information, and call a processing interface corresponding to the target topic;

[0089] Among them, the target topic can be used to indicate the processing scheme for data processing of classified information. The server can create a corresponding target topic based on the information type of the classified information. For example, when the classified information (information type) is the configuration information (scheduling information) of the scheduling file, the name of the corresponding target topic (data processing processing scheme) is scheduling file generation, and the content of the target topic is generating a scheduling file based on the configuration information of the scheduling file; when the classified information (information type) is the configuration information of the data table (creating data table information), the name of the corresponding target topic (data processing processing scheme) is table template generation, and the content of the target topic is generating a table template based on the configuration information of the data table; when the classified information (information type) is script code (developing script information), the name of the corresponding target topic (data processing processing scheme) is script performance review, and the content of the target topic is performing a performance review on the development script information. By creating a target topic corresponding to the classified information, the corresponding processing interface can be called. By creating a target topic and calling the processing interface corresponding to the target topic, the corresponding data processing can be carried out in an orderly and effective manner according to the various types of information in the information to be processed.

[0090] It should be noted that the server can create a target topic table in the form of a hash table of classified information. The target topic table can include the name of the target topic, the content of the processing object, and the processing flow (processing solution) to achieve the creation of target topics for classified information. For example, the target topic may include the name "Schedule File Generation", the content of the processing object, and the processing flow (processing solution) for "Generate a scheduling file based on the configuration information of the scheduling file". The server can also create target topics corresponding to scheduling configuration information, table creation configuration information, and development script information through the Kafka system.

[0091] The processing plan for data processing of classified information may include the information type of the classified information, the corresponding operation measure name and the operation measure content. For example: the processing plan is to create the configuration information of the data table (information type) - create a table template (operation measure name) - perform operation B according to information A in the configuration information of the scheduling file; or perform operation D (operation measure content) according to information C in the configuration information of the scheduling file.

[0092] 103. Preprocess the classified information through the processing interface to obtain processed information;

[0093] The server uses the processing interface corresponding to the information type of the classification information to generate a scheduling file, generate a table template, or perform script performance audits and exception monitoring for the corresponding classification. For example, if the classification information is scheduling configuration information and data table creation configuration information, the server calls the scheduling file processing interface to generate a scheduling file based on the scheduling configuration information, and calls the table template processing interface to generate a table template based on the data table creation configuration information. At the same time, the operation process of generating the scheduling file and the table template is monitored for exceptions. It should be emphasized that to further ensure the privacy and security of the above-mentioned processing information, the above-mentioned processing information can also be stored in a blockchain node.

[0094] Specifically, the server performs scheduling file generation processing, table creation template generation processing and script performance audit processing on the classified information through the processing interface, and obtains corresponding scheduling files, table creation templates and audit information; obtains operation log information of the scheduling files, table creation templates and audit information; performs exception monitoring processing on the operation log information to obtain exception information; and determines the scheduling files, table creation templates, audit information and exception information as processing information.

[0095] Optionally, when the classification information is scheduling information, the server obtains the target scheduling table structure and the first processing interface corresponding to the scheduling information; the scheduling information is filled into the target scheduling table structure through the first processing interface to obtain a scheduling file. For example: the scheduling information includes the scheduling module A, scheduling time B, scheduling task dependency C and scheduling task type D of the data warehouse, then the processing interface corresponding to the scheduling configuration information is called to fill A, B, C and D into the module item, time item, dependency item and task type item of the preset table template file (target scheduling table structure) respectively, thereby generating the corresponding scheduling file.

[0096] Optionally, when the classification information is information for creating a data table, the server obtains a second processing interface corresponding to the target data table structure and the information for creating the data table; the information for creating the data table is filled into the target data table structure through the second processing interface to obtain a table creation template. The server can fill the source library information, data table information (including table name, field name, type, length and other definition information for each field) and owner information of the data table into the target data table structure through the data management supervisor (DMS) or table creation statement DDL statement preset in the second processing interface, generate a corresponding table creation template, and perform data type conversion and data storage on the personalized table creation and source library table creation of the generated table creation template by connecting to the source library, thereby obtaining the final table creation template.

[0097] Optionally, the server can evaluate and score script performance and conduct batch audits to perform batch performance audits on the development script information in the batch of classified information, thereby obtaining scripts with good performance. The server performs abnormal monitoring on abnormal operations and abnormal information in the process of scheduling file generation, table template generation, and script performance audit. Among them, the server can perform abnormal monitoring on the information processing process of the classified information in parallel when processing information according to the classified information; it can also perform abnormal analysis on the classified information through a preset monitoring tool after processing information according to the classified information to implement abnormal monitoring. When performing abnormal analysis on the operation log information, the content of the abnormal analysis may include whether the duration of the operation is greater than a preset threshold and whether the information of the operation meets the corresponding conditional rules.

[0098] Optionally, when the classification information is development script information, the server obtains script performance audit rules from a preset rule library, as well as a third processing interface corresponding to the development script information; performs a performance audit on the development script information through the third processing interface and the script performance audit rules to obtain audit information. Specifically, the server obtains a preset performance statement and a preset degree table in the script performance audit rules; detects the development script information through the third processing interface and the preset performance statement to obtain target information, including a target script statement corresponding to the preset performance statement and a target number of target script statements; traverses the preset degree table based on the target information to obtain a target score corresponding to the target information, and determines the target script statement and target score as the audit information for the development script information.

[0099] Optionally, preset performance statements include, but are not limited to, statements containing select*, join, or order by, statement formats corresponding to hard-coded parameters (i.e., parameters are fixedly written in the development script information of this version), and statements containing Cartesian products. The target score may be the preset full score minus the deduction score corresponding to the target information. For example, if there are target script statements containing select* performance preset statements in the development script information, and the number of target script statements is 3, the deduction score for each target script statement containing select* performance preset statements is 1, and the preset full score is 10, then the development script information will be scored 7 points after the performance review.

[0100] Optionally, the target score may be the score of the development script information directly corresponding to the target information. For example, if the development script information contains two target script statements with an order by performance preset statement, and the number of target script statements is 2, and the score corresponding to the number of target script statements with an order by performance preset statement is 2, then the development script information will be scored 6 after the performance audit. To improve the efficiency of performance audits, multi-process technology is used to enhance the performance of performance audits.

[0101] Specifically, the server creates a hash value of a preset performance statement, a hash table of development script information, and multiple processes through a third processing interface; retrieves the hash table according to a preset dictionary, hash value, and multiple processes to obtain target information.

[0102] The server can create a dictionary object with preset performance statements based on hash values ​​from a pre-set dictionary. This dictionary object can then be used to perform multi-process dictionary queries and retrievals on the hash three-column table, effectively enabling batch audits of development script information. The server combines the hash value and hash table characteristics of the hash algorithm with the characteristics of the dictionary to perform performance audits on development script information, improving the accuracy and efficiency of script code performance audits. The server also creates multiple processes through a batch processing interface, performing parallel performance audits of development script information across multiple processes, improving the efficiency and performance of performance audits.

[0103] Specifically, after the server performs a performance audit on the development script information to obtain the audit information, before obtaining the scheduling file, table creation template and operation log information of the audit information, it obtains the historical development script information corresponding to the audit information from the preset script library; calculates the similarity between the historical development script information and the development script information; sorts the historical development script information in descending order according to the similarity value to obtain the target historical development script information ranked first; obtains the script optimization plan corresponding to the target historical development script information, and adds the script optimization plan to the audit information to obtain the final audit information.

[0104] After the server obtains the audit information that determines the target script statement and target score as the development script information, it obtains historical development scripts that are identical to the audit information or have a similarity greater than a preset threshold from the preset script library, obtains the script optimization plan of the historical opening script that is closest to the development script information (with the greatest similarity) from the historical opening scripts, and adds the script optimization plan to the audit information by adding a link or other adding methods to update the audit information. For example: the target script statement is E, the target score is 9.5 points, the preset threshold is 60%, and the historical development scripts are A1 (script statements are E1 and E2, with a score of 9.0 points), B1 (script statements are E1 and F, with a score of 8.0 points) and C1 (script statements are G and H, with a score of 6.5 points). The similarities of A1, B1 and C1 with the target script statement E and the score of 9.5 points are 90%, 75% and 55% respectively. Then, the script optimization plan of the historical opening script A1 is obtained, and the script optimization plan is added to the audit information by adding a link or other adding methods.

[0105] It should be noted that by adding script optimization plans to the audit information, while avoiding non-standard development script information and ensuring the quality of development script information, the diversity of information obtained from performance audits of development script information is improved, so as to reduce the time for users to obtain optimization reference information when optimizing development script information, enhance the automation of data warehouse development, and thus improve the efficiency of data warehouse development.

[0106] 104. Identify the information type of the processed information to obtain first target information and second target information. The first target information includes scheduling file information and table creation template information. The second target information includes audit information and exception information. The audit information is information obtained from script performance audit processing, and the exception information is information obtained from exception monitoring processing.

[0107] Information types include two types: one type is data table type information, and the other type is information obtained after audit and monitoring. Before the server identifies the information type of the processed information and classifies the processed information according to the information type, it can tag the content of the processed information to obtain label information. For example, it can tag the script exception item and the script exception reason on the exception information in the processed information, and extract the label information using a preset label extraction algorithm to obtain the first target information and / or the second target information.

[0108] The server can create a binary tree of the information type of the processed information and traverse the node information of the binary tree to identify the first target information and the second target information in the processed information, thereby identifying the information type of the processed information. By identifying the information type of the processed information, corresponding subsequent data operations can be effectively performed.

[0109] Optionally, after obtaining the processing information, the server may also store the processing information in a corresponding storage space, which is pre-marked with a type. The processing information is identified according to the type on the storage space, thereby obtaining the first target information and the second target information, so as to identify the information type of the processing information. For example, the preset storage space for the first information type in the preset database is C1 (scheduling file) and C2 (table creation template), and the preset storage space for the second information type in the preset database is C3 (audit information) and C4 (exception information). When the processing information of the scheduling file information is obtained, the scheduling file information is transferred to C1; when the processing information of the table creation template information is obtained, the table creation template information is transferred to C2; when the processing information of the audit information is obtained, the audit information is transferred to C3; when the processing information of the exception information is obtained, the exception information is stored in C4. It can be known that the processing information stored corresponding to C1 and C2 is the first target information (corresponding to the first information type), and the processing information stored corresponding to C3 and C4 is the second target information (corresponding to the second information type), thereby identifying the information type of the processing information and obtaining the first target information and the second target information.

[0110] 105. Connect the download interface and the modification interface corresponding to the first target information, and render the second target information on a preset display page.

[0111] After obtaining the first target information, the server connects the download interface and modification interface corresponding to the first target information, and connects the download interface and modification interface to the terminal's display page in the form of buttons or trigger indicators. The download interface allows downloading of the first information, and the modification interface allows direct configuration modification of the first information. After obtaining the second target information, the server converts the data format of the second target information into JSON format, transmits the converted second target information to a preset display page in the terminal, and renders the converted second target information on the user's display page using a preset rendering function.

[0112] It should be noted that by linking the download interface and the modification interface and rendering it to the display page on the user side, it is conducive to the automated operation of data warehouse development, and it is also conducive to users to understand abnormal information and then adjust the data warehouse development, thereby improving the accuracy of data warehouse development and thus improving the efficiency of data warehouse development.

[0113] In the embodiment of the present invention, by realizing automatic configuration of the data warehouse to generate scheduling files and table creation templates, and automatically performing script performance audits, manual operations in the development process are reduced, standardization and templateization of data table creation are achieved, and excessive operations of querying source table information are reduced. Script tasks that may affect script performance are predicted in advance and script codes are standardized, which reduces the workload of manual audits. By automatically processing the interface to generate scheduling files based on classification information, generate data tables to create templates, and audit script performance, the amount of manual operations is effectively reduced. Through exception monitoring, the accuracy of automated information processing is improved, thereby improving the efficiency of data warehouse development.

[0114] See also Figure 2 Another embodiment of the apparatus for automated data warehouse processing according to the present invention includes:

[0115] The first classification module 201 is used to obtain information to be processed, classify the information to be processed to obtain classified information, and the classified information includes scheduling information, data table creation information and / or development script information;

[0116] A creation module 202 is used to create a target topic of the classified information and call a processing interface corresponding to the target topic;

[0117] The processing module 203 is used to perform information preprocessing on the classified information through the processing interface to obtain processed information;

[0118] The second classification module 204 is configured to identify the type of the processed information and obtain first target information and second target information. The first target information includes scheduling file information and table creation template information. The second target information includes audit information and exception information. The audit information is information obtained through script performance audit processing, and the exception information is information obtained through exception monitoring processing.

[0119] The connection module 205 is used to connect the download interface and the modification interface corresponding to the first target information, and render the second target information on a preset display page.

[0120] Optionally, the processing module 203 specifically includes:

[0121] The processing submodule is used to perform scheduling file generation processing, table creation template generation processing and script performance audit processing on the classified information through the processing interface, and obtain the scheduling file, table creation template and audit information accordingly;

[0122] The first acquisition submodule is used to obtain operation log information of scheduling files, table creation templates and audit information;

[0123] The exception monitoring submodule is used to perform exception monitoring on the operation log information to obtain exception information;

[0124] The determination submodule is used to determine the scheduling file, table creation template, audit information and exception information as processing information.

[0125] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned processing information, the above-mentioned processing information can also be stored in a node of a blockchain.

[0126] Optionally, the processing submodule includes:

[0127] a first acquiring unit, configured to acquire a target scheduling table structure and a first processing interface corresponding to the scheduling information when the classification information is scheduling information;

[0128] A first filling unit is used to fill the scheduling information into the target scheduling table structure through the first processing interface to obtain a scheduling file;

[0129] A second acquiring unit, configured to acquire a target data table structure and a second processing interface corresponding to the data table creation information when the classification information is data table creation information;

[0130] A second filling unit is used to fill the data table creation information into the target data table structure through the second processing interface to obtain a table creation template;

[0131] a third acquiring unit, configured to acquire, when the classification information is development script information, a script performance audit rule from a preset rule library and a third processing interface corresponding to the development script information;

[0132] The performance audit unit is used to perform a performance audit on the development script information through the third processing interface and the script performance audit rule to obtain audit information.

[0133] Optionally, the processing module 203 further includes:

[0134] A first acquiring unit is configured to acquire historical development script information corresponding to the audit information from a preset script library;

[0135] A calculation unit, used to calculate the similarity between historical development script information and development script information;

[0136] A sorting unit, configured to sort the historical development script information in descending order of similarity values, and obtain target historical development script information ranked first;

[0137] An adding unit is used to obtain a script optimization plan corresponding to the target historical development script information, and add the script optimization plan to the audit information to obtain the final audit information.

[0138] Optional performance review units include:

[0139] An acquisition subunit is used to acquire a preset performance statement and a preset degree table in a script performance audit rule;

[0140] a detection subunit, configured to detect the development script information through the third processing interface and the preset performance statement to obtain target information, the target information including a target script statement corresponding to the preset performance statement and a target number of the target script statements;

[0141] A traversal subunit, configured to traverse a preset degree table according to target information to obtain a target score corresponding to the target information;

[0142] A determination subunit is used to determine the target script statement and the target score as the review information of the development script information.

[0143] Optionally, the detection subunit may also be specifically used for:

[0144] Through the third processing interface, a hash value of the preset performance statement, a hash table of the development script information and multiple processes are created respectively;

[0145] The target information is obtained by searching the hash table according to the preset dictionary, hash value and multi-process.

[0146] Optionally, the data warehouse automated processing device also includes:

[0147] A construction module is used to construct a user page element framework, connect the user page element framework with a preset data warehouse, and generate a rule base;

[0148] The encapsulation module is used to encapsulate preset script performance audit rules in a rule library through a preset open source web application framework. The script performance audit rules are used to indicate the rules for auditing whether the script code performance meets the preset requirements.

[0149] The functional implementation of each module and each unit in the above-mentioned data warehouse automatic processing device corresponds to each step in the above-mentioned data warehouse automatic processing method embodiment, and their functions and implementation processes are no longer repeated here.

[0150] In the embodiment of the present invention, by realizing automatic configuration of the data warehouse to generate scheduling files and table creation templates, and automatically performing script performance audits, manual operations in the development process are reduced, standardization and templateization of data table creation are achieved, and excessive operations of querying source table information are reduced. Script tasks that may affect script performance are predicted in advance and script codes are standardized, which reduces the workload of manual audits. By automatically processing the interface to generate scheduling files based on classification information, generate data tables to create templates, and audit script performance, the amount of manual operations is effectively reduced. Through exception monitoring, the accuracy of automated information processing is improved, thereby improving the efficiency of data warehouse development.

[0151] above Figure 2 The apparatus for automated data warehouse processing in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The equipment for automated data warehouse processing in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0152] Figure 3: This is a structural diagram of a data warehouse automation processing device provided by an embodiment of the present invention. The data warehouse automation processing device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and memory 320, and one or more storage media 330 (for example, one or more mass storage devices) for storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data warehouse automation processing device 300. Furthermore, the processor 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the data warehouse automation processing device 300.

[0153] The data warehouse automation processing device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 3 The device structure of the data warehouse automation processing shown does not constitute a limitation on the device of the data warehouse automation processing, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0154] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the method for automated data warehouse processing.

[0155] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0156] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0158] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automated processing of data warehouse, characterized in that: The data warehouse automation processing method includes: Acquiring information to be processed, and classifying the information to be processed to obtain classified information, wherein the classified information includes scheduling information, creation data table information, and / or development script information; Creating a target topic for the classified information and calling a processing interface corresponding to the target topic, wherein the target topic is used to indicate a processing scheme for performing data processing on the classified information; Performing information preprocessing on the classified information through the processing interface to obtain processed information; Identifying the information type of the processing information to obtain first target information and second target information, wherein the first target information includes scheduling file information and table creation template information, and the second target information includes audit information and exception information, wherein the audit information is information obtained by performing script performance audit processing, and the exception information is information obtained by performing exception monitoring processing; Connecting the download interface and the modification interface corresponding to the first target information, and rendering the second target information on a preset display page.

2. The method for automated data warehouse processing according to claim 1, wherein: The preprocessing of the classified information by the processing interface to obtain processed information includes: Through the processing interface, the classification information is processed by scheduling file generation processing, table creation template generation processing and script performance audit processing, and the scheduling file, table creation template and audit information are obtained accordingly; Obtaining operation log information of the scheduling file, the table creation template, and the audit information; Performing abnormal monitoring on the operation log information to obtain abnormal information; The scheduling file, the table creation template, the audit information and the exception information are determined as processing information.

3. The method for automated data warehouse processing according to claim 2, characterized in that: The processing interface is used to process the classification information, and the scheduling file generation processing, table creation template generation processing and script performance audit processing are performed respectively, and the scheduling file, table creation template and audit information are obtained accordingly, including: When the classification information is scheduling information, obtaining a target scheduling table structure and a first processing interface corresponding to the scheduling information; Filling the scheduling information into the target scheduling table structure through the first processing interface to obtain a scheduling file; When the classification information is data table creation information, obtaining a target data table structure and a second processing interface corresponding to the data table creation information; Filling the target data table structure with the data table creation information through the second processing interface to obtain a table creation template; When the classification information is development script information, obtaining script performance audit rules from a preset rule library, and obtaining a third processing interface corresponding to the development script information; The performance audit is performed on the development script information through the third processing interface and the script performance audit rule to obtain audit information.

4. The method for automated data warehouse processing according to claim 3, wherein: After performing a performance audit on the development script information through the third processing interface and the script performance audit rule and obtaining audit information, the method further includes: Acquire historical development script information corresponding to the audit information from a preset script library; Calculating the similarity between the historical development script information and the development script information; Sorting the historical development script information in descending order of similarity values ​​to obtain target historical development script information ranked first; A script optimization solution corresponding to the target historical development script information is obtained, and the script optimization solution is added to the audit information to obtain final audit information.

5. The method for automated data warehouse processing according to claim 3, wherein: The step of performing a performance audit on the development script information through the third processing interface and the script performance audit rule to obtain audit information includes: Obtaining the preset performance statement and preset degree table in the script performance audit rule; Detecting the development script information through the third processing interface and the preset performance statement to obtain target information, the target information including a target script statement corresponding to the preset performance statement and a target number of the target script statements; Traversing the preset degree table according to the target information to obtain a target score corresponding to the target information; The target script statement and the target score are determined as review information of the development script information.

6. The method for automated data warehouse processing according to claim 5, characterized in that: The detecting the development script information through the third processing interface and the preset performance statement to obtain target information includes: Creating, through the third processing interface, a hash value of the preset performance statement, a hash table of the development script information, and multiple processes respectively; The hash table is searched according to a preset dictionary, the hash value and the multiple processes to obtain target information.

7. The method for automated data warehouse processing according to any one of claims 1 to 6, characterized in that: The step of obtaining the information to be processed and classifying the information to be processed to obtain classified information includes: Constructing a user page element framework, connecting the user page element framework with a preset data warehouse, and generating a rule base; The preset script performance audit rules are encapsulated in the rule library through a preset open source web application framework, and the script performance audit rules are used to indicate rules for auditing whether the code performance of the script meets preset requirements.

8. A device for automated processing of data warehouse, characterized in that: The device for automatic data warehouse processing includes: A first classification module is used to obtain information to be processed, classify the information to be processed to obtain classified information, and the classified information includes scheduling information, data table creation information and / or development script information; a creation module, configured to create a target topic of the classified information and call a processing interface corresponding to the target topic, wherein the target topic is used to indicate a processing scheme for data processing of the classified information; a processing module, configured to perform information preprocessing on the classified information through the processing interface to obtain processed information; a second classification module for identifying an information type of the processing information to obtain first target information and second target information, wherein the first target information includes scheduling file information and table creation template information, and the second target information includes audit information and exception information, wherein the audit information is information obtained by performing script performance audit processing, and the exception information is information obtained by performing exception monitoring processing; A connection module is used to connect the download interface and the modification interface corresponding to the first target information, and render the second target information on a preset display page.

9. A device for automated processing of data warehouse, characterized in that: The data warehouse automation processing device includes: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a line; The at least one processor calls the instructions in the memory to enable the data warehouse automation processing device to execute the data warehouse automation processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automated data warehouse processing according to any one of claims 1 to 7 is implemented.

Images