Data processing method, device and equipment

Abstract

The invention relates to the technical field of databases, in particular to a data processing method, device and equipment. A data processing method comprises the steps that query statements in workflows to be processed are analyzed, keyword information of the workflows is obtained, and the keyword information comprises target table fields, query result fields, source table fields, query condition fields, grouping fields and sorting fields of the workflows; performing optimization processing on each workflow according to the keyword information of each workflow; and updating the data warehouse according to the optimization processing result of the workflow. Through the embodiment of the invention, the repeated calculation of the workflow is reduced, the calculation resources and the calculation cost are saved, the reasonability of the data processing task is improved, the maintainability is improved, the task robustness is improved, and the operation and maintenance cost is saved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and specifically to a data processing method, device and equipment. Background Art

[0002] The data warehouse uses SQL query statements to query and calculate the data in the table. However, for some more complex business logic, a single query statement cannot be used to complete the operation. The workflow is designed to solve this problem. It divides more complex businesses into multiple sub-parts and automatically executes each sub-part through multiple query statements according to certain predetermined rules to process the business logic.

[0003] In related technologies, the query statements in the business flow may be repeated, resulting in a waste of server resources. At the same time, the design of the business flow cannot know data governance at the data warehouse level, which further leads to problems such as engineers not following the specifications, handling temporary emergency needs, and complex blood relationships during the development of the data warehouse, which gradually deviates from the specifications. With the construction of the data warehouse and the increase in the number of workflows, manual inspection will also become very complicated, difficult to maintain, and poorly operable. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device and equipment to solve the problem in the prior art that query statements may be repeated, resulting in waste of server resources.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0006] Parsing the query statements in the workflow to be processed to obtain keyword information of each workflow, wherein the keyword information includes a target table field, a query result field, a source table field, a query condition field, a grouping field, and a sorting field of the workflow;

[0007] Optimizing each of the workflows according to keyword information of each of the workflows;

[0008] The data warehouse is updated according to the optimization processing result of the workflow.

[0009] Optionally, the optimizing each workflow according to the keyword information of each workflow includes:

[0010] When at least two workflows are computing workflows of the same source, merging the at least two workflows;

[0011] When at least two workflows are homologous progressive computing workflows, merging the at least two workflows;

[0012] When at least two workflows are homologous and similar computing workflows, merging the at least two workflows;

[0013] When at least two workflows are homologous but different-dimensional associative computing workflows, the at least two workflows are split, and the dimension table is reduced in dimension through the splitting.

[0014] Optionally, when the at least two workflows are homologous computing workflows, merging the at least two workflows includes:

[0015] When the source table fields, query condition fields, grouping fields, and sorting fields of at least two workflows are the same, determining that the at least two workflows are the same source and same computing workflows, and merging the at least two workflows into a target workflow based on the target table fields of the at least two workflows;

[0016] The merged target workflow is used to execute the query calculations of the at least two workflows in parallel.

[0017] Optionally, when at least two workflows are homologous progressive computing workflows, merging the at least two workflows includes:

[0018] When the source table fields, query condition fields and sorting fields of at least two workflows are the same, and the grouping fields of at least two workflows are in a progressive relationship, the at least two workflows are determined to be homologous progressive calculation workflows, and the at least two workflows are merged into a target workflow based on the target table fields and grouping fields of the at least two workflows;

[0019] The merged target workflow is used to sequentially execute query calculations of the at least two workflows according to a progressive relationship.

[0020] Optionally, when at least two workflows are homologous and similar computing workflows, merging the at least two workflows includes:

[0021] When the source table fields, query condition fields and sorting fields of at least two workflows are the same, but the grouping fields are different and there is no progressive relationship, it is determined that the at least two workflows are homologous similar computing workflows, and a first intermediate table is added according to the grouping fields of the at least two workflows;

[0022] Merging the at least two workflows into a target workflow according to the first intermediate table and the target table fields of the at least two workflows;

[0023] The merged target workflow is used to sequentially execute query calculations of the first intermediate table and the at least two workflows according to the first intermediate table and the progressive relationship between the at least two workflows.

[0024] Optionally, when the at least two workflows are homologous but heterodimensional associated computing workflows, splitting the at least two workflows includes:

[0025] When the source table fields of at least two workflows are the same, and the grouping fields in the associated dimension tables are different, the at least two workflows are determined to be the same-source but different-dimensional associated calculation workflows, and the second intermediate table is determined by the grouping fields of the associated maintenance tables;

[0026] Splitting the at least two workflows into at least three target workflows according to the second intermediate table and the target table fields of the at least two workflows;

[0027] The split multiple target workflows are used to execute query calculations of the at least two workflows according to the second intermediate table.

[0028] Optionally, determining whether the keyword information is the same includes:

[0029] Split the string contents in multiple fields to be determined into multiple arrays;

[0030] Rearrange the split arrays according to the preset string content order;

[0031] Whether multiple fields to be determined are the same is determined by comparing whether the contents of the reordered strings are the same.

[0032] Optionally, after parsing the workflow to be processed to obtain keyword information of each workflow, the method further includes:

[0033] When a workflow includes multiple query statements, multiple keyword information corresponding to each query statement is obtained according to the multiple query statements;

[0034] Determining the blood relationship of each of the query statements according to the multiple keywords;

[0035] De-duplicate query statements with the same blood relationship.

[0036] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0037] A parsing module, which parses the query statements in the workflow to be processed to obtain keyword information of each workflow, wherein the keyword information includes a target table field, a query result field, a source table field, a query condition field, a grouping field, and a sorting field of the workflow;

[0038] An optimization module, which optimizes each of the workflows according to the keyword information of each of the workflows;

[0039] An updating module updates the data warehouse according to the optimization processing result of the workflow.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0041] at least one processor; and

[0042] at least one memory in communication with the processor, wherein:

[0043] The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method as described in any one of the first aspects.

[0044] In the embodiment of the present invention, keyword information is obtained by parsing the workflow, and multiple workflows whose keyword information meets the preset conditions are optimized, thereby reducing repeated calculations of workflows, saving computing resources and computing costs, improving the rationality of data processing tasks, improving maintainability, improving task robustness, and saving operation and maintenance costs. Workflow checks can also be performed in a timely manner for incremental workflows and modified workflows, so that problems can be discovered and solved in a timely manner, and data quality can be guaranteed in real time.

[0045] At the same time, the optimized workflow can also clarify the construction of data warehouses, improve the standardization of data warehouses, improve the rationality of metadata and blood relationships, and improve data governance capabilities and the maturity of enterprise data management capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 Shown is a flow chart of a data processing method provided in an embodiment of the present application;

[0048] Figure 2 Shown is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0049] Figure 3 Shown is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0050] Figure 4 Shown is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0051] Figure 5Shown is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0052] Figure 6 The figure is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0053] Figure 7 Shown is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0055] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0056] like Figure 1 FIG. 1 is a data processing method provided by an embodiment of the present invention. Figure 1 The specific steps of the method include:

[0057] S101, parsing the query statements in the workflow to be processed to obtain keyword information of each workflow.

[0058] Specifically, the query statement is parsed by separating the character string keywords in the query statement to obtain the keyword information of the workflow on the query statement.

[0059] Among them, the string keywords in the query statement usually include INSERT INTO, SELECT, FROM, WHERE, GROUP BY, and ORDER BY. When the string keywords in the query statement are identified, separators are added to the string keywords to complete the identification of the corresponding content of the string keywords, so as to parse the query statement into keyword information.

[0060] By separating the above string keywords, the keyword information of the workflow can be obtained, including: target table field 'WTO', query result field 'WSELECT', source table field 'WFROM', query condition field 'WWHERE', grouping field 'WGROUP' and sorting field 'WORDER'.

[0061] Optionally, in some embodiments, a workflow to be processed may include multiple query statements, and the multiple query statements may be repeated in function, resulting in repeated calculations.

[0062] In an embodiment of the present invention, duplicate query statements are deduplicated by identifying the blood relationship of each query statement. Specifically, when a workflow contains multiple query statements, after parsing the multiple query statements to obtain keyword information corresponding to each query statement, the blood relationship of each query statement is determined based on the source table field and the target table field in the keyword information. When there are at least two query statements with the same blood relationship, that is, the source table field and the target table field are the same, deduplication is performed on the at least two query statements with the same blood relationship.

[0063] S102: Optimize each workflow according to the keyword information of each workflow.

[0064] Specifically, the keyword information of the workflow to be processed is identified to determine whether the keyword information of multiple workflows meets the preset conditions. When it is determined that the keyword information of multiple workflows meets the preset conditions, the workflow types of the multiple workflows are determined according to the satisfied conditions, and the multiple workflows of the same workflow type are merged and / or split based on the corresponding preset strategy. Optimization processing.

[0065] In the embodiment of the present invention, the workflow types include: same-source identical calculation workflow, same-source progressive calculation workflow, same-source similar calculation workflow, and same-source different-dimensional associated calculation workflow.

[0066] Optionally, whether each workflow is a workflow of the same source and same calculation is determined by the source table field 'WFROM', the query condition field 'WWHERE', the grouping field 'WGROUP' and the sorting field 'WORDER' of each workflow. When at least two workflows are workflows of the same source and same calculation, the at least two workflows are merged.

[0067] When the source table fields, query condition fields, grouping fields, and sorting fields of at least two workflows are the same, the at least two workflows are determined to be workflows with the same source and the same calculation. Based on the target table fields of the at least two workflows, at least two workflows belonging to the same source and the same calculation workflows are merged into one target workflow. The merged target workflow is used to execute the query calculations of the at least two workflows with the same source and the same calculation in parallel.

[0068] like Figure 2 As shown, it is a specific embodiment of a same source and same computing workflow provided by an embodiment of the present invention. Figure 2, the source table field of workflow 1 is table B, and the grouping field is name. The source table field of workflow 2 is table B, and the grouping field is name. The source table fields, query condition fields, grouping fields, and sorting fields of workflow 1 and workflow 2 are the same, so workflow 1 and workflow 2 are determined to be the same source and same calculation workflows. Based on the target table field T1 of workflow 1 and the target table field T2 of workflow 2, workflow 1 and workflow 2 are merged into the target workflow. In the target workflow, the grouping field is still name. By querying the result field, table T1 and table T2 are used as the target table respectively, and query statistics are performed based on the grouping field name, and the target tables T1 and T2 are determined from the source table B.

[0069] Optionally, whether each workflow is a homogeneous progressive computing workflow is determined by the source table field 'WFROM', the query condition field 'WWHERE', the sort field 'WORDER' and the group field 'WGROUP' of each workflow. When at least two workflows are homogeneous progressive computing workflows, the at least two workflows are merged.

[0070] Among them, when the source table field 'WFROM', the query condition field 'WWHERE' and the sort field 'WORDER' of at least two workflows are the same, and the grouping field 'WGROUP' of at least two workflows has a progressive relationship, that is, the target table field of one workflow contains the target table field of another workflow, it is determined that the at least two workflows are homologous progressive calculation workflows, and the latter workflow is the progression of the former workflow. At least two workflows are merged into one target workflow based on the target table field and the grouping field of at least two workflows. The merged target workflow is used to execute the query calculation of at least two workflows in sequence according to the progressive relationship, that is, the workflow with a larger grouping field range is executed first, and the workflow with a smaller grouping field range is executed progressively on this basis.

[0071] like Figure 3 As shown in FIG. 1 , a specific embodiment of a homologous progressive computing workflow provided by an embodiment of the present invention is shown. Figure 3, the source table field of workflow 1 is table B, and the grouping fields are name and gender. The source table field of workflow 2 is table B, and the grouping field is name. The source table fields, query condition fields, and sorting fields of workflow 1 and workflow 2 are the same. The grouping field of workflow 1 contains the grouping field of workflow 2, so it is determined that workflow 1 and workflow 2 are the same source progressive calculation workflows, and the workflow is the progressive of workflow 1. Based on the target table field T1, grouping fields name and gender of workflow 1, and the target table field T2, grouping field name of workflow 2, workflow 1 and workflow 2 are merged into the target workflow. In the target workflow, first query and calculate the source table B based on the grouping fields name and gender of workflow 1 through the query result field, and determine the target table T1 from the source table B. Then query and calculate the target table T1 based on the grouping field name of workflow 2 through the query result field, and determine the target table T2 from the target table T1.

[0072] Optionally, whether each workflow is a homogeneous similar computing workflow is determined by the source table field 'WFROM', the query condition field 'WWHERE', and the sort field 'WORDER' of each workflow. When at least two workflows are homogeneous similar computing workflows, the at least two workflows are merged.

[0073] Among them, when the source table field 'WFROM', the query condition field 'WWHERE' and the sorting field 'WORDER' of at least two workflows are the same, the grouping fields are different and there is no progressive relationship, it is determined that the at least two workflows are similar calculation workflows with the same source. A first intermediate table is added based on the grouping fields of the at least two workflows, and the at least two workflows are merged into a target workflow according to the first intermediate table and the target table fields of the at least two workflows. The merged workflow is used to execute the query calculation of the first intermediate table according to the grouping fields of the at least two workflows, and then execute the query calculation of the at least two workflows in sequence according to the progressive relationship between the first intermediate table and the at least two workflows.

[0074] like Figure 4 FIG. 1 is a specific embodiment of a homologous similar computing workflow provided by an embodiment of the present invention. Figure 4, the source table field of workflow 1 is table B, and the grouping fields are name and city. The source table field of workflow 2 is table B, and the grouping fields are name and gender. The source table fields, query condition fields, and sorting fields of workflow 1 and workflow 2 are the same, but the grouping fields are different and there is no progressive relationship, so workflow 1 and workflow 2 are determined to be similar computing workflows from the same source. The first intermediate table B1 is determined based on the grouping fields name and city of workflow 1 and the grouping fields name and gender of workflow 2, and then workflow 1 and workflow 2 are merged into the target workflow through the first intermediate table B1. In the target workflow, the query calculation is performed from table B through the grouping relationship name, gender, and city to obtain the first intermediate table B1. Based on the first intermediate table B1, the query calculation is performed with name and city as the grouping fields to obtain the target table T1, and the query calculation is performed with name and gender as the grouping fields to obtain the target table T2.

[0075] Optionally, whether each workflow is a homogeneous but heterogeneous associated computing workflow is determined by the source table field 'WFROM' of each workflow and the associated dimension table. When at least two workflows are homogeneous but heterogeneous associated computing workflows, the at least two workflows are split.

[0076] When the source table field 'WFROM' of at least two workflows is the same, and the grouping fields in the dimension table associated with the source table field are different, the at least two workflows are determined to be the same-source but different-dimensional associated calculation workflows. A second intermediate table is determined based on the grouping fields in the dimension table, and the at least two workflows are split into at least three target workflows through the second intermediate table. The at least three target workflows after the splitting include a target workflow for reducing the dimension table through the second intermediate table, and at least two target workflows corresponding to the at least two workflows and performing query calculations based on the intermediate table, so as to perform query calculations of the at least two workflows.

[0077] like Figure 5 FIG. 1 is a specific embodiment of a workflow of homologous heterogeneous association calculation provided by an embodiment of the present invention. Figure 5 , the source table field of workflow 1 is table B, and the grouping field in the associated dimension table W is city, which is used to associate the city with the first seven digits of the phone number stored in table B. The source table field of workflow 2 is table B, and the grouping field in the associated dimension table W is province, which is used to associate the province with the first seven digits of the phone number stored in table B. The source table fields of workflow 1 and workflow 2 are the same, but the grouping fields in the associated dimension table are different, so workflow 1 and workflow 2 are determined to be the same source but different dimension association calculation workflows, and the second intermediate table is determined by the grouping fields city and province in the dimension table. According to the second intermediate table, workflow 1 and workflow 2 are split into target workflow 0, target workflow 1, and target workflow 2.

[0078] In target workflow 0, the second intermediate table B1 is determined by the grouping fields city and province in dimension W associated with table B. In target workflow 1, the second intermediate table is queried and calculated by the grouping field city to obtain target table T1. In target workflow 2, the second intermediate table is queried and calculated by the grouping field province to obtain target table T2.

[0079] S103: Update the data warehouse according to the optimization processing result of the workflow.

[0080] Specifically, according to the optimization processing result of the workflow, a first intermediate table, a second intermediate table, etc. are generated in the data warehouse to implement the update of the data warehouse.

[0081] In the embodiment of the present invention, keyword information is obtained by parsing the workflow, and multiple workflows whose keyword information meets the preset conditions are optimized, thereby reducing repeated calculations of workflows, saving computing resources and computing costs, improving the rationality of data processing tasks, improving maintainability, improving task robustness, and saving operation and maintenance costs. Workflow checks can also be performed in a timely manner for incremental workflows and modified workflows, so that problems can be discovered and solved in a timely manner, and data quality can be guaranteed in real time.

[0082] At the same time, the optimized workflow can also clarify the construction of data warehouses, improve the standardization of data warehouses, improve the rationality of metadata and blood relationships, and improve data governance capabilities and the maturity of enterprise data management capabilities.

[0083] Optionally, in some embodiments, when executing S102, it is necessary to determine whether the string contents in each to-be-determined field of each workflow are consistent, for example, to determine whether the string contents in the source table field 'WFROM', the query condition field 'WWHERE', the grouping field 'WGROUP', and the sorting field 'WORDER' of each workflow are consistent, so as to determine whether each workflow meets the preset conditions. When determining whether the string contents of each field of each workflow are consistent, it is very likely that a recognition error caused by inconsistent recognition string contents may occur due to different sorting of string contents.

[0084] In an embodiment of the present invention, after the workflow is parsed, the string contents in the field need to be reordered. Specifically, the string contents are separated by a separator such as a comma and split into multiple arrays. The split string contents are reordered according to a preset string content order, and the reordered string contents are compared to determine whether they are the same, so as to determine whether multiple to-be-determined fields of multiple workflows are the same.

[0085] The preset string content order may be string ascending order or string descending order, etc.

[0086] Corresponding to the above data processing method, the present application embodiment also provides a data processing device. Figure 3 , is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device may include: a parsing module 601, an optimization module 602, and an updating module 603.

[0087] The parsing module 601 parses the query statements in the workflow to be processed to obtain keyword information of each workflow, wherein the keyword information includes the target table field, query result field, source table field, query condition field, grouping field and sorting field of the workflow.

[0088] The optimization module 602 optimizes each workflow according to the keyword information of each workflow.

[0089] The updating module 603 updates the data warehouse according to the optimization processing result of the workflow.

[0090] Figure 7 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present specification. Figure 7 As shown, the electronic device may include at least one processor; and at least one memory communicatively connected to the processing unit, wherein the memory stores program instructions that can be executed by the processing unit, and the processor calls the program instructions to execute the data processing method provided in this embodiment.

[0091] The electronic device can be a device capable of intelligently communicating with a user, such as a cloud server, and the specific form of the electronic device is not limited in the embodiments of this specification. It can be understood that the electronic device here is the machine mentioned in the method embodiment.

[0092] Figure 7 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present description is shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of this specification.

[0093] like Figure 7 As shown, the electronic device is in the form of a general computing device. The components of the electronic device may include, but are not limited to: one or more processors 710, a communication interface 720, a memory 730, and a communication bus 740 connecting different system components (including the memory 730, the communication interface 720 and the processor 710).

[0094] The communication bus 740 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0095] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0096] The memory 730 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 730 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of each embodiment of the present specification.

[0097] A program / utility having a set (at least one) of program modules may be stored in memory 730, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described in this specification.

[0098] The processor 710 executes various functional applications and data processing by running the programs stored in the memory 730, such as implementing the data processing method provided in the embodiments shown in this specification.

[0099] An embodiment of the present specification provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the data processing method provided by the embodiment shown in the present specification.

[0100] The above-mentioned non-transitory computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (Read Only Memory; hereinafter referred to as: ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory; hereinafter referred to as: EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0101] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0102] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0103] Computer program code for performing the operations of this specification may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this specification, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0106] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of this specification includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of this specification belong.

[0107] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0108] It should be noted that the terminals involved in the embodiments of this specification may include but are not limited to personal computers (Personal Computer; hereinafter referred to as: PC), personal digital assistants (Personal Digital Assistant; hereinafter referred to as: PDA), wireless handheld devices, tablet computers (Tablet Computer), mobile phones, MP3 players, MP4 players, etc.

[0109] In the embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0111] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of this specification.

[0112] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A data processing method, characterized in that: The method comprises: Parsing the query statements in the workflow to be processed to obtain keyword information of each workflow, wherein the keyword information includes a target table field, a query result field, a source table field, a query condition field, a grouping field, and a sorting field of the workflow; Optimizing each of the workflows according to keyword information of each of the workflows; The data warehouse is updated according to the optimization processing result of the workflow.

2. The method according to claim 1, characterized in that The optimizing process of each workflow according to the keyword information of each workflow includes: When at least two workflows are computing workflows of the same source, merging the at least two workflows; When at least two workflows are homologous progressive computing workflows, merging the at least two workflows; When at least two workflows are homologous and similar computing workflows, merging the at least two workflows; When at least two workflows are homologous but different-dimensional associated computing workflows, the at least two workflows are split, and the dimension table is reduced in dimension through the splitting.

3. The method according to claim 2, characterized in that When the at least two workflows are the same source and the same computing workflows, merging the at least two workflows comprises: When the source table fields, query condition fields, grouping fields, and sorting fields of at least two workflows are the same, determining that the at least two workflows are the same source and same computing workflows, and merging the at least two workflows into a target workflow based on the target table fields of the at least two workflows; The merged target workflow is used to execute the query calculations of the at least two workflows in parallel.

4. The method according to claim 2, characterized in that: When at least two workflows are homologous progressive computing workflows, merging the at least two workflows comprises: When the source table fields, query condition fields and sorting fields of at least two workflows are the same, and the grouping fields of at least two workflows are in a progressive relationship, the at least two workflows are determined to be homologous progressive calculation workflows, and the at least two workflows are merged into a target workflow based on the target table fields and grouping fields of the at least two workflows; The merged target workflow is used to sequentially execute query calculations of the at least two workflows according to a progressive relationship.

5. The method according to claim 2, characterized in that: When at least two workflows are homologous and similar computing workflows, merging the at least two workflows includes: When the source table fields, query condition fields and sorting fields of at least two workflows are the same, but the grouping fields are different and there is no progressive relationship, it is determined that the at least two workflows are homologous similar computing workflows, and a first intermediate table is added according to the grouping fields of the at least two workflows; Merging the at least two workflows into a target workflow according to the first intermediate table and the target table fields of the at least two workflows; The merged target workflow is used to sequentially execute query calculations of the first intermediate table and the at least two workflows according to the first intermediate table and the progressive relationship between the at least two workflows.

6. The method according to claim 2, characterized in that When the at least two workflows are homologous but different-dimensional associated computing workflows, splitting the at least two workflows comprises: When the source table fields of at least two workflows are the same, and the grouping fields in the associated dimension tables are different, the at least two workflows are determined to be the same-source but different-dimensional associated calculation workflows, and the second intermediate table is determined by the grouping fields of the associated maintenance tables; Splitting the at least two workflows into at least three target workflows according to the second intermediate table and the target table fields of the at least two workflows; The split multiple target workflows are used to execute query calculations of the at least two workflows according to the second intermediate table.

7. The method according to any one of claims 3 to 6, determining whether the keyword information is the same comprises: Split the string contents in multiple fields to be determined into multiple arrays; Rearrange the split arrays according to the preset string content order; Whether multiple fields to be determined are the same is determined by comparing whether the contents of the reordered strings are the same.

8. The method according to claim 1, characterized in that: After parsing the workflow to be processed to obtain keyword information of each workflow, the method further includes: When a workflow includes multiple query statements, multiple keyword information corresponding to each query statement is obtained according to the multiple query statements; Determining the blood relationship of each of the query statements according to the multiple keywords; De-duplicate query statements with the same blood relationship.

9. A data processing device, characterized in that: The device comprises: A parsing module, which parses the query statements in the workflow to be processed to obtain keyword information of each workflow, wherein the keyword information includes a target table field, a query result field, a source table field, a query condition field, a grouping field, and a sorting field of the workflow; An optimization module, which optimizes each of the workflows according to the keyword information of each of the workflows; An updating module updates the data warehouse according to the optimization processing result of the workflow.

10. An electronic device, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.

Images