Method and device for processing data in data warehouse

Abstract

The present application discloses a method and device for processing data in a data warehouse, comprising: adding the name and type of the newly added field in a corresponding partition table according to information of a newly added field of a first business in a field addition request; obtaining a first business code corresponding to an original field of the first business in a first partition under the partition table, generating a second business code according to the first business code and field information recorded in the partition table and writing the code into the first partition, wherein the second business code includes the original field of the first business and an empty string for representing the newly added field; receiving a third business code corresponding to the newly added field uploaded by a user, generating a fourth business code according to the third business code and field information recorded in the partition table, creating a second partition under the partition table and writing the fourth business code, wherein the fourth business code includes the newly added field of the first business and an empty string for representing the original field; and summarizing the second business code and the fourth business code to obtain a complete business code for the first business.

Description

Technical Field

[0001] The present application belongs to the technical field of big data processing, and specifically relates to a method and device for processing data in a data warehouse. Background Art

[0002] A data warehouse is a strategic collection of all types of data support for decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes, and can provide services such as guiding business process improvements, monitoring time, cost, quality, and control for enterprises. According to the process of data inflow and outflow, the data warehouse architecture can be roughly divided into three layers: ODS (Operation Data Store) layer, DW (DataWarehouse) layer, and DA (Data Application) layer.

[0003] At present, in the offline processing technology of big data warehouse, the design of the DW layer is positioned for detailed summary. When there is a new business, the existing technology continuously adds business logic to a code and adds it to the original business code of the DW layer.

[0004] However, as the business volume grows, the number of fields increases, and the actual processing code gradually accumulates to thousands to tens of thousands of lines, and is constantly added. In addition, more and more fields need to be compatible. At the same time, when adding fields, the order of data insertion must be considered, and the caliber is becoming increasingly difficult to trace, resulting in high maintenance costs for the data warehouse. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for processing data in a data warehouse, which can solve the problem of high maintenance cost of the data warehouse existing in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for processing data in a data warehouse, the method comprising:

[0007] When receiving a request for adding a business field for a first business in the data warehouse, adding a name and a type of the newly added field in a partition table of the data warehouse according to information of the newly added field carried in the request for adding the field;

[0008] Obtaining a first service code in a first partition under the partition table, generating a second service code according to the first service code and field information of the first service recorded in the partition table, and writing the second service code into the first partition, wherein the first service code is a service code corresponding to an original field of the first service, and the second service code includes the original field of the first service and an empty string used to represent a newly added field;

[0009] Receive a third service code corresponding to a newly added field of the first service uploaded by a user, generate a fourth service code according to the third service code and field information of the first service recorded in the partition table, create a second partition under the partition table, and write the fourth service code into the second partition, wherein the fourth service code includes the newly added field of the first service and an empty string used to represent the original field;

[0010] The second service code in the first partition and the fourth service code in the second partition are aggregated to obtain a complete service code of the first service after the newly added field.

[0011] In a second aspect, an embodiment of the present application provides a device for processing data in a data warehouse, the device comprising:

[0012] A modification module, configured to, upon receiving a request for adding a business field for a first business in a data warehouse, add a name and a type of the newly added field in a partition table of the data warehouse according to information of the newly added field carried in the request for adding the field;

[0013] a first processing module, configured to obtain a first service code in a first partition under the partition table, generate a second service code according to the first service code and field information of the first service recorded in the partition table, and write the second service code into the first partition, wherein the first service code is a service code corresponding to an original field of the first service, and the second service code includes the original field of the first service and an empty string used to represent a newly added field;

[0014] a second processing module, configured to receive a third service code corresponding to a newly added field of the first service uploaded by a user, generate a fourth service code according to the third service code and field information of the first service recorded in the partition table, create a second partition under the partition table, and write the fourth service code into the second partition, wherein the fourth service code includes the newly added field of the first service and an empty string used to represent the original field;

[0015] The summarizing module is used to summarize the second service code in the first partition and the fourth service code in the second partition to obtain the complete service code of the first service after the newly added field.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0020] In an embodiment of the present application, when a request for adding a business field for a first business in a data warehouse is received, the table structure of the corresponding partition table is changed; based on the changed partition table, for the first business, the first business code of the original field is automatically rewritten as a second business code containing the original field and an empty string for representing the newly added field, and the third business code of the newly added field written by the user is automatically rewritten as a fourth business code containing the newly added field and an empty string for representing the original field, and stored in different partitions, the second business code and the fourth business code in different partitions are aggregated to obtain the complete business code of the first business after the newly added field is added.

[0021] Compared with the prior art, in the embodiment of the present application, when a business field needs to be added, the user only needs to write the corresponding business code for the newly added field and submit it, without considering field compatibility, insertion order and traceability. The data warehouse can automatically generate all the business codes for the first business based on the business code written by the user, thereby reducing the maintenance cost of the data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an example diagram of the process of adding a new business field in the prior art;

[0023] Figure 2It is a flow chart of a method for processing data in a data warehouse provided by an embodiment of the present application;

[0024] Figure 3 This is the first example diagram of the process of adding a new service field provided in the embodiment of the present application;

[0025] Figure 4 This is a second example diagram of a process for adding a new service field provided in an embodiment of the present application;

[0026] Figure 5 It is a structural block diagram of a data processing device in a data warehouse provided by an embodiment of the present application;

[0027] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of the hardware structure of an electronic device implementing various embodiments of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0033] At present, in the offline processing technology of big data warehouse, the design of the DW layer is positioned at detailed summary. The existing solutions for new business are to continuously add business logic to a piece of code and add it to the existing maintenance code.

[0034] by Figure 1 For example, the data warehouse includes the ODS layer, the DW layer and the DA layer. The partition table of the DW layer contains N partitions, and each partition stores the business code of a business. For example, partition 1 stores the business code of business 1, partition 2 stores the business code of business 2, partition 3 stores the business code of business 3, and so on. Partition N stores the business code of business N.

[0035] When business 1 needs to add M new business fields, in the prior art, the user needs to insert the business codes of the M newly added fields into the business code of partition 1. In addition, it is also necessary to insert empty characters of the same length into partitions 2 to N to ensure that the number of characters inserted into the N partitions is the same, where N and M are both positive integers.

[0036] However, as the business increment increases, the number of fields increases, and the actual processing code gradually accumulates to thousands to tens of thousands of lines, and is constantly added. In addition, more and more fields need to be compatible. At the same time, when adding fields, the order of data insertion must be considered, and the caliber is becoming increasingly difficult to trace. For maintenance personnel, the cost is rising exponentially. It can be seen that the existing solution has defects in business compatibility and maintainability.

[0037] The embodiments of the present application provide a method and device for processing data in a data warehouse to solve the technical problems of coping with additional business logic and high maintenance costs.

[0038] The following is a detailed description of the method for processing data in a data warehouse provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0039] Figure 2 is a flow chart of a method for processing data in a data warehouse provided by an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps: step 201, step 202, step 203 and step 204, wherein:

[0040] In step 201, when a business field addition request for a first business in a data warehouse is received, the name and type of the new field are added to a partition table of the data warehouse according to the information of the new field carried in the field addition request.

[0041] In the embodiment of the present application, the relationship between the partition table and the partition is similar to that between a directory and a subdirectory. A target may include multiple subdirectories. Similarly, a partition table may include multiple partitions.

[0042] In the prior art, the data warehouse includes: ODS layer, DW layer and DA layer, wherein the DW layer includes a partition table, which includes multiple partitions, and each partition stores the business code of the business. When a new business field needs to be added, it is mainly processed on the DW layer.

[0043] In the embodiment of the present application, two scenarios may be included: In the first scenario, the data warehouse includes: ODS layer, DW layer and DA layer, wherein the DW layer includes a partition table, and the partition table includes multiple partitions. When it is necessary to add a business field, the DW layer is processed and steps 201 to 204 are executed.

[0044] In the second scenario, the data warehouse includes: ODS layer, middle layer, DW layer and DA layer, among which the middle layer is a newly added structural layer. The content contained in the middle layer is the same as that in the DW layer, and also includes a partition table. The partition table includes multiple partitions. When it is necessary to add a business field, the middle layer is first processed, and steps 201 to 203 are executed. Then, the DW layer is processed and step 204 is executed.

[0045] In the prior art, when a business in a data warehouse needs to add a new field, it is necessary to change the table structure of the partition table of the DW layer, specifically, to add the name and type of the new field in the partition table. Similarly, in the embodiment of the present application, when a business in a data warehouse needs to add a new field, it is also necessary to change the table structure of the partition table.

[0046] For the first scenario, when the business in the data warehouse needs to add a new field, the table structure of the partition table of the DW layer needs to be changed. For the second scenario, when the business in the data warehouse needs to add a new field, the table structure of the partition table of the middle layer needs to be changed.

[0047] In step 202, the first business code in the first partition under the partition table is obtained, and a second business code is generated according to the first business code and the field information of the first business recorded in the partition table, and the second business code is written into the first partition, wherein the first business code is the business code corresponding to the original field of the first business, and the second business code includes the original field of the first business and an empty string used to represent the newly added field.

[0048] In the embodiment of the present application, the first partition is the partition where the business code of the first business is located, and the number of first partitions can be 1 or more.

[0049] In the embodiment of the present application, the first service code includes original fields of the first service, and the second service code includes original fields of the first service and an empty string used to represent a newly added field.

[0050] In the embodiment of the present application, in order to reduce data redundancy, after the second service code is generated, the first service code in the first partition may be deleted first, and then the second service code may be written into the first partition.

[0051] In the embodiment of the present application, a field auto-fill tool can be used to convert the first business code into a second business code and write it into the first partition. Specifically, the first business code is uploaded to the field auto-fill tool through a specific parameter transmission method, and the second business code can be generated after being processed by the field auto-fill tool. The specific parameter transmission method can be implemented by a parameter transmission method in the prior art, which is not limited here.

[0052] In the embodiment of the present application, for the input code, the field automatic filling tool can achieve the following functions: automatically fill in the fields that have not been queried and inserted, and can write queries and inserts out of order, and can support dynamic partitioning.

[0053] In an embodiment of the present application, the specific processing process of the field automatic filling tool may include the following steps: parsing the field name and type of the first business in the partition table, and generating a sixth business code based on the parsed field name and type, wherein the data in the sixth business code except for the numeric type and the structure type is an empty string; parsing the information of the original field of the first business in the first business code, inserting the information of the original field into the sixth business code, and obtaining the second business code.

[0054] In step 203, the third business code corresponding to the newly added field of the first business uploaded by the user is received, and a fourth business code is generated according to the third business code and the field information of the first business recorded in the partition table. A second partition is created under the partition table, and the fourth business code is written into the second partition, wherein the fourth business code includes the newly added field of the first business and an empty string used to represent the original field.

[0055] In the embodiment of the present application, the third business code is an SQL code written by the user based on the newly added fields of the business.

[0056] In an embodiment of the present application, the business code of the newly added field and the business code of the original field are stored in different partitions, so that when a new field is needed, the user only needs to pay attention to the newly added field, write a machine-recognizable SQL code based on the newly added field, and submit it. The machine can automatically implement the addition of business code based on the code submitted by the user, without the user having to pay attention to the original field or consider field compatibility.

[0057] In the embodiment of the present application, the third service code includes a newly added field of the first service, and the fourth service code includes a newly added field of the first service and an empty string used to represent the original field.

[0058] In the prior art, when a business in a data warehouse needs to add a new field, the business code of the new field needs to be inserted into the business code of the original field in the DW layer. When inserting, it needs to be inserted sequentially. When there are many fields, the insertion process is time-consuming because the insertion order needs to be considered.

[0059] In order to solve the above problems, in the embodiment of the present application, when writing the second business code, the user can write it according to certain rules. Specifically, the third business code is an SQL code written according to the preset code writing rules, wherein the preset code writing rules are: each newly added field is assigned to the corresponding field in the partition table through as. That is, the insertion principle of hive is reversed from field sequential insertion to field specified insertion, so data processing logic can be written regardless of any sorting, and can be written in a dynamic partitioning manner.

[0060] Accordingly, in an implementation manner provided in the present application, the above step 203 may include the following steps:

[0061] Parse the names and types of all fields of the first business in the partition table, and generate a fifth business code based on the names and types of the parsed fields, wherein data in the fifth business code other than numeric types and structure types is an empty string; parse the content between the first select and the first from in the third business code to obtain information of the newly added fields; insert the information of the newly added fields into the fifth business code according to the information indicated by as in the third business code to obtain a fourth business code.

[0062] In the embodiment of the present application, the field automatic filling tool can also be used to convert the third business code into the fourth business code and write it into the second partition. Specifically, the third business code is uploaded to the field automatic filling tool, and the fourth business code can be generated after being processed by the field automatic filling tool.

[0063] In the embodiment of the present application, for the first scenario, the first partition and the second partition are both in the DW layer. For the second scenario, the first partition and the second partition are both in the middle layer.

[0064] In step 204, the second service code in the first partition and the fourth service code in the second partition are summarized to obtain a complete service code of the first service after the newly added field.

[0065] In the embodiment of the present application, when summarizing, a GROUP BY operation can be performed on the second business code and the fourth business code to obtain the complete business code of the first business after the newly added field, so as to replace the multi-result layer join to reduce shuffle. Specifically, the above step 204 can include the following steps: by using the GROUP BY function, delete the empty string in the second business code and the fourth business code, merge the remaining content in the second business code and the fourth business code, and obtain the complete business code of the first business after the newly added field.

[0066] With respect to the first situation mentioned in step 201, in an implementation manner provided in the present application, the above step 204 may specifically include the following steps:

[0067] Create a third partition under the partition table;

[0068] In the third partition, the second service code in the first partition and the fourth service code in the second partition are summarized to obtain a complete service code of the first service after the newly added field.

[0069] In an example, Figure 3 As shown, the data warehouse includes the ODS layer, the DW layer (also called the middle layer, that is, the middle layer is the DW layer) and the DA layer. The partition table of the DW layer contains N partitions, and each partition stores the business code of a business. For example, partition 1 stores the business code of business 1, partition 2 stores the business code of business 2, partition 3 stores the business code of business 3, and so on, partition N stores the business code of business N.

[0070] When business 1 needs to add M business fields, the user needs to write the corresponding SQL code based on the M business fields, that is, the third business code. Then the user uploads the written SQL code to the field automatic filling tool mentioned in step 203. The field automatic filling tool processes the SQL code into the fourth business code, creates partition N+1, that is, the second partition, and writes the fourth business code into partition N+1.

[0071] Similarly, since the business code of business 1 is stored in partition 1, partition 1 is the first partition, and the business code in partition 1, that is, the first business code, is read, and then the first business code is uploaded to the field automatic filling tool mentioned in step 202. The field automatic filling tool processes the first business code into the second business code, deletes the first business code in partition 1, and then writes the second business code into partition 1;

[0072] Finally, a TYPE partition, that is, the third partition, is created, and the second business code and the fourth business code are summarized in the TYPE partition to obtain a complete business code of the first business after the newly added field, where N and M are both positive integers.

[0073] It can be seen that in the embodiment of the present application, a new partition is created under the partition table to store the business code of the newly added field. For the partition where the business without the newly added field is located (for example Figure 3 Partitions 2 to N in the partition do not need to be padded with characters, which can save storage space.

[0074] For the second situation mentioned in step 201, according to the process of data inflow and outflow, the data warehouse includes: an operational data storage layer, an intermediate layer, a data warehouse layer and a data application layer, and the partition table is stored in the intermediate layer. In another embodiment provided by the present application, the above step 204 may specifically include the following steps:

[0075] Sending the second business code in the first partition of the intermediate layer and the fourth business code in the second partition to the data warehouse layer;

[0076] In the data warehouse layer, the second business code in the first partition and the fourth business code in the second partition are summarized to obtain a complete business code of the first business after the newly added field.

[0077] In an example, Figure 4 As shown, the data warehouse includes an ODS layer, a middle layer, a DW layer (the middle layer and the DW layer are two layers) and a DA layer. The partition table of the middle layer contains N partitions, and each partition stores the business code of a business. For example, partition 1 stores the business code of business 1, partition 2 stores the business code of business 2, partition 3 stores the business code of business 3, and so on, partition N stores the business code of business N.

[0078] When business 1 needs to add M business fields, the user needs to write the corresponding SQL code based on the M business fields, that is, the third business code. Then the user uploads the written SQL code to the field automatic filling tool mentioned in step 203. The field automatic filling tool processes the SQL code into the fourth business code, creates partition N+1, that is, the second partition, and writes the fourth business code into partition N+1.

[0079] Similarly, since the business code of business 1 is stored in partition 1, partition 1 is the first partition, and the business code in partition 1, that is, the first business code, is read, and then the first business code is uploaded to the field automatic filling tool mentioned in step 202. The field automatic filling tool processes the first business code into the second business code, deletes the first business code in partition 1, and then writes the second business code into partition 1;

[0080] Finally, the second service code and the fourth service code are summarized in the DW layer to obtain a complete service code of the first service after the newly added field, where N and M are both positive integers.

[0081] It can be seen that in the embodiment of the present application, a new partition is created under the partition table of the middle layer to store the business code of the newly added field. For the partition where the business without the newly added field is located (for example Figure 4 Partitions 2 to N in the partition do not need to be padded with characters, which can save storage space.

[0082] For example, the browser business needs to add 20 fields, and the partition table has 150 fields for the browser business. When writing SQL code, the user only needs to write the code for these 20 fields, and the code volume can be reduced to about 300 lines.

[0083] In the embodiment of the present application, the first partition and the second partition further include a Type field, wherein the Type field is used to indicate the type of data accessed in the partition in which it is located. That is, in addition to the commonly used day and hour partition fields, a Type field can be added to the partition of the partition table to meet time requirements, for example, as shown in Table 1.

[0084] name type describe day string Date Partition hour string Hourly partitions Type string Data type, 0 is summary data

[0085] Table 1

[0086] For scenarios that require output timeliness, since the processing time and output time of each feature are different, in an embodiment of the present application, the current processing task result can be written into the corresponding Type field to meet timeliness.

[0087] For example, for the feature location, if you need to pay attention to its output timeliness, you can write it as follows: day = 2021-10-16 / hour = _ HIVE_DEFAULT_PARTITION_ / Type = location

[0088] For common scenarios such as feature mining and quality analysis, the overall spliced ​​data features are generally required. In the embodiment of the present application, a certain Type field can be specified as summary to provide summary data.

[0089] For example, when summarization is required, it can be written as follows:

[0090] day=2021-10-16 / hour=_HIVE_DEFAULT_PARTITION_ / Type=0.

[0091] It can be seen that in the embodiment of the present application, the caliber is disassembled into tasks for easy analysis, and there is no need to pay attention to other fields when adding features. The ideas are concentrated, and the tedious field supplement process is omitted. When adding fields, all tasks except summary do not need to be changed, which solves the problems of difficult operation and maintenance of caliber, poor compatibility and scalability as a whole.

[0092] It can be seen from the above embodiment that in this embodiment, when a request for adding a business field for the first business in the data warehouse is received, the table structure of the corresponding partition table is changed; on the basis of the changed partition table, for the first business, the first business code of the original field is automatically rewritten as a second business code containing the original field and an empty string for representing the newly added field, and the third business code of the newly added field written by the user is automatically rewritten as a fourth business code containing the newly added field and an empty string for representing the original field, and stored in different partitions, the second business code and the fourth business code in different partitions are aggregated to obtain the complete business code of the first business after the newly added field is added.

[0093] Compared with the prior art, in the embodiment of the present application, when a business field needs to be added, the user only needs to write the corresponding business code for the newly added field and submit it, without considering field compatibility, insertion order and traceability. The data warehouse can automatically generate all the business codes for the first business based on the business code written by the user, thereby reducing the maintenance cost of the data warehouse.

[0094] The method for processing data in a data warehouse provided in the embodiment of the present application can be executed by a data processing device in the data warehouse. The method for processing data in a data warehouse is executed by a data processing device in the data warehouse as an example to illustrate the data processing device in the data warehouse provided in the embodiment of the present application.

[0095] Figure 5 is a structural block diagram of a data processing device in a data warehouse provided by an embodiment of the present application, such as Figure 5 As shown, the data processing device 500 in the data warehouse may include: a modification module 501, a first processing module 502, a second processing module 503 and a summary module 504, wherein:

[0096] The modification module 501 is used to, when receiving a business field addition request for a first business in the data warehouse, add the name and type of the newly added field in the partition table of the data warehouse according to the information of the newly added field carried in the field addition request;

[0097] The first processing module 502 is used to obtain a first service code in the first partition under the partition table, generate a second service code according to the first service code and field information of the first service recorded in the partition table, and write the second service code into the first partition, wherein the first service code is a service code corresponding to an original field of the first service, and the second service code includes the original field of the first service and an empty string used to represent a newly added field;

[0098] The second processing module 503 is used to receive the third service code corresponding to the newly added field of the first service uploaded by the user, generate a fourth service code according to the third service code and the field information of the first service recorded in the partition table, create a second partition under the partition table, and write the fourth service code into the second partition, wherein the fourth service code includes the newly added field of the first service and an empty string used to represent the original field;

[0099] The summarizing module 504 is used to summarize the second service code in the first partition and the fourth service code in the second partition to obtain a complete service code of the first service after the newly added field.

[0100] It can be seen from the above embodiment that in this embodiment, when a request for adding a business field for the first business in the data warehouse is received, the table structure of the corresponding partition table is changed; on the basis of the changed partition table, for the first business, the first business code of the original field is automatically rewritten as a second business code containing the original field and an empty string for representing the newly added field, and the third business code of the newly added field written by the user is automatically rewritten as a fourth business code containing the newly added field and an empty string for representing the original field, and stored in different partitions, the second business code and the fourth business code in different partitions are aggregated to obtain the complete business code of the first business after the newly added field is added.

[0101] Compared with the prior art, in the embodiment of the present application, when a business field needs to be added, the user only needs to write the corresponding business code for the newly added field and submit it, without considering field compatibility, insertion order and traceability. The data warehouse can automatically generate all the business codes for the first business based on the business code written by the user, thereby reducing the maintenance cost of the data warehouse.

[0102] Optionally, as an embodiment, the summarizing module 504 may include:

[0103] Creating a submodule, used to create a third partition under the partition table;

[0104] The first summarizing submodule is used to summarize the second service code in the first partition and the fourth service code in the second partition in the third partition to obtain a complete service code of the first service after the newly added field.

[0105] Optionally, as an embodiment, according to the process of data inflow and outflow, the data warehouse includes: an operational data storage layer, an intermediate layer, a data warehouse layer and a data application layer, and the partition table is stored in the intermediate layer;

[0106] The aggregation module 504 may include:

[0107] A sending submodule, used for sending the second business code in the first partition and the fourth business code in the second partition of the intermediate layer to the data warehouse layer;

[0108] The second summary submodule is used to summarize the second business code in the first partition and the fourth business code in the second partition in the data warehouse layer to obtain the complete business code of the first business after the newly added field.

[0109] Optionally, as an embodiment, the summarizing module 504 may include:

[0110] The third summary submodule is used to delete the empty strings in the second business code and the fourth business code through the GROUP BY function, merge the remaining content in the second business code and the fourth business code, and obtain the complete business code of the first business after the newly added field.

[0111] Optionally, as an embodiment, the third business code is an SQL code written according to a preset code writing rule, wherein the preset code writing rule is: each newly added field is assigned to a corresponding field in the partition table through as;

[0112] The second processing module 503 may include:

[0113] A first parsing submodule, configured to parse the names and types of all fields of the first business in the partition table, and generate a fifth business code based on the names and types of the parsed fields, wherein data other than a numeric type and a structure type in the fifth business code is an empty string;

[0114] The second parsing submodule is used to parse the content between the first select and the first from in the third business code to obtain the information of the newly added field;

[0115] The insertion submodule is used to insert the information of the newly added field into the fifth business code according to the information indicated by as in the third business code to obtain the fourth business code.

[0116] Optionally, as an embodiment, the first partition and the second partition further include a Type field, wherein the Type field is used to indicate the type of data accessed in the partition where the data is located.

[0117] The processing device of data in the data warehouse in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0118] The device for processing data in the data warehouse in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0119] The data processing device in the data warehouse provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0120] Alternatively, if Figure 6 As shown, an embodiment of the present application further provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores programs or instructions that can be executed on the processor 601, and when the program or instructions are executed by the processor 601, the various steps of the embodiment of the method for processing data in the above-mentioned data warehouse are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0121] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0122] Figure 7 It is a schematic diagram of the hardware structure of an electronic device implementing various embodiments of the present application.

[0123] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0124] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0125] The processor 710 is configured to, upon receiving a request for adding a business field for a first business in a data warehouse, add a name and type of the newly added field in a partition table of the data warehouse according to information of the newly added field carried in the request for adding the field; obtain a first business code in a first partition under the partition table, generate a second business code according to the first business code and field information of the first business recorded in the partition table, and write the second business code into the first partition, wherein the first business code is a business code corresponding to an original field of the first business, and the second business code includes the original field of the first business and an empty string for representing the newly added field; receive a third business code corresponding to the newly added field of the first business uploaded by a user, generate a fourth business code according to the third business code and field information of the first business recorded in the partition table, create a second partition under the partition table, and write the fourth business code into the second partition, wherein the fourth business code includes the newly added field of the first business and an empty string for representing the original field; and summarize the second business code in the first partition and the fourth business code in the second partition to obtain a complete business code of the first business after the newly added field is added.

[0126] It can be seen that in the embodiment of the present application, when a business field needs to be added, the user only needs to write the corresponding business code for the newly added field and submit it, without considering field compatibility, insertion order and traceability. The data warehouse can automatically generate all the business codes for the first business based on the business code written by the user, thereby reducing the maintenance cost of the data warehouse.

[0127] Optionally, as an embodiment, the processor 710 is also used to create a third partition under the partition table; in the third partition, the second business code in the first partition and the fourth business code in the second partition are aggregated to obtain the complete business code of the first business after the newly added field.

[0128] Optionally, as an embodiment, according to the process of data inflow and outflow, the data warehouse includes: an operational data storage layer, an intermediate layer, a data warehouse layer and a data application layer, and the partition table is stored in the intermediate layer;

[0129] Processor 710 is also used to send the second business code in the first partition of the intermediate layer and the fourth business code in the second partition to the data warehouse layer; in the data warehouse layer, the second business code in the first partition and the fourth business code in the second partition are summarized to obtain the complete business code of the first business after the newly added field.

[0130] Optionally, as an embodiment, the processor 710 is also used to delete the empty strings in the second business code and the fourth business code through the GROUP BY function, merge the remaining content in the second business code and the fourth business code, and obtain the complete business code of the first business after the newly added field.

[0131] Optionally, as an embodiment, the third business code is an SQL code written according to a preset code writing rule, wherein the preset code writing rule is: each newly added field is assigned to a corresponding field in the partition table through as;

[0132] Processor 710 is also used to parse the names and types of all fields of the first business in the partition table, and generate a fifth business code based on the names and types of the parsed fields, wherein the data in the fifth business code except for the numeric type and the structure type is an empty string; parse the content between the first select and the first from in the third business code to obtain information of the newly added field; and insert the information of the newly added field into the fifth business code according to the information indicated by as in the third business code to obtain a fourth business code.

[0133] Optionally, as an embodiment, the first partition and the second partition further include a Type field, wherein the Type field is used to indicate the type of data accessed in the partition where the data is located.

[0134] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0135] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0136] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0137] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the embodiment of the method for processing data in the above-mentioned data warehouse is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0138] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0139] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data processing method embodiment in the data warehouse, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0141] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the embodiment of the method for processing data in the data warehouse as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0142] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for processing data in a data warehouse, characterized in that: The method comprises: When receiving a request for adding a business field for a first business in the data warehouse, adding a name and a type of the newly added field in a partition table of the data warehouse according to information of the newly added field carried in the request for adding the field; Obtaining a first service code in a first partition under the partition table, generating a second service code according to the first service code and field information of the first service recorded in the partition table, and writing the second service code into the first partition, wherein the first service code is a service code corresponding to an original field of the first service, and the second service code includes the original field of the first service and an empty string used to represent a newly added field; Receive a third service code corresponding to a newly added field of the first service uploaded by a user, generate a fourth service code according to the third service code and field information of the first service recorded in the partition table, create a second partition under the partition table, and write the fourth service code into the second partition, wherein the fourth service code includes the newly added field of the first service and an empty string used to represent the original field; The second service code in the first partition and the fourth service code in the second partition are aggregated to obtain a complete service code of the first service after the newly added field.

2. The method according to claim 1, characterized in that The aggregating the second service code in the first partition and the fourth service code in the second partition to obtain a complete service code of the first service after the newly added field includes: Creating a third partition under the partition table; In the third partition, the second service code in the first partition and the fourth service code in the second partition are summarized to obtain a complete service code of the first service after the newly added field.

3. The method according to claim 1, characterized in that According to the process of data inflow and outflow, the data warehouse includes: an operational data storage layer, an intermediate layer, a data warehouse layer and a data application layer, and the partition table is stored in the intermediate layer; The aggregating the second service code in the first partition and the fourth service code in the second partition to obtain a complete service code of the first service after the newly added field includes: Sending the second business code in the first partition and the fourth business code in the second partition of the intermediate layer to the data warehouse layer; In the data warehouse layer, the second business code in the first partition and the fourth business code in the second partition are summarized to obtain a complete business code for the first business after the newly added field.

4. The method according to any one of claims 1 to 3, characterized in that The aggregating the second service code in the first partition and the fourth service code in the second partition to obtain a complete service code of the first service after the newly added field includes: Through the GROUP BY function, the empty strings in the second business code and the fourth business code are deleted, and the remaining contents in the second business code and the fourth business code are merged to obtain the complete business code of the first business after the newly added field.

5. The method according to claim 1, characterized in that The third business code is an SQL code written according to a preset code writing rule, wherein the preset code writing rule is: each newly added field is assigned to a corresponding field in the partition table through as; The generating a fourth service code according to the third service code and the field information of the first service recorded in the partition table includes: Parsing the names and types of all fields of the first business in the partition table, and generating a fifth business code based on the names and types of the parsed fields, wherein data other than numeric types and structure types in the fifth business code is an empty string; Parse the content between the first select and the first from in the third business code to obtain information about the newly added field; According to the information indicated by as in the third service code, the information of the newly added field is inserted into the fifth service code to obtain a fourth service code.

6. A data processing device in a data warehouse, characterized in that: The device comprises: A modification module, configured to, upon receiving a request for adding a business field for a first business in a data warehouse, add a name and a type of the newly added field in a partition table of the data warehouse according to information of the newly added field carried in the request for adding the field; a first processing module, configured to obtain a first service code in a first partition under the partition table, generate a second service code according to the first service code and field information of the first service recorded in the partition table, and write the second service code into the first partition, wherein the first service code is a service code corresponding to an original field of the first service, and the second service code includes the original field of the first service and an empty string used to represent a newly added field; a second processing module, configured to receive a third service code corresponding to a newly added field of the first service uploaded by a user, generate a fourth service code according to the third service code and field information of the first service recorded in the partition table, create a second partition under the partition table, and write the fourth service code into the second partition, wherein the fourth service code includes the newly added field of the first service and an empty string used to represent the original field; The summarizing module is used to summarize the second service code in the first partition and the fourth service code in the second partition to obtain the complete service code of the first service after the newly added field.

7. The device according to claim 6, characterized in that The summary module includes: Creating a submodule, used to create a third partition under the partition table; The first summarizing submodule is used to summarize the second service code in the first partition and the fourth service code in the second partition in the third partition to obtain a complete service code of the first service after the newly added field.

8. The device according to claim 6, characterized in that According to the process of data inflow and outflow, the data warehouse includes: an operational data storage layer, an intermediate layer, a data warehouse layer and a data application layer, and the partition table is stored in the intermediate layer; The summary module includes: A sending submodule, used for sending the second business code in the first partition and the fourth business code in the second partition of the intermediate layer to the data warehouse layer; The second summary submodule is used to summarize the second business code in the first partition and the fourth business code in the second partition in the data warehouse layer to obtain the complete business code of the first business after the newly added field.

9. The device according to any one of claims 6 to 8, characterized in that The summary module includes: The third summary submodule is used to delete the empty strings in the second business code and the fourth business code through the GROUP BY function, merge the remaining contents in the second business code and the fourth business code, and obtain the complete business code of the first business after the newly added field.

10. The device according to claim 6, characterized in that The third business code is an SQL code written according to a preset code writing rule, wherein the preset code writing rule is: each newly added field is assigned to a corresponding field in the partition table through as; The second processing module comprises: A first parsing submodule is used to parse the names and types of all fields of the first business in the partition table, and generate a fifth business code based on the names and types of the parsed fields, wherein data other than numeric types and structure types in the fifth business code is an empty string; The second parsing submodule is used to parse the content between the first select and the first from in the third business code to obtain the information of the newly added field; The insertion submodule is used to insert the information of the newly added field into the fifth business code according to the information indicated by as in the third business code to obtain the fourth business code.

Images