A data processing method, device, equipment and computer storage medium

Abstract

The present application discloses a data processing method, device, equipment and computer storage medium. The method includes: obtaining a first data file; importing the first data file into a Hive table to generate first record data; comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table; setting the closed date of the target record data corresponding to the target identifier in the second record data to the date of obtaining the first data file, the open date of the first record data to the date of obtaining the first data file, and the closed date of the first record to infinity; mapping the first record data, the open date of the first record data, the closed date of the first record and the closed date of the target record data to the HBase table. Using the data processing method, device, equipment and computer storage medium provided by the present application, large data processing of PB or above can be performed on a single cluster, reducing the complexity of cluster management.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data processing method, device, equipment and computer storage medium. Background Art

[0002] In the classic dimensional modeling scenario of data warehouse, there is a concept called slowly changing dimensions (SCD). SCD was proposed because the attributes of dimensions are not static, but will change slowly over time. Such dimensions that change over time are generally accompanied by changes in the status of historical information. We generally call the scenario of processing historical change information of dimension tables the modeling application scenario of processing SCD.

[0003] In the prior art, the SCD scenario in data warehouse dimensional modeling is usually handled by using the Massively Parallel Processing (MPP) architecture technology based on PostgreSQL. However, since the MPP architecture technology based on PostgreSQL is limited in cluster scale, the theoretical upper limit of the number of nodes in a single cluster is 512. When processing data above the petabyte (PB) level, multiple clusters need to cooperate, which increases the complexity of cluster management. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and computer storage medium, which can at least solve the problem in the prior art that multi-cluster cooperation is required when processing data of a scale above PB level.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Obtaining a first data file;

[0007] Import the first data file into the Hive table to generate first record data;

[0008] Compare the first identifier of the first record data with the second identifier of each second record data stored in the Hive table;

[0009] When the target identifier in the first identifier is consistent with the target identifier in the second identifier, the closing date of the target record data corresponding to the target identifier in the second record data is set to the date of obtaining the first data file, the opening date of the first record data is set to the date of obtaining the first data file, and the closing date of the first record is set to infinity;

[0010] The first record data, the opening date of the first record data, the closing date of the first record, and the closing date of the target record data are mapped into an HBase table.

[0011] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0012] A first acquisition module, used to acquire a first data file;

[0013] A first generating module, used for importing the first data file into a Hive table to generate first record data;

[0014] A comparison module, used for comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table;

[0015] A first setting module is used to set the closing date of the target record data corresponding to the target identifier in the second record data to the date of obtaining the first data file, the opening date of the first record data to the date of obtaining the first data file, and the closing date of the first record to infinity when the first identifier is consistent with the target identifier in the second identifier;

[0016] The first mapping module is used to map the first record data, the opening date of the first record data, the closing date of the first record, and the closing date of the target record data into an HBase table.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0018] When the processor executes the computer program instructions, the data processing method as shown in any one of the embodiments of the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the data processing method shown in any one of the embodiments of the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer product, characterized in that when instructions in the computer product are executed by a processor of a server, the server is enabled to execute the data processing method shown in any one of the embodiments of the first aspect.

[0021] The data processing method, device, equipment and computer storage medium of the embodiment of the present application generate first record data by importing the acquired first data file into the Hive table, and then compare the first identifier of the first record data with the second identifier of each second record data stored in the Hive table. When the target identifier in the first identifier is consistent with the target identifier in the second identifier, the closed date of the target record data corresponding to the target identifier in the second record data is set to the date of acquiring the first data file, the open date of the first record data is the date of acquiring the first data file, and the closed date of the first record data is infinite, and the first record data, the open date of the first record data, the closed date of the first record and the closed date of the target record data are mapped to the HBase table. Since Hive and HBase are both data warehouses based on Hadoop (a distributed system infrastructure developed by the Apache Foundation), and the Hadoop platform can reach nearly 10,000 nodes in a single cluster, it is possible to process large data at the PB level or above on a single cluster, reducing the complexity of cluster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment;

[0024] Figure 2 is a flow chart of another data processing method according to an exemplary embodiment;

[0025] Figure 3a is a flow chart of another data processing method according to an exemplary embodiment;

[0026] Figure 3b is a flow chart of yet another data processing method according to an exemplary embodiment;

[0027] Figure 3c is a flow chart of yet another data processing method according to an exemplary embodiment;

[0028] Figure 4 is a structural schematic diagram of a data processing device according to an exemplary embodiment;

[0029] Figure 5 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0032] In conjunction with the accompanying drawings, the mirror package file verification method, device, electronic device and computer storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0033] The data processing method provided in this application can be applied to the data table provided in this application, that is, the associated data table of Hive and Hbase. Therefore, before performing data processing, it is necessary to create the data table and import the initial data, which is explained in detail below.

[0034] Figure 1 A flow chart of a data processing method provided by an embodiment of the present application is shown.

[0035] In a possible embodiment, before obtaining the first data file, Figure 1 As shown, the data processing method may include: S110-S120, which are specifically as follows:

[0036] S110: Create a Hive table.

[0037] Here, you can create a Hive table first. Since Hive can be a data warehouse based on Hadoop, and the Hadoop platform can reach nearly 10,000 nodes in a single cluster, data processing based on the Hive table can reduce the complexity of cluster management. The Hive table can include external tables and internal tables.

[0038] S120, through the base administrator HBase storage handler, map the table name of the Hive table to the table name of the HBase table, map the column name of the Hive table to the column name of the HBase table, and map the column cluster of the Hive table to the column cluster of the HBase table.

[0039] Here, Hive tables and HBase tables can be associated through HBase storage processors, and HBase can also be a data warehouse based on Hadoop. Since Hive tables do not support update and delete operations, Hive tables alone cannot handle SCD modeling application scenarios. They need to be associated with HBase tables and use the version management function of HBase tables to keep all historical record data.

[0040] In an example, a contact dimension table can be created. Specifically, row_key can be set as a proxy primary key, and YYYYMMDD-<10-digit Contact Code with leading zeros> can be set as the encoding format of the proxy primary key value. The encoding format of the key value can be composed of a date including year, month, and day and a ten-digit contact number with leading zeros. The date can be the date when the corresponding contact record data is generated. You can also set info_code as a natural primary key, representing the contact number; info_name as a dimension attribute, representing the contact name; info_email as a dimension attribute, representing the contact email; info_state as a dimension attribute, representing the state where the contact is located; valid_from as the start date attribute of the data zipper, representing the opening date; valid_to as the end date attribute of the data zipper, representing the closing date. In addition, an HBase storage processor is used to associate the Hive table with the HBase table. Specifically, the TBLPROPERTIES statement can be used to specify that the table name of the HIVE table is mapped to the table name of the HBASE table, and the SERDEPROPERTIES statement can be used to specify that the column name of the HIVE table is mapped to the column name of the HBASE table, and the column cluster of the HIVE table is mapped to the column cluster of the HBASE table.

[0041] Thus, through the above process, the HIVE table and the HBASE table can be associated, and the HBASE table can be used to make up for the disadvantage that the HIVE table does not support update and delete, so that all historical record data can be retained.

[0042] Based on this, in an optional implementation, after the above S120, the method may further include:

[0043] Obtaining a second data file;

[0044] Import the second data file into the Hive table to generate the sixth record data;

[0045] Set the opening date of the sixth record data to be infinitesimal and the closing date of the sixth record data to be infinite;

[0046] The sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data are mapped to the HBase table.

[0047] Here, the second data file may be a file storing the original data. After the associated data table of Hive and Hbase is created, the original data, i.e., the second data file, may be imported into the Hive table to generate the sixth record data, and the sixth record data may be subjected to data zipper processing. The opening date may be the start date of the data zipper, and the closing date may be the end date of the data zipper. Infinitesimal may be the earliest historical date that a machine can recognize, for example: January 1, 1900. Infinity may be the latest future date that a machine can recognize, for example: January 1, 9999.

[0048] Since, in S120, the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table, after the second data file is imported into the Hive table and closed-link processing is performed, the sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data in the Hive table will be mapped to the HBase table.

[0049] In one example, the original contact data can be imported. Specifically, scd2_contacts_initial_stage can be set as a Hive external table to mount the original contact's Comma-Separated Values ​​(CSV) data file; the INSERT statement can be used to import the CSV data file into the contact's Hive internal table, such as scd2_contacts_target, to generate the sixth record data, and the opening date of the sixth record data can be set to 1900-01-01, and the closing date can be initialized to 9999-12-31; in addition, the Hive external table can also include scd2_contacts_update_stage, which is used to mount the incrementally updated contact-related CSV data file.

[0050] In this way, the associated data table of HIVE and HBASE can be initialized, and the original data can be imported into the associated data table for storage.

[0051] Further, in an optional implementation manner, the mapping of the sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data to the HBase table may specifically include:

[0052] Map the column cluster corresponding to the sixth record data in the Hive table to any empty column cluster in the HBase table to generate the seventh record data;

[0053] Map the column cluster corresponding to the opening date of the sixth record data in the Hive table to the column cluster corresponding to the opening date of the seventh record data in the HBase table;

[0054] Map the column cluster corresponding to the closing date of the sixth record data in the Hive table to the column cluster corresponding to the closing date of the seventh record data in the HBase table.

[0055] Here, since the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table in S120, after the second data file is imported into the Hive table and closed-link processing is performed, the column cluster corresponding to the sixth record data in the Hive table will be mapped to any empty column cluster in the HBase table to generate the seventh record data, the column cluster corresponding to the open-link date of the sixth record data in the Hive table will be mapped to the column cluster corresponding to the open-link date of the seventh record data in the HBase table, and the column cluster corresponding to the closed-link date of the sixth record data in the Hive table will be mapped to the column cluster corresponding to the closed-link date of the seventh record data in the HBase table.

[0056] In this way, the version management function of the HBase table can be fully utilized to ensure that all original data information can be retained in the HBase table without being lost.

[0057] Figure 2 A flow chart of another data processing method provided by an embodiment of the present application is shown.

[0058] like Figure 2 As shown, the data processing method may include the following steps:

[0059] First, S210, obtaining a first data file;

[0060] Next, S220, import the first data file into a Hive table to generate first record data;

[0061] Again, S230, comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table;

[0062] Then, S240, when the first identifier is consistent with the target identifier in the second identifier, set the closing date of the target record data corresponding to the target identifier in the second record data to the date of obtaining the first data file, the opening date of the first record data to the date of obtaining the first data file, and the closing date of the first record to infinity;

[0063] Finally, S250, the first record data, the opening date of the first record data, the closing date of the first record data, and the closing date of the target record data are mapped into an HBase table.

[0064] Thus, by importing the acquired first data file into the Hive table, generating the first record data, and then comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table, when the first identifier is consistent with the target identifier in the second identifier, setting the closed date of the target record data corresponding to the target identifier in the second record data to the date of acquiring the first data file, the open date of the first record data to the date of acquiring the first data file, and the closed date of the first record data to infinity, and mapping the first record data, the open date of the first record data, the closed date of the first record, and the closed date of the target record data to the HBase table. Since Hive and HBase are both data warehouses based on Hadoop (a distributed system infrastructure developed by the Apache Foundation), and the Hadoop platform can reach nearly 10,000 nodes in a single cluster, it is possible to process large data at the PB level or above on a single cluster, reducing the complexity of cluster management.

[0065] Below, before explaining the above steps S210-S250 in detail, the principle of the data processing method provided in the embodiment of the present application is briefly explained.

[0066] Usually when dealing with SCD scenarios, you can use Figure 3a or Figure 3b The method shown.

[0067] Figure 3a The method shown is a method of directly overwriting the original value, which may specifically include: S311-S314, wherein:

[0068] S311, obtaining new data.

[0069] S312, determining whether there is a historical record of the newly added data in the data table.

[0070] Here, if it exists, execute S313, if it does not exist, execute S314.

[0071] S313, loading a new record.

[0072] Here, if there is no historical record of the newly added data in the data table, a new record is directly added according to the newly added data.

[0073] S314, replace the historical record.

[0074] Here, if there is a historical record of the newly added data in the data table, a new record is generated according to the newly added data, and the historical record is replaced by the new record, that is, only the latest record is saved.

[0075] therefore, Figure 3a The method shown does not retain historical data and cannot analyze historical change information.

[0076] Figure 3b The method shown is a method for adding an attribute column, which may specifically include: S321-S325,

[0077] in:

[0078] S321, obtaining new data.

[0079] S322, determine whether there is a historical record of the newly added data in the data table.

[0080] Here, if it exists, execute S323, if it does not exist, execute S325.

[0081] S323, determining the changed attributes of the newly added data.

[0082] S324, modify the changed attribute and add a column to record the value of the attribute before the change.

[0083] Here, if there is a historical record of the newly added data in the data table, the attributes of the newly added data that have changed compared to the historical record can be modified, and a column can be added to record the attribute value before the modification.

[0084] S325, loading a new record.

[0085] Here, if there is no historical record of the newly added data in the data table, a new record is directly added according to the newly added data.

[0086] therefore, Figure 3b Although the method shown can analyze the current and previous changed attribute values ​​at the same time, it only retains the last change information and cannot retain all historical information.

[0087] Based on this, the embodiment of the present application provides a method for adding a dimension row, such as Figure 3c As shown, it may specifically include: S331-S335, wherein:

[0088] S331, obtain new data.

[0089] S332, determine whether there is a historical record of the newly added data in the data table.

[0090] Here, if it exists, execute S333, if it does not exist, execute S335.

[0091] S333, add a new record.

[0092] Here, if there is a historical record of the newly added data in the data table, a new record is generated.

[0093] S334, association history.

[0094] Here, if there is a historical record of the newly added data in the data table, after the new record is generated, the new record and the historical record can be associated with each other through the primary key.

[0095] S335, loading a new record.

[0096] Here, if there is no historical record of the newly added data in the data table, a new record is directly added according to the newly added data.

[0097] Should Figure 3c In the method shown, when a dimension attribute changes, a new record can be generated and associated with the historical record through the primary key, so that all historical records can be retained.

[0098] Based on the above explanation of the principle of the data processing method adopted in the embodiment of the present application, the above steps S210-S250 are described in detail as follows:

[0099] First, regarding S210, the first data file may be a CSV file for storing newly added data, and the newly added data may be newly added status data of the original object, or may be newly added objects and their status data.

[0100] Secondly, involving S220, the first data file can be imported into the Hive table through the Insert statement to generate first record data. If the new data in the first data file is the new status data of the original object, the generated first record data can be the new record data of the original object; if the new data in the first data file is the new object and its status data, the generated first record data can be the new object and its record data.

[0101] Again, regarding S230, the first identifier can be a unique identifier of the object corresponding to the first record data in the Hive table, and the second identifier can be a unique identifier of the object corresponding to the second record data in the Hive table. One object corresponds to only one identifier, and one object can correspond to multiple record data. By comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table, it can be determined whether the first record data is a newly added record data of the original object, or a newly added object and its record data. Specifically, if there is an identifier in the second identifier that is consistent with the first identifier, then the first record data is the newly added record data of the original object; if there is no identifier in the second identifier that is consistent with the first identifier, then the first record data is a newly added object and its record data.

[0102] Then, in S240, when the first identifier is consistent with the target identifier in the second identifier, it can be indicated that the first record data is newly added record data of the original object, and the first record data and the target record data corresponding to the target identifier can be closed-linked.

[0103] Finally, regarding S250, since the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table in S120, after the first record data and the target record data are closed-linked, the first record data, the opening date of the first record data, the closing date of the first record, and the closing date of the target record data can be mapped to the HBase table.

[0104] In combination with S210-S250, in an example, by comparing the identifier of the original object in the Hive table and the identifier of the newly added data record, it can be determined that the newly added data record is a new data record of the original object. Therefore, the newly added record data and the historical record data of the target object corresponding to the newly added data record can be closed-linked.

[0105] Specifically, taking the contact record data table as an example, the historical record data of the target contact corresponding to the newly added data record is closed, and the closed date of the historical record data is set to the current date, that is, the date of obtaining the first data file, and the open date remains unchanged.

[0106] At the same time, the newly added record data is closed-linked, and the opening date of the newly added record data is set to the current date, that is, the date of obtaining the first data file, and the closing date of the newly added record data is set to 9999-12-31, that is, infinity.

[0107] In this way, the number of HBase versions can be fully utilized, so that the maximum number of versions of each row key is 2, namely the initial loading version and the historical information version, to avoid bringing too heavy a burden to the system. Among them, the initial loading version has an infinitesimal opening date and an infinity as the closing date, and the historical information version has an infinitesimal opening date and an infinity as the closing date.

[0108] Based on this, in an optional implementation manner, the above S250 may specifically include:

[0109] Map the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table to generate third record data corresponding to the first record data;

[0110] Map the column cluster corresponding to the open date of the first record data in the Hive table to the column cluster corresponding to the open date of the third record data in the HBase table;

[0111] Map the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the third record in the HBase table;

[0112] Map the column cluster corresponding to the closing date of the target record data in the Hive table to the column cluster corresponding to the closing date of the fourth record data in the HBase table.

[0113] Here, since the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table in S120, after the first record data and the target record data are closed, the column cluster corresponding to the first record data in the Hive table will be mapped to any empty column cluster in the HBase table to generate third record data corresponding to the first record data, the column cluster corresponding to the open chain date of the first record data in the Hive table will be mapped to the column cluster corresponding to the open chain date of the third record data in the HBase table, the column cluster corresponding to the closed chain date of the first record in the Hive table will be mapped to the column cluster corresponding to the closed chain date of the third record data in the HBase table, and the column cluster corresponding to the closed chain date of the target record data in the Hive table will be mapped to the column cluster corresponding to the closed chain date of the fourth record data in the HBase table. The fourth record data is the record data corresponding to the target record data generated after the column cluster corresponding to the target record data in the Hive table is mapped to any empty column cluster in the HBase table.

[0114] In this way, the version management function of the HBase table can be fully utilized to ensure that all historical record data and newly added record data can be retained in the HBase table and will not be lost.

[0115] In addition, in an optional implementation, after S230, the data processing method may further include:

[0116] In the case where any identifier in the first identifier and the second identifier are inconsistent, the opening date of the first record data is set to be infinitesimal and the closing date of the first record data is set to be infinite;

[0117] The first record data, the opening date of the first record data, and the closing date of the first record are mapped to the HBase table.

[0118] Here, when any identifier of the first identifier and the second identifier are inconsistent, it can be indicated that the first record data is a newly added object and its record data, and the first record data can be closed-link processed.

[0119] In one example, by comparing the identifier of the original object in the Hive table with the identifier of the newly added data record, it can be determined that the newly added data record is a newly added object and its record data, so the newly added record data can be closed-link processed.

[0120] Specifically, taking the contact record data table as an example, a closed link process is performed on the newly added data record, and the opening date of the newly added record data is set to 1900-01-01, which is infinitesimal, and the closing date of the newly added record data is set to 9999-12-31, which is infinite.

[0121] In this way, the record data of objects that are not in the original Hive table can be stored in the Hive table and mapped to the HBase table to complete the storage of the record data of the newly added objects.

[0122] Based on this, in an optional implementation manner, the mapping of the first record data, the opening date of the first record data, and the closing date of the first record to the HBase table may specifically include:

[0123] Map the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table to generate fifth record data corresponding to the first record data;

[0124] Map the column cluster corresponding to the open date of the first record data in the Hive table to the column cluster corresponding to the open date of the fifth record data in the HBase table;

[0125] Map the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the fifth record in the HBase table.

[0126] Here, since the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table in S120, after the first record data is closed-linked, the column cluster corresponding to the first record data in the Hive table will be mapped to any empty column cluster in the HBase table to generate the fifth record data corresponding to the first record data, the column cluster corresponding to the open-link date of the first record data in the Hive table will be mapped to the column cluster corresponding to the open-link date of the fifth record data in the HBase table, and the column cluster corresponding to the closed-link date of the first record in the Hive table will be mapped to the column cluster corresponding to the closed-link date of the fifth record data in the HBase table.

[0127] In this way, the version management function of the HBase table can be fully utilized to ensure that the record data of all newly added objects can be retained in the HBase table and will not be lost.

[0128] In addition, in an optional implementation, all of the above-mentioned opening and closing dates can be set through an Insert statement.

[0129] In this way, there is no need to use traditional update operations in SQL to close the data chain, thus making up for the disadvantage that Hive cannot close the data chain because it does not support update operations.

[0130] Based on the same inventive concept, the present application also provides a data processing device. Figure 4The data processing device provided in the embodiment of the present application is described in detail.

[0131] Figure 4 It is a structural block diagram of a data processing device according to an exemplary embodiment.

[0132] like Figure 4 As shown, the data processing device may include:

[0133] A first acquisition module 401, used to acquire a first data file;

[0134] A first generating module 402, used to import the first data file into a Hive table to generate first record data;

[0135] A comparison module 403, configured to compare the first identifier of the first record data with the second identifier of each second record data stored in the Hive table;

[0136] The first setting module 404 is used to set the closing date of the target record data corresponding to the target identifier in the second record data to the date of obtaining the first data file, the opening date of the first record data to the date of obtaining the first data file, and the closing date of the first record to infinity when the first identifier is consistent with the target identifier in the second identifier;

[0137] The first mapping module 405 is used to map the first record data, the opening date of the first record data, the closing date of the first record and the closing date of the target record data into an HBase table.

[0138] In one embodiment, the first mapping module 405 may specifically include:

[0139] A first mapping submodule, used to map the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table, to generate third record data corresponding to the first record data;

[0140] The second mapping submodule is used to map the column cluster corresponding to the opening date of the first record data in the Hive table to the column cluster corresponding to the opening date of the third record data in the HBase table;

[0141] A third mapping submodule is used to map the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the third record data in the HBase table;

[0142] The fourth mapping submodule is used to map the column cluster corresponding to the closed date of the target record data in the Hive table to the column cluster corresponding to the closed date of the fourth record data in the HBase table. The fourth record data is the record data corresponding to the target record data generated after mapping the column cluster corresponding to the target record data in the Hive table to any empty column cluster in the HBase table.

[0143] In one of the embodiments, the data processing device may further include:

[0144] A second setting module is used to set the opening date of the first record data to be infinitesimal and the closing date of the first record data to be infinite after comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table, if any identifier of the first identifier and the second identifier are inconsistent;

[0145] The second mapping module is used to map the first record data, the opening date of the first record data, and the closing date of the first record into the HBase table.

[0146] In one embodiment, the second mapping module may specifically include:

[0147] A fifth mapping submodule, used to map the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table, to generate fifth record data corresponding to the first record data;

[0148] A sixth mapping submodule, used to map the column cluster corresponding to the opening date of the first record data in the Hive table to the column cluster corresponding to the opening date of the fifth record data in the HBase table;

[0149] The seventh mapping submodule is used to map the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the fifth record data in the HBase table.

[0150] In one of the embodiments, the data processing device may further include:

[0151] Create a module to create a Hive table before obtaining data files;

[0152] The third mapping module is used to map the table name of the Hive table to the table name of the HBase table, the column name of the Hive table to the column name of the HBase table, and the column cluster of the Hive table to the column cluster of the HBase table through the base administrator HBase storage processor.

[0153] In one embodiment, the data processing device may further include:

[0154] A second acquisition module is used to acquire a second data file after mapping the table name of the Hive table to the table name of the HBase table, the column name of the Hive table to the column name of the HBase table, and the column cluster of the Hive table to the column cluster of the HBase table through the base administrator HBase storage processor;

[0155] A second generating module, used for importing the second data file into a Hive table to generate sixth record data;

[0156] A third setting module is used to set the opening date of the sixth record data to be infinitesimal and the closing date of the sixth record data to be infinite;

[0157] The fourth mapping module is used to map the sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data into the HBase table.

[0158] In one embodiment, the fourth mapping module may specifically include:

[0159] An eighth mapping submodule is used to map the column cluster corresponding to the sixth record data in the Hive table to any empty column cluster in the HBase table to generate the seventh record data;

[0160] A ninth mapping submodule, used to map the column cluster corresponding to the opening date of the sixth record data in the Hive table to the column cluster corresponding to the opening date of the seventh record data in the HBase table;

[0161] The tenth mapping submodule is used to map the column cluster corresponding to the closed chain date of the sixth record data in the Hive table to the column cluster corresponding to the closed chain date of the seventh record data in the HBase table.

[0162] In one embodiment, the opening date and closing date are set via an Insert statement.

[0163] Thus, by importing the acquired first data file into the Hive table, generating the first record data, and then comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table, when the first identifier is consistent with the target identifier in the second identifier, setting the closed date of the target record data corresponding to the target identifier in the second record data to the date of acquiring the first data file, the open date of the first record data to the date of acquiring the first data file, and the closed date of the first record data to infinity, and mapping the first record data, the open date of the first record data, the closed date of the first record, and the closed date of the target record data to the HBase table. Since Hive and HBase are both data warehouses based on Hadoop (a distributed system infrastructure developed by the Apache Foundation), and the Hadoop platform can reach nearly 10,000 nodes in a single cluster, it is possible to process large data at the PB level or above on a single cluster, reducing the complexity of cluster management.

[0164] Figure 5 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment.

[0165] like Figure 5 As shown, the electronic device 5 is a structural diagram of an exemplary hardware architecture of an electronic device that can implement the data processing method and data processing device in the embodiments of the present application. The electronic device may refer to the electronic device in the embodiments of the present application.

[0166] The electronic device 5 may include a processor 501 and a memory 502 storing computer program instructions.

[0167] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0168] The memory 502 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 502 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 502 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 502 is a non-volatile solid-state memory. In a particular embodiment, the memory 502 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0169] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement Figure 1 The method in the embodiment shown in FIG3 is implemented and the corresponding technical effect is achieved, which will not be described again for the sake of brevity.

[0170] In one embodiment, the electronic device 5 may further include a transceiver 503 and a bus 504. Figure 5 As shown, the processor 501, the memory 502 and the transceiver 503 are connected via a bus 504 and communicate with each other.

[0171] Bus 504 includes hardware, software or both.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral control interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 504 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0172] The embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored, and the computer executable instructions are used to implement the data processing method described in the embodiment of the present application.

[0173] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to the various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the data processing method recorded in the embodiments of the present application.

[0174] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a first data file; Importing the first data file into a Hive table to generate first record data; Compare the first identifier of the first record data with the second identifier of each second record data stored in the Hive table; In the case where the first identifier is consistent with the target identifier in the second identifier, the closing date of the target record data corresponding to the target identifier in the second record data is set to the date of obtaining the first data file, the opening date of the first record data is set to the date of obtaining the first data file, and the closing date of the first record is set to infinity; The first record data, the opening date of the first record data, the closing date of the first record, and the closing date of the target record data are mapped into an HBase table.

2. The method according to claim 1, characterized in that The mapping of the first record data, the opening date of the first record data, the closing date of the first record, the second record data, and the closing date of the target record data into an HBase table includes: Mapping the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table to generate third record data corresponding to the first record data; Mapping the column cluster corresponding to the opening date of the first record data in the Hive table to the column cluster corresponding to the opening date of the third record data in the HBase table; Mapping the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the third record data in the HBase table; Map the column cluster corresponding to the closed date of the target record data in the Hive table to the column cluster corresponding to the closed date of the fourth record data in the HBase table, wherein the fourth record data is record data corresponding to the target record data generated after mapping the column cluster corresponding to the target record data in the Hive table to any empty column cluster in the HBase table.

3. The method according to claim 1, characterized in that After comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table, the method further includes: When any identifier of the first identifier and the second identifier are inconsistent, the opening date of the first record data is set to be infinitesimal and the closing date of the first record data is set to be infinite; The first record data, the opening date of the first record data, and the closing date of the first record are mapped to the HBase table.

4. The method according to claim 3, characterized in that The mapping of the first record data, the opening date of the first record data, and the closing date of the first record to the HBase table includes: Mapping the column cluster corresponding to the first record data in the Hive table to any empty column cluster in the HBase table to generate fifth record data corresponding to the first record data; Mapping the column cluster corresponding to the opening date of the first record data in the Hive table to the column cluster corresponding to the opening date of the fifth record data in the HBase table; Map the column cluster corresponding to the closing date of the first record in the Hive table to the column cluster corresponding to the closing date of the fifth record data in the HBase table.

5. The method according to claim 1, characterized in that Before obtaining the first data file, the method further includes: Create the Hive table; Through the base administrator HBase storage processor, the table name of the Hive table is mapped to the table name of the HBase table, the column name of the Hive table is mapped to the column name of the HBase table, and the column cluster of the Hive table is mapped to the column cluster of the HBase table.

6. The method according to claim 5, characterized in that After mapping the table name of the Hive table to the table name of the HBase table, mapping the column name of the Hive table to the column name of the HBase table, and mapping the column cluster of the Hive table to the column cluster of the HBase table through the base administrator HBase storage processor, the method further includes: Obtaining a second data file; Importing the second data file into the Hive table to generate sixth record data; Setting the opening date of the sixth record data to be infinitesimal and the closing date of the sixth record data to be infinite; The sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data are mapped to the HBase table.

7. The method according to claim 6, characterized in that Mapping the sixth record data, the opening date of the sixth record data, and the closing date of the sixth record data to the HBase table includes: Mapping the column cluster corresponding to the sixth record data in the Hive table to any empty column cluster in the HBase table to generate the seventh record data; Mapping the column cluster corresponding to the opening date of the sixth record data in the Hive table to the column cluster corresponding to the opening date of the seventh record data in the HBase table; Map the column cluster corresponding to the closed date of the sixth record data in the Hive table to the column cluster corresponding to the closed date of the seventh record data in the HBase table.

8. The method according to any one of claims 1 to 7, characterized in that: The opening date and the closing date are set by means of an Insert statement.

9. A data processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a first data file; A first generating module, used for importing the first data file into a Hive table to generate first record data; A comparison module, used for comparing the first identifier of the first record data with the second identifier of each second record data stored in the Hive table; A first setting module is used to set the closing date of the target record data corresponding to the target identifier in the second record data to the date of obtaining the first data file, the opening date of the first record data to the date of obtaining the first data file, and the closing date of the first record to infinity when the first identifier is consistent with the target identifier in the second identifier; The first mapping module is used to map the first record data, the opening date of the first record data, the closing date of the first record and the closing date of the target record data into an HBase table.

10. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data processing method according to any one of claims 1 to 8 is implemented.

11. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the data processing method according to any one of claims 1 to 8 is implemented.

12. A computer product, characterized in that: When the instructions in the computer product are executed by a processor of a server, the server is enabled to execute the data processing method according to any one of claims 1 to 8.

Images