Big data zipper table processing method, device, equipment and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for processing large-scale zipper tables, relating to the field of data processing. The method includes: acquiring incremental data from a zipper table, wherein the primary key of the zipper table includes a customer identifier and a product identifier, and the information change key of the zipper table includes at least one of consumption details, revenue details, and asset details; obtaining a Spark-based concurrency level for processing the zipper table based on the storage space occupied by the incremental data; and processing the zipper table based on the incremental data corresponding to each concurrency level, as well as the primary key and information change key of the zipper table. The method of this application addresses the shortcomings of existing systems in meeting user needs in terms of low cost and high efficiency.

Description

Technical Field

[0001] The present application relates to data processing, and in particular to a method, device, equipment and storage medium for processing a large data zipper table. Background Art

[0002] As financial systems become more integrated and specialized, users' requirements for data services provided by financial systems are becoming increasingly broad and demanding.

[0003] The number of daily changes in data in the financial system is in the billions. According to the traditional implementation method, each product system calculates the end-of-day holdings of customers every day and then summarizes them. The daily data volume reaches tens of billions, which consumes a lot of storage resources, increases the cost of data use, and also reduces the stability of the system. The amount of data that needs to be maintained increases exponentially with the number of daily customers, and the difficulty of maintenance continues to increase. In fact, only part of the data in the financial system changes, and the frequency of change of different data is different. For example, different data changes in daily, monthly, and annual cycles.

[0004] The existing system does not take into account the characteristics of data in the financial system and is lacking in meeting user needs at low cost and high efficiency. Summary of the invention

[0005] The present application provides a big data zipper table processing method, device, equipment and storage medium to solve the problem that the existing system does not take into account the data characteristics of the financial system and is lacking in meeting user needs in terms of low cost and high efficiency.

[0006] On the one hand, the present application provides a big data zipper table processing method, comprising:

[0007] Acquire incremental data of a zipper table, wherein the primary key of the zipper table includes a customer identifier and a product identifier, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details;

[0008] According to the storage space occupied by the incremental data, obtain the number of concurrency based on the Spark framework for processing the zipper table;

[0009] The zipper table is processed according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table.

[0010] Optionally, obtaining the number of concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data includes:

[0011] According to the storage space occupied by the incremental data and the processing volume of the incremental data corresponding to each concurrent thread, the concurrency number based on the Spark framework for processing the zipper table is obtained; wherein the processing volume corresponding to each concurrent thread is the same.

[0012] Optionally, the processing of the zipper table according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table includes:

[0013] For the incremental data corresponding to each concurrent thread, prefix matching is performed using the primary key to obtain target data under the information change key that has information changes relative to the open chain data;

[0014] A closed-chain operation is performed on the open-chain data corresponding to the target data, new open-chain data is generated according to the target data, and the new open-chain data is stored in an HBase database.

[0015] Optionally, before obtaining the incremental data of the zipper table, the method further includes:

[0016] The zipper table is constructed, and the zipper table also includes a start time and an end time, wherein the end time is set to a preset time for indicating the latest data, and the start closure time is set to the time when the data changes.

[0017] Optionally, the storing the new open-chain data in an HBase database, the method further comprises:

[0018] The new open chain data is stored in the HBase database by bulkload.

[0019] Optionally, storing the new open chain data in the HBase database by bulkload includes:

[0020] Generate an Hfile file using the new open chain data according to the region distribution of the HBase database;

[0021] Register the Hfile file to the HBase database.

[0022] Optionally, the method further comprises:

[0023] According to the unzip date field of the same primary key in the zipper table, multiple form data are obtained from bottom to top;

[0024] Historical trend statistics are performed based on the multiple form data.

[0025] On the other hand, the present application provides a large data zipper table processing device, comprising:

[0026] An acquisition module, used for acquiring incremental data of a zipper table, wherein the primary key of the zipper table includes a customer identifier and a product identifier, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details;

[0027] A calculation module, used for obtaining the number of concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data;

[0028] The processing module processes the zipper table according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table.

[0029] In a possible implementation, the computing module is further specifically configured to:

[0030] According to the storage space occupied by the incremental data and the processing volume of the incremental data corresponding to each concurrent thread, the concurrency number based on the Spark framework for processing the zipper table is obtained; wherein the processing volume corresponding to each concurrent thread is the same.

[0031] In a possible implementation, the processing module is further specifically configured to:

[0032] For the incremental data corresponding to each concurrent thread, prefix matching is performed using the primary key to obtain target data under the information change key that has information changes relative to the open chain data;

[0033] A closed-chain operation is performed on the open-chain data corresponding to the target data, new open-chain data is generated according to the target data, and the new open-chain data is stored in an HBase database.

[0034] In a possible implementation, the acquisition module is further specifically used for:

[0035] The zipper table is constructed, and the zipper table also includes a start time and an end time, wherein the end time is set to a preset time for indicating the latest data, and the start time is set to the time when the data changes.

[0036] In a possible implementation, the processing module is further specifically configured to:

[0037] The new open chain data is stored in the HBase database by bulkload.

[0038] In a possible implementation, the processing module is further specifically configured to:

[0039] Generate an Hfile file using the new open chain data according to the region distribution of the HBase database;

[0040] Register the Hfile file to the HBase database.

[0041] In a possible implementation, the acquisition module is further specifically used for:

[0042] According to the unzip date field of the same primary key in the zipper table, multiple form data are obtained from bottom to top;

[0043] Historical trend statistics are performed based on the multiple form data.

[0044] According to a third aspect of the present application, an electronic device is provided, including:

[0045] Processor and memory;

[0046] Memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes any one of the methods in the first aspect.

[0048] In a fourth aspect of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement a method for determining a driver program for a hardware peripheral as described in any one of the first aspects.

[0049] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the first aspects.

[0050] This embodiment provides a method, device, equipment and storage medium for processing a zipper table of big data. The method obtains the incremental data of the zipper table and obtains the concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data; the zipper table is processed according to the incremental data corresponding to each concurrency and the primary key and information change key of the zipper table. The method solves the storage and processing of a large amount of data in the financial system by using the zipper table, and speeds up the processing of the zipper table of big data by calculating the concurrency based on the Spark framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 A specific application scenario diagram of the big data zipper table processing method provided in this application;

[0053] Figure 2 The big data zipper table processing method provided in the embodiment of the present application Figure 1 ;

[0054] Figure 3 The big data zipper table processing method provided in the embodiment of the present application Figure 2 ;

[0055] Figure 4 A schematic diagram of the structure of a large data zipper table processing device provided in an embodiment of the present application;

[0056] Figure 5 A hardware structure diagram of a big data zipper table processing device provided in an embodiment of the present application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] Figure 1 This is a specific application scenario diagram of the big data zipper table processing method provided in this application. Figure 1 As shown, the application scenario includes: a financial system 101, data A, data B, data C, and a database 102. Exemplarily, data A, data B, and data C represent data generated on three different dates in the financial system, respectively. If data A, data B, and data C are stored in the database 102 without processing, a large amount of storage space will be occupied, and IO port resources will also be occupied, and the maintenance of a large amount of data requires more workload.

[0060] The present application provides a big data zipper table processing method, which processes big data in a financial system by using a zipper table. At the same time, when processing incremental data, the efficiency of data processing is improved by calculating the number of concurrency based on the Spark framework, thereby saving storage space and IO port resources and reducing the workload required for maintaining large amounts of data.

[0061] The big data zipper table processing method provided in this application is intended to solve the above technical problems in the prior art.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] Figure 2 The big data zipper table processing method provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the method of this embodiment includes:

[0064] S201, obtaining incremental data of a zipper table, where the primary key of the zipper table includes a customer ID and a product ID, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details;

[0065] A zipper table is a table that maintains historical status and the latest status data. Depending on the zipper granularity, the zipper table is actually equivalent to a snapshot, but it has been optimized and some unchanged records have been removed. The zipper table can be used to easily restore customer records at the zipper time point.

[0066] In this embodiment, the zipper table is created based on the data of the financial system. Based on business needs, the financial system provides customers with equal-revenue trend charts of various data through different channels, summarizes the data change trends of customers in various products and systems of the financial system, and displays them according to different dimensions. The current way in which the financial system provides services is to calculate the latest data of each product detail every day based on the business date, and then summarize and calculate all products, resulting in billions of incremental data added every day, which in turn leads to insufficient computing resources and storage resources of the system, increasing the burden on the financial system. In the financial system, by using a zipper table, historical data is closed and the latest data is opened based on only a small number of change records every day. Only the data of the change date is stored, and the data from the opening date to the closing date is static data and does not occupy the storage space of the financial system.

[0067] In a zipper table, the primary key refers to a column or a combination of columns whose values ​​can uniquely identify each row in the table, and through it, the entity integrity of the table can be enforced. The primary key is mainly used for foreign key associations in other tables, as well as modification and deletion of the current record. The primary key acts as an index during search to increase the speed of database operations.

[0068] In this embodiment, the primary key of the zipper table includes the customer ID and the product ID. The data is located by the customer ID and the product ID, and the customer ID is the primary one. The product ID under each customer ID has the same prefix as the customer ID. The customer ID refers to a string of characters or numbers that is unique to each customer and is used for customer coding. The product ID refers to a string of characters or numbers that is unique to a product in a financial system. Here, the product includes products and systems. Products include deposit certificates, etc., and systems include deposit systems, loan purchase systems, and housing reform systems, etc.

[0069] In the zipper table, the information change key usually selects the fields that change and the variables that change slowly, and cannot select the fields that change every day and have no comparative meaning, such as date, etc. In this embodiment, in order to cover a more comprehensive scope, the information change key of the zipper table includes at least one of consumption details, income details, and asset details.

[0070] In this embodiment, the incremental data is the daily changing data of each product system in the financial system. For some data, the customer's holdings are low-frequency changes, the net value table is daily changes, and the net value data is small. Optionally, in this embodiment, the shares in the share table can be stored in a zipper.

[0071] In a possible implementation, before obtaining the incremental data of the zipper table, a zipper table is constructed, and the zipper table also includes a start time and an end time, wherein the end time is set to a preset time for indicating the latest data, and the start time is set to the time when the data changes;

[0072] Zipper tables usually also include: start time and end time. By comparing the primary key with historical data, it is determined whether the current data has changed from the historical data. If it has changed or added, the corresponding open and close operations are performed. The end time is set to the preset time, usually set to the maximum time 29991231, and the start time is set to the time when the data changes, which can be the business time.

[0073] S202, according to the storage space occupied by the incremental data, obtaining the number of concurrency based on the Spark framework for processing the zipper table;

[0074] The Spark framework is a fast and general computing engine designed for large-scale data processing. The Spark framework uses a general parallel framework similar to Hadoop MapReduce. The intermediate output results can be stored in memory, eliminating the need to read and write the Hadoop distributed file system. Among them, MapReduce is a computing model, framework and platform for parallel processing of big data. The concurrency (concurrency) of the zipper list based on the Spark framework refers to the number of incremental data requests that the financial system can process at the same time, which also reflects the load capacity of the financial system.

[0075] In this embodiment, the financial system uses the Spark framework. In order to improve the processing speed of incremental data, considering the large amount of incremental data in the financial system, it is necessary to concurrently process the incremental data and the indexing process of the original zipper table. The incremental data files are divided according to the storage space occupied by the incremental quantity to prepare for subsequent work.

[0076] S203: Process the zipper table according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table.

[0077] In this embodiment, the open chain data in the zipper table is first queried using the primary key as an index, that is, the data whose end time is a preset time, and then the incremental data is compared with it to filter out the data in the zipper table that needs to be closed, and the closing operation is performed according to the information change key.

[0078] This embodiment provides a method for processing a zipper table of big data. The method obtains the incremental data of the zipper table and obtains the concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data; the zipper table is processed according to the incremental data corresponding to each concurrency and the primary key and information change key of the zipper table. The method solves the storage and processing of a large amount of data in the financial system by using the zipper table, and speeds up the processing of the zipper table of big data by calculating the concurrency based on the Spark framework.

[0079] Figure 3 The big data zipper table processing method provided in the embodiment of the present application Figure 2 .like Figure 3 As shown, in this embodiment Figure 2 Based on the illustrated embodiment, a method for processing a large data zipper table is described in detail. The method includes:

[0080] S301, according to the storage space occupied by the incremental data and the processing amount of the incremental data corresponding to each concurrent thread, obtain the number of concurrency based on the Spark framework for processing the zipper table; wherein the processing amount corresponding to each concurrent thread is the same;

[0081] In this embodiment, the concurrency of the zipper table based on the Spark framework is determined by two factors: the storage space occupied by the incremental data and the processing volume of the incremental data corresponding to each concurrent thread. That is, the processing volume of the incremental data corresponding to each concurrent thread can be specified first, and then the storage space occupied by the incremental data is compared with the processing volume of the incremental data corresponding to each concurrent thread to obtain the concurrency. It is also possible to predetermine that the storage space occupied by the incremental data corresponds to different concurrency numbers, that is, to use the storage space occupied by the incremental data as the independent variable of the piecewise function, and the concurrency number as the dependent variable of the piecewise function, and determine the concurrency number through the piecewise function.

[0082] Those skilled in the art can understand that the concurrency of the zipper table based on the Spark framework can also consider other factors, such as the hardware configuration of the database, and the calculation method is not limited to the above two methods. At the same time, the same processing volume corresponding to each concurrent thread is not a sufficient condition.

[0083] S302: For the incremental data corresponding to each concurrent thread, prefix matching is performed using the primary key to obtain target data under the information change key that has information changes relative to the open chain data;

[0084] In this embodiment, each customer has a customer ID, and each product under the name of each customer corresponds to a single product ID, and the prefix of the product ID is the same as the prefix of the customer ID. When prefix matching is performed with the primary key, the open-chain data is obtained according to the end time. For example, Table 1 is a zipper table. As shown in Table 1, the customer ID of customer A is 010001, and customer A has three products. The product ID of product A is 010010, the product ID of product B is 010020, and the product ID of product C is 010030, that is, the customer ID and product ID prefix of customer A are both 01. When processing incremental data, prefix matching is performed with 01. In the data whose end time is the preset time, that is, 29991231, the change data of customer A's consumption details, income details, and asset details are found. In Table 1, the customer ID, product A, and product B are open-chain data, so the data with information changes in the incremental data, that is, the target data, is found in these three data.

[0085] Table 1

[0086]

[0087] S303, performing a closing operation on the open chain data corresponding to the target data, and generating new open chain data according to the target data;

[0088] In this embodiment, when the target data is determined in the incremental data, the open chain data in the zipper table is also determined. At this time, the open chain data is no longer the latest data, so a closed chain operation is required, that is, the end time of the open chain data is changed to the business date. At the same time, a new file needs to be created to store the new open chain data. The open chain data here is the target data in the incremental data. At the same time, the start time of the open chain data needs to be set to the business hours, and the end time needs to be set to the preset time 29991231.

[0089] S304, generating an Hfile file through new open chain data according to the region distribution of the HBase database;

[0090] HBase database is a non-relational database (NoSQL) that was created to solve the storage problem of large-scale semi-structured and unstructured complex data. HBase database is a storage platform that can access and retrieve data randomly, storing structured and semi-structured data. Therefore, general websites can store web page content and log information in HBase database.

[0091] The logical storage structure of the HBase database includes tables, row keys, column families, and timestamps. Data in the HBase database is stored in the form of tables. In an HBase table, each row represents a data object, and each row is uniquely identified by a row key, which can be any string. A column family in HBase is a collection of columns. All column members in a column family have the same prefix, and the name of the column family must be a displayable string. Logically, HBase tables are stored in a column-oriented manner. In physical storage, HBase tables have the concept of Region partitions, which merge each data into a Region partition and store it by region.

[0092] In this embodiment, the row keys of the HBase table are the customer ID and the product ID. In physical storage, the HBase table is stored according to the Region partition. Therefore, according to the Region distribution of the HBase database, an Hfile file is generated. The Hfile file is the file organization form of HBase storage data.

[0093] S305, registering the Hfile file to the HBase database;

[0094] In the actual production environment, there is such a scenario: user data is located in the Hadoop distributed file system, and the business needs to regularly import this part of the massive data into the HBase database to perform random query update operations. If this scenario is processed by calling the write API, it is very likely to bring great write pressure to the RegionServer, including: affecting the stability of the cluster; consuming a large amount of CPU resources, bandwidth resources, memory resources, and IO resources, and competing with other businesses for resources; in some scenarios, causing cluster blocking. In view of the above problems, the HBase database provides another method to write data to the HBase cluster: bulkload technology. Bulkload first uses MapReduce to convert the cluster data to be written into Hfile files, and then directly loads these Hfile files into the online cluster.

[0095] In this embodiment, bulkload technology is used to register the Hfile file to the HBase database. After the open chain data corresponding to the target data is closed, new open chain data is generated according to the target data, and the new open chain data is written into the HBase database through the Hfile file, the changed zipper table in the database is shown in Table 2, where the closed chain data is the data corresponding to the end time of 20220907 and 20220908, and the open chain data is the data corresponding to the end time of 29991231, where 20220908 is the time of this business.

[0096] Table 2

[0097]

[0098] S306, acquiring multiple form data from bottom to top according to the open date field of the same primary key in the zipper table;

[0099] In this embodiment, the opening date is actually a character, and the characters are the same. Therefore, when obtaining the historical data of a product or system of a certain customer, the form data must be obtained from bottom to top, which is the data in the order from recent to far in time.

[0100] S307: Perform historical trend statistics based on the multiple form data.

[0101] In this embodiment, after obtaining multiple form data, when performing historical trend statistics on a product or system of a single customer, only data with an end time of a preset time can be counted, or data with an end time recursively can be counted, and it is not limited to open-link data.

[0102] This embodiment provides a method for processing a big data zipper table, which obtains the concurrency based on the Spark framework for processing the zipper table through the storage space occupied by the incremental data and the processing amount of the incremental data corresponding to each concurrent thread; performs prefix matching with the primary key for the incremental data corresponding to each concurrent thread, and obtains the target data with information changes relative to the open chain data under the information change key; performs a closed chain operation on the open chain data corresponding to the target data, and generates new open chain data according to the target data; generates an Hfile file through the open chain data according to the region distribution of the HBase database; registers the Hfile file to the HBase database; obtains multiple form data from bottom to top according to the open chain date field of the same primary key in the zipper table; and performs historical trend statistics according to the multiple form data. The method improves the processing efficiency of incremental data by calculating the concurrency based on the Spark framework, simplifies the data storage process by using the Hfile file, reduces the database pressure, performs historical trend statistics according to the open chain date field, and simply and quickly classifies the big data of the financial system, while improving the efficiency of historical trend statistics.

[0103] Figure 4 This is a schematic diagram of the structure of a large data zipper table processing device provided in an embodiment of the present application. The device of this embodiment can be in the form of software and / or hardware. Figure 4 As shown, a large data zipper table processing device 400 provided in an embodiment of the present application includes an acquisition module 401, a calculation module 402 and a processing module 403.

[0104] The acquisition module 401 is used to acquire incremental data of the zipper table, the primary key of the zipper table includes the customer identifier and the product identifier, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details;

[0105] The calculation module 402 is used to obtain the number of concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data;

[0106] The processing module 403 is used to process the zipper table according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table.

[0107] In a possible implementation, the computing module is further specifically configured to:

[0108] According to the storage space occupied by the incremental data and the processing volume of the incremental data corresponding to each concurrent thread, the concurrency number based on the Spark framework for processing the zipper table is obtained; wherein the processing volume corresponding to each concurrent thread is the same.

[0109] In a possible implementation, the processing module is further specifically configured to:

[0110] For the incremental data corresponding to each concurrent thread, prefix matching is performed with the primary key to obtain the target data under the information change key that has information changes relative to the open chain data;

[0111] Perform a closed-chain operation on the open-chain data corresponding to the target data, generate new open-chain data based on the target data, and store the new open-chain data in the HBase database.

[0112] In a possible implementation, the acquisition module is further specifically used for:

[0113] Construct a zipper table, which also includes a start time and an end time, wherein the end time is set to a preset time to indicate the latest data, and the start closure time is set to the time when the data changes.

[0114] In a possible implementation, the processing module is further specifically configured to:

[0115] The new open chain data is stored in the HBase database through bulkload.

[0116] In a possible implementation, the processing module is further specifically configured to:

[0117] Generate Hfile files with new open chain data according to the region distribution of HBase database;

[0118] Register the Hfile file to the HBase database.

[0119] In a possible implementation, the acquisition module is further specifically used for:

[0120] Get multiple form data from bottom to top based on the open date field of the same primary key in the zipper table;

[0121] Perform historical trend statistics based on multiple form data.

[0122] The big data zipper table processing device provided in this embodiment can be used to execute the above method embodiments. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0123] Figure 5 The hardware structure diagram of the big data zipper table processing device provided in the embodiment of the present application. Figure 5 As shown, the big data zipper table processing device 500 includes:

[0124] Processor 501 and memory 502;

[0125] Memory stores computer-executable instructions;

[0126] The processor executes the computer execution instructions stored in the memory 502, so that the electronic device executes the large data zipper table processing method as described above.

[0127] It should be understood that the processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor. The memory 502 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk, or an optical disk.

[0128] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement a large data zipper table processing method.

[0129] The embodiment of the present application also provides a computer program product, including a computer program, which implements a large data zipper table processing method when executed by a processor.

[0130] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0131] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0132] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing a large data zipper table, characterized in that: include: Acquire incremental data of a zipper table, wherein the primary key of the zipper table includes a customer ID and a product ID, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details; wherein the product ID under each customer ID has the same prefix as the customer ID; According to the storage space occupied by the incremental data, obtain the number of concurrency based on the Spark framework for processing the zipper table; For the incremental data corresponding to each concurrent thread, prefix matching is performed using the primary key to obtain target data under the information change key that has information changes relative to the open chain data; Performing a closing operation on the open chain data corresponding to the target data, generating new open chain data according to the target data, and storing the new open chain data in an HBase database; The storing of the new open chain data in the HBase database further comprises: Generate an Hfile file using the new open chain data according to the region distribution of the HBase database; Register the Hfile file to the HBase database.

2. The method according to claim 1, characterized in that The obtaining, according to the storage space occupied by the incremental data, a concurrency number based on the Spark framework for processing the zipper table includes: According to the storage space occupied by the incremental data and the processing volume of the incremental data corresponding to each concurrent thread, the concurrency number based on the Spark framework for processing the zipper table is obtained; wherein the processing volume corresponding to each concurrent thread is the same.

3. The method according to claim 1, characterized in that Before obtaining the incremental data of the zipper table, the method further includes: The zipper table is constructed, and the zipper table also includes a start time and an end time, wherein the end time is set to a preset time for indicating the latest data, and the start time is set to the time when the data changes.

4. The method according to claim 3, characterized in that: The method further comprises: According to the unzip date field of the same primary key in the zipper table, multiple form data are obtained from bottom to top; Historical trend statistics are performed based on the multiple form data.

5. A big data zipper table processing device, characterized in that: include: An acquisition module, used for acquiring incremental data of a zipper table, wherein the primary key of the zipper table includes a customer ID and a product ID, and the information change key of the zipper table includes at least one of consumption details, income details, and asset details; wherein the product ID under each customer ID has the same prefix as the customer ID; A calculation module, used for obtaining the number of concurrency based on the Spark framework for processing the zipper table according to the storage space occupied by the incremental data; A processing module processes the zipper table according to the incremental data corresponding to each concurrent number and the primary key and information change key of the zipper table; The processing module is specifically used to perform prefix matching with the primary key for the incremental data corresponding to each concurrent thread, and obtain target data under the information change key that has information changes relative to the open chain data; Performing a closing operation on the open chain data corresponding to the target data, generating new open chain data according to the target data, and storing the new open chain data in an HBase database; The processing module is further specifically used to generate an Hfile file through the new open chain data according to the region distribution of the HBase database; Register the Hfile file to the HBase database.

6. An electronic device comprising: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement a large data zipper table processing method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images