Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the steps that a target full scale is acquired from a data warehouse, and the target full scale comprises historical data of a first period; the target full scale is split to obtain a plurality of sub-tables, the sub-tables comprise historical data of a second period, and the second period is smaller than the first period; performing zipper processing on the historical data of the second period to obtain a local zipper result corresponding to each sub-table; and integrating the plurality of local zipper results to obtain a target zipper table. According to the invention, the manufacturing efficiency of the zipper table can be greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of data tables, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] A zipper table is a table that records all the changes in information from the beginning to the current state of things. Compared with a full table, a zipper table records the life cycle of data through the zipper start time and the zipper end time, reduces duplicate records, and can save storage space to the maximum extent.

[0003] In the related art, a zipper table is made by using a daily comparison method. If the number of sections is large, for example, there is a full year of data, each section needs to be compared once when making the zipper table, that is, 365 comparisons are required, which results in a long time consumption and low efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can greatly improve the efficiency of zipper table production.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method, including:

[0007] Obtaining a target full-scale table from a data warehouse, wherein the target full-scale table includes historical data of a first period;

[0008] Splitting the target full table to obtain a plurality of sub-tables, wherein the sub-tables include historical data of a second period, wherein the second period is smaller than the first period;

[0009] Performing zipper processing on the historical data in the second period to obtain a local zipper result corresponding to each of the sub-tables;

[0010] The plurality of local zipper results are integrated to obtain a target zipper table.

[0011] The present application provides a data processing device, including:

[0012] An acquisition module, configured to acquire a target full-scale table from a data warehouse, wherein the target full-scale table includes historical data of a first period;

[0013] A splitting module, configured to split the target full table to obtain a plurality of sub-tables, wherein the sub-tables include historical data of a second period, wherein the second period is smaller than the first period;

[0014] A processing module, used for performing zipper processing on the historical data in the second period to obtain a local zipper result corresponding to each of the sub-tables;

[0015] The integration module is used to integrate the multiple local zipper results to obtain a target zipper table.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] A memory for storing executable instructions;

[0018] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the data processing method provided in the embodiment of the present application when executed by a processor.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By splitting the target full scale and performing zipper processing on the multiple sub-tables obtained by the splitting, the local zipper results corresponding to each sub-table can be quickly obtained, thereby improving the efficiency of making the zipper table. Subsequently, the local zipper results corresponding to the multiple sub-tables can be integrated to quickly obtain the complete zipper table. That is to say, compared with the daily comparison method provided by the related technology for making a zipper table, the technical solution provided by the embodiment of the present application splits the target full scale into multiple sub-tables and performs zipper production on each sub-table. In this way, the processing pressure of the electronic device can be alleviated. After obtaining the local zipper results corresponding to the multiple sub-tables, the local zipper results corresponding to the multiple sub-tables can be integrated into a complete zipper table. In this way, the efficiency of making the zipper table can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a full representation of the stored user phone number information provided by the embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of the production process of a zipper watch provided by the related art;

[0025] Figure 3 is a schematic diagram of the architecture of a data processing system 100 provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application;

[0028] Figure 6 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0029] Figure 7 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0030] Figure 8 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0031] Fig. 9 It is a process schematic diagram of the data processing method provided in an embodiment of the present application;

[0032] Fig.10 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0033] Fig.11 It is a schematic diagram of the principle of the data processing method provided in the embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of dividing long-period data into different batches provided in an embodiment of the present application;

[0035] Fig.13 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0036] Fig.14 This is a schematic diagram of Hash data provided in an embodiment of the present application;

[0037] Fig.15 This is a schematic diagram of data after rank sorting provided in an embodiment of the present application;

[0038] Fig.16 It is a schematic diagram of subtracting rank values ​​provided in an embodiment of the present application;

[0039] Fig.17 is a schematic diagram of retaining data with a delta value other than 0 provided in an embodiment of the present application;

[0040] Fig.18 It is a schematic diagram of performing date offset on adjacent data provided in an embodiment of the present application;

[0041] Fig.19 It is a schematic diagram of the partial zipper results of 4 batches provided in the examples of the present application;

[0042] Fig. 20 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0043] Fig.21 This is a schematic diagram of deleting data with the same hash value provided by an embodiment of the present application;

[0044] Fig. 22 This is a schematic diagram of the result after deleting redundant data provided by an embodiment of the present application;

[0045] Fig.23 is a schematic diagram of adjusting the end time of an ineffective zipper provided in an embodiment of the present application;

[0046] Fig.24 It is a schematic diagram of the zipper result provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0048] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0050] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0053] 1) Full table: A table that records all data within the data synchronization cycle, regardless of whether the data has changed.

[0054] 2) Zipper table: A table that records all changes in information from the beginning to the current state of a thing. Compared with a full table, a zipper table records the life cycle of data through the zipper start time and the zipper end time, reduces duplicate records, and can save storage space to the maximum extent.

[0055] 3) Zipper start time: represents the start time when the data takes effect.

[0056] 4) Zipper end time: represents the end time when the data takes effect, used in conjunction with the zipper start time.

[0057] 5) Section date: the date on which business system data is extracted from the data warehouse.

[0058] 6) Section: a snapshot of the data. For the full scale, each section includes the full data up to the date of that section.

[0059] For consumer finance companies, they serve a large number of customers and have a massive amount of customer data, transaction data, and status data. Different data need to adopt different storage strategies in a targeted manner to find a balance between efficient service applications and cost savings. Compared with full tables, zipper tables can retain equivalent information while consuming less space. Therefore, zipper tables, as tables commonly used in data warehouses to record information about changes in things, are widely used in applications in various industries. The following introduces the production process of zipper tables provided by related technologies.

[0060] like Figure 1 As shown, Table A is a full table storing user phone number information. Figure 2 The production process of the zipper table in the solution provided by the related technology is shown, which mainly includes the following three steps:

[0061] 1. Zipper information initialization: such as Figure 2 .a, you can use Figure 1 The earliest section time, that is, the user information of the 2020-01-01 section is used as the first row of data, start_dt is the start time of the zipper, that is, 2020-01-01; end_dt is the end time of the zipper, for example, it can be filled with 9999-12-31, which means that the data in this row is in effect at the latest point in time.

[0062] 2. Closed chain operation: On each batch run date, the user information of that date is compared with the existing effective data in the zipper table (i.e., end_dt = "9999-12-31" data). If the information changes, the zipper end time end_dt is set as the batch run date. Figure 2 As shown in .b, the user's telephone number in the 2020-01-02 section changes, so end_dt in the zipper table is changed from 9999-12-31 to 2020-01-02.

[0063] 3. Open new chain operation: There are two situations that will cause the zipper table to open a new chain. 1) If there is a closed chain of data, a new chain needs to be added as valid data at the same time, such as Figure 2 As shown in .b, a new data with a phone number of 111-222 is added, the zipper start time start_dt is 2020-01-2, and the zipper end time end_dt is 999-12-31. 2) If a new user is added, a new valid zipper data for the user is added, just like the initialization information in step 1.

[0064] However, in the process of implementing the embodiments of the present application, the applicant discovered that the solution provided by the related technology has low efficiency in zipper production for historical data. If the number of sections is large, for example, there is a full year of data (i.e., from 2020-01-01 to 2020-12-31, including 366 section dates), the production of the zipper table requires comparing each section once, that is, 365 comparisons are required (for example, the 2020-01-01 information can be used for initialization, and then the daily data is compared with the zipper table), which is time-consuming and inefficient.

[0065] In view of this, the embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium and computer program product, which can greatly improve the efficiency of making zipper tables. The electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as a server, or as a terminal device, or implemented by a server and a terminal device in collaboration. The following is an example of the data processing method provided by the embodiments of the present application being implemented by a server and a terminal device in collaboration.

[0066] For example, see Figure 3 , Figure 3 1 is a schematic diagram of the architecture of a data processing system 100 provided in an embodiment of the present application, in order to realize an application that supports a significant improvement in the efficiency of making zipper tables, such as Figure 3As shown, the data processing system 100 includes: a server 200, a network 300, a terminal device 400 and a data warehouse 500, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, the terminal device 400 is a user-associated terminal device, a client 410 runs on the terminal device 400, and the server 200 can be a background server of the client 410.

[0067] In some embodiments, the client 410 may send a zipper table creation request to the server 200 through the network 300 , wherein the zipper table creation request may carry an identifier of a target full table (eg, full table 1 selected by the user). After receiving the zipper table creation request sent by the client 410, the server 200 can first extract the carried identifier from the zipper table creation request, and obtain the matched target full table from the data warehouse 500 based on the identifier, wherein the target full table can record the historical data of the target object (for example, user 1) within a first period (for example, one year). Then, the server 200 can split the obtained target full table to obtain multiple sub-tables, wherein each sub-table records the historical data of the target object within a second period (for example, one month or one quarter), wherein the second period is smaller than the first period. Subsequently, the server 200 can perform zipper creation on the historical data within the second period recorded in each sub-table to obtain local zipper results corresponding to each sub-table. Finally, the server 200 can integrate multiple local zipper results to obtain a target zipper table, and return the integrated target zipper table to the client 410 through the network 300 for presentation in the human-computer interaction interface of the client 410. In this way, the efficiency of zipper table creation can be greatly improved.

[0068] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0069] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0070] For example, Figure 3The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0071] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a server as an example, see Figure 4 , Figure 4 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, Figure 4 The electronic device 600 shown includes: at least one processor 610, a memory 640, and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. It is understood that the bus system 630 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 630 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 630 is not described in detail. Figure 4 Various buses are labeled as bus system 630 .

[0072] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The memory 640 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 640 may optionally include one or more storage devices that are physically remote from the processor 610.

[0074] The memory 640 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 640 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, memory 640 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0076] Operating system 641, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0077] A network communication module 642, for reaching other computing devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0078] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 4 The data processing device 643 stored in the memory 640 is shown, which can be software in the form of a program or plug-in, including the following software modules: an acquisition module 6431, a splitting module 6432, a processing module 6433, an integration module 6434, and a determination module 6435. These modules are logical, and thus can be arbitrarily combined or further split according to the functions implemented. It should be noted that in Figure 4 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the data processing device 643 excludes the implementation that only includes the acquisition module 6431, the splitting module 6432, the processing module 6433 and the integration module 6434. The functions of each module will be explained below.

[0079] The data processing method provided by the embodiment of the present application will be specifically described below in conjunction with the exemplary application and implementation of the server provided by the embodiment of the present application. Before describing the data processing method provided by the embodiment of the present application, the application scenario of the data processing method provided by the embodiment of the present application is first introduced.

[0080] For example, see Figure 5 , Figure 5is a schematic diagram of an application scenario of the data processing method provided in the embodiment of the present application, such as Figure 5 As shown, after the server receives a zipper table creation request sent by a user (such as a data analyst) through a terminal device, it can first extract the carried identifier from the zipper table creation request, and obtain the target full table from the data warehouse based on the identifier, wherein the target full table can record the historical data (i.e., 365 historical data) of the target object (such as user 1) within a first period (such as 1 year), then the server can split the obtained target full table (such as full table 1) into multiple sub-tables, wherein each sub-table records the historical data of the target object within a second period (such as 1 month or 1 quarter), then the server can zipper the historical data within the second period recorded in each sub-table to obtain the local zipper results corresponding to each sub-table, finally, the server can integrate the multiple local zipper results corresponding to the multiple sub-tables one by one to obtain the target zipper table, and return the integrated target zipper table to the terminal device associated with the data analyst, so that the terminal device calls the client's human-computer interaction interface for presentation, thereby greatly improving the efficiency of zipper table creation.

[0081] The following will be combined Figure 6 The data processing method provided in the embodiment of the present application is described in detail.

[0082] For example, see Figure 6 , Figure 6 is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 6 The steps shown are explained.

[0083] In step 101, a target full scale is obtained from a data warehouse.

[0084] In some embodiments, when the server receives a zipper table creation request sent by a data analyst through a terminal device, it can first extract the carried identifier from the zipper table creation request, and then, based on the identifier, obtain the target full table (for example, full table 1) that matches the identifier from the data warehouse, wherein the target full table may include the historical data of the target object (for example, user 1) within the first period (for example, 1 year), for example, the target full table may record 365 pieces of historical data of user 1 from 2021.01.01 to 2021.12.31 for the whole year.

[0085] In step 102, the target full scale is split into multiple sub-scales.

[0086] In some embodiments, step 102 can be implemented in the following manner: determine a batch value (i.e., batch value) based on the amount of historical data in the first cycle, where the batch value is positively correlated with the amount of data; split the target full table into multiple sub-tables with the same number as the batch value. For example, when the batch value is 4, the target full table can be split into 4 sub-tables, where each sub-table records the historical data of the target object within 1 quarter. For example, assuming that the target full table records the historical data of user 1 from 2021.01.01 to 2021.12.31 throughout the year 365 pieces of historical data, the above data can be divided into 4 batches by quarter, where each batch includes 3 months of historical data; when the batch value is 12, the target full table can be split into 12 sub-tables, where each sub-table records the historical data of the target object within 1 month. For example, assuming that the target full table records 365 pieces of historical data of user 1 from 2021.01.01 to 2021.12.31, the above data can be divided into 12 batches by month, where each batch includes 1 month of historical data.

[0087] For example, the above-mentioned determination of the batch value based on the amount of historical data in the first period can be achieved in the following manner: obtaining the amount of historical data in the first period and the performance parameters (such as computing power) of the electronic device used to process the historical data in the first period; determining the batch value based on the amount of historical data in the first period and the performance parameters of the electronic device. In other words, the batch value can be comprehensively determined based on the amount of data and the performance parameters of the electronic device. If the amount of data is large and the performance parameters of the electronic device are weak, then the batch value should be set larger, that is, the larger the batch value, the smaller the amount of data to be processed by each electronic device. Conversely, the batch value can be set smaller to make full use of the performance parameters of the electronic device.

[0088] In step 103, the historical data of the second period is subjected to zipper processing to obtain a local zipper result corresponding to each sub-table.

[0089] In some embodiments, the zipper process may include: hash processing, sorting, subtraction processing, and offset operations on the historical data of the second cycle.

[0090] In some embodiments, the historical data (i.e., 30 pieces of historical data) within the second period (e.g., 1 month) may include the historical data of the target object within multiple third periods (e.g., 1 day), wherein the third period is smaller than the second period. Figure 7 , Figure 7 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 7 As shown, Figure 6 Step 103 shown may be performed by Figure 7 Steps 1031 to 1035 shown in the figure are implemented by combining Figure 7 The steps shown are explained.

[0091] In step 1031, hash processing is performed on the historical data of each third period to obtain a hash value corresponding to each third period.

[0092] Here, the hash value can be used to characterize the identification of the data of the target object (eg, user 1) within each third period (eg, 1 day).

[0093] In some embodiments, the historical data may include field values ​​corresponding to multiple fields respectively, and step 1031 can be implemented in the following manner: determine the target field from the multiple fields included in the historical data within the third period; combine the field values ​​corresponding to the multiple target fields respectively, and hash the combined result to obtain the hash value corresponding to the third period.

[0094] For example, taking historical data as financial data, the characteristics of financial data determine that there are often many target fields, and field-by-field comparison is very time-consuming and inefficient. Therefore, the embodiment of the present application can first select the target field (for example, including telephone number, principal balance, interest balance, penalty interest balance, etc.) from multiple fields of financial data, and combine the field values ​​corresponding to the selected multiple target fields, and then perform hash calculation on the combined result to obtain the corresponding hash value. For example, algorithms such as MD5, SHA256, and SHA512 can be used to perform hash calculation on the combined result, and the embodiment of the present application does not specifically limit this.

[0095] In addition, it should be noted that the above target fields are not randomly selected, but are selected based on actual needs. For example, the original account table has 20 fields, and the business needs only require 5 of them. Then these 5 fields are the target fields. Therefore, the target field is not a specific field, but a required field set, which is selected according to the specific situation.

[0096] In step 1032, the multiple hash values ​​are sorted to obtain a ranking corresponding to each third period.

[0097] In some embodiments, after obtaining the hash value representing the data in each third period (e.g., 1 day), multiple hash values ​​may be ranked. The purpose of the ranking is to map the same hash value to the same value (i.e., the corresponding ranking is the same), and different hash values ​​correspond to different rankings. For example, the rank function or dense_rank function in the database language (SQL, Structured Query Language) may be used to rank multiple hash values ​​to obtain the ranking (i.e., rank value) corresponding to each third period.

[0098] In step 1033, the rankings of two adjacent third periods are subtracted in sequence to obtain the difference value corresponding to each third period.

[0099] In some embodiments, the multiple third periods may be N third periods, where N is a positive integer greater than 1, and step 1033 may be implemented in the following manner: sort the N third periods in ascending order according to date to obtain a sorting result; determine the difference corresponding to the first third period in the sorting result as 1, and iterate i to perform the following processing, where i is an integer variable and satisfies 2≤i≤N: subtract the ranking of the i-th third period from the ranking of the i-1th third period, and use the obtained difference as the difference corresponding to the i-th third period, for example, the difference may be recorded as delta.

[0100] For example, take the third period as 1 day, see Fig.16 , Fig.16 It is a schematic diagram of subtracting the rank value provided in the embodiment of the present application, such as Fig.16 As shown, the delta value corresponding to the data in the first row (i.e., the first third period) can be recorded as 1. Then, the rank value corresponding to the data in the second row (i.e., the second third period) can be subtracted from the rank value corresponding to the data in the first row, and the difference obtained (i.e., 2-1=1) can be used as the delta value corresponding to the data in the second row. Subsequently, the rank value corresponding to the data in the third row (i.e., the third third period) can be subtracted from the rank value corresponding to the data in the second row, and the difference obtained (i.e., 2-2=0) can be used as the delta value corresponding to the data in the third row, and so on, to obtain the delta value corresponding to each row (i.e., each third period) of data.

[0101] In step 1034, an offset operation is performed on the historical data of the target third period to obtain a local zipper result corresponding to the sub-table.

[0102] Here, the difference corresponding to the target third period is not zero. That is to say, after obtaining the difference (i.e., delta value) corresponding to each third period (e.g., 1 day), the historical data of the target object (e.g., user 1) in the third period with a delta value of 0 can be eliminated. This is because the delta value is not 0, which means that the historical data recorded in the current row (e.g., 2020-01-03) is different from the historical data recorded in the previous row (e.g., 2020-01-02), that is, the information has changed, so it is retained. When the delta value is 0, it means that the historical data recorded in the current row is consistent with the historical data recorded in the previous row, that is, no information about data changes is provided, so it needs to be eliminated from the historical data of the second period (e.g., 1 month).

[0103] In some embodiments, the local zipper result may include the zipper start time and zipper end time corresponding to the historical data of the target object in multiple target third periods, and the multiple target third periods may be M third periods, where M is a positive integer greater than 1 and M is less than N. The above step 1035 can be implemented in the following way: Iterate j to perform the following processing, where j is an integer variable and satisfies 1≤j≤M: use the start date corresponding to the j-th target third period as the zipper start time corresponding to the historical data of the target object in the j-th target third period; use the start date corresponding to the j+1-th target third period as the zipper end time corresponding to the historical data of the target object in the j-th target third period; where, when j=M, use the start date corresponding to the M-th target third period as the zipper start time corresponding to the historical data of the target object in the M-th target third period, and use the set date (for example, 2099-12-31) as the zipper end time corresponding to the historical data of the target object in the M-th target third period.

[0104] For example, take the third period as 1 day, see Fig.18 , Fig.18 Schematic diagram of performing date offset on adjacent data provided by an embodiment of the present application, such as Fig.18 As shown, after removing the data with a delta value of 0, the adjacent data can be offset by date. For example, the section date of the X+1th row can be assigned to the Xth row, that is, the section date of the Xth row is the zipper start time start_dt of the historical data in the Xth row, and the section date of the X+1th row is the zipper end time end_dt of the historical data in the Xth row. In particular, since the data is sorted in ascending order by section date, the zipper end time of the data in the last row can be set to 2099-12-31, that is, it is in effect. In this way, the local zipper results corresponding to each sub-table can be generated.

[0105] In step 104, multiple local zipper results are integrated to obtain a target zipper table.

[0106] In some embodiments, see Figure 8 , Figure 8 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 8 As shown, Figure 6 Step 104 shown may be performed by Figure 8 Steps 1041 to 1044 shown are implemented by combining Figure 8 The steps shown are explained.

[0107] In step 1041, multiple local zipper results are integrated to obtain an integrated result.

[0108] In some embodiments, since the partial zipper results of each batch are scattered in different sub-tables, the data in different sub-tables need to be integrated before the next operation. For example, the union all method in SQL can be used to aggregate the partial zipper results scattered in different sub-tables.

[0109] In step 1042, the integration result is sorted in ascending order according to the date corresponding to the date field to obtain a sorting result.

[0110] In some embodiments, the historical data may include a date field. After integrating multiple partial zipper results to obtain an integrated result, the historical data in the integrated result may be sorted in ascending order according to the date corresponding to the date field.

[0111] In step 1043, duplicate historical data with the same hash value in the sorting result is deleted to obtain a subset of historical data.

[0112] Here, the historical data subset includes the historical data remaining after deleting duplicate historical data in the sorting result.

[0113] In some embodiments, each piece of historical data in the sorting result may correspond to a unique hash value, and step 1043 may be implemented in the following manner: for multiple pieces of historical data with the same hash value in the sorting result, retain the piece of historical data with the earliest zipper start time. For example, assuming that the hash values ​​of historical data 1 and historical data 2 in the sorting result are the same, and the zipper start time of historical data 1 is earlier than the zipper start time of historical data 2, only historical data 1 may be retained in the sorting result, i.e., historical data 2 may be deleted from the sorting result.

[0114] For example, for the integrated data, if the Hash value of the data is the same, it means that the field information is completely consistent and has not changed, so the data sorted first is retained and the data sorted later is deleted. This is because the zipper start time of the data sorted first is earlier than the zipper start time of the data sorted later. For example, the lag function in SQL can be used to obtain the Hash value of the current data. If it is the same as the Hash value of the previous data, the current data is marked for deletion. In special cases, if the Hash values ​​of multiple consecutive data are the same, the processing method is still the same. When each data is compared with the data sorted first, it will mark itself for deletion, and only the data with the earliest zipper start time will be retained.

[0115] In step 1044, the ineffective zipper end time of some historical data in the historical data subset is corrected to obtain a target zipper table.

[0116] Here, part of the historical data refers to the historical data whose zipper end time is a set date, that is, the specific historical data described below.

[0117] In some embodiments, step 1044 can be implemented in the following manner: for specific historical data in the historical data subset, when the zipper start time of the next historical data of the specific historical data is not a null value, the zipper start time of the next historical data is used as the zipper end time of the specific historical data; when the zipper start time of the next historical data of the specific historical data is a null value, the zipper end time of the specific historical data is kept unchanged, wherein the specific historical data is the historical data whose zipper end time is a set date.

[0118] For example, see Fig.23 , Fig.23 is a schematic diagram of adjusting the end time of the ineffective zipper provided in an embodiment of the present application, such as Fig.23 As shown, since the partial zipper results produced by multiple batches are integrated, there will be a situation where the zipper end time end_dt of multiple data with the same primary key (such as user id) is 2099-12-31, so it is necessary to correct the invalid zipper end time. The specific process is as follows: Assuming that the zipper end time end_dt of the Xth row of data is 2099-12-31, take the zipper start time start_dt of the X+1th row of data. If the zipper start time start_dt of the X+1th row of time is not a null value, then use it as the result to overwrite the zipper end time end_dt of the Xth row of data; if the start_dt of the X+1th row of data is a null value, then the end_dt of the Xth row of data will not be changed. For example Fig.23If the zipper end time end_dt of the 3rd row of data is 2099-12-31, the zipper start time start_dt of the 4th row of data can overwrite the end_dt of the 3rd row of data. If the start_dt of the 6th row of data is a null value, the end_dt of the 5th row of data will not be changed, that is, the value of the end_dt of the 5th row of data remains unchanged.

[0119] The data processing method provided in the embodiment of the present application can quickly obtain the local zipper results corresponding to each sub-table by splitting the target full-scale table and zipping each sub-table obtained by the split, and then integrate the local zipper results corresponding to each sub-table to obtain the target zipper table. In other words, by breaking down the task into smaller parts, the processing pressure of the cluster is alleviated, and then the intermediate results are integrated into a complete zipper table, thereby avoiding the low efficiency problem of daily comparison in related technologies and greatly improving the efficiency of zipper table production.

[0120] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0121] In the context of big data, the data time period spans a long period, ranging from a few months to several years; the data volume is huge, with hundreds of millions or even tens of billions of data. When processing long-term and massive data, the cluster is usually unable to process the entire amount of data at one time, and the efficiency of running batches of data on a daily basis is too slow. In view of this, the embodiment of the present application provides a set of standard processing procedures to break down and refine the tasks, alleviate the processing pressure of the cluster, and then integrate the intermediate results to obtain the full amount of zipper data of the target. The data processing method provided in the embodiment of the present application is described in detail below.

[0122] In some embodiments, see Fig. 9 , Fig. 9 is a process diagram of the data processing method provided in the embodiment of the present application, such as Fig. 9 As shown, the data processing method provided in the embodiment of the present application mainly includes three modules, namely, a historical data batch module, a zipper quick production module and a zipper integration module. The functions of these three modules are specifically described below.

[0123] In some embodiments, the historical data batching module is mainly used to process long-period data in batches. For example, assuming that the time range of the zipper table to be constructed is from 2021.01.01 to 2021.12.31 throughout the year, processing 365 days of data at one time will put a lot of pressure on the cluster. Therefore, the long-period data can be divided into batches to break down large tasks into smaller ones and improve batch running efficiency.

[0124] For example, you can manually divide the continuous long-period data into batches. For example, you can divide the whole year from 2021.01.01 to 2021.12.31 into 12 batches by month, or divide it into 4 batches by quarter. Fig.10 The process of dividing continuous long-period data is explained in detail.

[0125] In step 201, the total batch value of each batch is determined.

[0126] In some embodiments, the number of batches to be processed can be determined first, that is, the batch value. This value is a configuration parameter and needs to be comprehensively determined based on the cluster situation and data volume. If the data volume is large and the cluster computing power is small, the batch value should be set larger. The larger the batch value, the smaller the amount of data processed in each batch, and the longer it takes to process all batches. Otherwise, the batch value should be set smaller.

[0127] It should be noted that because different companies and different scenarios have different requirements for resources for table calculations, it is difficult to have a fixed standard for determining the batch value. Developers need to determine the number of batches based on specific circumstances. The basic principle is that if resources are sufficient, small batches are used (small batches mean more data in each batch), and if resources are tight, large batches are used (i.e. large batches mean less data in each batch).

[0128] In step 202, the ntile function in SQL is used to pass the batch value in step 201 into the function, and the batch value corresponding to each date is output.

[0129] In some embodiments, the ntile function in SQL can be used to pass the batch value determined in step 201 into the function, and the output result is the batch result corresponding to each date, that is, each date is labeled with a batch in ascending or descending order. Fig.11 As shown, for the convenience of display, the embodiment of the present application batches the date range from 2021-01-01 to 2021-01-16, and divides the results into batch = 4 batches. Among them, the function of the ntile function is to group (or bucket) mark the data set. For example, assuming there are 100 rows of data, and you want to divide the data set into 10 groups, the ntile function will output the grouping results of each detailed data, for example, it can be based on the results after ascending or descending order.

[0130] In step 203, the output results of step 202 are aggregated by batches to obtain the minimum date and the maximum date, wherein the minimum date is the start date of each batch and the maximum date is the end date.

[0131] In some embodiments, for the batch labels corresponding to each date output in step 202, an aggregation operation can be performed according to the batches, and the maximum date and the minimum date corresponding to each batch are calculated to obtain a closed interval range [minimum date, maximum date], and this date is the final corresponding [start_dt, end_dt] of each batch. It should be noted that the aggregation operation here is to directly aggregate (group) the batch field, and then use the max() function and the min() function to calculate the maximum and minimum values.

[0132] In step 204, the start time and end time of each batch are calculated.

[0133] In some embodiments, see Fig.12 , Fig.12 is a schematic diagram of dividing long-period data into different batches provided in an embodiment of the present application, such as Fig.12 As shown, the data for the whole year from 2021.01.01 to 2021.12.31 can be divided into 12 batches by month, or into 4 batches by quarter.

[0134] In some embodiments, the zipper rapid production module is mainly used to process the block data output by the historical data batch module to generate local zipper results. Zipper data of an indefinite interval can be obtained by running the batch only once, thereby avoiding the defects of daily calculation in the conventional zipper generation mode.

[0135] For example, see Fig.13 , Fig.13 is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Fig.13 The steps shown are explained.

[0136] In step 301, in each batch, the target field information is integrated and hashed.

[0137] In some embodiments, the characteristics of financial data determine that there are often many target fields, and field-by-field comparison is time-consuming and inefficient. Therefore, the embodiment of the present application can combine the target fields and calculate the hash value, where the hash value is equivalent to the unique ID of each piece of data. The above combination can be implemented using the concat function in SQL to convert different information into text form. Hash is a set of algorithms. Common hash algorithms can be MD5, SHA256, SHA512, etc. If the target fields are all the same, then the hash values ​​are the same. If there is one piece of information that is different, then the hash values ​​are different. Fig.14 As shown in the figure, the data framed by the same style of lines have the same hash value. Fig.14The hash value shown in is just a simplified example, and the actual hash value is more complex.

[0138] In addition, it should be noted that the above target fields are not randomly selected, but are selected based on actual needs. For example, the original account table has 20 fields, and the business needs only require 5 of them. Then these 5 fields are the target fields. Therefore, the target field is not a specific field, but a required field set, which is selected according to the specific situation.

[0139] In step 302, the hash results are sorted by rank.

[0140] In some embodiments, Fig.15 As shown, after calculating the Hash value of the target field combination, the Hash result can be ranked. That is, after having the Hash value representing each data, the Hash result can be ranked. The purpose of sorting is to map the same Hash value to a numerical value, and different data has different rank values.

[0141] It should be noted that the above rank sorting can be implemented through the rank function in SQL, or through the dense_rank function.

[0142] In step 303, the adjacent data rank results are subtracted and the difference is recorded as delta.

[0143] In some embodiments, Fig.16 As shown, after the Hash results are ranked, the difference between the ranks of adjacent data can be taken. For example, the data can be sorted in ascending order first by date, and then the difference between the rank of each row of data after sorting and the rank of the previous row of data is calculated, and the result is recorded as delta, where the delta of the first row of data is 1-0=1.

[0144] In step 304, delta non-zero data is extracted.

[0145] In some embodiments, Fig.17 As shown in the figure, after obtaining the delta corresponding to each row of data, the data with non-zero delta can be extracted. Non-zero delta indicates that the data in this row is different from the data in the previous row, that is, the information has changed, so it is retained. Similarly, if delta is 0, it means that the data in this row is consistent with the data in the previous row, and no information about data changes is provided, so it is removed.

[0146] In step 305, adjacent data are offset in date.

[0147] In some embodiments, Fig.18As shown in the figure, after removing the data with delta 0, the date offset can be performed on the adjacent data in the remaining data (for simplicity, Fig.18 The middle field information is hidden in the zipper. For example, the section date of the X+1th row of data can be assigned to the Xth row of data, where the section date of the Xth row of data is the zipper start time, and the section date of the X+1th row is the zipper end time. In particular, since the data is arranged in ascending order according to the section date, the zipper end time of the last row of data can be set to 2099-12-31, which is the effective state.

[0148] In step 306, the local zipper result is output.

[0149] In some embodiments, Fig.19 As shown, for the time interval from 2021-01-01 to 2021-04-30, a long period of 4 months of data (for simplicity, Fig.19 The information field in the middle is hidden), it can be divided into 4 batches, that is, one batch for each natural month, and the corresponding 4 partial zipper results can be generated after the zipper fast production module.

[0150] In some embodiments, the zipper integration module is mainly used to process the local zipper data output by the zipper rapid production module, integrate the local zipper data, remove redundant data, correct the zipper end time end_dt of critical non-valid data, and finally output the complete full zipper data.

[0151] For example, see Fig. 20 , Fig. 20 is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Fig. 20 The steps shown are explained.

[0152] In step 401, multiple local zipper results are integrated.

[0153] In some embodiments, since the output results of the zipper rapid production module are partial zipper results of each batch, that is, the data is scattered in different tables, it is necessary to first collect the data in different tables together before proceeding to the next step of operation. For example, the union all method in SQL can be used to collect the data in different tables (that is, partial zipper results) together.

[0154] In step 402, the integration results are sorted in ascending order.

[0155] In some embodiments, after the data in different tables are integrated together, the integrated results may be sorted in ascending order according to date, that is, data with earlier dates are sorted higher.

[0156] In step 403, duplicate data with the same Hash value is deleted.

[0157] In some embodiments, Fig.21 As shown, for the data sorted in ascending order by date, if the Hash value of the data is the same, it means that the field information is exactly the same and has not changed, then the data sorted first will be retained and the data sorted later will be deleted. This is because the zipper start time start_dt of the data sorted first is earlier than the start_dt of the data sorted later. The specific implementation is as follows: You can use the lag function in SQL to get the Hash value of the data sorted first. If it is the same as the Hash value of the previous data, you can delete this data. In special cases, if the Hash values ​​of multiple consecutive data are the same, the processing method is still the same. When the Hash value of each data is compared with the Hash value of the data sorted first, it will mark itself as deleted. In the end, only the data with the smallest start_dt will be retained. The data results after deletion are as follows. Fig. 22 shown.

[0158] In step 404, the ineffective zipper end time is corrected.

[0159] In step 405, the complete zipper table is output.

[0160] In some embodiments, Fig.23 As shown, since the partial zipper results produced in multiple batches are integrated, there will be a situation where the zipper end time end_dt of multiple data with the same primary key is 2099-12-31. Therefore, it is necessary to correct the invalid zipper end time. The specific process is as follows: Assuming that the zipper end time end_dt of the Xth row of data is 2099-12-31, take the zipper start time start_dt of the X+1th row of data. If the zipper start time start_dt of the X+1th row of data is not a null value, then use this as the result to overwrite the zipper end time end_dt of the Xth row of data. If the zipper start time start_dt of the X+1th row of data is a null value, the zipper end time end_dt of the Xth row of data will not be changed. The specific implementation can be achieved by using the lead function in SQL to obtain the start_dt of the X+1th row of data. The result after processing is as follows Fig.24 shown.

[0161] In other words, the embodiment of the present application proposes a systematic solution for the rapid production of historical zipper tables in big data scenarios, which can balance efficiency and cluster resources, and can flexibly adjust the batch size based on the needs of different scenarios to adapt to its own needs. At the same time, a local zipper rapid production algorithm is also proposed to avoid the low efficiency problem of daily comparison in related technologies. In addition, a local zipper result integration algorithm is also proposed to turn big problems into small problems, and then summarize small results (i.e., local zipper results) into complete results (i.e., complete zipper tables).

[0162] In summary, the data processing method provided in the embodiment of the present application has the following beneficial effects:

[0163] 1. The embodiment of the present application proposes a systematic solution for the rapid construction of historical zipper tables in big data scenarios. It is divided into three modules and processed sequentially, which can greatly improve the efficiency of historical zipper table production. At the same time, it can be flexibly processed in batches under different cluster resource constraints, breaking large into small, and then merging small into large, to obtain the target result.

[0164] 2. The embodiment of the present application proposes a calculation method for a zipper table quick production module. Due to the characteristics of financial data, there are many information fields that need to be recorded. Therefore, it is proposed to use Hash values ​​to represent each row of specific data, and then rank sorting is performed on the Hash results. In essence, it is to map the data numerically (that is, mapping text to numerical values). The mapped numerical results can be calculated and compared with adjacent data, and then a zipper table can be generated based on the comparison results.

[0165] 3. The embodiment of the present application proposes a zipper integration module, which integrates the local zipper results of the zipper rapid production module to generate the final target result.

[0166] The following is a description of an exemplary structure of the data processing device 643 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 4 As shown, the software modules stored in the data processing device 643 of the memory 640 may include: an acquisition module 6431 , a splitting module 6432 , a processing module 6433 and an integration module 6434 .

[0167] The acquisition module 6431 is used to acquire the target full table from the data warehouse, wherein the target full table includes the historical data of the first period; the splitting module 6432 is used to split the target full table to obtain multiple sub-tables, wherein the sub-tables include the historical data of the second period, wherein the second period is smaller than the first period; the processing module 6433 is used to perform zipper processing on the historical data of the second period to obtain the local zipper results corresponding to each sub-table; the integration module 6434 is used to integrate the multiple local zipper results to obtain the target zipper table. In some embodiments, the zipper processing includes: hash processing, sorting, subtraction processing and offset operation of the historical data of the second period.

[0168] In some embodiments, the data processing device 643 also includes a determination module 6435, which is used to determine the batch value based on the data volume of the historical data of the first cycle, wherein the batch value is positively correlated with the data volume; and a splitting module 6432, which is used to split the target full-volume table into multiple sub-tables of the same number as the batch value.

[0169] In some embodiments, the acquisition module 6431 is further used to acquire the data volume of the historical data of the first cycle and the performance parameters of the electronic device used to process the historical data of the first cycle; the determination module 6435 is further used to determine the batch value based on the data volume of the historical data of the first cycle and the performance parameters of the electronic device.

[0170] In some embodiments, the historical data of the second period includes historical data of multiple third periods, wherein the third period is smaller than the second period; the processing module 6433 is further used to perform hash processing on the historical data of each third period to obtain a hash value corresponding to each third period; sort the multiple hash values ​​to obtain a ranking corresponding to each third period; subtract the rankings of two adjacent third periods in turn to obtain a difference corresponding to each third period; perform an offset operation on the historical data of the target third period to obtain a local zipper result corresponding to the sub-table, wherein the difference corresponding to the target third period is not zero.

[0171] In some embodiments, the historical data includes field values ​​corresponding to multiple fields respectively; the processing module 6433 is also used to determine the target field from the multiple fields included in the historical data of the third period; the field values ​​corresponding to the multiple target fields are combined, and the combined result is hashed to obtain the hash value corresponding to the third period.

[0172] In some embodiments, the multiple third periods are N third periods, where N is a positive integer greater than 1; the processing module 6433 is also used to sort the N third periods in ascending order according to date to obtain a sorting result; the difference corresponding to the first third period in the sorting result is determined to be 1, and the following processing is performed iteratively i, where i is an integer variable and satisfies 2≤i≤N: the ranking of the i-th third period is subtracted from the ranking of the i-1-th third period, and the obtained difference is used as the difference corresponding to the i-th third period.

[0173] In some embodiments, the local zipper result includes the zipper start time and zipper end time corresponding to the historical data of multiple target third periods, and the multiple target third periods include M third periods, where M is a positive integer greater than 1 and M is less than N; the processing module 6433 is also used to iterate j to perform the following processing, where j is an integer variable and satisfies 1≤j≤M: the start date corresponding to the j-th target third period is used as the zipper start time corresponding to the historical data of the target object in the j-th target third period; the start date corresponding to the j+1-th target third period is used as the zipper end time corresponding to the historical data of the target object in the j-th target third period; where, when j=M, the set date is used as the zipper end time corresponding to the historical data of the target object in the M-th target third period.

[0174] In some embodiments, the integration module 6434 is also used to integrate multiple local zipper results to obtain an integrated result; sort the integrated result in ascending order by date to obtain a sorted result; delete duplicate historical data with the same hash value in the sorted result to obtain a subset of historical data; correct the invalid zipper end time of some historical data in the subset of historical data to obtain a target zipper table.

[0175] In some embodiments, each piece of historical data in the sorting result corresponds to a unique hash value; the integration module 6434 is further used to retain the piece of historical data with the earliest zip start time for multiple pieces of historical data with the same hash value in the sorting result.

[0176] In some embodiments, the integration module 6434 is also used to, for specific historical data in the historical data subset, use the zipper start time of the next historical data as the zipper end time of the specific historical data when the zipper start time of the next historical data of the specific historical data is not a null value; and keep the zipper end time of the specific historical data unchanged when the zipper start time of the next historical data of the specific historical data is a null value, wherein the specific historical data is the historical data whose zipper end time is a set date.

[0177] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 6 , Figure 7 ,or Figure 8 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0178] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the data processing method described above in the embodiment of the present application.

[0179] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the data processing method provided by the present application embodiment, for example, Figure 6 , Figure 7 ,or Figure 8 The data processing method is shown.

[0180] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0181] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0182] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0183] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a target full-scale table from a data warehouse, wherein the target full-scale table includes historical data of a first period; Splitting the target full table to obtain a plurality of sub-tables, wherein the sub-tables include historical data of a second period, wherein the second period is smaller than the first period; Performing zipper processing on the historical data of the second period to obtain a local zipper result corresponding to each of the sub-tables; The plurality of local zipper results are integrated to obtain a target zipper table.

2. The method according to claim 1, characterized in that The target full scale is split to obtain multiple sub-scales, including: Determine a batch value according to the data volume of the historical data of the first cycle, wherein the batch value is positively correlated with the data volume; The target full table is split into a plurality of sub-tables having the same number as the batch values.

3. The method according to claim 2, characterized in that The step of determining the batch value according to the amount of historical data in the first cycle includes: Acquire the data volume of the historical data of the first period and the performance parameters of the electronic device used to process the historical data of the first period; The batch value is determined according to the data volume of the historical data of the first cycle and the performance parameters of the electronic device.

4. The method according to claim 1, characterized in that The historical data of the second period includes historical data of a plurality of third periods, wherein the third period is smaller than the second period; The performing zipper processing on the historical data of the second period to obtain a local zipper result corresponding to each of the sub-tables includes: Performing hash processing on the historical data of each of the third periods to obtain a hash value corresponding to each of the third periods; Sorting the plurality of hash values ​​to obtain a ranking corresponding to each of the third periods; Subtracting the rankings of two adjacent third periods in sequence to obtain a difference value corresponding to each third period; An offset operation is performed on the historical data of the target third period to obtain a local zipper result corresponding to the sub-table, wherein the difference corresponding to the target third period is not zero.

5. The method according to claim 4, characterized in that The historical data includes field values ​​corresponding to a plurality of fields respectively; The performing hash processing on the historical data of each of the third periods to obtain a hash value corresponding to each of the third periods includes: Determine a target field from a plurality of fields included in the historical data of the third period; The field values ​​corresponding to the plurality of target fields are combined, and a hash process is performed on the combined result to obtain a hash value corresponding to the third cycle.

6. The method according to claim 4, characterized in that The plurality of third periods are N third periods, where N is a positive integer greater than 1; The sequentially subtracting the rankings of two adjacent third periods to obtain the difference value corresponding to each third period includes: Sorting the N third periods in ascending order according to date to obtain a sorting result; The difference value corresponding to the first third period in the sorting result is determined to be 1, and the following processing is performed iteratively i, where i is an integer variable and satisfies 2≤i≤N: The ranking of the ith third period is subtracted from the ranking of the (i-1)th third period, and the obtained difference is used as the difference corresponding to the ith third period.

7. The method according to claim 4, characterized in that The local zipper result includes the zipper start time and the zipper end time respectively corresponding to the historical data of multiple target third periods, and the multiple target third periods include M third periods, where M is a positive integer greater than 1, and M is less than N; The performing of the offset operation on the historical data of the third period of the target to obtain the local zipper result corresponding to the sub-table includes: Iteration j performs the following processing, where j is an integer variable and satisfies 1≤j≤M: The starting date corresponding to the j-th target third period is used as the zipper start time corresponding to the historical data of the j-th target third period; The start date corresponding to the third period of the j+1th target is used as the zipper end time corresponding to the historical data of the third period of the jth target; When j=M, the set date is used as the zipper end time corresponding to the historical data of the third period of the Mth target.

8. The method according to claim 1, characterized in that The historical data includes a date field; the multiple local zipper results are integrated to obtain a target zipper table, including: Integrate the plurality of local zipper results to obtain an integrated result; Sorting the integration result in ascending order according to the date corresponding to the date field to obtain a sorting result; Deleting duplicate historical data with the same hash value in the sorting result to obtain a subset of historical data; The ineffective zipper end time of some historical data in the historical data subset is corrected to obtain a target zipper table.

9. The method according to claim 8, characterized in that Each piece of the historical data in the sorting result corresponds to a unique hash value; The deleting of repeated historical data with the same hash value in the sorting result includes: For multiple pieces of historical data with the same hash value in the sorting result, the piece of historical data with the earliest zip start time is retained.

10. The method according to claim 8, characterized in that The step of correcting the ineffective zipper end time of some historical data in the historical data subset to obtain a target zipper table includes: For specific historical data in the historical data subset, when the zipper start time of the next historical data of the specific historical data is not a null value, the zipper start time of the next historical data is used as the zipper end time of the specific historical data; when the zipper start time of the next historical data of the specific historical data is a null value, the zipper end time of the specific historical data is kept unchanged, wherein the specific historical data is the historical data whose zipper end time is a set date.

11. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a target full-scale table from a data warehouse, wherein the target full-scale table includes historical data of a first period; A splitting module, configured to split the target full table to obtain a plurality of sub-tables, wherein the sub-tables include historical data of a second period, wherein the second period is smaller than the first period; A processing module, used for performing zipper processing on the historical data in the second period to obtain a local zipper result corresponding to each of the sub-tables; The integration module is used to integrate the multiple local zipper results to obtain a target zipper table.

12. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the data processing method according to any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

Images