Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a data generation request which is used for requesting to generate total data in a first time period; determining multiple rows of target zipper data in a zipper table, wherein an intersection exists between the effective time period of the target zipper data and the first time period; a full data set corresponding to each row of target zipper data is determined, the full data set comprises data belonging to a second time period in the target zipper data, and the second time period is an intersection time period of an effective time period of the target zipper data and the first time period; and merging the total data corresponding to each row of target zipper data to obtain the total data in the first time period. According to the embodiment of the invention, the processing efficiency of obtaining total data through the zipper table can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] The full table is used to record all data within the data synchronization cycle, regardless of whether the data has changed. The zipper table is used to record all change information of things from the beginning to the current state. Compared with the full table, the zipper table only adds new records after the data changes, and does not need to store the same data. It can reduce duplicate records, retain equivalent information and consume less space, saving storage space. The zipper table is a storage method that exchanges time for space. Therefore, as a table commonly used in data warehouses to record change information of things, zipper tables are widely used in various industries. However, the efficiency of obtaining historical data through zipper tables in related technologies is low. Summary of the invention

[0003] The present disclosure provides a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, the present disclosure provides a data processing method, the data processing method comprising:

[0005] Obtaining a data generation request, where the data generation request is used to request generation of full data within a first time period;

[0006] Determine a plurality of rows of target zipper data in the zipper table, wherein the effective time period of the target zipper data and the first time period have an intersection;

[0007] Determine a full data set corresponding to each row of the target zipper data, the full data set including data of the target zipper data that belongs to a second time period, the second time period being an intersection time period of an effective time period of the target zipper data and the first time period;

[0008] The full data set corresponding to each row of the target zipper data is merged to obtain the full data in the first time period.

[0009] In a second aspect, the present disclosure provides a data processing device, the data processing device comprising:

[0010] An acquisition module, used to acquire a data generation request, where the data generation request is used to request generation of full data within a first time period;

[0011] A first determination module, configured to determine a plurality of rows of target zipper data in a zipper table, wherein an effective period of the target zipper data and the first period have an intersection;

[0012] A second determination module is used to determine a full data set corresponding to each row of target zipper data, wherein the full data set corresponds to data in the target zipper data that belongs to a second time period, and the second time period is an intersection time period of an effective time period of the target zipper data and the first time period;

[0013] A merging module is used to merge the full data set corresponding to each row of the target zipper data to obtain the full data in the first time period.

[0014] In a third aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing method.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0016] In a fifth aspect, the present disclosure provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0017] The data processing method provided by the embodiment of the present disclosure can determine multiple rows of zipper data in the zipper table that intersect with the first time period, thereby narrowing the calculation scope to reduce the amount of calculation; and then determine the full data set corresponding to each row of zipper data and in the time period that intersects with the first time period, and merge the full data set to obtain the full data of the first time period, thereby improving the processing efficiency of obtaining the full data in a continuous historical time period through the zipper table.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 is a schematic diagram of a zipper table in the related art;

[0021] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the time axis of a zipper table provided in an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a zipper list screening process provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of the zipper table screening results provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of a time dimension table provided for an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of expanding zipper data into full data provided by an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of a target zipper data group provided by an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of parallel processing of data groups provided by an embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of merging a full data set provided in an embodiment of the present disclosure;

[0030] Fig.11 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0031] Fig.12 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0032] Fig.13 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0035] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0036] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0038] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution complies with relevant national laws and regulations (for example, the "Information Security Technology Personal Information Security Specification", etc.). For example: corresponding prescribed measures are taken for personal information access control; prescribed restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; when using personal information, clear identity orientation is eliminated to avoid precise positioning of specific individuals.

[0039] Figure 1 FIG. 1 is a schematic diagram of a zipper table in the related art. Figure 1As shown, the zipper table includes 4 fields, namely business field 1, business field 2, zipper start time and zipper end time, wherein each row of data in the zipper table is a row of zipper data, the zipper start time refers to the start time when the zipper data takes effect, and the zipper end time refers to the end time when the zipper data takes effect. The effective period of the zipper data is from the zipper start time to the zipper end time, including the zipper start time but excluding the zipper end time. For example, the zipper start time of the first row of zipper data is 2020-06-01, the zipper end time is 202-06-07, and the effective period is from 2020-06-01 to 202-06-06. Figure 1 The zipper table in the figure is only an example. As required, the zipper table may also include other fields, and the number of business fields may also be other values, which is not limited in the present disclosure.

[0040] However, when it is necessary to obtain the full data for a continuous period of historical time through a zipper table, each time point in the continuous period of historical time needs to be brought into the zipper table, and the effective zipper data needs to be compared one by one to find the effective zipper data, and the historical data corresponding to the time point is obtained according to the effective zipper data, so as to obtain the full data for a continuous period of historical time, which is inefficient.

[0041] For example, when you need to Figure 1 When obtaining the historical data of 2020-06-10 from the zipper table in , you need to bring 2020-06-10 into the zipper table, compare it with the zipper start time and zipper end time of each row of zipper data, find the zipper data with a zipper start time less than or equal to 2020-06-10 and a zipper end time greater than 2020-06-10, and finally find the second row of zipper data from the top, that is, business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 500, zipper start time is 2020-06-07, and zipper end time is 2020-08-15. Therefore, the historical data of 2020-06-10 includes: business data 1 corresponding to business field 1 is A, and business data 2 corresponding to business field 2 is 500. For another example, when you need to use Figure 1 When the zipper table in obtains the full data in the historical continuous time period [2020-10-01, 2020-10-31], each time point in [2020-10-01, 2020-10-31], that is, 2020-10-01, 2020-10-02, ..., 2020-10-31, these 31 time points are brought into the zipper table, and compared one by one to find the effective zipper data. Each time point needs to be brought into the zipper table for calculation once, and a total of 31 calculations are required, which has a large amount of calculations, a long processing time, and low processing efficiency. Moreover, each time point needs to be compared with the zipper start time and zipper end time of each row of zipper data in the zipper table, resulting in many invalid calculations.

[0042] It can be seen that the related technology has the problem of large amount of calculation and low efficiency in obtaining the full amount of data in a continuous historical period through a zipper table.

[0043] In order to solve the problem of large amount of calculation and low efficiency in obtaining the full amount of data in a continuous historical period through a zipper table, the embodiments of the present disclosure provide a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0044] According to the data processing method of the embodiment of the present disclosure, it is possible to determine multiple rows of zipper data in the zipper table that intersect with the first time period, thereby narrowing the calculation scope to reduce the amount of calculation; and then determine the full data set corresponding to each row of zipper data and in the time period that intersects with the first time period, and merge the full data set to obtain the full data of the first time period, thereby improving the processing efficiency of obtaining the full data in a continuous historical time period through the zipper table.

[0045] The data processing method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0046] The data processing method of the embodiment of the present disclosure can be applied to a variety of application scenarios, including but not limited to the following application scenarios that require obtaining full data within a continuous historical period:

[0047] Application scenario 1

[0048] Historical data correction scenario. For example, when it is found that there are erroneous records in historical data (such as errors in loan balance, overdue date, repayment date, etc.), it is necessary to restore the zipper table data to the full data and perform data correction on the full data.

[0049] Application scenario 2

[0050] Scenarios where regulatory requirements change and historical data needs to be refreshed. For example, regulators may have different requirements for financial companies at different time points, such as disclosing data according to requirement A in 2020 and data according to requirement B in 2021, and historical disclosed data needs to be updated synchronously.

[0051] As long as the scenario requires frequent acquisition of historical data, the embodiments of the present disclosure are applicable. Compared with the full amount of data, the efficiency of frequently acquiring historical information from the zipper table is very low, so it is necessary to generate the full amount of data for a continuous period of historical time through the zipper table through the data processing method of the embodiments of the present disclosure.

[0052] The fields of each row of zipper data in the zipper table of the embodiment of the present disclosure include a business field, a zipper start time field, and a zipper end time field. Among them, the business field includes different attribute fields in the zipper table structure, and the business field does not include the zipper start time field and the zipper end time field.

[0053] There may be multiple business fields, and the business fields in the zipper table of different scenarios may contain different contents, including but not limited to the business fields in the zipper table of the following scenarios:

[0054] Zipper Watch Scene 1

[0055] Financial zipper table scenario. For example, the business field may include a customer ID, and the business field may also include one or more of a resource balance, an interest balance, a penalty interest balance, a credit limit, an overdue date, and a settlement date. The business field may also include other information as needed.

[0056] Zipper Watch Scene 2

[0057] Customer information zipper table scenario. For example, the business field may include a customer ID, and the business field may also include one or more of work unit, address, and contact information. The business field may also include other information as needed.

[0058] Zipper Watch Scene 3

[0059] E-commerce zipper table scenario. For example, the price of a product is adjusted at different times. The business field may include the product ID, the business field may also include one or more of the product price, product listing time, and product delisting time. The business field may also include other information as needed.

[0060] Among them, various information in the zipper table, especially customer information, is obtained after obtaining authorization and consent.

[0061] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 2 , the method comprising:

[0062] Step 201: Obtain a data generation request, where the data generation request is used to request generation of full data within a first time period.

[0063] In some embodiments, the first time period refers to the time period corresponding to the full amount of data to be generated, wherein the two endpoints of the first time period may be dates, months, hours, or other time units, and the present disclosure does not limit this. For example, if the full amount of data within the historical continuous time period [2020-11-15, 2021-01-31] ​​needs to be generated by the data processing method provided in the embodiment of the present disclosure, then the first time period is [2020-11-15, 2021-01-31].

[0064] Step 202 : determining a plurality of rows of target zipper data in the zipper table, wherein the effective time period of the target zipper data overlaps with the first time period.

[0065] In some embodiments, the zipper table is a table used to record all change information of things from the beginning to the current state. Each row of zipper data in the zipper table is a row of zipper data, the zipper start time refers to the start time when the zipper data takes effect, the zipper end time refers to the end time when the zipper data takes effect, and the effective period of the zipper data is from the zipper start time to the zipper end time, including the zipper start time but excluding the zipper end time.

[0066] Figure 3 A schematic diagram of the time axis of a zipper table provided in an embodiment of the present disclosure. Figure 1 Only the zipper data of A corresponding to business field 1 is shown. As needed, business data 1 corresponding to business field 1 can also be other values ​​such as B, C, D, etc., indicating that the actual amount of data will be large, not just object A. Figure 3 Lieutenant General Figure 1 The partial zipper data of business data 1 corresponding to business field 1 is A is displayed on the timeline, and the partial zipper data of business data 1 corresponding to business field 1 is B, C, and D is also displayed on the timeline. The dotted box is the result displayed on the timeline for the first period. Figure 3 The first period is [2020-11-15, 2021-01-31] ​​for example. Figure 1 The data shown in the example are used for explanation, but not for limitation. Figure 1 The zipper table in only includes the zipper data of the business data 1 corresponding to the business field 1 being A. The business data 1 corresponding to the business field 1 may also be other values ​​such as B, C, D, etc. as required. Figure 3It can be seen that among the zipper data of business field 1 corresponding to business data 1, some zipper data have an effective period that intersects with the first period [2020-11-15, 2021-01-31]. This part of the zipper data is the target zipper data. The effective periods of these target zipper data are [2020-11-10, 2020-11-30), [2020-11-30, 2020-12-27), [2020-12-27, 2021-01-03), [2021-01-03, 2021-01-07), [2021-01-07, 2021-01-10), [2021-01-10, 2021-01-25) and [2021-01-25, 2021-02-02).

[0067] In some embodiments, the zipper start time of the target zipper data is less than or equal to the target end time of the first period, and the zipper end time of the target zipper data is greater than or equal to the target start time of the first period;

[0068] The target starting time is the left endpoint of the first time period, and the target ending time is the right endpoint of the first time period.

[0069] The zipper start time, zipper end time, target start time and target end time can be dates, months, hours or other time units, and the present disclosure does not limit this. The present disclosure takes the zipper start time, zipper end time, target start time and target end time as dates for explanation, but does not limit the time units of the zipper start time, zipper end time, target start time and target end time.

[0070] For example, in the first period [2020-11-15, 2021-01-31], the target start time is 2020-11-15, the target end time is 2021-01-31, and the effective period of a row of target zipper data is [2020-11-10, 2020-11-30), the zipper start time is 2020-11-10, and the zipper end time is 2020-11-30.

[0071] Figure 4 Schematic diagram of the zipper list screening process provided by the embodiment of the present disclosure. Figure 1 The zipper start time of each row of target zipper data in the zipper table is compared with the target end time. The target end time is 2021-01-31. The zipper start time that is less than or equal to the target end time 2021-01-31 is filtered out. Figure 4 The dashed-lined portion of the Zipper Start Time column is the Zipper Start Time that is less than or equal to the target end time 2021-01-31. Figure 1 The zipper end time of each row of target zipper data in the zipper table is compared with the target start time. The target start time is 2020-11-15. The zipper end time that is greater than or equal to the target start time 2020-11-15 is filtered out. Figure 4 The part selected by the dotted box in the zipper end time column is the zipper end time that is greater than or equal to the target start time 2020-11-15. Figure 4 The intersection of the zipper data corresponding to the parts selected by the two dotted boxes is the target zipper data.

[0072] Figure 5 A schematic diagram of the zipper table screening results provided in an embodiment of the present disclosure. Figure 5 The part selected by the dotted box is the target zipper data. Figure 4 The parts in the two dashed boxes correspond to the intersection of the zipper data.

[0073] Through the two filtering conditions that the zipper start time is less than or equal to the target end time, and the zipper end time is greater than or equal to the target start time, multiple rows of target zipper data are filtered out from the zipper table, so that the target zipper data can be quickly obtained and the calculation scope can be narrowed. There is no need to expand the data of each row of zipper data in the zipper table. Only the target zipper data needs to be expanded, and irrelevant zipper data is eliminated, thereby reducing the amount of zipper data involved in subsequent data expansion, and reducing the amount of calculation for obtaining the full amount of data in a continuous historical period through the zipper table, thereby saving calculation time and calculation cost, and thus improving processing efficiency.

[0074] Step 203 , determining a full data set corresponding to each row of the target zipper data, wherein the full data set includes data of the target zipper data that belongs to a second time period, and the second time period is an intersection time period of an effective time period of the target zipper data and the first time period.

[0075] That is, for each row of target zipper data, the intersection period of the effective period of the target zipper data and the first period, that is, the second period, can be determined first. Figure 5 The target zipper data (A; 5000; 2020-11-10; 2020-11-30) in the query has an effective period of [2020-11-10, 2020-11-30), and the first period is [2020-11-15, 2021-01-31], then the second period is [2020-11-15, 2020-11-30).

[0076] In some embodiments, the second time period can be divided according to preset time units (such as days) to obtain the time value of the target zipper data in the second time period. For example, in the above example, the second time period is [2020-11-15, 2020-11-30), which corresponds to 15 time values ​​from 2020-11-15 to 2020-11-29.

[0077] In some embodiments, the number of time values ​​of the target zipper data in the second time period can be used as the number of rows of the full data corresponding to the target zipper data, and the time value can be used as the field value of the time field of the full data; the field value of the business field of the target zipper data can be used as the field value of the business field of the full data.

[0078] In this way, multiple rows of full data corresponding to the row of target zipper data can be obtained as the full data set corresponding to the target zipper data.

[0079] Step 204 , merging the full data set corresponding to each row of the target zipper data to obtain the full data within the first time period.

[0080] In some embodiments, after obtaining the full data set corresponding to each row of target zipper data, the full data set may be merged in chronological order to obtain the full data within the first time period, thereby completing the process of obtaining the full data.

[0081] According to the data processing method of the embodiment of the present disclosure, it is possible to determine multiple rows of zipper data in the zipper table that intersect with the first time period, thereby narrowing the calculation scope to reduce the amount of calculation; and then determine the full data set corresponding to each row of zipper data and in the time period that intersects with the first time period, and merge the full data set to obtain the full data of the first time period, thereby improving the processing efficiency of obtaining the full data in a continuous historical time period through the zipper table.

[0082] The data processing method according to the embodiment of the present disclosure is described below.

[0083] As described above, a data generation request for requesting to generate full data in a first time period may be obtained in step 201, and multiple rows of target zipper data that intersect with the first time period in the zipper table may be determined in step 202. Further, a full data set corresponding to each row of the target zipper data may be determined in step 203.

[0084] In some embodiments, step 203 may include:

[0085] Determine, according to the time value of the target zipper data in the second time period, a field value of the time field of each row of the full data set;

[0086] Determine, according to the field value of the business field of the target zipper data, the field value of the business field of each row of the full data set;

[0087] The full data set is obtained according to the time field and the business field of each row of full data.

[0088] For example, for each row of target zipper data, the intersection period of the effective period of the target zipper data and the first period, that is, the second period, may be determined first. Figure 5 The target zipper data (A; 5000; 2020-11-10; 2020-11-30) in the query has an effective period of [2020-11-10, 2020-11-30), and the first period is [2020-11-15, 2021-01-31], then the second period is [2020-11-15, 2020-11-30).

[0089] In some embodiments, the data processing method according to the embodiment of the present disclosure further includes:

[0090] The second time period is divided according to preset time units to obtain a time value of the target zipper data in the second time period.

[0091] That is to say, the second time period can be divided according to a preset time unit (such as day) to obtain the time value of the target zipper data in the second time period, wherein the preset time unit is consistent with the time unit of the zipper start time and the zipper end time of the target zipper data, such as date, month, hour, etc. For example, in the above example, the second time period is [2020-11-15, 2020-11-30), which corresponds to 15 time values ​​from 2020-11-15 to 2020-11-29.

[0092] In some embodiments, each row of target zipper data may be expanded according to the time value, so that each time value of the target zipper data in the second time period corresponds to a row of full data, thereby obtaining a full data set corresponding to the target zipper data.

[0093] In some embodiments, data expansion refers to the process of dividing the effective period or intersection period of the target zipper data into time units and mapping them into full data corresponding to multiple time values ​​(also called cross-sectional time), where each time value corresponds to a row of full data.

[0094] In the example, the time value refers to the time when the data warehouse extracts business system data. For example, a row of target zipper data includes business data 1 corresponding to business field 1, which is A, and business data 2 corresponding to business field 2, which is 8000. The zipper start time is 2021-01-07, and the zipper end time is 2021-01-10. The effective period of the target zipper data is [2021-01-07, 2021-01-10). The time value is the effective period of the target zipper data [2021-01-07, 2021-01-10) divided according to the unit time (one day), that is, the time value includes 2021-01-07, 2021-01-08 and 2021-01-09. The row of target zipper data is expanded, and the full data obtained includes full data 1, full data 2 and full data 3.

[0095] In some embodiments, the number of time values ​​of the target zipper data in the second time period can be used as the number of rows of the full data corresponding to the target zipper data, and the time value can be used as the field value of the time field of the full data; the field value of the business field of the target zipper data can be used as the field value of the business field of the full data.

[0096] In the example, the full data 1 includes the field value of business data 1 corresponding to business field 1, which is A, the field value of business data 2 corresponding to business field 2 is 8000, and the field value of the time field is 2021-01-07; the full data 2 includes the field value of business data 1 corresponding to business field 1, which is A, the field value of business data 2 corresponding to business field 2 is 8000, and the field value of the time field is 2021-01-08; the full data 3 includes the field value of business data 1 corresponding to business field 1, which is A, the field value of business data 2 corresponding to business field 2 is 8000, and the field value of the time field is 2021-01-09.

[0097] In this way, multiple rows of full data corresponding to the row of target zipper data can be obtained as the full data set corresponding to the target zipper data.

[0098] In some embodiments, step 203 may be implemented in a variety of ways, including but not limited to the following ways:

[0099] Method 1

[0100] In the second period, the target zipper data is expanded according to the time dimension table to generate full data, where each row in the time dimension table corresponds to a time point, and the difference between the time points of two adjacent rows is the unit time.

[0101] In the example, the time dimension table refers to a dimension table that records time. The time dimension table includes all time points within the time range required for the business and time-related information. The time dimension table is stored in rows in the database, that is, each row in the time dimension table corresponds to a time point. In the embodiment of the present disclosure, only the time point corresponding to each row in the time dimension table is used, and time-related information is not used. The unit time can be one day, one hour, one month, or other values. Since the difference between the time points of two adjacent rows is the unit time, each row in the time dimension table can correspond to a specific date, a specific hour, a specific month, or other values. The present disclosure uses the example of each row in the time dimension table corresponding to a specific date for explanation, but does not limit this.

[0102] Figure 6 A schematic diagram of a time dimension table provided for an embodiment of the present disclosure. Figure 6 In the time dimension table, the date and time related information is included. Figure 6 The time-related information is represented by xxx, and the actual time-related information is not listed. The time-related information includes the year, month, quarter, week number, and day of the week to which the date belongs.

[0103] In the second time period, the target zipper data is expanded according to the time dimension table to generate full data. Since the time dimension table already includes specific time points, the target zipper data is expanded according to the time dimension table. It is possible to obtain multiple time values ​​mapped to the effective time period of the target zipper data at one time according to the time dimension table, and then generate historical data corresponding to multiple consecutive time values ​​at one time according to the target zipper data, that is, full data, which improves the processing efficiency of obtaining full data within a continuous historical time period through the zipper table and reduces the processing time.

[0104] Method 2

[0105] In the second time period, the zipper start time of the target zipper data is incremented sequentially according to the unit time until it is incremented to the zipper end time of the target zipper data to obtain a time value, wherein the time value is in the second time period and the time value does not include the zipper end time of the target zipper data; the target zipper data is expanded according to the time value to generate full data.

[0106] For example, a row of target zipper data includes business data 1 corresponding to business field 1, which is A, and business data 2 corresponding to business field 2, which is 8000. The zipper start time is 2021-01-07, the zipper end time is 2021-01-10, and the second period is [2020-01-07, 2021-01-10]. In the second period, the zipper start time of the target zipper data is incremented sequentially according to the unit time until it is incremented to the zipper end time of the target zipper data to obtain the time value, that is, the zipper start time 2021-01-07 is incremented by 1 day each time until it is incremented to 2021-01-10 to obtain the time value. 2021-01-07 plus 1 day is 2021-01-08, 2021-01-08 plus 1 day is 2021-01-09, and 2021-01-09 plus 1 day is 2021-01-1 0. Since the time value is within the second time period and the time value does not include the zipper end time of the target zipper data, the time values ​​include 2021-01-07, 2021-01-08 and 2021-01-09. This row of target zipper data is expanded, and the historical data obtained includes historical data 1, historical data 2 and historical data 3. Historical data 1 includes business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 8000, and the time value is 2021-01-07. Historical data 2 includes business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 8000, and the time value is 2021-01-08. Historical data 3 includes business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 8000, and the time value is 2021-01-09.

[0107] In some embodiments, during the second period, the target zipper data is expanded according to the time dimension table to generate full data. There are multiple implementation methods, including but not limited to the following methods:

[0108] Method 1

[0109] In the case where the target zip data is a row,

[0110] Traverse the time points from the time dimension table, and select each first target time that meets the first preset condition, wherein the first preset condition means that the first target time is greater than or equal to the target start time, and the first target time is less than or equal to the target end time, and the first target time is greater than or equal to the zipper start time of the target zipper data, and the first target time is less than the zipper end time of the target zipper data; expand the target zipper data according to each first target time to generate full data, wherein the fields in the full data include business fields and time fields, the fields in the target zipper data include business fields, zipper start time fields, and zipper end time fields, the business data corresponding to the business fields in the full data and the target zipper data are consistent, and the data corresponding to the time field is the first target time.

[0111] When the target zipper data is a row, you only need to expand the data of this row of target zipper data according to the time dimension table to generate the full data. There are two parts of logic for selecting the first target time from the time dimension table. One part of the logic is to limit the first target time to be within the first time period, specifically, the first target time is greater than or equal to the target start time, and the first target time is less than or equal to the target end time. The other part of the logic is to limit the first target time to be within the time interval corresponding to the target zipper data, specifically, the first target time is greater than or equal to the zipper start time of the target zipper data, and the first target time is less than the zipper end time of the target zipper data.

[0112] In the example, the business field includes different attribute fields in the zipper table structure, and the business field does not include the zipper start time field and the zipper end time field.

[0113] Figure 7 A schematic diagram of expanding zipper data into full data provided in an embodiment of the present disclosure.

[0114] like Figure 7As shown, the target zipper data includes business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 5000, the zipper start time is 2020-11-10, the zipper end time is 2020-11-30, the second time period is [2020-11-15, 2020-11-29], the first target time is greater than or equal to the target start time 2020-11-15, and the first target time is less than or equal to the target end time 2020-11-29, and the first target time is greater than or equal to The zipper start time of the target zipper data is 2020-11-10, and the first target time is less than the zipper end time of the target zipper data 2020-11-30. Therefore, the time points are traversed from the time dimension table, and each first target time that meets the first preset condition is selected, including 2020-11-15, 2020-11-16, 2020-11-17, 2020-11-18, 2020-11-19, 2020-11-20, 2020-11-21, 2020-1 There are 15 time points in total, including 2020-11-22, 2020-11-23, 2020-11-24, 2020-11-25, 2020-11-26, 2020-11-27, 2020-11-28, and 2020-11-29. The business data corresponding to the business fields in the full data and the target zipper data are consistent, and the data corresponding to the time field is the first target time. Therefore, the full data includes 15 data records, and the time values ​​are 2020-11-15, 2020-11-16, 2020-11-17, 2020-11-18, 2020-11-19, 2020-11-20, 2020-11-21, 2020-11-22, 2020-11-23, 2020-11-24, 2020-11-25, 2020-11-26, 2020-11-27, 2020-11-28, and 2020-11-29, the business data 1 corresponding to the business field 1 is all A, and the business data 2 corresponding to the business field 2 is all 5000.

[0115] When the target zipper data is a row, the time points are traversed from the time dimension table, each first target time that meets the first preset condition is selected, and the target zipper data is expanded according to each first target time to generate full data. The full data can be generated at one time according to the target zipper data and each first target time selected from the traversed time points in the time dimension table, which improves the processing efficiency of obtaining full data in a continuous historical period through the zipper table and reduces the processing time.

[0116] Method 2

[0117] When the target zipper data is multiple lines and the execution subject is a single electronic device,

[0118] Traverse the target zipper data, and perform the following operations for any row of target zipper data: traverse the time points from the time dimension table, select each second target time that meets the second preset condition, the second preset condition means that the second target time is greater than or equal to the target start time, and the second target time is less than or equal to the target end time, and the second target time is greater than or equal to the zipper start time of any row of target zipper data, and the second target time is less than the zipper end time of any row of target zipper data; perform data expansion on any row of target zipper data according to each second target time to generate a full data set corresponding to the target zipper data, wherein the fields of the full data in the full data set include business fields and time fields, the fields in any row of target zipper data include business fields, zipper start time fields, and zipper end time fields, the full data in the full data set is consistent with the business data corresponding to the business fields in the target zipper data, and the field value corresponding to the time field is the second target time;

[0119] Furthermore, in step 204, the full data set corresponding to each row of target zipper data is merged to obtain the full data within the first time period.

[0120] When the target zipper data is multiple rows and the execution subject is a single electronic device, it is necessary to first traverse the target zipper data, expand each row of the target zipper data, generate the corresponding full data set, and then merge the full data sets corresponding to each row of the target zipper data to generate the full data within the first time period.

[0121] For example: the target zipper data is Figure 5As shown, there are 7 rows of target zipper data, one row of target zipper data includes business data 1 corresponding to business field 1 is A, business data 2 corresponding to business field 2 is 5000, the zipper start time is 2020-11-10, the zipper end time is 2020-11-30, the second time period is [2020-11-15, 2021-01-31], the second target time is greater than or equal to the target start time 2020-11-15, and the second target time is less than or equal to the target end time 2021-01-31, and the second target time The zipper start time 2020-11-10 is greater than or equal to the target zipper data row, and the second target time is less than the zipper end time 2020-11-30 of the target zipper data row. Therefore, the time points are traversed from the time dimension table, and each second target time that meets the second preset condition is selected, including 2020-11-15, 2020-11-16, 2020-11-17, 2020-11-18, 2020-11-19, 2020-11-20, 2020-11-21, 2020-11-2 2. There are 15 time points in total, including 2020-11-23, 2020-11-24, 2020-11-25, 2020-11-26, 2020-11-27, 2020-11-28, and 2020-11-29. The business fields of the full data are consistent with the business data corresponding to the business fields in the target zipper data of this row. The field value corresponding to the time field is the second target time. Therefore, the full data set corresponding to this row of target zipper data includes 15 data records, and the time values ​​are 2020-11-15, 2020- 11-16, 2020-11-17, 2020-11-18, 2020-11-19, 2020-11-20, 2020-11-21, 2020-11-22, 2020-11-23, 2020-11-24, 2020-11-25, 2020-11-26, 2020-11-27, 2020-11-28, 2020-11-29, the business data 1 corresponding to the business field 1 is A, and the business data 2 corresponding to the business field 2 is 5000. For details, see Figure 7 The expansion process of the remaining 6 rows of target zipper data is similar to that of this row of target zipper data. The full data set corresponding to these 7 rows of target zipper data is merged to generate the full data in the first period.

[0122] In the case where the number of target zipper data is multiple rows and the execution subject is a single electronic device, the target zipper data is first traversed, and each row of target zipper data is expanded to generate a full data set, and then the full data sets corresponding to each row of target zipper data are merged to generate full data. The full data set can be generated once for each second target time selected from the target zipper data and the traversal time point in the time dimension table, and then the full data sets corresponding to each row of target zipper data are merged to generate full data in the first time period, which improves the processing efficiency of obtaining full data in a continuous historical period through the zipper table and reduces the processing time. Moreover, in the case where the number of target zipper data is at least two rows and the execution subject is a single electronic device, the target zipper data is first traversed, and each row of target zipper data is expanded to generate a full data set, and then the full data sets corresponding to each row of target zipper data are merged to generate full data, which is more suitable for scenarios where the number of target zipper data is relatively small and the amount of full data is relatively small. The data processing method is executed by a single electronic device, which reduces the interaction between each electronic device and improves the processing efficiency.

[0123] Method 3

[0124] When the number of target zipper data is multiple rows and is processed by multiple computing nodes, the data processing method according to the embodiment of the present disclosure further includes:

[0125] Grouping multiple rows of target zipper data to obtain multiple data groups;

[0126] Send each data group to a computing node;

[0127] In step 203, the full data set corresponding to each row of target zipper data is determined, including: receiving the full data set sent by each computing node, wherein the full data set is obtained by the computing node calculating the target zipper data of the data group.

[0128] That is to say, in the case of processing by multiple computing nodes, the target zipper data can be grouped first to obtain multiple data groups; and each data group can be sent to a computing node, so that the target zipper data of different data groups can be calculated by different computing nodes. In other words, the target zipper data in the data group is expanded according to the time dimension table to generate a full data set corresponding to the target zipper data in the data group.

[0129] In some embodiments, the full data set sent by each computing node may be received in step 203, and the full data set corresponding to each row of target zipper data may be merged in chronological order in step 204 to obtain the full data within the first time period, thereby completing the process of obtaining the full data.

[0130] Figure 8A schematic diagram of a target zipper data group provided by an embodiment of the present disclosure. Figure 8 There are 7 rows of target zipper data in total, and these 7 rows of target zipper data are divided into 3 groups to obtain 3 data groups. Data group 1 includes 3 rows of target zipper data, data group 2 includes 2 rows of target zipper data, and data group 3 includes 2 rows of target zipper data. Here, the number of groups 3 can be adjusted based on actual conditions, and the number of groups can be increased or decreased.

[0131] Fig. 9 A schematic diagram of parallel processing of data groups provided by an embodiment of the present disclosure. Fig. 9 In the process, data group 1 is sent to computing node 1, and computing node 1 outputs the full data set of computing node 1. Data group 2 is sent to computing node 2, and computing node 2 outputs the full data set of computing node 2. Data group 3 is sent to computing node 3, and computing node 3 outputs the full data set of computing node 3.

[0132] Fig.10 A schematic diagram of merging the full data set provided in the embodiment of the present disclosure. Fig.10 In the example, the target zipper data expands from 7 rows to 78 rows of full data.

[0133] In the context of big data, the data time period spans a long period, ranging from a few months to several years; the data volume is huge, with hundreds of millions or even tens of billions of data. Only one computing node cannot meet the timeliness requirements of the application. Therefore, the target zipper data is grouped to obtain multiple data groups. Different computing nodes are used to expand the data of different data groups according to the time dimension table to generate a full data set. The full data set corresponding to each computing node is merged to generate the full data. This can utilize the multi-node computing power of the big data cluster, and different computing nodes can expand the data efficiently in parallel to improve computing efficiency. It can make full use of the advantages of parallel computing in distributed systems, and then merge the full data set corresponding to each computing node to generate the full data, which improves the processing efficiency of obtaining the full data in a continuous historical period through the zipper table and reduces the processing time. It is more suitable for scenarios where the number of target zipper data is large and the amount of full data is large.

[0134] In some embodiments, in method three, the step of grouping the multiple rows of target zipper data to obtain multiple data groups includes: determining the number of groups; and evenly grouping the multiple rows of target zipper data according to the number of groups to obtain multiple data groups.

[0135] In some embodiments, the step of determining the number of groups includes:

[0136] The number of groups is obtained according to the data volume of the target zipper data, the maximum data volume processed by each computing node, and the number of computing nodes scheduled in parallel.

[0137] In the example, assume that the maximum amount of data that each computing node can process is 2G. The full amount of data in the first period includes approximately 100 million rows of data, and the amount of the full amount of data is 128G. To ensure the efficiency of parallel processing, it is ideal to divide it into 128 / 2=64 groups, and the number of groups can be 64. If the number of computing nodes that can be scheduled in parallel is 128, it can also be divided into 128 groups. Each computing node processes 1G of data, and the efficiency will be higher. The number of groups can be 128. At the same time, the number of target zipper data also needs to be considered to ensure that each computing node can be assigned the target zipper data. If the amount of data of the full amount of data is large, and the more computing nodes that can be scheduled in parallel, the more groups there are, the higher the parallel efficiency will be.

[0138] In this way, the number of groups can be obtained according to the data volume of the target zipper data, the maximum data volume processed by each computing node, and the number of computing nodes scheduled in parallel, so as to maximize the parallel efficiency, evenly group the target zipper data according to the number of groups, and try to balance the data volume processed by each computing node to avoid performance shortcomings, further improve the parallel efficiency, and thus improve the processing efficiency of obtaining the full amount of data in a continuous historical period through the zipper table.

[0139] In some embodiments, the target zipper data is evenly grouped according to the number of groups to obtain multiple data groups. The ntile function (an analysis function) in SQL can be used. The ntile function sorts the target data set according to the agreed field information and then groups them evenly. The number of groups can be specified arbitrarily. The grouping result here corresponds to each computing node.

[0140] For example, the ntile function can first sort the target zipper data in ascending order by the zipper start time field. If the zipper start time of multiple rows of target zipper data is the same, then sort them in ascending order by the zipper end time field. After obtaining an ordered data, it is evenly grouped.

[0141] In practical applications, whether or not to sort does not affect the final result, because even if it is in disorder, as long as the grouping is uniform, each row of target zipper data will not be calculated repeatedly.

[0142] In some embodiments, the full data set sent by each computing node is merged to generate the full data in the first time period, and the merger can be performed through a union operation (an operation in SQL used to merge result sets of multiple statements).

[0143] In some embodiments, Fig.11 A flowchart of a data processing method provided by an embodiment of the present disclosure. Fig.11 , the method comprising:

[0144] Step 1101, obtaining a first time period corresponding to the full amount of data to be generated.

[0145] Step 1102, compare the zipper start time, zipper end time of each row of target zipper data in the zipper table with the target start time and target end time to obtain the target zipper data, wherein the zipper start time of the target zipper data is less than or equal to the target end time, and the zipper end time of the target zipper data is greater than or equal to the target start time.

[0146] The target starting time is the left endpoint of the first time period, and the target ending time is the right endpoint of the first time period.

[0147] Step 1103, grouping the target zipper data to obtain multiple data groups; and sending different data groups to different computing nodes respectively.

[0148] Step 1104, obtaining the full data set sent by each computing node, wherein the full data set is the full data generated by the computing node by performing data expansion on the target zipper data in the data group within the first time period according to the time dimension table.

[0149] Step 1105: merge the full data set sent by each computing node to generate the full data within the first time period.

[0150] Through the two filtering conditions that the zipper start time is less than or equal to the target end time, and the zipper end time is greater than or equal to the target start time, the target zipper data is filtered out from the zipper table, so that the target zipper data can be obtained quickly and the calculation scope can be narrowed. There is no need to expand the data of each row of zipper data in the zipper table. Only the target zipper data needs to be expanded, and irrelevant zipper data is eliminated, thereby reducing the amount of zipper data involved in subsequent data expansion, and reducing the amount of calculation for obtaining the full amount of data in a continuous historical period through the zipper table, saving calculation time and calculation cost, thereby improving processing efficiency. The target zipper data is grouped to obtain multiple data groups. Different data groups are expanded according to the time dimension table through different computing nodes to generate a full data set. The full data sets corresponding to each computing node are merged to generate the full data within the first time period. The multi-node computing power of the big data cluster can be utilized. Different computing nodes can expand data in parallel and efficiently to improve computing efficiency. The advantages of parallel computing in distributed systems are fully utilized. The full data sets corresponding to each computing node are then merged to generate the full data within a time period, which improves the processing efficiency of obtaining the full data within a continuous historical period through the zipper table and reduces the processing time.

[0151] In summary, the data processing method provided by the embodiment of the present disclosure expands the target zipper data according to the unit time to generate the full data. Compared with the related art that can only obtain the historical data corresponding to one time point at a time, the embodiment of the present disclosure can generate the historical data corresponding to multiple consecutive time points at one time according to the target zipper data, that is, the full data, which improves the processing efficiency of obtaining the full data within a continuous period of historical time through the zipper table and reduces the processing time. Moreover, the data processing method provided by the embodiment of the present disclosure obtains the target zipper data by comparing the effective time period of each row of zipper data in the zipper table with the first time period, wherein the effective time period of the target zipper data and the first time period have an intersection, thereby narrowing the calculation range and eliminating the need to expand the data of each row of zipper data in the zipper table, thereby reducing the amount of calculation for obtaining the full data within a continuous period of historical time through the zipper table, thereby improving the processing efficiency.

[0152] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0153] In addition, the present disclosure also provides a data processing device, an electronic device, and a computer program product, all of which can be used to implement any data processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0154] Fig.12 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0155] Reference Fig.12 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0156] An acquisition module 1201 is used to acquire a data generation request, where the data generation request is used to request generation of full data within a first time period;

[0157] A first determining module 1202 is used to determine a plurality of rows of target zipper data in a zipper table, wherein the effective period of the target zipper data and the first period have an intersection;

[0158] A second determination module 1203 is used to determine a full data set corresponding to each row of target zipper data, wherein the full data set corresponds to data in the target zipper data that belongs to a second time period, and the second time period is an intersection time period of an effective time period of the target zipper data and the first time period;

[0159] The merging module 1204 is configured to merge the full data set corresponding to each row of the target zipper data to obtain the full data within the first time period.

[0160] In some embodiments, for each row of the target zipper data, the second determining module 1203 is used to:

[0161] According to the time value of the target zipper data in the second time period, the field value of the time field of each row of the full data set is determined; according to the field value of the business field of the target zipper data, the field value of the business field of each row of the full data set is determined; according to the time field and business field of each row of the full data, the full data set is obtained.

[0162] In some embodiments, the device further includes: a time division module, configured to divide the second time period according to a preset time unit to obtain a time value of the target zipper data in the second time period.

[0163] In some embodiments, before the second determining module 1203, the apparatus further includes:

[0164] A grouping module, used for grouping the target zipper data of multiple rows to obtain multiple data groups;

[0165] A sending module, used for sending each data group to a computing node;

[0166] The second determination module 1203 is used to receive a full data set sent by each computing node, where the full data set is obtained by the computing node calculating the target zipper data of the data group.

[0167] In some embodiments, the grouping module is used to: determine the number of groups; and evenly group the multiple rows of target zipper data according to the number of groups to obtain multiple data groups.

[0168] In some embodiments, the grouping module is used to obtain the number of groups according to the data volume of the target zipper data, the maximum data volume processed by each computing node, and the number of computing nodes scheduled in parallel.

[0169] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0170] Fig.13 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0171] Reference Fig.13 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 1301; at least one memory 1302, and one or more I / O interfaces 1303, connected between the processor 1301 and the memory 1302; wherein the memory 1302 stores one or more computer programs that can be executed by the at least one processor 1301, and the one or more computer programs are executed by the at least one processor 1301 so that the at least one processor 1301 can execute the above-mentioned data processing method.

[0172] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0173] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0174] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0175] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0176] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0177] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0178] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0179] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0180] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0183] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0184] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Obtaining a data generation request, where the data generation request is used to request generation of full data within a first time period; Determine a plurality of rows of target zipper data in the zipper table, wherein the effective time period of the target zipper data and the first time period have an intersection; Determine a full data set corresponding to each row of the target zipper data, the full data set including data of the target zipper data that belongs to a second time period, the second time period being an intersection time period of an effective time period of the target zipper data and the first time period; The full data set corresponding to each row of the target zipper data is merged to obtain the full data in the first time period.

2. The data processing method according to claim 1, characterized in that: For each row of the target zipper data, determining a full data set corresponding to the target zipper data includes: Determine, according to the time value of the target zipper data in the second time period, a field value of the time field of each row of the full data set; Determine, according to the field value of the business field of the target zipper data, the field value of the business field of each row of the full data set; The full data set is obtained according to the time field and the business field of each row of full data.

3. The data processing method according to claim 2, characterized in that: The method further comprises: The second time period is divided according to preset time units to obtain a time value of the target zipper data in the second time period.

4. The data processing method according to claim 1, characterized in that: Before determining the full data set corresponding to each row of target zipper data, the method further includes: Grouping multiple rows of target zipper data to obtain multiple data groups; Send each data group to a computing node; The determining of the full data set corresponding to each row of target zipper data includes: receiving the full data set sent by each computing node, where the full data set is obtained by the computing node calculating the target zipper data of the data group.

5. The data processing method according to claim 4, characterized in that: The step of grouping the target zipper data in multiple rows to obtain multiple data groups includes: Determine the number of groups; According to the grouping quantity, multiple rows of the target zipper data are evenly grouped to obtain multiple data groups.

6. The data processing method according to claim 5, characterized in that: The determining of the number of groups comprises: The number of groups is obtained according to the data volume of the target zipper data, the maximum data volume processed by each computing node, and the number of computing nodes scheduled in parallel.

7. A data processing device, characterized in that: include: An acquisition module, used to acquire a data generation request, where the data generation request is used to request generation of full data within a first time period; A first determination module, configured to determine a plurality of rows of target zipper data in a zipper table, wherein an effective period of the target zipper data and the first period have an intersection; A second determination module is used to determine a full data set corresponding to each row of target zipper data, wherein the full data set corresponds to data in the target zipper data that belongs to a second time period, and the second time period is an intersection time period of an effective time period of the target zipper data and the first time period; A merging module is used to merge the full data set corresponding to each row of the target zipper data to obtain the full data in the first time period.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the data processing method as described in any one of claims 1 to 6.

Images