A data delivery method and apparatus

Abstract

Embodiments of the present application disclose a data delivery method and device, a storage medium and an electronic device, wherein the method comprises: receiving a data acquisition request from a data requester, analyzing the data acquisition request to obtain condition dimension information, delivery dimension information and a data filtering rule; performing condition matching in a public dimension wide table based on the condition dimension information, so as to determine at least one associated object based on a matching result of the condition matching; obtaining delivery data of each associated object based on the delivery dimension information, and generating full-amount period data based on the delivery data of each associated object; and performing data filtering on the full-amount period data according to the data filtering rule, so as to obtain delivery full-amount data of a current period, and delivering the delivery full-amount data of the current period to the data requester in a determined delivery mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and in particular to a data delivery method and device, a computer readable storage medium, an electronic device, and a computer program product. BACKGROUND

[0002] At present, in the scene of large-scale periodic customization and high fault tolerance, database delivery work is a very tricky job. Using the traditional MySQL (relational database management system) database and incremental way exposes more and more problems: on the one hand, limited by the performance constraints of MySQL, the full data delivery at the same time can only serve one destination. Using the way of listening to the changes of the full table to synchronize and real-time synchronize each destination, the incremental data in the delivery period cannot be effectively merged, which leads to the playback pressure of the destination is larger and the downstream has a serious risk of downtime; on the other hand, for the growing scene demand, it is difficult to realize the customized data delivery demand and the incremental output logic lacks verification, in addition, error repair often follows the full pull-in way and the problem of data validity and safety of itself and many other problems are difficult to solve.

[0003] In the prior art, there is a lack of technical solutions for applying big data technology to database delivery scenarios, especially incremental data delivery scenarios, but a reliable and easy-to-implement database delivery system is very necessary. SUMMARY

[0004] In view of this, in view of the performance limitations and business demands that cannot be supported of the traditional MySQL delivery scheme, relying on hive (data warehouse tool) and related scheduling systems for system reconstruction optimization, a method and device for realizing customized large-scale data full and incremental delivery are provided, which realizes functions including main function customization delivery, multi-user full delivery at the same time, incremental data time limit within the merger delivery, error data rapid repair, complex analysis requirement support, incremental full data quality monitoring and alarm, problem backtracking and positioning, and the like.

[0005] Based on one aspect of an embodiment of the present application, a data delivery method is provided, which comprises:

[0006] receiving a data acquisition request from a data requester, and analyzing the data acquisition request to obtain condition dimension information, delivery dimension information and data filtering rules;

[0007] performing condition matching in a public dimension wide table based on the condition dimension information, so as to determine at least one associated object based on the matching result of the condition matching;

[0008] obtain delivery data of each associated object based on the delivery dimension information, and generate full-cycle data based on the delivery data of each associated object; and

[0009] filter the full-cycle data according to the data filtering rule, to obtain delivery full-cycle data of the current cycle, and deliver the delivery full-cycle data of the current cycle to the data requester in the determined delivery mode.

[0010] Optionally, before the receiving of the data acquisition request from the data requester, the method further comprises:

[0011] statistically processing all data tables in the database to determine all associated objects and all attributes involved in the database;

[0012] creating a public dimension wide table, in which each attribute of all attributes is taken as a dimension, and a single record is set for each associated object in the public dimension wide table;

[0013] The value of the dimension is used to indicate whether the associated object has data content corresponding to the dimension.

[0014] Optionally, the method further comprises, when detecting that a new attribute is added in the database, adding a dimension corresponding to the new attribute in the public dimension wide table.

[0015] Optionally, the difference between the update frequency of the public dimension wide table and the update frequency of the full-cycle data of the database is less than or equal to a difference threshold.

[0016] When updating the public dimension wide table, the public dimension wide table is divided into a plurality of sub-tables, each sub-table is updated in a parallel update manner, and the updated sub-tables are combined to form an updated public dimension wide table.

[0017] Optionally, the method further comprises determining an increment data delivery time granularity t i of the i th data requester in all data requesters.

[0018] Setting the granularity T of the production cycle of the data in the database, where T≤Min(t i ).

[0019] Optionally, the delivering of the delivery full-cycle data of the current cycle to the data requester in the determined delivery mode comprises:

[0020] When the determined delivery mode is full-cycle data delivery, the delivery full-cycle data of the current cycle is directly delivered to the data requester.

[0021] Optionally, the delivering the delivery fullness data of the current period to the data requester according to the determined delivery mode comprises:

[0022] When the determined delivery mode is the incremental data delivery, determining whether the data requester needs data repair;

[0023] If it is determined that the data requester needs data repair, taking the delivery fullness data of the current period as a first data set and taking the delivery fullness data currently stored in the data requester as a second data set;

[0024] Comparing the first data set and the second data set to determine incremental data;

[0025] Delivering the incremental data to the data requester to enable the data requester to generate updated delivery fullness data based on the incremental data and the delivery fullness data currently stored;

[0026] Optionally, the delivering the delivery fullness data of the current period to the data requester according to the determined delivery mode comprises:

[0027] When the determined delivery mode is the incremental data delivery, determining whether the data requester needs data repair;

[0028] If it is determined that the data requester needs data repair, taking the delivery fullness data of the current period as a first data set and taking the delivery fullness data of the last period as a second data set; and

[0029] Comparing the first data set and the second data set to determine incremental data;

[0030] Delivering the incremental data to the data requester to enable the data requester to generate updated delivery fullness data based on the incremental data and the delivery fullness data of the last period.

[0031] Optionally, the comparing the first data set and the second data set to determine incremental data comprises:

[0032] Determining a difference set of data record identification fields of the first data set and the second data set, and constructing data contents belonging to the difference set of data record identification fields in the second data set into insertion data increments in an incremental format;

[0033] Determining a difference set of data record identification fields of the first data set and the second data set, and constructing data contents belonging to the difference set of data record identification fields in the first data set into deletion data increments in an incremental format;

[0034] determine a data set of data inconsistency existing in the rest of the fields in the intersection of the data record identification fields of the first data set and the second data set, and construct the data content belonging to the data set of data inconsistency in the second data set into an update data increment in an incremental format;

[0035] combine the insertion data increment, the deletion data increment and the update data increment into the incremental data.

[0036] Optionally, further comprising, performing data volume statistics on the insertion data increment, the deletion data increment and the update data increment, and triggering an alarm and stopping data delivery when the statistical data volume is greater than a volume threshold.

[0037] Optionally, further comprising, determining a data volume fluctuation rate of the insertion data increment, the deletion data increment and the update data increment, and triggering an alarm and stopping data delivery when the data volume fluctuation rate is greater than a fluctuation rate threshold.

[0038] Optionally, further comprising, determining the generation time of the full-period data, the delivered full-period data and the incremental data respectively;

[0039] determining the duration of the full-period data, the delivered full-period data and the incremental data respectively based on the difference between the current time and the generation time;

[0040] deleting the full-period data, the delivered full-period data or the incremental data when the duration of the full-period data, the delivered full-period data or the incremental data is greater than a time threshold.

[0041] Optionally, after generating the full-period data based on the delivery data of each associated object, further comprising:

[0042] adding an update time to each field in each data table of the full-period data.

[0043] According to another aspect of the present application, a data delivery device is provided, the device comprising:

[0044] a parsing unit configured to receive a data acquisition request from a data requester, and parse the data acquisition request to obtain condition dimension information, delivery dimension information and data filtering rules;

[0045] a matching unit configured to perform condition matching in a public dimension wide table based on the condition dimension information, and determine at least one associated object based on the matching result of the condition matching;

[0046] a generating unit configured to acquire delivery data of each associated object based on the delivery dimension information, and generate full-period data based on the delivery data of each associated object; and

[0047] The delivery unit is used to filter the full-cycle data according to the data filtering rules to obtain the full-cycle delivery data for the current cycle, and deliver the full-cycle delivery data for the current cycle to the data requester according to the determined delivery method.

[0048] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program for performing the methods described in any of the above embodiments.

[0049] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor and a memory; wherein...

[0050] The memory is used to store the processor-executable instructions;

[0051] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the above embodiments.

[0052] According to another aspect of the present invention, a computer program product is provided, including computer-readable code, which, when run on a device, enables a processor in the device to perform the method described in any of the above embodiments.

[0053] Based on the data delivery method and apparatus, computer-readable storage medium, electronic device, and computer program product provided in the above embodiments of the present invention, the incremental production logic and data repair logic are unified. The implementation logic of the data delivery apparatus can complete these two important processes. Data repair only requires the full state and the latest data state of the destination to be repaired as the two inputs for producing incremental data. Secondly, the technical solution of the present invention merges useless increments within the production cycle, ensuring that there is at most one incremental change record for the same record within the production cycle. This ensures the production safety of each destination and further reduces the pressure on the destination.

[0054] According to the technical solution presented here, on the one hand, the overall process tasks are decoupled and clearly divided, enabling acceptance of highly customized and data analysis requirements; on the other hand, data quality monitoring is integrated to prevent the potential for data pollution to escalate, and support for retrospective analysis and localization of incremental data issues becomes highly user-friendly. When problems occur, they can be quickly located by simply searching through multiple delivery statuses at the destination, facilitating problem resolution. Attached Figure Description

[0055] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0056] Figure 1 This is a flowchart of a data delivery method provided by an exemplary embodiment of the present invention;

[0057] Figure 2 This is a flowchart of a data processing method within a data production cycle provided by an exemplary embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the structure of a data delivery device provided in an exemplary embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0060] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0061] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0062] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0063] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0064] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0065] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0066] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0067] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0068] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0069] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0070] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0071] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0072] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0073] This invention provides a data delivery method, comprising: determining conditional dimension information, delivery dimension information, and data filtering rules based on a data acquisition request; subsequently performing conditional matching in a common dimension wide table based on the conditional dimension information to determine at least one associated object based on the matching result; obtaining delivery data for each associated object based on the delivery dimension information; and generating full-cycle data based on the delivery data for each associated object. Next, the full-cycle data is filtered according to the data filtering rules to obtain the full-cycle delivery data for the current cycle; and the full-cycle delivery data for the current cycle is delivered to the data requester according to a determined delivery method. Through this data delivery method, this invention can achieve functions such as customized data delivery, simultaneous full delivery to multiple users, merged delivery of incremental data within its validity period, and rapid repair of erroneous data.

[0074] Figure 1 This is a flowchart of a data delivery method provided by an exemplary embodiment of the present invention. Figure 1 As shown, the data delivery methods include:

[0075] Step 101: Receive a data acquisition request from the data requester, and parse the request to obtain conditional dimension information, delivery dimension information, and data filtering rules. Preferably, the conditional dimension information is used to limit the dimension conditions in the public dimension wide table. For example, if the data requester wants to query shareholder information of all companies or enterprises with invention patent applications, then the conditional dimension information is the invention patent applications. After identifying all companies or enterprises with invention patent applications, the delivery dimension information indicates the shareholder information of these companies or enterprises that needs to be acquired; that is, the delivery dimension information indicates the dimension information, attribute information, or field information that is desired. The data filtering rules are used to filter the actual data of the shareholder information of these companies or enterprises (e.g., selecting data within the current period) to obtain the data content of the dimension information, attribute information, or field information required by the data acquisition request.

[0076] In one embodiment, before receiving a data retrieval request from a data requester, the method further includes: performing statistics on all data tables in the database to determine all related objects and all attributes involved in the database. Typically, to provide the most comprehensive data possible to different data requesters, a large number of data tables are stored in the database, and each data table can be used to store different types of data. Preferably, in one embodiment, statistics on all related objects and all attributes involved in the database are used as the basis for creating a common dimension wide table.

[0077] In one embodiment, a common dimension wide table is created based on all related objects and all attributes involved in the database. Each attribute in the common dimension wide table is treated as a dimension, and a single record is set for each related object. For example, the data columns (or attributes, dimensions) of the common dimension wide table can be all attributes involved in the database; that is, the common dimension wide table can determine the association relationship between each related object and any attribute among all attributes involved in the database, such as whether the association relationship exists or not. The data column of the common dimension wide table is a single record for each related object (e.g., company, enterprise, individual, etc.), used to indicate the association relationship with each attribute or dimension. Therefore, the common dimension wide table can record or indicate the association relationship between each related object and each attribute or dimension; that is, the value of the dimension is used to indicate whether the related object has data content corresponding to the dimension (or attribute).

[0078] In a specific example, to meet the diverse data customization needs of the target end, this embodiment constructs a common dimension wide table (EntityWideTable) for related entities to be used for conditional data filtering. Data content operations are implemented through an update and maintenance program for the common dimension wide table (EntityWideTable). The definition of field indicators (or fields, where each field or field indicator describes an indicator of the associated object) in the common dimension wide table (EntityWideTable) includes: quantification of customized filtering field indicators and the output of the common dimension wide table. Based on support for multiple filtering conditions, the field indicators of the common dimension wide table are automatically expanded. Statistical indicators of entity-attribute are used as atomic indicators for statistical analysis of the common dimension wide table. All atomic indicators are numeric types, while other indicators that cannot be expressed as numeric types are converted according to event semantics. For example, when encoding regional conditions, whether the name contains a string is mapped to 1-0 in the field indicator enumeration value, etc.

[0079] In one embodiment, the method further includes adding a dimension corresponding to the newly added attribute to the public dimension wide table when a new attribute is detected in the database. Since the data columns of the public dimension wide table can contain all attributes involved in the database, when a new attribute is detected in the database, it needs to be reflected in the public dimension wide table, i.e., adding a dimension corresponding to the newly added attribute. Furthermore, after adding the dimension corresponding to the newly added attribute, the association relationship between each associated object and this dimension needs to be populated in the public dimension wide table. In a specific instance, the field or metric update method of the public dimension wide table EntityWideTable is as follows: for a given customized metric A, it is converted into the corresponding field metric ColA. If the field metric ColA has already been replaced by a field with the same function or content in the public dimension wide table EntityWideTable, then the columns of the public dimension wide table EntityWideTable are not expanded; otherwise, a new field metric ColA is added, and the update rules for the ColA column data are configured.

[0080] In one embodiment, the difference between the update frequency of the common dimension wide table and the update frequency of the full database is less than or equal to a difference threshold. To ensure that the dimension information of the common dimension wide table and the full database is synchronized as much as possible, it is necessary to avoid situations where the difference between the update frequency of the common dimension wide table and the update frequency of the full database is too large. For example, if the difference between the update frequency of the common dimension wide table and the update frequency of the full database is greater than the difference threshold, newly added dimensions or attributes in the full database may not be reflected in the common dimension wide table, resulting in incomplete full or incremental data provided based on the data retrieval request from the data requester; for example, the data content corresponding to the newly added dimensions or attributes may be missing. Furthermore, if the difference between the update frequency of the common dimension wide table and the update frequency of the full database is greater than the difference threshold, it may also result in the deletion of a specific dimension or attribute in the full database, while this specific dimension or attribute still exists in the common dimension wide table. In this case, it becomes impossible to provide full or incremental data for the specific dimension or attribute.

[0081] In a specific example, the field data update method for the common dimension wide table EntityWideTable is as follows: The update frequency of the field data in the common dimension wide table does not necessarily need to be exactly the same as the update frequency of the full-cycle data. However, the difference between the two should not be too large, for example, less than or equal to a difference threshold. In a preferred embodiment, the update frequency of the field data in the common dimension wide table is the same as the update frequency of the full-cycle data. When updating the field data of the common dimension wide table EntityWideTable, the common dimension wide table can be split into multiple sub-tables, and updates can be performed on a sub-table basis to achieve parallel acceleration of updates. Alternatively, the final update scheme of the synthesized common dimension wide table can be used to update the data according to the update cycle.

[0082] In one embodiment, to improve the update efficiency of the common dimension wide table, when updating the common dimension wide table, it is divided into multiple sub-tables, and each sub-table is updated in parallel. The updated sub-tables are then combined to form the updated common dimension wide table. Since the table size of the common dimension wide table can be very large, updating it through table partitioning can significantly improve update efficiency. In one embodiment, the method further includes determining the delivery time granularity t of the incremental data for the i-th data requester among all data requesters. i Where i is a natural number, and the granularity T of the data production cycle in the database is set, where T≤Min(t) i That is, the granularity of the data production cycle in the database is less than or equal to the delivery time granularity of incremental data from any data requester. In a specific instance, in the single dimension of data production and delivery, a full-cycle data PreFullTable is generated according to the data production cycle. The granularity of the data production cycle is not greater than the minimum granularity cycle requirement for incremental data delivery. That is, if the full data production cycle T has an incremental delivery time granularity of {t1, t2, t3, ...} for each destination, then T must satisfy T <= Min{t1, t2, t3, ...}.

[0083] Step 102: Perform conditional matching in the public dimension wide table based on the conditional dimension information, thereby identifying at least one associated object based on the matching results. For example, if a data requester wants to query shareholder information for all companies or enterprises with invention patent applications, then the conditional dimension information is the invention patent applications. In the public dimension wide table, determine the dimensions or attributes related to the invention patent applications, and identify the associated objects involved in the records with invention patent applications from the dimensions or attributes related to the invention patent applications, as at least one associated object in the conditional matching results. For example, by performing conditional matching in the public dimension wide table based on the conditional dimension information, each associated object with an invention patent application (e.g., a company, enterprise, or individual) is identified.

[0084] Step 103: Obtain delivery data for each associated object based on delivery dimension information, and generate full-cycle data based on the delivery data of each associated object. For example, if a data requester wants to query shareholder information for all companies or enterprises with invention patent applications, then the delivery dimension information is the shareholder information. For example, through the processing in step 102, each associated object with an invention patent application is identified. Subsequently, the shareholder information of each associated object with an invention patent application is obtained to constitute the delivery data. Optionally, the public dimension wide table belongs to a metadata table or a data record table because the data content in the public dimension wide table can indicate whether each associated object has a specific dimension or attribute, and the specific data table where the specific dimension or attribute is located. Delivery data can be obtained through the specific data table where the specific dimension or attribute is located, and full-cycle data can be generated based on the delivery data.

[0085] Step 104: Filter the full-cycle data according to data filtering rules to obtain the full delivery data for the current cycle. Deliver the full delivery data for the current cycle to the data requester according to the determined delivery method. Preferably, the full-cycle data can be all data for one or more dimensions or attributes, for example, from earliest to latest in time. Since the data requester may need data for one or more dimensions or attributes over all time lengths, or data for one or more dimensions or attributes over at least one cycle (e.g., a period of time or a data update cycle), this application filters the full-cycle data according to data filtering rules to obtain the full delivery data for the current cycle.

[0086] In one specific embodiment, producing a Full Table of full-cycle data includes: In the single-dimensional direction of delivery, the full-cycle data needs to be filtered according to the data filtering requirements of the destination. Specifically, the PreFull Table of full-cycle data is associated with the EntityWideTable based on entity fields, filtered by data metrics in the result table, and stored as the Full Table of delivered full-cycle data. Furthermore, if full-cycle data delivery is required, then the full-cycle data is delivered (Note: the Full Table data format is the same as the PreFull Table).

[0087] In one embodiment, delivering the full volume of delivery data for the current period to the data requester according to the determined delivery method includes: when the determined delivery method is full data delivery, directly delivering the full volume of delivery data for the current period to the data requester.

[0088] In one embodiment, delivering the full delivery data for the current period to the data requester according to the determined delivery method includes: when the determined delivery method is incremental data delivery, determining whether the data requester needs to perform data repair; if it is determined that the data requester needs to perform data repair, then using the full delivery data for the current period as a first dataset and using the full delivery data currently stored in the data requester as a second dataset; comparing the first dataset and the second dataset to determine incremental data; and delivering the incremental data to the data requester so that the data requester generates updated full delivery data based on the incremental data and the currently stored full delivery data.

[0089] In one embodiment, delivering the full delivery data of the current period to the data requester according to the determined delivery method includes: when the determined delivery method is incremental data delivery, determining whether the data requester needs to perform data repair; if it is determined that the data requester needs to perform data repair, then using the full delivery data of the current period as a first dataset and the full delivery data of the previous period as a second dataset; comparing the first dataset and the second dataset to determine incremental data; and delivering the incremental data to the data requester so that the data requester generates updated full delivery data based on the incremental data and the full delivery data of the previous period.

[0090] In one embodiment, comparing a first dataset and a second dataset to determine incremental data includes: determining the difference set of data record identifier fields between the first dataset and the second dataset, and constructing the data content in the second dataset belonging to the difference set of data record identifier fields as the insertion data increment according to the incremental format; determining the difference set of data record identifier fields between the first dataset and the second dataset, and constructing the data content in the first dataset belonging to the difference set of data record identifier fields as the deletion data increment according to the incremental format; determining the set of data inconsistencies in the intersection of the data record identifier fields of the first dataset and the second dataset, excluding the identifier field, and constructing the data content in the second dataset belonging to the inconsistent data set as the update data increment according to the incremental format; and combining the insertion data increment, the deletion data increment, and the update data increment to form incremental data.

[0091] In a specific example, when generating incremental data MincDataTable, if data repair is required, the FullTable state of the current period containing the data at the destination end is recorded as FullTableB; otherwise, the FullTable state of the previously generated (previous period) delivered full data is recorded as FullTableB. The incremental data MincDataTable is generated using a full diff (difference) device or process that inputs FullTableA and FullTableB. Based on the same data structure of FullTableA and FullTableB, the data is compared according to the record identifiers. The steps to generate MincDataTable are as follows:

[0092] To generate the INSERT increment in MincDataTable, extract the difference between the data record identifier fields of FullTableB and FullTableA, and construct the INSERT increment by taking all fields of the records in FullTableB that belong to this difference set according to the incremental format convention.

[0093] To generate the DELETE increment in MincDataTable, extract the difference set of the data record identifier fields between FullTableA and FullTableB, and construct the DELETE increment by taking all fields of the records in FullTableA that belong to this difference set according to the incremental format convention.

[0094] To generate the UPDATE increment in MincDataTable, extract the set of records in the intersection of the data record identifier fields of FullTableA and FullTableB where the remaining fields (excluding the identifier field) are inconsistent. Then, take all fields of the records in FullTableB that belong to this set and construct the UPDATE increment according to the incremental format convention (Note: If additional information about the preceding changes is needed, it can be added from the records in FullTableA).

[0095] Merge INSERT increments, DELETE increments, and UPDATE increments to form a complete incremental data MincDataTable.

[0096] In one embodiment, the method further includes performing data volume statistics on the incremental data insertion, deletion, and update. When the statistically calculated data volume exceeds a threshold, an alarm is triggered and data delivery is stopped. In another embodiment, the method further includes determining the volatility of the incremental data insertion, deletion, and update. When the volatility exceeds a volatility threshold, an alarm is triggered and data delivery is stopped. In a specific instance, the generated incremental data MincDataTable is validated. For example, validation is performed on metrics of interest to the target end according to scenario requirements. This embodiment sets volume statistics and periodic volatility threshold monitoring for the three types of incremental data. If any anomalies are detected, an alarm is triggered and the downstream task chain is blocked.

[0097] In one embodiment, the method further includes determining the generation time of each of the full-cycle data, the delivered full data, and the incremental data; determining the duration of each of the full-cycle data, the delivered full data, and the incremental data based on the difference between the current time and the generation time; and deleting the full-cycle data, the delivered full data, or the incremental data when the duration of the full-cycle data, the delivered full data, or the incremental data exceeds a time threshold. In a specific instance, expired data is maintained to prevent wasted storage and performance consumption caused by expired data. The relevant data that needs to be maintained for expiration are: the full-cycle data PreFullTable, the delivered full data FullTable, and the incremental data MincDataTable.

[0098] In one embodiment, after generating full-cycle data based on the delivery data of each associated object, the method further includes adding an update time to each field in each data table of the full-cycle data. In a specific instance, the tables used to generate the full-cycle data PreFullTable can have the latest update time added to each field based on the existing business logic. By using the update times of the fields used, it is determined whether to generate incremental data and what kind of incremental data to generate, thereby optimizing the generation process. For example, incremental data can be generated without going through a diff device or process.

[0099] Figure 2 This is a flowchart of a data processing method within a data production cycle provided by an exemplary embodiment of the present invention. The data processing method begins at step 201.

[0100] In step 202, the process of generating the full-cycle data PreFullTable begins. The full-cycle data PreFullTable is generated according to the data production cycle in the single-dimensional direction of production delivery. The granularity of the data production cycle must not exceed the minimum granularity cycle requirement for the delivery increment, i.e., the full-cycle data production cycle T, and the incremental delivery time granularity of each destination {t1,t2,t3,...}, then T must satisfy T<=Min{t1,t2,t3,...}.

[0101] In step 203, it is determined whether the common dimension wide table needs to be updated. If yes, proceed to step 204; otherwise, proceed to step 205. Before determining whether the common dimension wide table needs to be updated, the process also includes constructing the relevant entity common dimension wide table, EntityWideTable. To meet the different data customization needs of the destination, the most common method is to construct the relevant entity common dimension wide table, EntityWideTable, for filtering. This part of the operation is implemented through the EntityWideTable update and maintenance program. The definition method of the field indicators of the common dimension wide table, EntityWideTable: the quantification of customized filtering indicators and the output of the common dimension wide table. This invention accumulates the characteristics of automatic expansion of common dimension table fields based on the existing filtering conditions. Entity-attribute statistical indicators are used as atomic indicators for dimension table statistics. All of them are numerical types. Other indicators that cannot be expressed as numerical types are converted according to the event semantics. For example, the regional conditions are encoded, and whether the name contains a certain string is mapped to 1-0 in the field enumeration value, etc.

[0102] In step 204, the common dimension wide table EntityWideTable of the relevant entities is updated, followed by step 205. The update method for the field metrics in the common dimension wide table EntityWideTable is as follows: For a given customized metric A, it is converted into the corresponding field metric ColA. If ColA has already been replaced by a field with the same function in EntityWideTable, the EntityWideTable column is not expanded; otherwise, a new ColA is added, and the data update rules for the ColA column are configured. Regarding the data update method for the field data in the common dimension wide table EntityWideTable: In terms of the update cycle, the data frequency of the common dimension wide table does not need to be completely consistent with the full cycle, but the difference between the two should not be too large. This invention chooses that both cycles be the same. In terms of update operations, a scheme of splitting the wide table into multiple sub-tables for parallel acceleration calculation and finally merging them into a wide table is adopted to update the data according to the update cycle.

[0103] In step 205, associate the common dimension wide table EntityWideTable.

[0104] In step 206, the full data FullTable is generated and delivered.

[0105] In step 207, determine if there is a need for full data. If yes, proceed to step 208 to deliver the full data; otherwise, proceed to step 209. In the single-dimensional delivery direction, the full-cycle data needs to be filtered according to the target data filtering requirements. Specifically, the full-cycle data PreFullTable is associated with the common dimension table EntityWideTable based on entity fields, filtered by data metrics in the result table, and stored as the delivered full-cycle data FullTable. If full delivery is required, it will proceed. The data format of the FullTable is the same as that of the PreFullTable.

[0106] In step 209, determine whether the destination requires data repair. If yes, proceed to step 211; otherwise, proceed to step 210. If data repair is required, record the FullTable status of the period containing the destination data as FullTableB; otherwise, record the FullTable status of the previously produced full data as FullTableB. Use FullTableA and FullTableB to input the full diff device to generate an incremental MincDataTable. Based on the identical structure of FullTableA and FullTableB, the MincDataTable is generated by comparing data record identifiers as follows:

[0107] To generate the INSERT increment in MincDataTable, extract the difference between the data record identifier fields of FullTableB and FullTableA, and construct the INSERT increment by taking all fields of the records in FullTableB that belong to this difference set according to the incremental format convention.

[0108] To generate the DELETE increment in MincDataTable, extract the difference set of the data record identifier fields between FullTableA and FullTableB, and construct the DELETE increment by taking all fields of the records in FullTableA that belong to this difference set according to the incremental format convention.

[0109] To generate the UPDATE increment in MincDataTable, extract the set of records in the intersection of the data record identifier fields of FullTableA and FullTableB where the remaining fields (excluding the identifier field) are inconsistent. Take all fields of the records in FullTableB that belong to this set and construct the UPDATE increment according to the incremental format convention (if additional information about the preceding changes is needed, it is added from the records in FullTableA).

[0110] Merge INSERT increments, DELETE increments, and UPDATE increments to form a complete incremental data MincDataTable.

[0111] In step 210, the current cycle's full-volume FullTableA and the previous cycle's full-volume FullTableB are set as the two inputs to the incremental data output module.

[0112] In step 211, the current cycle delivery full tableA and the destination full tableB are set as the two inputs to the output incremental data module.

[0113] In step 212, input FullTableA and FullTableB to generate incremental data MincDataTable.

[0114] In step 213, expired data is cleaned up.

[0115] Step 214, End.

[0116] In addition, the data processing method also includes data validation of the generated incremental data MincDataTable. Validation is performed on the target-side metrics according to scenario requirements. Currently, for the three types of incremental data, only magnitude statistics and periodic fluctuation rate threshold monitoring are set. Anomalies trigger alarms and block downstream task chains. The data processing method also includes maintaining expired data. The relevant data requiring expiration maintenance include: full-cycle data PreFullTable, delivered full-cycle data FullTable, and incremental data MincDataTable, to prevent wasted storage and performance consumption caused by expired data. The table used for generating full-cycle data PreFullTable can have its latest update time added to each field based on the existing business logic. By using the update time of the fields used, it is determined whether to generate incremental data and what type of incremental data to generate, thereby optimizing the generation process. For example, incremental data can be generated without a diffing device or process.

[0117] Figure 3 This is a schematic diagram of a data delivery apparatus provided in an exemplary embodiment of the present invention. The data delivery apparatus includes:

[0118] The parsing unit 301 is used to receive a data acquisition request from the data requester, and parse the data acquisition request to obtain condition dimension information, delivery dimension information and data filtering rules.

[0119] Matching unit 302 is used to perform conditional matching in a common dimension wide table based on conditional dimension information, thereby determining at least one associated object based on the matching result of the conditional matching.

[0120] Generation unit 303 is used to obtain delivery data for each associated object based on delivery dimension information, and to generate full-cycle data based on the delivery data of each associated object; and

[0121] Delivery unit 304 is used to filter the full-cycle data according to data filtering rules to obtain the full-cycle delivery data for the current period, and deliver the full-cycle delivery data for the current period to the data requester according to the determined delivery method. Delivery unit 304 is also used to add an update time to each field in each data table of the full-cycle data.

[0122] The preprocessing unit 305 is used to perform statistics on all data tables in the database to determine all related objects and all attributes involved in the database; and to create a common dimension wide table, in which each attribute in all attributes is treated as a dimension, and a single record is set for each related object in the common dimension wide table; wherein the value of the dimension is used to indicate whether the related object has data content corresponding to the dimension.

[0123] In one embodiment, when a newly added attribute is detected in the database, the preprocessing unit 305 adds a dimension corresponding to the newly added attribute to the common dimension wide table. The difference between the update frequency of the common dimension wide table and the update frequency of the full database is less than or equal to a difference threshold. When updating the common dimension wide table, the preprocessing unit 305 divides the common dimension wide table into multiple sub-tables, updates each sub-table in parallel, and combines the updated sub-tables into an updated common dimension wide table. The preprocessing unit 305 is also used to determine the incremental data delivery time granularity t for the i-th data requester among all data requesters. i , where i is a natural number; and set the granularity T of the data production cycle in the database, where T≤Min(t i ).

[0124] In one embodiment, delivery unit 304 is used to deliver the full data of the current period directly to the data requester when the determined delivery method is full data delivery.

[0125] In one embodiment, when the determined delivery method is incremental data delivery, the delivery unit 304 determines whether the data requester needs to perform data repair; if it is determined that the data requester needs to perform data repair, the full delivery data of the current period is used as the first dataset, and the full delivery data currently stored in the data requester is used as the second dataset; the first dataset and the second dataset are compared to determine the incremental data; the incremental data is delivered to the data requester so that the data requester can generate updated full delivery data based on the incremental data and the currently stored full delivery data.

[0126] In one embodiment, delivery unit 304 is configured to determine whether the data requester needs to perform data repair when the determined delivery method is incremental data delivery; if it is determined that the data requester needs to perform data repair, then the full delivery data of the current period is used as the first dataset, and the full delivery data of the previous period is used as the second dataset; and the first dataset and the second dataset are compared to determine the incremental data; and the incremental data is delivered to the data requester so that the data requester can generate updated full delivery data based on the incremental data and the full delivery data of the previous period.

[0127] In one embodiment, delivery unit 304 is further configured to: determine the difference set of data record identifier fields between the first dataset and the second dataset; construct the data content in the second dataset belonging to the difference set of data record identifier fields into an insert data increment according to an incremental format; determine the difference set of data record identifier fields between the first dataset and the second dataset; construct the data content in the first dataset belonging to the difference set of data record identifier fields into a delete data increment according to an incremental format; determine the data set in the intersection of the data record identifier fields of the first dataset and the second dataset that has inconsistent data among the fields other than the identifier field; construct the data content in the second dataset belonging to the inconsistent data set into an update data increment according to an incremental format; and combine the insert data increment, delete data increment, and update data increment into incremental data.

[0128] Monitoring unit 306 is used to perform data volume statistics on inserted data increments, deleted data increments, and updated data increments. When the statistically calculated data volume exceeds a volume threshold, an alarm is triggered and data delivery is stopped. Monitoring unit 306 is also used to determine the quantity volatility of inserted data increments, deleted data increments, and updated data increments. When the data volume volatility exceeds a volatility threshold, an alarm is triggered and data delivery is stopped. Monitoring unit 306 is also used to determine the generation time of full-cycle data, delivered full data, and incremental data; based on the difference between the current time and the generation time, determine the duration of each of the full-cycle data, delivered full data, and incremental data; when the duration of full-cycle data, delivered full data, or incremental data exceeds a time threshold, the full-cycle data, delivered full data, or incremental data is deleted.

[0129] Figure 4 This is a schematic diagram of an electronic device provided in an exemplary embodiment of the present invention. Figure 4 As shown, the electronic device 40 includes one or more processors 410 and memory 420.

[0130] The processor 410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0131] Memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 410 may execute the program instructions to implement the data delivery methods of the software programs described in the various embodiments of the present invention above, and / or other desired functions. In one example, the electronic device may also include an input device 430 and an output device 440, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0132] In addition, the input device 430 may also include, for example, a keyboard, a mouse, etc.

[0133] The output device 440 can output various information to the outside. The output device 440 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0134] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0135] In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products and computer-readable storage media, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the data delivery methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0136] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0137] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the data delivery methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0138] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0139] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0141] The block diagrams of devices, apparatuses, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0142] The methods and apparatus of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0143] It should also be noted that in the apparatus, device, and method of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0144] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A data delivery method, characterized in that, The method includes: Receive a data acquisition request from a data requester, and parse the data acquisition request to obtain condition dimension information, delivery dimension information, and data filtering rules; Based on the conditional dimension information, conditional matching is performed in the public dimension wide table, thereby determining at least one associated object based on the matching result of the conditional matching. Based on the aforementioned delivery dimension information, delivery data for each associated object is obtained, and full-cycle data is generated based on the delivery data for each associated object; and The full-cycle data is filtered according to the data filtering rules to obtain the full delivery data for the current cycle. The full delivery data for the current cycle is then delivered to the data requester according to the determined delivery method, including: When the determined delivery method is incremental data delivery, determine whether the data requester needs to perform data repair; If it is determined that the data requester needs data repair, then the full delivery data of the current period is used as the first dataset, and the full delivery data currently stored in the data requester is used as the second dataset. The first and second datasets are compared to determine the incremental data. The incremental data is delivered to the data requester so that the data requester can generate updated full delivery data based on the incremental data and the currently stored full delivery data.

2. The method according to claim 1, characterized in that, Before receiving the data acquisition request from the data requester, the method further includes: Perform statistics on all data tables in the database to determine all related objects and all attributes involved in the database; Create a common dimension wide table, in which each attribute of all attributes is treated as a dimension, and set a single record for each associated object in the common dimension wide table; The dimension value is used to indicate whether the associated object has data content corresponding to the dimension.

3. The method according to claim 2, characterized in that, It also includes adding a dimension corresponding to the newly added attribute to the public dimension wide table when a newly added attribute is detected in the database.

4. The method according to claim 2, characterized in that, in, The difference between the update frequency of the common dimension wide table and the update frequency of the full data of the database is less than or equal to the difference threshold. Specifically, when updating the common dimension wide table, the common dimension wide table is divided into multiple sub-tables, each sub-table is updated in parallel, and the updated sub-tables are combined to form the updated common dimension wide table.

5. The method according to claim 2, characterized in that, It also includes determining the delivery time granularity ti of the incremental data for the i-th data requester among all data requesters, where i is a natural number; Set the granularity T of the data production cycle in the database, where T≤Min(ti).

6. The method according to claim 1, characterized in that, The step of delivering the full volume of delivery data for the current period to the data requester according to the determined delivery method includes: When the determined delivery method is full data delivery, the full data for the current period will be delivered directly to the data requester.

7. The method according to claim 1, characterized in that, The step of delivering the full volume of delivery data for the current period to the data requester according to the determined delivery method includes: When the determined delivery method is incremental data delivery, determine whether the data requester needs to perform data repair; If it is determined that the data requester needs data repair, then the full delivery data of the current period is used as the first dataset, and the full delivery data of the previous period is used as the second dataset; and the first dataset and the second dataset are compared to determine the incremental data. The incremental data is delivered to the data requester so that the data requester can generate updated full delivery data based on the incremental data and the full delivery data of the previous period.

8. The method according to claim 1 or 7, characterized in that, The step of comparing the first dataset and the second dataset to determine the incremental data includes: Determine the difference set of data record identifier fields between the first dataset and the second dataset, and construct the data content in the second dataset that belongs to the difference set of data record identifier fields as the data insertion increment according to the incremental format; Determine the difference set of data record identifier fields between the first dataset and the second dataset, and construct the data content in the first dataset that belongs to the difference set of data record identifier fields as the data deletion increment according to the incremental format; Identify a set of data inconsistencies within the intersection of the data record identifier fields of the first and second datasets (excluding the identifier field). Construct the update data increment from the data content of this inconsistent set in the second dataset using an incremental format. The incremental data is composed of the inserted data increment, the deleted data increment, and the updated data increment.

9. The method according to claim 8, characterized in that, It also includes performing data volume statistics on the incremental insertion, incremental deletion, and incremental update of data. When the statistical data volume exceeds the volume threshold, an alarm is triggered and data delivery is stopped.

10. The method according to claim 9, characterized in that, It also includes determining the volatility of the number of inserted data increments, deleted data increments, and updated data increments. When the data volatility exceeds a volatility threshold, an alarm is triggered and data delivery is stopped.

11. The method according to claim 1, characterized in that, This also includes determining the generation time of full-cycle data, the delivery of full data, and incremental data; Based on the difference between the current time and the generation time, determine the duration of each of the full-cycle data, the delivery of full data, and the incremental data; When the duration of full-cycle data, full-data delivery, or incremental data exceeds the time threshold, the full-cycle data, full-data delivery, or incremental data will be deleted.

12. The method according to claim 1, characterized in that, After generating full-cycle data based on the delivery data of each associated object, the process also includes: Add an update time to each field in each data table of the full-cycle data.

13. A data delivery device, characterized in that, The device includes: The parsing unit is used to receive a data acquisition request from a data requester and parse the data acquisition request to obtain condition dimension information, delivery dimension information and data filtering rules. A matching unit is used to perform conditional matching in a common dimension wide table based on the conditional dimension information, thereby determining at least one associated object based on the matching result of the conditional matching. A generation unit is configured to acquire delivery data for each associated object based on the delivery dimension information, and generate full-cycle data based on the delivery data of each associated object; and A delivery unit is configured to filter the full-cycle data according to the data filtering rules to obtain the full-cycle delivery data for the current cycle, and deliver the full-cycle delivery data for the current cycle to the data requester according to a determined delivery method, including: When the determined delivery method is incremental data delivery, determine whether the data requester needs to perform data repair; If it is determined that the data requester needs data repair, then the full delivery data of the current period is used as the first dataset, and the full delivery data currently stored in the data requester is used as the second dataset. The first and second datasets are compared to determine the incremental data. The incremental data is delivered to the data requester so that the data requester can generate updated full delivery data based on the incremental data and the currently stored full delivery data.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method according to any one of claims 1-12.

15. An electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-12.

Images