Data storage method and device for real-time data analysis

Abstract

The embodiment of the application discloses a data storage method and device for real-time data analysis, platform data generated by a resource platform and media data generated by a resource putting media are acquired; the platform data and the media data are written into a database, and a wide table data structure is used to store the platform data and the media data to obtain a data table during data writing; wherein, the platform data and the media data are managed through a data key and a source field in the data table; the platform data and the media data stored in the data table are aggregated and calculated according to the data key, and an aggregated data table is obtained, the wide table data structure is used as a basic data storage structure, the media data is written into the database in a full amount mode, the platform data is written into the database in an incremental mode, and a differentiated scene processing mode is realized; the data in the data table is aggregated and calculated, so that the data aggregation degree is improved, and the real-time performance of data query is guaranteed.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, specifically to a data storage method and apparatus for real-time data analysis. Background Technology

[0002] In current technologies, the common solution for real-time analysis of massive amounts of data is to use relational databases. Star schemas, snowflake schemas, and Online Analytical Processing (OLAP) databases are representative wide-table models within relational databases. For relational database solutions, fact table data and data from various dimension tables are stored in separate relational tables. For scenarios with large amounts of fact table data, database sharding and table partitioning are necessary. When querying data, it is required to perform join queries on multiple partitioned tables containing related dimension tables and fact table data.

[0003] While the above solution offers flexible data updates for each table, allowing for row-level updates, it requires multi-table joins for complex aggregation queries, resulting in complex query logic. As the volume of business data increases, the efficiency of aggregation queries decreases, making it difficult to meet the needs of real-time analysis of massive amounts of data. Summary of the Invention

[0004] In view of the above problems, this application proposes a data storage method, apparatus, computing device and computer storage medium for real-time data analysis, to solve the following problems: existing data table aggregation query efficiency is low and cannot meet the needs of real-time analysis of massive data.

[0005] According to one aspect of the embodiments of this application, a data storage method for real-time data analysis is provided, comprising:

[0006] Acquire platform data generated by the resource platform and media data generated by the media outlets where resources are deployed;

[0007] Platform data and media data are written into the database, and a wide table data structure is used to store the platform data and media data during the data writing process to obtain a data table; the platform data and media data are managed in the data table through data keys and source fields.

[0008] Based on the data key, the platform data and media data stored in the data table are aggregated and calculated to obtain the aggregated data table.

[0009] Furthermore, acquiring platform data generated by the resource platform and media data generated by the resource delivery media further includes:

[0010] Acquire platform data generated by incremental updates from the resource platform;

[0011] The resource provider retrieves all media data from the media being delivered via a scheduled task using a media interface.

[0012] Furthermore, before acquiring the platform data generated incrementally by the resource platform, the method also includes:

[0013] By processing the log data of the resource platform using real-time data processing tools, the incremental platform data generated by the resource platform can be determined.

[0014] Furthermore, platform data and media data are written into the database, and a wide table data structure is used to store the platform data and media data during the data writing process to obtain a data table that further includes:

[0015] Platform data is written to the database incrementally. During the data writing process, for each piece of platform data, the corresponding data key and the platform data field in the source field are set in the data table according to the data content of the platform data, and the media data field in the source field corresponding to the platform data is set to the preset value.

[0016] Media data is written to the database in full. During the media data writing process, for each piece of media data, a corresponding data key and a media data field in the source field are set for the media data in the data table according to the data content of the media data, and the platform data field in the source field of the media data is set to a preset value.

[0017] Furthermore, the data key includes: dimensional data and partitions;

[0018] For each platform's data, setting a corresponding data key in the data table based on the data content of the platform's data further includes: setting dimension data in the corresponding data key in the data table based on the data content of the platform's data, and setting the partition in the corresponding data key of the platform's data as the platform identifier;

[0019] For each media data, setting a corresponding data key in the data table based on the data content of the media data further includes: setting dimension data in the corresponding data key of the media data based on the data content of the media data, and setting the partition in the corresponding data key of the media data as the media identifier.

[0020] Furthermore, writing all media data into the database in a full manner includes:

[0021] A temporary partition is created in the database, media data is written to the temporary partition, and the temporary partition is used to replace the target partition corresponding to the media data to complete the update of the media data in the database.

[0022] Furthermore, the source field is a field in numeric form;

[0023] Based on the data key, the platform data and media data stored in the data table are aggregated and calculated to obtain the aggregated data table, which further includes:

[0024] At preset intervals, query the data table for platform data and media data with the same data key;

[0025] The source fields corresponding to platform data with the same data key are summed to form the corresponding aggregated platform data;

[0026] The source fields corresponding to media data with the same data key are summed to form the corresponding aggregated media data.

[0027] According to another aspect of the embodiments of this application, a data storage device for real-time data analysis is provided, comprising:

[0028] The data acquisition module is used to acquire platform data generated by the resource platform and media data generated by the resource delivery media.

[0029] The data storage module is used to write platform data and media data into the database. During the data writing process, a wide table data structure is used to store the platform data and media data to obtain a data table. The platform data and media data are managed in the data table through data keys and source fields.

[0030] The data aggregation module is used to perform aggregation calculations on the platform data and media data stored in the data table according to the data key, and obtain the aggregated data table.

[0031] According to another aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0032] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the data storage method described above for real-time data analysis.

[0033] According to another aspect of the embodiments of this application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform an operation corresponding to the data storage method for real-time data analysis described above.

[0034] According to an embodiment of this application, a data storage method and apparatus for real-time data analysis are provided. The method involves acquiring platform data generated by a resource platform and media data generated by resource delivery media; writing the platform data and media data into a database, and using a wide table data structure to store the platform data and media data during the data writing process to obtain a data table; managing the platform data and media data in the data table using data keys and source fields; and performing aggregation calculations on the platform data and media data stored in the data table according to the data keys to obtain an aggregated data table. This application uses a wide table data structure as the basic data storage structure, which can improve data query efficiency. Furthermore, media data is written to the database in a full manner, while platform data is written to the database incrementally, enabling differentiated scenario processing. For real-time analysis needs, the data in the data table is periodically aggregated, thereby effectively improving the data aggregation degree and achieving complex aggregation of massive amounts of data. Data queries on the aggregated data ensure the real-time nature of the data query.

[0035] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of the embodiments of this application are described below. Attached Figure Description

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0037] Figure 1 A schematic flowchart of a data storage method for real-time data analysis according to an embodiment of this application is shown;

[0038] Figure 2 A flowchart illustrating a data storage method for real-time data analysis according to another embodiment of this application is shown;

[0039] Figure 3a A schematic diagram of the wide table data structure according to an embodiment of this application is shown;

[0040] Figure 3b This illustration shows a schematic diagram of the platform data zero-padding operation process according to an embodiment of this application;

[0041] Figure 3c This illustration shows a schematic diagram of creating a temporary partition during the writing process of media data, according to an embodiment of this application.

[0042] Figure 3d This illustration shows a schematic diagram of the platform data before data aggregation in an embodiment of this application.

[0043] Figure 3e This illustration shows a schematic diagram of the platform data after data aggregation, as shown in an embodiment of this application.

[0044] Figure 4 A structural block diagram of a data storage device for real-time data analysis according to an embodiment of this application is shown;

[0045] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] Online Analytical Processing (OLAP) databases are software technologies that enable analysts to quickly, consistently, and interactively view information from various perspectives to achieve a deep understanding of the data. They use wide table data structures to store fact table data.

[0048] Fact table: A result table generated based on a certain dimension after data aggregation; it is a specific statistical table, such as system logs or sales records. The records in the fact table will continuously increase dynamically, and the volume occupied is usually much larger than that of other tables; in this application, it is uniformly referred to as a data table.

[0049] Dimension tables, also known as lookup tables, are used to store the attribute values ​​of dimensions. They can be associated with fact tables to extract data from the fact tables; examples include date tables and location tables.

[0050] Incremental (Append) writing: Data generated sequentially over time with no overlap is written to the database in a step-by-step appending manner.

[0051] Full write: also known as full replacement write, in the time dimension, the data generated earlier is a subset of the data generated later. During the data writing process, the data written later needs to overwrite the data written earlier.

[0052] Figure 1A schematic flowchart of a data storage method for real-time data analysis according to an embodiment of this application is shown, such as... Figure 1 As shown, the method includes the following steps:

[0053] Step S110: Obtain platform data generated by the resource platform and media data generated by the resource delivery media.

[0054] Specifically, while the wide table data structure of existing OLAP databases can avoid multi-table joins, it cannot achieve real-time data updates. A resource platform refers to a platform used to manage the resources provided by resource providers for distribution. These resources can specifically include content such as advertisements, images, and videos. Resource distribution media refers to media channels used to distribute and display resources; for example, game application platforms, e-book reading platforms, and online shopping platforms can be used as resource distribution media. Taking advertisements as an example, the resource provider can be the advertiser. The advertiser publishes their desired advertisements on the resource platform. The resource distribution media connects with the resource platform, retrieves the advertisements published by the advertiser from the resource platform, and distributes and displays the advertisements on their corresponding pages (such as game app pages) using its own resource distribution methods, thereby achieving resource distribution and enabling users to easily access resource content.

[0055] Resource management involves two processes: publishing resources through a resource platform and distributing resources through media outlets. Therefore, resource management data includes platform data generated by the resource platform and media data generated by the media outlets. Platform data refers to data obtained from the resource platform related to resource providers publishing resources, such as resource material data, the number of resources published, the publication time, the resource provider, the distribution requirements, and the distribution price. Media data refers to data obtained from the media outlets related to resource distribution, such as the amount of resources distributed, the duration of resource exposure, the number of clicks on resources, and the conversion rate.

[0056] Since different types of data have different characteristics, they are written to the database in different ways. Therefore, in this embodiment, it is necessary to obtain platform data generated by the resource platform and media data generated by the resource delivery media, and to use different methods to write different types of data to the database.

[0057] Step S120: Write platform data and media data into the database, and use a wide table data structure to store platform data and media data to obtain a data table during the data writing process.

[0058] Specifically, in this embodiment, a wide table data structure of an OLAP database can be used to uniformly store platform data and media data. Since platform data and media data are obtained from the resource platform and the resource delivery media respectively, i.e., their sources are different, the wide table data structure can manage platform data and media data through data keys and source fields, with the source field identifying the data's origin. In this step, platform data and media data can be written to the database using different writing methods based on their characteristics, thus adapting data writing to differentiated scenarios. For example, platform data can be written to the database incrementally, while media data can be written to the database in full.

[0059] Step S130: According to the data key, perform aggregation calculation on the platform data and media data stored in the data table to obtain the aggregated data table.

[0060] Data aggregation operations can be performed on a schedule, that is, at preset intervals, overlapping or identical data in the data table are aggregated. Generally, data aggregation methods can include data accumulation, deduplication, etc. In this application, platform data and media data are managed based on data keys and source fields. Therefore, platform data and media data with the same data keys can be queried in the data table; the source fields corresponding to the platform data with the same data keys are accumulated to form the corresponding aggregated platform data; the source fields corresponding to the media data with the same data keys are accumulated to form the corresponding aggregated media data.

[0061] According to an embodiment of this application, a data storage method for real-time data analysis is provided. This method involves acquiring platform data generated by a resource platform and media data generated by resource delivery media; writing the platform data and media data into a database; and using a wide table data structure to store the platform data and media data during the data writing process to obtain a data table. The data table manages the platform data and media data through data keys and source fields. Aggregation calculations are performed on the platform data and media data stored in the data table according to the data keys to obtain an aggregated data table. This application uses a wide table data structure of an OLAP database as the basic data storage structure, and writes media data to the database in a full manner while writing platform data incrementally, achieving differentiated scenario processing. For real-time analysis needs, the data in the data table is periodically aggregated, effectively improving the data aggregation degree, achieving complex aggregation of massive amounts of data, and ensuring the real-time nature of data queries.

[0062] Figure 2 A flowchart illustrating a data storage method for real-time data analysis according to another embodiment of this application is shown, such as... Figure 2As shown, the method includes the following steps:

[0063] Step S210: Process the log data of the resource platform using real-time data processing tools to determine the platform data generated incrementally by the resource platform.

[0064] Specifically, taking game advertising as an example, the advertising business involves two processes: publishing resources through a game advertising platform (i.e., a resource platform) and placing resources through advertising media (i.e., resource placement media). Advertising business data includes platform data generated by the advertising platform and media data generated by the advertising media. Platform data generated by the game advertising platform refers to data related to the advertising provider's ad publishing, such as ad creative data, ad publishing quantity, ad publishing time, the corresponding advertising provider, the corresponding placement requirements, and the corresponding placement price. For acquiring platform data, real-time data processing tools can be used to perform real-time calculations and analysis on the game advertising platform's log data, which is then used as incremental platform data generated by the game advertising platform. These real-time data processing tools can include open-source stream processing frameworks, such as the Apache Flink stream data processing engine.

[0065] Step S220: Obtain platform data generated incrementally by the resource platform; obtain media data fully retrieved by the resource provider from the resource delivery media through a scheduled task using the media interface.

[0066] In this step, the platform data generated incrementally by the resource platform is obtained through incremental (Append) writing. For media data, the media interface can be used to pull the full amount of data from the resource placement media through a scheduled task. Taking game advertisements as an example, the advertiser (i.e. the resource provider) can pull the full amount of media data from the advertisement placement media through the interface provided by the media through a pre-set scheduled task.

[0067] It should be noted that data synchronization generally falls into two categories: full replacement write and incremental write. Full replacement write involves writing all data to the database daily (e.g., avoiding peak business hours) or periodically. Full write can be achieved by directly overwriting the entire database, using new data to overwrite old data, or by executing update logic (checking before overwriting; if the new and old data are inconsistent, then updating). Since a resource platform manages resources from multiple providers, the analysis system used for real-time data analysis in this embodiment can interface with the resource platform to write platform data generated by the resource platform into the analysis system's database. To reduce the writing pressure on the platform data, incremental write can be used. However, resource delivery media are diverse; the same resource from the same provider may be delivered through different media. Due to the diversity of resource delivery media and the complexity of media data, to ensure the accuracy of the media data stored in the database, a full replacement write can be used. Furthermore, the media data can be retrieved from the resource delivery media's end to the resource provider's local machine by the resource provider through a scheduled task using the media interface provided by the resource delivery media. The media data can be written to the database in one go through the full replacement write method, without the need for multiple retrievals.

[0068] Step S230: Write platform data to the database incrementally. During the platform data writing process, for each piece of platform data, set the corresponding data key and the corresponding platform data field in the source field of the data table according to the data content of the platform data, and set the media data field in the source field of the platform data to the preset value.

[0069] Specifically, incremental writes are based on full replacement writes. This means that incremental writes require first copying all the data and then updating it synchronously using incremental writes. Incremental writes refer to capturing data from a specific point in time (update time) or checkpoint for synchronization, rather than performing random full synchronization. Since the platform data generated by the resource platform is produced by processing the resource platform's log data, and log data updates frequently, incremental writes can better improve write efficiency and save database resources.

[0070] Step S240: Write the media data to the database in full. During the media data writing process, for each piece of media data, set the corresponding data key and the corresponding media data field in the source field of the media data in the data table according to the data content of the media data, and set the platform data field in the source field of the media data to the preset value.

[0071] Figure 3a A schematic diagram of the wide table data structure of an embodiment of this application is shown, such as... Figure 3a As shown, platform data and media data are managed in the data table through data keys and source fields. The data key can include dimension data and partitions; the source fields include platform data fields and media data fields. Specifically, compared to narrow tables, wide tables, as the name suggests, are database tables with more fields. They typically refer to a database table where business-related metrics, dimensions, and attributes are linked together. Because different content is stored in the same table, the main impact is data redundancy. The corresponding advantage is improved query performance and convenience. Therefore, this embodiment uses a wide table data structure to improve database query performance; furthermore, to overcome data redundancy, data aggregation operations are performed.

[0072] In an optional implementation, step S230 further includes: setting dimension data in the corresponding data key of the platform data in the data table according to the data content of the platform data, and setting the partition in the corresponding data key of the platform data as a platform identifier; wherein, the platform identifier can be indicated by the platform identifier; step S240 further includes: setting dimension data in the corresponding data key of the media data in the data table according to the data content of the media data, and setting the partition in the corresponding data key of the media data as a media identifier; wherein, the media identifier can be indicated by the media identifier.

[0073] Figure 3b This illustration shows a schematic diagram of the platform data zero-padding operation processing according to an embodiment of this application, such as... Figure 3b As shown, the log data from the resource platform is processed by a real-time data processing tool to obtain platform data, which is then incrementally written to the database. Taking a preset value of 0 as an example, all media data fields in the platform data table are set to 0, and the partition field in the data key is set to the platform identifier ("platform"). Similarly, when media data is written to the database, all platform data fields in the media data table are set to 0, and the partition field in the data key is set to the media identifier ("media"). It should be noted that when writing platform data or media data to the database, the source fields of the other data need to be padded with zeros. For example, when the preset value is 0, after writing platform data, the media data field of the platform data written to the data table needs to be padded with zeros; after writing media data, the platform data field of the media data written to the data table needs to be padded with zeros.

[0074] In an optional implementation, step S240 further includes: creating a temporary partition in the database, writing media data into the temporary partition, and replacing the target partition corresponding to the media data with the temporary partition to complete the update of the media data in the database.

[0075] Specifically, the media data is retrieved from the resource delivery media's end to the resource provider's local machine by the resource provider through a scheduled task using the media interface provided by the resource delivery media, and then written to the database in full. Since OLAP databases do not support row-level data updates, in the scenario of full data replacement and writing, only partition replacement can be used to update the data. Figure 3c This illustration shows a schematic diagram of creating a temporary partition during the writing process of media data, as per an embodiment of this application. Figure 3c As shown, for writing media data, a temporary partition is created in the database, the latest media data is written to the temporary partition, and then the temporary partition is used to replace the target partition, thus completing the data update.

[0076] Step S250: According to the data key, perform aggregation calculations on the platform data and media data stored in the data table to obtain the aggregated data table.

[0077] In one optional implementation, the source field is a numerical field; step S250 further includes: querying the data table for platform data and media data with the same data key at preset intervals; accumulating the source fields corresponding to the platform data with the same data key to form the corresponding aggregated platform data; accumulating the source fields corresponding to the media data with the same data key to form the corresponding aggregated media data.

[0078] Specifically, by pre-setting a periodic data aggregation calculation interval in the database, and querying the data table at pre-set intervals for platform data and media data with the same key, the aggregation degree of data within a partition can be improved, the amount of data in the table can be reduced, and the efficiency of data query can be improved by periodically aggregating platform data and media data with the same key. Figure 3d This application illustrates a schematic diagram of platform data before data aggregation, as shown in the embodiment of this application. Figure 3d As shown, the data in the data key, partitioned by the platform identifier, contains 6 data entries. The dimensional data can include channel dimensions, group dimensions, etc. Figure 3dAs can be seen, there are 3 records with the key "channel1, group1", with platform data field values ​​of 1, 3 and 4 respectively, and media data field values ​​of 0 after zero padding; there are 2 records with the key "channel2, group1", with platform data field values ​​of 1 and 3 respectively, and media data field values ​​of 0 after zero padding; there is 1 record with the key "channel3, group1", with platform data field value of 4 and media data field value of 0.

[0079] Furthermore, by aggregating platform data with the same key at the row granularity level, the corresponding platform data fields are summed, and the corresponding media data fields are summed. Figure 3e This application illustrates a schematic diagram of platform data after data aggregation, as shown in the embodiment of the present application. Figure 3e As shown, after data aggregation, the number of records in the data table has decreased from 6 to 3, and the field values ​​of the platform data fields have changed from... Figure 3d The numbers 1, 3, and 4 in the first three rows before aggregation become 1 + 3 + 4 = 8 in the first row after aggregation. Figure 3d The 1 in the fourth row and the 3 in the fifth row become the sum of 1 + 3 = 4 in the second row after aggregation. Figure 3d The 4 in the sixth row becomes the 4 in the third row after aggregation. Since the amount of data is effectively reduced after data aggregation, the amount of data traversed during the data query process is also effectively reduced. Therefore, the increase in the degree of data aggregation reduces the amount of data to be aggregated and calculated during the data query process, thus improving query efficiency.

[0080] According to an embodiment of this application, a data storage method for real-time data analysis is provided. This method acquires platform data generated by a resource platform and media data generated by resource-delivered media. The platform data and media data are written into a database, and a wide-table data structure is used during the data writing process to store the platform data and media data, resulting in a data table. The data table manages the platform data and media data using data keys and source fields. Aggregation calculations are performed on the platform data and media data stored in the data table according to the data keys to obtain an aggregated data table. This method uses a wide-table data structure as the basic data storage structure, which can improve data query efficiency. Media data is written to the database in a full manner, while platform data is written incrementally, enabling differentiated scenario processing. For real-time analysis needs, the data in the data table is periodically aggregated, effectively improving the data aggregation degree and enabling complex aggregation of massive amounts of data. Data queries on the aggregated data ensure real-time data query performance.

[0081] Figure 4A structural block diagram of a data storage device for real-time data analysis according to an embodiment of this application is shown, as follows: Figure 4 As shown, the device includes: a data acquisition module 410, a data storage module 420, and a data aggregation module 430.

[0082] The data acquisition module 410 is used to acquire platform data generated by the resource platform and media data generated by the resource delivery media.

[0083] In an optional implementation, the data acquisition module 410 is further configured to: acquire platform data generated incrementally by the resource platform; and acquire media data fully retrieved by the resource provider from the resource delivery media through a scheduled task using the media interface.

[0084] In an optional implementation, the data acquisition module 410 is further configured to: process the log data of the resource platform using a real-time data processing tool to determine the platform data generated incrementally by the resource platform.

[0085] The data storage module 420 is used to write platform data and media data into the database. During the data writing process, a wide table data structure is used to store the platform data and media data to obtain a data table. The platform data and media data are managed in the data table through data keys and source fields.

[0086] In an optional implementation, the data storage module 420 is further configured to: write platform data to the database incrementally; during the platform data writing process, for each piece of platform data, set a corresponding data key and a platform data field in the corresponding source field in the data table according to the data content of the platform data, and set the media data field in the corresponding source field of the platform data to a preset value; and write media data to the database in full; during the media data writing process, for each piece of media data, set a corresponding data key and a media data field in the corresponding source field in the data table according to the data content of the media data, and set the platform data field in the corresponding source field of the media data to a preset value.

[0087] In one optional implementation, the data key includes: dimension data and partition; the data storage module 420 is further configured to: set the dimension data in the corresponding data key for the platform data according to the data content of the platform data in the data table, and set the partition in the corresponding data key of the platform data as the platform identifier; set the dimension data in the corresponding data key for the media data according to the data content of the media data in the data table, and set the partition in the corresponding data key of the media data as the media identifier.

[0088] In an optional implementation, the data storage module 420 is further configured to: create a temporary partition in the database, write media data into the temporary partition, and replace the target partition corresponding to the media data with the temporary partition to complete the update of the media data in the database.

[0089] The data aggregation module 430 is used to perform aggregation calculations on the platform data and media data stored in the data table according to the data key, and obtain the aggregated data table.

[0090] In one optional implementation, the source field is a numerical field; the data aggregation module 430 is further configured to: query platform data and media data with the same data key in the data table at preset intervals; accumulate the source fields corresponding to the platform data with the same data key to form the corresponding aggregated platform data; and accumulate the source fields corresponding to the media data with the same data key to form the corresponding aggregated media data.

[0091] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0092] According to an embodiment of this application, a data storage device for real-time data analysis acquires platform data generated by a resource platform and media data generated by resource delivery media. The platform data and media data are written into a database, and a wide table data structure is used to store the platform data and media data during the data writing process to obtain a data table. The data table manages the platform data and media data through data keys and source fields. Aggregation calculations are performed on the platform data and media data stored in the data table according to the data keys to obtain an aggregated data table. This device uses a wide table data structure as the basic data storage structure, which can improve data query efficiency. Media data is written to the database in a full manner, while platform data is written to the database incrementally, enabling differentiated scenario processing. For real-time analysis needs, the data in the data table is periodically aggregated, thereby effectively improving the data aggregation degree and achieving complex aggregation of massive amounts of data. Data queries on the aggregated data can ensure the real-time nature of the data query.

[0093] This application also provides a non-volatile computer storage medium that stores at least one executable instruction, which can execute a data storage method for real-time data analysis in any of the above method embodiments.

[0094] Figure 5 The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0095] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 608.

[0096] in:

[0097] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0098] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0099] The processor 502 is used to execute program 510, specifically to perform the relevant steps in the above-described data storage method embodiment for real-time data analysis.

[0100] Specifically, program 510 may include program code that includes computer operation instructions.

[0101] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0102] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0103] Specifically, program 510 can be used to cause processor 502 to execute a data storage method for real-time data analysis in any of the above method embodiments. The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and units in the above-described data storage method embodiment for real-time data analysis, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0104] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the contents of the embodiments of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best implementation of the embodiments of this application.

[0105] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0106] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of the present application, various features of the present application embodiments are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed embodiments of the present application require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the present application.

[0107] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0108] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are meant to be within the scope of the embodiments of this application and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0109] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of this application. The embodiments of this application can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0110] It should be noted that the above embodiments are illustrative of the embodiments of this application and not limiting of the embodiments of this application, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of this application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A data storage method for real-time data analysis, comprising: Acquire platform data generated by the resource platform and media data generated by the media outlets where resources are deployed; The platform data is incrementally written to the database, while the media data is fully written. During the data writing process, a wide table data structure is used to store the platform data and the media data, resulting in a data table. The data table manages the platform data and the media data through data keys and source fields. The source fields are numerical fields and include both platform data fields and media data fields. When writing either platform data or media data to the database, the source fields of the other data need to be supplemented. According to the data key, the platform data and the media data stored in the data table are aggregated and calculated to obtain the aggregated data table; The step of aggregating and calculating the platform data and media data stored in the data table according to the data key to obtain the aggregated data table further includes: At preset intervals, query the data table for platform data and media data with the same data key; The source fields corresponding to platform data with the same data key are summed to form the corresponding aggregated platform data; The source fields corresponding to media data with the same data key are summed to form the corresponding aggregated media data.

2. The method according to claim 1, wherein, The platform data generated by the resource acquisition platform and the media data generated by the resource delivery media further include: Obtain the platform data generated incrementally by the resource platform; The resource provider retrieves all media data from the media being delivered via a scheduled task using a media interface.

3. The method according to claim 2, wherein, Before acquiring the platform data generated incrementally by the resource platform, the method further includes: The log data of the resource platform is processed by real-time data processing tools to determine the incremental platform data generated by the resource platform.

4. The method according to any one of claims 1-3, wherein, The step of using a wide table data structure to store the platform data and the media data to obtain a data table during the data writing process further includes: During the platform data writing process, for each piece of platform data, a corresponding data key and a corresponding platform data field in the source field are set for the platform data in the data table according to the data content of the platform data, and the media data field in the source field corresponding to the platform data is set to a preset value. During the media data writing process, for each piece of media data, a corresponding data key and a corresponding media data field in the source field are set for the media data in the data table according to the data content of the media data, and the platform data field in the source field corresponding to the media data is set to a preset value.

5. The method according to claim 4, wherein, The data key includes: dimensional data and partitions; The step of setting a corresponding data key for each platform data in the data table according to the data content of the platform data further includes: setting dimension data in the corresponding data key of the platform data in the data table according to the data content of the platform data, and setting the partition in the corresponding data key of the platform data as the platform identifier; The step of setting a corresponding data key for each media data in the data table based on the data content of the media data further includes: setting dimension data in the corresponding data key of the media data in the data table based on the data content of the media data, and setting the partition in the corresponding data key of the media data as a media identifier.

6. The method according to claim 4, wherein, The step of writing the media data into the database in a full manner further includes: A temporary partition is created in the database, the media data is written to the temporary partition, and the temporary partition is used to replace the target partition corresponding to the media data to complete the update of the media data in the database.

7. A data storage device for real-time data analysis, comprising: The data acquisition module is used to acquire platform data generated by the resource platform and media data generated by the resource delivery media. The data storage module is used to incrementally write platform data to the database and fully write media data to the database. During the data writing process, a wide table data structure is used to store the platform data and media data to obtain a data table. The platform data and media data are managed in the data table through data keys and source fields. The source fields are numerical fields and include both platform data fields and media data fields. During the process of writing platform data or media data to the database, when writing data from one side, the source fields of the other side need to be supplemented. The data aggregation module is used to perform aggregation calculations on the platform data and the media data stored in the data table according to the data key, so as to obtain an aggregated data table; The data aggregation module is further used for: At preset intervals, query the data table for platform data and media data with the same data key; The source fields corresponding to platform data with the same data key are summed to form the corresponding aggregated platform data; The source fields corresponding to media data with the same data key are summed to form the corresponding aggregated media data.

8. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to a data storage method for real-time data analysis as described in any one of claims 1-6.

9. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to a data storage method for real-time data analysis as claimed in any one of claims 1-6.

10. A computer program product comprising at least one executable instruction that causes a processor to perform an operation corresponding to the data storage method for real-time data analysis as described in any one of claims 1-6.

Images