A method and system for high temporal efficiency two-stage data aggregation

Abstract

The application provides a high-time-effectiveness two-stage data aggregation method and system, including collecting log record data and associated information through a streaming processing program, performing aggregation calculation in a streaming calculation memory to obtain a first aggregation data table; performing aggregation calculation on the first aggregation data table and inventory data through a data lake primary key table merging engine to obtain a second aggregation data table and generate a change log queue; and obtaining the second aggregation data table from the change log queue and synchronously updating and updating the database in batches. Through two-stage aggregation of the streaming calculation window and the data lake primary key table, the application can reduce the amount of aggregated data, reduce the database write pressure, maintain the data timeliness at the hour level or even the minute level, and stably generate inventory-level associated data of hundreds of millions of levels.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and system for highly timely two-stage data aggregation. Background Technology

[0002] Data aggregation refers to the process of merging data from different data sources to obtain more complete information about an object. Its core is to form a complete description by integrating scattered information, merging different statements about the same object scattered in different places, and obtaining complete information about the object.

[0003] Traditional techniques often involve collecting log data and then directly writing the log data, along with the relationship data between different objects extracted from the log data, into the database. If the relationship data between objects is updated directly into the database every time, it will affect the update speed and scale of the database, leading to a performance bottleneck.

[0004] In scenarios with massive amounts of data, traditional solutions often choose databases that support high concurrency and high throughput aggregation to extract the correlation data between two objects from log data, perform aggregation calculations on the correlation data, and store it. However, when faced with hundreds of billions of log data per day, such databases rely on server hardware with high costs and can only perform aggregation during off-peak hours at night. This greatly sacrifices the timeliness of data aggregation, and the correlation data is only visible the next day, commonly known as T+1.

[0005] To address the aforementioned issues, this application proposes a method and system for highly timely two-stage data aggregation. Summary of the Invention

[0006] This application addresses one or more technical deficiencies in the prior art by proposing the following technical solution.

[0007] Based on the first aspect of this application, a highly timely two-stage data aggregation method is proposed, comprising: S1: Collect log data and its associated information through a streaming processing program, perform aggregation calculations in the streaming computing memory, and obtain the first aggregated data table; S2: The first aggregated data table and the existing data are aggregated and calculated using the data lake primary key table merging engine to obtain the second aggregated data table and generate a change log queue; S3: Retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

[0008] This application introduces a two-stage aggregation mechanism, first performing preliminary aggregation in streaming computing memory, and then performing secondary aggregation using the data lake primary key table. This significantly reduces the amount of data written to the database, lowers the dependence on database performance, and maintains high data timeliness.

[0009] Furthermore, the formula for the aggregation calculation is: ; Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

[0010] This step ensures the accuracy and consistency of the aggregation results, providing a reliable data foundation for subsequent data analysis and business decisions.

[0011] Furthermore, the first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

[0012] This step helps to trace the time range and frequency of relationships between log data objects, enhancing the analytical value of the data and enabling the discovery and application of more complex relationships.

[0013] Furthermore, the streaming process performs aggregation calculations in a preset first time period, while the data lake primary key table merging engine performs aggregation calculations in a preset second time period.

[0014] Furthermore, the first time period is a minute-level time period, and the second time period is an hour-level time period.

[0015] This step ensures both rapid visibility of log data and compression and consolidation of log data, maintaining an effective balance between timeliness and system performance to meet the needs of different business scenarios.

[0016] Based on a second aspect of this application, a system for highly timely two-stage data aggregation is also proposed, comprising: Streaming module: Collects log data and its associated information through a streaming processing program, performs aggregation calculations in the streaming computing memory, and obtains the first aggregated data table; Data Lake Module: This module uses the primary key table merging engine of the data lake to perform aggregation calculations on the first aggregated data table and existing data, resulting in a second aggregated data table and generating a change log queue. Batch update module: retrieves the second aggregated data table from the change log queue and updates and synchronizes it to the database in batches.

[0017] Furthermore, the formula for the aggregation calculation is: ; Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

[0018] Furthermore, the first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

[0019] Furthermore, the streaming process performs aggregation calculations in a predetermined first time period, and the data lake primary key table merging engine performs aggregation calculations in a predetermined second time period; the first time period is a minute-level time period, and the second time period is an hour-level time period.

[0020] Based on a third aspect of this application, a computer program product is also proposed, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0021] The technical effect of this application is that it extracts the aggregation of related data between objects from massive log records. Through two-stage aggregation, it can stably generate hundreds of billions of related data. The amount of data after aggregation is reduced by more than 70% compared with the amount of data before aggregation. This can significantly reduce the write pressure on the database and improve the timeliness of data, which can be maintained at the hourly or even minute level. Attached Figure Description

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of a highly efficient two-stage data aggregation method provided according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram illustrating the processing logic of a highly efficient two-stage data aggregation method provided according to an embodiment of this application.

[0025] Figure 3 This is a block diagram of a highly efficient two-stage data aggregation system provided according to an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of this application. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 A flowchart illustrating a time-efficient two-stage data aggregation method according to this application is shown, including: S1: Collect log data and its associated information through a streaming processing program, perform aggregation calculations in the streaming computing memory, and obtain the first aggregated data table; S2: The first aggregated data table and the existing data are aggregated and calculated using the data lake primary key table merging engine to obtain the second aggregated data table and generate a change log queue; S3: Retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

[0030] It should be noted that this application introduces a two-stage aggregation mechanism, first performing preliminary aggregation in streaming computing memory, and then performing secondary aggregation using the data lake primary key table. This significantly reduces the amount of data written to the database, lowers the dependence on database performance, and maintains high data timeliness.

[0031] It should be noted that the formula for the aggregation calculation is as follows: ; Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

[0032] It should be noted that this step ensures the accuracy and consistency of the aggregation results, providing a reliable data foundation for subsequent data analysis and business decisions.

[0033] It should be noted that the first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

[0034] It should be noted that this step helps to trace the time range and frequency of associations between log record data objects, thereby enhancing the analytical value of the data and enabling the discovery and application of more complex relationships.

[0035] It should be noted that the streaming process performs aggregation calculations in the first preset time period, while the data lake primary key table merging engine performs aggregation calculations in the second preset time period.

[0036] It should be noted that the first time period is a minute-level time period, and the second time period is an hour-level time period.

[0037] It should be noted that this step ensures both the rapid visibility of log data and the compression and integration of log data, maintaining an effective balance between timeliness and system performance to meet the needs of different business scenarios.

[0038] It should be noted that the aggregation of a shorter first time period (e.g., 5 minutes) is achieved first through the time window of streaming computing, and then the aggregation of a second time period (e.g., 1 hour) is achieved using the merging engine of the data lake primary key table. Effective aggregation is achieved before the related data is written to the database, which greatly reduces the performance requirements of the database. The time length of the two stages can be adjusted according to computing resources, and the timeliness of the data can be controlled to a certain extent. The specific processing logic diagram is as follows: Figure 2 As shown: Step 1: Collect log data and its associated information using a streaming process; Step 2: Perform window aggregation calculations in the streaming computing memory to obtain the first aggregated data table; Step 3: Use the data lake primary key table merging engine to perform aggregation calculations on the first aggregated data table and the existing data to obtain the second aggregated data table and generate a change log queue; Step 4: Retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

[0039] In a specific embodiment, the streaming process extracts associated data from the log data, including object identifier a, object identifier b, and associated business time, as shown in Table 1; Table 1 Related Data Table ; In the first time period of window aggregation, the aggregation calculation is performed in small batches in the streaming computing memory using the streaming computing time window (taking 5 minutes as an example). After aggregation, the number of records is reduced, and the objects and times of aggregation within the time window, as well as the first and last times of association, are written into the first aggregation data table. For example, in the time period from 8:01 to 8:05 on August 2, 2025, object identifier 13666666666 and object identifier 101 are associated 3 times, and object identifier 13666666666 and object identifier 102 are associated 2 times. The aggregation calculation in the first time period is performed in the streaming computing memory, without relying on external cache and storage, and the data is not written to disk. Compared with the second time period, the data aggregation is more efficient. However, due to the limitation of the streaming computing memory size, the first time period is only suitable for aggregation of shorter time periods. The aggregation process is shown in Table 2. In the second time period of aggregation in the primary key table of the data lake, the merging engine of the primary key table of the data lake is used for aggregation. Taking 1 hour as an example, the aggregation calculation of incremental data after the aggregation of the first time period (as shown in Table 3) and large batches of existing stock data (as shown in Table 4) is implemented in the primary key table. The incremental data table records the number of aggregations of related objects in each time window of the first time period, and the stock data table records the number of aggregations of previously existing related objects. The incremental data table and the stock data table are aggregated and calculated, and a change log queue is generated for the extraction of the second aggregated data table, that is, the latest data aggregation result is obtained, as shown in Table 5. Table 2 First Stage Polymerization Process Table ; Table 3 Incremental Data Table ; Table 4. Inventory Data Table ; Table 5 Second Aggregated Data Table ; Consume the data from this incremental update from the change log queue, perform batch updates, and synchronize them to the database; Compared with the data volume in Table 1, it can be seen that the data volume has decreased significantly after two aggregations, thereby reducing the write pressure on the database. Optionally, in actual engineering implementation, the length of the two time periods, namely the time window of the first time period streaming computation and the time window of the second time period data lake primary key table aggregation, can be adjusted according to the business timeliness requirements and available computing resources.

[0040] The following is for reference. Figure 3 It demonstrates a highly timely two-stage data aggregation system, comprising a streaming module 3-1, a data lake module 3-2, and a batch update module 3-3.

[0041] In a specific embodiment, the streaming module 3-1 is configured to: collect log record data and its associated information through a streaming processing program, perform aggregation calculations in the streaming computing memory, and obtain a first aggregated data table.

[0042] In a specific embodiment, the data lake module 3-2 is configured to: perform aggregation calculations on the first aggregated data table and the existing data through the data lake primary key table merging engine to obtain a second aggregated data table and generate a change log queue.

[0043] In a specific embodiment, the batch update module 3-3 is configured to: retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

[0044] It should be noted that the formula for the aggregation calculation is as follows: ; Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

[0045] It should be noted that the first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

[0046] It should be noted that the streaming process performs aggregation calculations in a predetermined first time period, while the data lake primary key table merging engine performs aggregation calculations in a predetermined second time period; the first time period is a minute-level time period, and the second time period is an hour-level time period.

[0047] It should be noted that this application extracts the aggregation of relational data between objects from massive log records. Through two-stage aggregation, it can stably generate hundreds of billions of relational data entries. The amount of data after aggregation is reduced by more than 70% compared to the amount of data before aggregation. This can significantly reduce the write pressure on the database, improve data timeliness, and keep the data visible at the hourly or even minute level.

[0048] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0049] like Figure 4As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for system operation. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0050] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a liquid crystal display (LCD) and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card and a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0051] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0052] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0054] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0055] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: collect log record data and its associated information through a streaming processing program, perform aggregation calculations in streaming computing memory to obtain a first aggregated data table; perform aggregation calculations on the first aggregated data table and existing data through a data lake primary key table merging engine to obtain a second aggregated data table and generate a change log queue; and retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

[0056] Finally, it should be noted that the above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for highly timely two-stage data aggregation, characterized in that, include: S1: Collect log data and its associated information through a streaming processing program, perform aggregation calculations in the streaming computing memory, and obtain the first aggregated data table; S2: The first aggregated data table and the existing data are aggregated and calculated using the data lake primary key table merging engine to obtain the second aggregated data table and generate a change log queue; S3: Retrieve the second aggregated data table from the change log queue and batch update and synchronize it to the database.

2. The method according to claim 1, characterized in that, The formula for the aggregation calculation is: ； Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

3. The method according to claim 1, characterized in that, The first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

4. The method according to claim 1, characterized in that, The streaming process performs aggregation calculations in a preset first time period, while the data lake primary key table merging engine performs aggregation calculations in a preset second time period.

5. The method according to claim 4, characterized in that, The first time period is a minute-level time period, and the second time period is an hour-level time period.

6. A system for highly timely two-stage data aggregation, characterized in that, include: Streaming module: Collects log data and its associated information through a streaming processing program, performs aggregation calculations in the streaming computing memory, and obtains the first aggregated data table; Data Lake Module: This module uses the primary key table merging engine of the data lake to perform aggregation calculations on the first aggregated data table and existing data, resulting in a second aggregated data table and generating a change log queue. Batch update module: retrieves the second aggregated data table from the change log queue and updates and synchronizes it to the database in batches.

7. The system according to claim 6, characterized in that, The formula for the aggregation calculation is: ； Where a and b both represent object identifiers; t represents the time window; I represents the indicator function; a i and b i Let t represent the identifiers of object a and object b in the i-th associated record, respectively; i This represents the time when the i-th related information was generated; n is the total number of log records within the time window. Represents the time window, t start Indicates the window start time, t end Indicates the window end time; , Represents the Kronecker function, when a = a i When or b=b i The value is 1 if the condition is met, and 0 otherwise. This indicates the number of times object identifier a and object identifier b are associated within the time window t.

8. The system according to claim 6, characterized in that, The first aggregated data table and the second aggregated data table include object identifier, number of associations, first time and last time.

9. The system according to claim 6, characterized in that, The streaming process performs aggregation calculations in a predetermined first time period, and the data lake primary key table merging engine performs aggregation calculations in a predetermined second time period; the first time period is a minute-level time period, and the second time period is an hour-level time period.

10. A computer program product having one or more computer programs stored thereon, characterized in that, When the computer program is executed by a computer processor, the method as described in any one of claims 1-5 is performed.

Images