Report data processing method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a report data processing method, device and equipment and a storage medium, at least applied to the field of block chain, wherein the method comprises: performing log analysis on newly added log information in a current time period to obtain at least one incremental statistical data of a target business in a first time dimension; obtaining at least one historical statistical data of the target business in a preset historical time period before the current time period; the historical statistical data is statistical data in a second time dimension, and a time interval corresponding to the second time dimension is greater than a time interval corresponding to the first time dimension; performing aggregation processing on all historical statistical data with the second time dimension and all incremental statistical data with the first time dimension to obtain an aggregation result of the target business in the current time period; and updating the aggregation result to a business report corresponding to the target business. Through the present application, the data jump problem in report statistics can be solved, and the data quality of the report service is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet, and relate to but are not limited to a report data processing method and device, equipment and a storage medium. BACKGROUND

[0002] In a common architecture of report data calculation, there are usually two ways of real-time calculation and offline coverage calculation, the real-time calculation can guarantee low delay of data, and the offline coverage calculation can guarantee accuracy of data.

[0003] In the related art, in order to guarantee the accuracy of report data, both real-time calculation and offline coverage calculation are usually used. However, two sets of systems are needed for the two calculation methods, and two sets of codes need to be maintained, which has a high cost of development and operation and maintenance; and in some report data calculation with a backtracking scene, the report calculation architecture in the related art needs to be taken over by the offline data stream at a certain moment because the result of the real-time data stream is unreliable during real-time calculation, so the update frequency of data will be consistent with the frequency of offline update, and the report calculation architecture will cause data jump, affecting the data quality of the report service. SUMMARY

[0004] Embodiments of the present application provide a report data processing method, device, equipment and storage medium, which are at least applied to the field of block chain, can reduce the cost of development and operation and maintenance, and solve the problem of data jump in the backtracking scene, and improve the data quality of the report service.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a report data processing method, which comprises the following steps:

[0007] Obtaining newly added log information of a target business in a current time period;

[0008] Performing log analysis on the newly added log information to obtain at least one incremental statistical data of the target business in a first time dimension;

[0009] Obtaining at least one historical statistical data of the target business in a preset historical time period before the current time period; wherein the historical statistical data is statistical data in a second time dimension, and the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension;

[0010] Performing aggregation processing on all historical statistical data with the second time dimension and all incremental statistical data with the first time dimension to obtain an aggregation result of the target business in the current time period;

[0011] The aggregation result is updated into a business report corresponding to the target business.

[0012] Embodiments of the present application provide a report data processing device, which comprises:

[0013] The first obtaining module is configured to obtain newly added log information of the target business in a current time period;

[0014] The log analysis module is configured to perform log analysis on the newly added log information to obtain at least one incremental statistical data of the target business in a first time dimension;

[0015] The second obtaining module is configured to obtain at least one historical statistical data of the target business in a preset historical time period before the current time period; wherein the historical statistical data is statistical data in a second time dimension, and a time interval corresponding to the second time dimension is greater than a time interval corresponding to the first time dimension;

[0016] The aggregation processing module is configured to perform aggregation processing on all historical statistical data having the second time dimension and all incremental statistical data having the first time dimension to obtain an aggregation result of the target business in the current time period;

[0017] The updating module is configured to update the aggregation result into a business report corresponding to the target business.

[0018] Embodiments of the present application provide a report data processing device, which comprises:

[0019] The memory is configured to store executable instructions; and the processor is configured to execute the executable instructions stored in the memory to implement the report data processing method.

[0020] Embodiments of the present application provide a computer program product or computer program, which comprises executable instructions stored in a computer readable storage medium; wherein the processor of the report data processing device reads the executable instructions from the computer readable storage medium and executes the executable instructions to implement the report data processing method.

[0021] Embodiments of the present application provide a computer readable storage medium, which stores executable instructions, and is configured to cause the processor to execute the executable instructions to implement the report data processing method.

[0022] The embodiments of the present application have the following beneficial effects: by performing log analysis on the newly added log information in the current time period, obtaining at least one incremental statistical data of the target service in the first time dimension, and obtaining at least one historical statistical data of the target service in the second time dimension in the preset historical time period, then performing aggregation processing on all the historical statistical data and all the incremental statistical data to obtain the aggregation result of the target service, thereby updating the business report of the target service by using the aggregation result, and realizing the report calculation of the target service. In this way, since the report calculation framework corresponding to the whole report data processing method does not need to be realized by two sets of systems, the development and operation and maintenance costs are greatly reduced, and resource consumption is reduced. Moreover, for the scene with data backtracking, since the historical statistical data and the statistical data in the current time period are fully aggregated, the problem of data jump can be solved, and the data quality of the report service is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an implementation path diagram of Lambda architecture;

[0024] Figure 2 is a report scene with backtracking provided by the embodiments of the present application;

[0025] Figure 3 is a data processing process diagram of the report scene with backtracking provided by the embodiments of the present application;

[0026] Figure 4 is an optional architecture diagram of a report data processing system provided by the embodiments of the present application;

[0027] Figure 5 is a structure diagram of a report data processing device provided by the embodiments of the present application;

[0028] Figure 6 is an optional flow diagram of a report data processing method provided by the embodiments of the present application;

[0029] Figure 7 is another optional flow diagram of a report data processing method provided by the embodiments of the present application;

[0030] Figure 8 is an implementation flow diagram of a hierarchical aggregation processing method provided by the embodiments of the present application;

[0031] Figure 9 is an advertisement delivery report product interface diagram of the technical architecture implementation of the embodiments of the present application;

[0032] Figure 10 is a data report diagram obtained after selecting the options shown in Figure 9 ​

[0033] Figure 11 is an architecture diagram of a report computing framework provided by an embodiment of the present application;

[0034] Figure 12 is a schematic diagram of a report computing framework and some internal logic provided by an embodiment of the present application;

[0035] Figure 13 is a schematic diagram of core aggregation logic of a report data computing system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0037] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as understood by those skilled in the art to which the embodiments of the present application belong. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] In the related art, in order to ensure the accuracy of report data, real-time calculation and offline coverage calculation are usually used at the same time. Among them, Lambda architecture is a common architecture for report calculation. Lambda architecture is a data processing architecture that simultaneously takes advantage of stream processing and batch processing when dealing with large-scale data. Lambda architecture provides comprehensive and accurate data through batch processing and low-latency data through stream processing, so as to balance delay, throughput and fault tolerance. In order to meet the downstream data query, the results of batch processing and stream processing are merged. In the report data processing process, real-time calculation ensures low latency of data, and offline coverage calculation ensures accuracy of data. The real-time calculation channel usually uses a message queue as an input source, for example, kafka (an open source stream processing platform), pulsar (a distributed message publishing / subscription delivery platform), uses storm (an open source WebSerivce test tool) or flink (an open source stream processing application framework) as a real-time calculation framework, and the output result directly updates the underlying report library in real time. Offline calculation usually uses HDFS (a distributed file system designed to run on general hardware) hour / day partitioned data as an input source, uses MapReduce (a programming model for parallel operation of large-scale data sets (more than 1TB)) as an offline calculation framework, and the output result batch updates the report library. The implementation path of Lambda architecture is shown in Figure 1 The message queue 101 is first sent to the HDFS system 102 and the storm / flink framework 103 in two paths, real-time calculation is performed through the storm / flink framework 103 to obtain real-time calculation results, and the message queue 101 is forwarded to the MapReduce framework 104 through the HDFS system 102 for offline calculation to obtain offline calculation results. Finally, the real-time calculation results and the offline calculation results are updated to the report library 105 in a manner that the offline calculation results cover the real-time calculation results.

[0039] The Lambda architecture system is simple to implement, but the problems are also very obvious. First, two sets of systems need to maintain two sets of codes, and the development and operation and maintenance cost is high, which easily leads to inconsistent results of the two sets of systems. Second, the two sets of systems need to be deployed respectively, and the resource consumption is huge. Finally, in some report scenarios that need to have a backtracking scene, Lambda architecture will cause data jump, affecting the data quality of the report service. Especially the last point, which is particularly prominent in the advertising business report system.

[0040] Based on the problems existing in the related art, the embodiment of the present application provides a report data processing method, which is a real-time and batch integrated report aggregation computing framework based on a spark streaming framework, and simultaneously solves the problems of data delay and accuracy, and is widely used in an advertisement external report system.

[0041] In the report data processing method provided by the embodiment of the present application, first, the newly added log information of a target service in a current time period is obtained; the newly added log information is parsed to obtain at least one incremental statistical data of the target service in a first time dimension; then, at least one historical statistical data of the target service in a preset historical time period before the current time period is obtained; the historical statistical data is statistical data in a second time dimension, and a time interval corresponding to the second time dimension is greater than a time interval corresponding to the first time dimension; then, all the historical statistical data and all the incremental statistical data are aggregated to obtain an aggregation result of the target service in the current time period; finally, the aggregation result is updated to a business report corresponding to the target service. In this way, since the report computing framework corresponding to the entire report data processing method does not need to be implemented by two systems to process the process, the development and operation and maintenance costs are greatly reduced, and resource consumption is reduced; and for a data backtracking scene, since the historical statistical data and the statistical data in the current time period are fully aggregated, the problem of data jump can be solved, and the data quality of the report service is improved.

[0042] Here, it is explained what the report scene with backtracking is. First, consider the following example, Figure 2 is a report scene with backtracking provided by the embodiment of the present application, as shown in Figure 2 , a user sees an advertisement on September 1 and clicks it, adds a product to be purchased into a shopping cart after jumping to a landing page, and performs an order operation on September 29. For the order behavior on September 29, if the order quantity on September 29 is 1 according to the reporting time statistics, but if the order quantity on September 1 is 1 according to the charge time of the advertisement corresponding to the order. This is equivalent to changing the data on September 1 on September 29, which is a report with a backtracking scene, and both of the two kinds of indicators are explicitly required. In particular, the second billing indicator, the user can intuitively feel how many advertisements are consumed on September 1, and how many orders these advertisements bring to the advertiser.

[0043] The report data processing method of the embodiment of the present application is to solve Figure 2 the data processing process of the report scene with backtracking as shown in the figure, to ensure the accuracy of the report data. Figure 3 is a data processing process of a report scene with backtracking provided by the embodiment of the present application, as shown inFigure 3 As shown, assuming the system continuously receives order data with a billing time of 8:00 AM, the real-time calculation channel (i.e., the real-time channel report) ignores the delay, while the offline channel report has a 3-hour delay. In other words, for orders with a billing time between 8:00 AM and 9:00 AM, the real-time channel has no delay, but the offline channel has a 3-hour delay. Figure 3 The horizontal timeline represents the number of orders placed at 8 AM that advertisers see when viewing real-time or offline data at different natural times. For example, at 11 AM, the real-time table shows 400 orders, while the offline table shows 100 orders. This is just an example; the system doesn't explicitly show the concepts of "real-time" and "offline" to advertisers. Advertisers only see one value when logging in. Because the real-time stream may lose data, it's essential that the offline stream take over at a certain point in time. For instance, if it's decided at 12 PM that the offline stream takes over the real-time stream, the number of orders seen by customers at 8 AM will change to 100, 200, 300, 400, 500, 200... Of course, special methods can be used, such as having the real-time stream only update data from the last two hours, to reduce the jumps caused by this takeover. However, if this method isn't well-coordinated, it can still cause the metrics to regress.

[0044] The following describes exemplary applications of the report data processing device according to embodiments of this application. The report data processing device provided in this application can be implemented as a terminal or as a server. In one implementation, the report data processing device provided in this application can be implemented as any terminal capable of generating log information and forming report data, such as a laptop, tablet, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), smart robot, smart home appliance, and smart vehicle device. In another implementation, the report data processing device provided in this application can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application embodiment. The following describes exemplary applications when the report data processing device is implemented as a server.

[0045] See Figure 4 , Figure 4This is an optional architecture diagram of the report data processing system provided in this application embodiment. To support any report processing application and accurately generate report data corresponding to log information through the report processing application, at least one report processing application is installed on the terminal in this application embodiment. The report processing application can be any application capable of generating streaming data, such as an advertising application. The report data processing system 10 includes at least a terminal 100, a network 200, and a server 300. The server 300 is the server for the report processing application, which can be the server for the advertising application or a third-party server independent of the advertising application. This third-party server is used to aggregate and process the log information generated in the advertising application to obtain report data for the target business in the advertising application and generate a business report for the target business. The server 300 can constitute the report data processing device in this application embodiment. The terminal 100 is connected to the server 300 through the network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. When processing report data, terminal 100 collects new log information of the target business within the current time period through the client of the report processing application, and sends the new log information to server 300 via network 200. Server 300 parses the new log information to obtain incremental statistical data of the target business in the first time dimension. Simultaneously, server 300 obtains historical statistical data of the target business within a preset historical time period prior to the current time period. This historical statistical data is statistical data in the second time dimension, where the time interval is greater than that in the first time dimension. All historical statistical data and all incremental statistical data are aggregated to obtain the aggregated result of the target business in the current time period. Finally, the aggregated result is updated in the business report corresponding to the target business. After generating the business report corresponding to the target business, the business report is fed back to terminal 100.

[0046] In some embodiments, the report data processing process can also be implemented by the terminal 100. That is, the terminal acts as the execution subject to implement the report data processing method of this application embodiment, determine the aggregation result of the target business in the current time period, and update the aggregation result to the business report corresponding to the target business.

[0047] The report data processing method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the aforementioned server 300 can be a cloud server. The cloud server can parse the newly added log information to obtain incremental statistical data of the target business in the first time dimension; or, the cloud server can obtain historical statistical data of the target business within a preset historical time period before the current time period; or, the cloud server can aggregate all historical statistical data and all incremental statistical data to obtain the aggregated result of the target business in the current time period; or, the cloud server can update the aggregated result to the corresponding business report of the target business, etc.

[0048] In some embodiments, a cloud storage system may also be included, which can store newly added log information and historical statistical data within a preset historical time period. Alternatively, business reports corresponding to the target business may also be stored in the cloud storage system. In this way, when the business reports of the target business are updated again in a subsequent manner, the data required for calculation can be directly obtained from the cloud server, thereby enabling accurate and rapid calculation of the business reports of the target business.

[0049] It's important to clarify that cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied in the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0050] The report data processing system 10 involved in this application embodiment can also be a distributed system of a blockchain system. This distributed system can be a distributed network of nodes (any form of computing device in the network, such as servers or user terminals) and clients, forming a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer. In this application embodiment, the functions of each node in the blockchain system include: 1) Routing: a basic function of nodes to support communication between nodes. In addition to routing, nodes may also have the following functions: 2) Application: deployed in the blockchain to implement specific business according to actual business needs, recording data related to the implementation of functions to form record data, carrying digital signatures in the record data to indicate the source of the task data, and sending the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block. 3) A blockchain consists of a series of blocks linked together in chronological order of their creation. Once a new block is added to the blockchain, it cannot be removed. Each block records the data submitted by nodes within the blockchain system. 4) Consensus is a process within a blockchain network used to reach agreement among multiple involved nodes on the transactions within a block. A block that reaches agreement is appended to the end of the blockchain. Mechanisms for achieving consensus include Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof-of-Stake (DPoS), and Proof of Elapsed Time (PoET).

[0051] In this embodiment of the application, in the distributed system of the blockchain system, each node can record the business report corresponding to the target business. When updating the business report recorded in the node of the blockchain system, an aggregation result for updating the next node can be generated based on the historical statistical data stored in the previous node. Based on the generated aggregation result, the data in the next node is updated to generate the latest business report of the target business and store the business report in the next node to avoid data tampering and ensure data accuracy.

[0052] Figure 5 This is a schematic diagram of the structure of the report data processing device provided in the embodiments of this application.Figure 5 The report data processing device shown includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components of the report data processing device are coupled together via a bus system 340. It is understood that the bus system 340 is used to implement communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general labeled all buses as Bus System 340.

[0053] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0054] User interface 330 includes one or more output devices 331 that enable the presentation of media content, and one or more input devices 332.

[0055] Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0056] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0057] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0058] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0059] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 5 A report data processing device 354 stored in memory 350 is shown. This device 354 can be a report data processing unit within a report data processing device, and can be software in the form of programs and plug-ins. It includes the following software modules: a first acquisition module 3541, a log parsing module 3542, a second acquisition module 3543, an aggregation processing module 3544, and an update module 3545. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0060] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the report data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0061] The report data processing methods provided in the embodiments of this application can be executed by a report data processing device. The report data processing device can be any terminal capable of generating log information and forming report data, or it can be a server. That is, the report data processing methods in the embodiments of this application can be executed by a terminal, by a server, or by interaction between a terminal and a server.

[0062] See Figure 6 , Figure 6 This is an optional flowchart illustrating the report data processing method provided in this application embodiment. The following will be combined with... Figure 6 The steps shown will be explained. It should be noted that... Figure 6The report data processing method in this example is illustrated by using the server as the execution entity.

[0063] Step S601: Obtain new log information of the target business within the current time period.

[0064] Here, new log information of the target business of the report processing application installed on the terminal can be obtained. In other words, for any business of the report processing application, the report data processing method of this application embodiment can be used to generate and update the business report of that business.

[0065] The target business can be any business within the reporting application. For example, if the reporting application is an advertising application, then the target business can be the advertising business.

[0066] The current time period refers to a period of time at present, such as the current duration of one minute, one hour, or one day.

[0067] It's important to note that the reporting application generates log information in real time during operation; that is, log information is generated at every moment. New log information refers to log information generated within the current time period. This log information records streaming data characterizing the target business, such as request operations, impressions, clicks, conversions, and interactions. By parsing the new log information, we can determine the data such as request operation volume, impressions, clicks, conversions, and interactions related to the target business within the current time period.

[0068] Step S602: Parse the newly added log information to obtain at least one incremental statistical data of the target business in the first time dimension.

[0069] In this embodiment of the application, each log information corresponds to a timestamp. When determining the incremental statistical data of the target business in the first time dimension, any of the following methods can be used:

[0070] Method 1: Before parsing newly added log information, the new log information within the current time period can be divided based on the timestamp of each log message. This involves dividing the new log information within the current time period into multiple new log segments using equal time intervals. Each new log segment corresponds to a time segment, and the duration of this time segment is shorter than the duration of the current time period. After obtaining multiple new log segments, each segment is parsed sequentially to obtain corresponding statistical data segments. Each statistical data segment represents incremental statistical data in the first time dimension. It should be noted that the time interval corresponding to the first time dimension is the same as the time interval used when dividing the new log information.

[0071] Method 2: Parse the newly added log information to obtain the current complete statistical data for the target business within the current time period. Each data point in the current complete statistical data has the same timestamp as the corresponding log information. Then, divide the current complete statistical data. During division, the current complete statistical data can be divided based on the time interval corresponding to the first time dimension, resulting in at least one incremental statistical data point.

[0072] In other words, in Method 1 above, the newly added log information is first divided, and then the log is parsed; in Method 2 above, the newly added log information is first parsed, and then divided.

[0073] In this embodiment, the incremental statistical data can be any type of data related to the target business, such as request volume, impressions, clicks, conversions, interactions, etc. When parsing newly added log information, the type of incremental statistical data to be parsed can be specified, as well as the time interval corresponding to the first time dimension. Then, the incremental statistical data of the specified type can be parsed from the newly added log information. For example, when the target business is a target advertisement, it can be specified that the conversion volume of the target advertisement needs to be parsed, and the first time dimension can be specified as one minute. The resulting incremental statistical data is the conversion volume for each minute within the current time period.

[0074] In this embodiment of the application, since the incremental statistical data is the data obtained after parsing the newly added log information in the current time period, the incremental statistical data in the current time period can be real-time data.

[0075] Step S603: Obtain at least one historical statistical data point for the target business within a preset historical time period prior to the current time period. The historical statistical data point is a statistical data point in a second time dimension, where the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension.

[0076] Here, historical statistical data refers to the statistical data obtained by parsing historical log information within a preset historical time period. Since log information is generated at each moment, the log information at the current moment or within a short period of time can be parsed to obtain statistical data over a relatively small time interval, as time progresses and log information accumulates, the statistical data will also accumulate. Therefore, when the statistical data over a relatively small time interval reaches a certain amount, these statistical data can be re-analyzed to obtain statistical data over a relatively large time interval, which is to obtain historical statistical data.

[0077] For example, you can first count the conversions within multiple one-minute intervals. When you have 60 conversions within one minute, you can sum up these 60 conversions to get the conversions within one hour. At this point, one minute is a smaller time interval, and one hour is a larger time interval.

[0078] In this embodiment of the application, when determining the historical statistical data within a preset historical time period, the above method can be used: first, statistical data on the time dimension of the first time interval is calculated, then statistical data on the time dimension of the second time interval is calculated, since the first time interval is less than the second time interval; then statistical data on the time dimension of the third time interval is calculated, since the second time interval is less than the third time interval, and so on, until statistical data under multiple time intervals are obtained according to the preset conditions, or until a certain time period is calculated.

[0079] In this embodiment, the historical statistical data within the preset historical time period can be obtained either by real-time statistical calculation within the preset historical time period or by updating it after real-time statistics combined with offline data. Therefore, the historical statistical data within the preset historical time period is a statistical data that truly reflects the data level of the preset historical time period, obtained by combining real-time data and offline data.

[0080] Step S604: Aggregate all historical statistical data with the second time dimension and all statistical data with the first time dimension to obtain the aggregated result of the target business in the current time period.

[0081] In this embodiment of the application, all historical statistical data with a second time dimension and all statistical data with a first time dimension are aggregated. That is, all historical statistical data and statistical data are fully aggregated. Full aggregation means that all the acquired historical statistical data and statistical data within the current time period are included in the calculation. In other words, full aggregation means that all historical statistical data within a preset historical time period and all statistical data within the current time period are aggregated. All historical statistical data and statistical data are statistically summed to obtain the statistical summation result of the target business in the current time period, which is the aggregation result. The aggregation result can represent all the statistical data of the target business within the preset historical time period and the current time period.

[0082] It should be noted that there can be multiple second time dimensions, meaning there can be multiple second time dimensions with different time intervals. Furthermore, since the time interval corresponding to the second time dimension is longer than the time interval corresponding to the first time dimension, the time interval corresponding to each of the multiple second time dimensions is longer than the time interval corresponding to the first time dimension. In this embodiment, the historical statistical data are different on different second time dimensions. During full aggregation calculation, the statistical data corresponding to different time periods on different second time dimensions are extracted for calculation. That is, the historical statistical data on existing second time dimensions can be aggregated again (sequentially aggregating the statistical data on multiple second time dimensions constitutes a hierarchical aggregation process) to obtain historical statistical data on second time dimensions with larger time intervals. The hierarchical aggregation process will be described in detail below.

[0083] Step S605: Update the aggregation results to the business report corresponding to the target business.

[0084] Here, when updating business reports, the current time can be mapped to the aggregation results and then updated in the business report. Alternatively, the historical aggregation results within a historical time period in the business report can be overwritten, and the aggregation results of the current time period can be used to replace the historical aggregation results. In other words, the latest statistical data of the target business is presented in the business report.

[0085] The report data processing method provided in this application provides a method for parsing newly added log information within the current time period to obtain statistical data of the target business in the first time dimension, and historical statistical data of the target business in the second time dimension within a preset historical time period. Then, it performs full aggregation processing on all historical statistical data with the second time dimension and all statistical data with the first time dimension to obtain the aggregation result of the target business. The aggregation result is then used to update the business report of the target business, thus realizing report calculation for the target business. In this way, since the report calculation framework corresponding to the entire report data processing method does not require two separate systems to implement the processing, it greatly reduces development and maintenance costs and resource consumption. Furthermore, for scenarios with data backtracking, since it performs full aggregation processing on historical statistical data and statistical data within the current time period, it considers both historical statistical data within the preset historical time period and statistical data within the current time period, i.e., it simultaneously considers historical statistical data within the preset historical time period and incremental data relative to the preset historical time period. Therefore, it can solve the problem of data jumps and improve the data quality of the report service.

[0086] In some embodiments, the report data processing system includes at least a terminal and a server. The terminal is equipped with an advertising application. During the process of delivering a target advertisement (i.e. a target business), the advertising application generates a series of log information. The log information includes streaming data that characterizes the target advertisement, such as request operations, impressions, clicks, conversions, interactions, and other streaming data.

[0087] The following uses an advertising scenario where the target business is advertising placement as an example to illustrate the report data processing method of this application embodiment. The report data processing method of this application embodiment can be implemented through a report calculation framework, which is deployed on a server. Therefore, the report data processing method of this application embodiment can be implemented through a server.

[0088] Figure 7 This is another optional flowchart illustrating the report data processing method provided in the embodiments of this application, such as... Figure 7 As shown, the method includes the following steps:

[0089] In step S701, the terminal collects streaming data for the target business through the client of the advertising application and generates log information based on the streaming data.

[0090] Here, the target business's streaming data is a sequence of data that arrives sequentially, in large quantities, rapidly, and continuously, used to represent any operation performed on the target business. The target business's streaming data is a dynamic data set that grows infinitely over time.

[0091] In step S702, the terminal sends the newly added log information within the current time period to the server.

[0092] In some embodiments, newly added log information within the current time period can be sent to a distributed file system. In this way, when the server retrieves newly added log information, it can obtain the newly added log information of the target business within the current time period from the distributed file system. The distributed file system is a file system that can be accessed multiple times to request log information of the target business, and some log information in the distributed file system can be deleted.

[0093] In step S703, the server parses the newly added log information to obtain at least one incremental statistical data of the target business in the first time dimension.

[0094] Step S704: The server determines the time dimension with a time interval greater than the first time dimension from the time dimensions with different time intervals as the second time dimension; wherein, the number of second time dimensions is one or more.

[0095] In this embodiment, multiple time dimensions with different time intervals can be provided in advance; wherein, under each time dimension, the target service has aggregated statistical data corresponding to the time dimension. Here, the aggregated statistical data is statistical data for any operation of the target service.

[0096] For example, when there are three second time dimensions, namely 10 minutes, one hour, and one day, the aggregated statistics for the conversion volume of the target ad are: the conversion volume per 10 minutes in the 10-minute dimension, the conversion volume per hour in the one-hour dimension, and the conversion volume per day in the one-day dimension.

[0097] Step S705: The server obtains the aggregated statistical data of the target business corresponding to each second time dimension within the historical time period.

[0098] Step S706: The server performs hierarchical aggregation processing on the aggregated statistical data of each second time dimension to obtain the corresponding hierarchical aggregation results under the second time dimension.

[0099] based on Figure 7 The embodiments shown in this application further provide a hierarchical aggregation processing method. Figure 8 This is a schematic diagram illustrating the implementation flow of the hierarchical aggregation processing method provided in the embodiments of this application, as follows: Figure 8 As shown, the method includes the following steps (i.e., step S706 can be implemented through the following steps):

[0100] Step S801: When there are multiple second time dimensions, determine the time interval corresponding to each second time dimension.

[0101] Step S802: Sort the multiple second time dimensions in ascending order of time intervals to form a time dimension sequence.

[0102] For example, when there are three second time dimensions, namely 10 minutes, one hour, and one day, the time dimension sequence formed by these three second time dimensions is: 10 minutes time dimension, one hour time dimension, and one day time dimension.

[0103] Step S803: According to the time dimension sequence, the aggregated statistical data of the previous second time dimension is aggregated again in each second time dimension to obtain the hierarchical aggregation result of the target business in the corresponding second time dimension.

[0104] Here, the hierarchical aggregation result in the current second time dimension is obtained by aggregating statistical data from the prior second time dimension. In other words, the result in the current second time dimension is obtained by aggregating statistical data from the prior second time dimension adjacent to the current second time dimension.

[0105] It should be noted that, as the time interval increases, for the Nth and (N-1)th second time dimensions in the time dimension sequence, the data collection duration corresponding to the statistical data in the (N-1)th second time dimension is greater than or equal to the data collection duration corresponding to the statistical data in the Nth second time dimension.

[0106] For example, the time interval corresponding to the (N-1)th second time dimension is 10 minutes, and the time interval corresponding to the Nth second time dimension is 15 minutes. If the total data collection time is 50 minutes, statistical aggregation on the (N-1)th second time dimension (data collected every 10 minutes) yields 5 statistical data points. Statistical aggregation on the Nth second time dimension (data collected every 15 minutes) results in only 3 complete and consecutive 15-minute intervals, thus yielding 3 statistical data points. The data corresponding to the last 5 minutes of these 50 minutes is not subject to statistical aggregation on the Nth second time dimension. Furthermore, if the time dimension sequence only includes the Nth and (N-1)th second time dimensions, the historical statistical data within the preset historical time period includes: the 3 statistical data points obtained from the Nth second time dimension and the data corresponding to the last 5 minutes. In other words, the historical statistical data within the preset historical time period is the statistical data of all data collected over the total data collection period. However, if there are statistical data in the second time dimension corresponding to a larger time interval, the statistical data obtained from the second time dimension corresponding to the larger time interval will be used first. That is to say, when obtaining historical statistical data within the preset historical time period, statistical data in multiple second time dimensions within the preset historical time period is obtained. Furthermore, among the statistical data in any two second time dimensions, the statistical data in the second time dimension with a larger time interval has a higher priority than the statistical data in the second time dimension with a smaller time interval. When selecting historical statistical data within the preset historical time period, statistical data can be selected from different second time dimensions according to priority. Ultimately, it is only necessary to ensure that the obtained historical statistical data is data within the preset historical time period.

[0107] In some embodiments, step S803 can be implemented by the following steps S8031 and S8032 (not shown in the figure):

[0108] Step S8031: When the aggregated statistical data is aggregated again in the Nth second time dimension of the time dimension sequence, the aggregated statistical data obtained in the (N-1)th second time dimension is determined as the aggregated statistical data in the previous second time dimension.

[0109] Step S8032: The aggregated statistical data obtained in the (N-1)th second time dimension is aggregated again to obtain the aggregated statistical data of the target business in the Nth second time dimension. Here, N is an integer greater than 1.

[0110] In some embodiments, each re-aggregation process corresponds to an aggregation task; correspondingly, the report data processing method provided in this application embodiment may further include at least one of the following processing methods:

[0111] Method 1: When the current task status of any aggregated task changes, a preset scheduler is used to manage the task status of the aggregated task; the current task status includes: pending execution, executing, executed and failed.

[0112] Method 2: When the number of aggregated tasks in the pending state at the same time and under the same task execution environment exceeds the number threshold, a preset scheduler is used to schedule the aggregated tasks in the pending state.

[0113] Method 3: When the current task status of any aggregate task is a task failure state, a preset scheduler is used to perform task recovery processing on the aggregate task, and the current task status of the recovered aggregate task is adjusted to the pending execution state.

[0114] Please continue to refer to Figure 7 The method also includes the following steps:

[0115] Step S707: The server determines the historical statistical data within the preset historical time period based on the hierarchical aggregation results under each second time dimension.

[0116] In this embodiment, the historical statistical data is statistical data in a second time dimension, where the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension. There can be multiple historical statistical data sets.

[0117] In some embodiments, step S707 can be implemented by the following steps S7071 to S7073 (not shown in the figure):

[0118] Step S7071: Determine the starting and ending statistical time points corresponding to the hierarchical aggregation results on each second time dimension.

[0119] Step S7072: Based on the start and end statistical time points, determine the hierarchical aggregation result on the (N-1)th second time dimension, and the incremental data relative to the hierarchical aggregation result on the Nth second time dimension.

[0120] Here, since the time interval of the Nth second time dimension is greater than the time interval of the (N-1)th second time dimension, the number of statistical data in the Nth second time dimension is less than or equal to the number of statistical data in the (N-1)th second time dimension.

[0121] When the number of statistical data in the Nth second time dimension is equal to the number of statistical data in the (N-1)th second time dimension (i.e., the number of hierarchical aggregation results), the incremental data of the hierarchical aggregation results in the (N-1)th second time dimension relative to the hierarchical aggregation results in the Nth second time dimension is 0; when the number of statistical data in the Nth second time dimension is less than the number of statistical data in the (N-1)th second time dimension, the incremental data of the hierarchical aggregation results in the (N-1)th second time dimension relative to the hierarchical aggregation results in the Nth second time dimension is the additional statistical data in the hierarchical aggregation results in the (N-1)th second time dimension relative to the hierarchical aggregation results in the Nth second time dimension.

[0122] Step S7073: All incremental data in the second time dimension are identified as historical statistical data within a preset historical time period.

[0123] Here, when the second time dimension is the last time dimension in the time dimension sequence, the incremental data in the second time dimension is the result of all hierarchical aggregations in the second time dimension.

[0124] In some embodiments, the level of the Nth second time dimension is higher than the level of the (N-1)th second time dimension. Correspondingly, in step S707, determining the historical statistical data within the preset historical time period based on the hierarchical aggregation results under each second time dimension can be achieved by using the hierarchical aggregation results of the higher-level second time dimension to determine the historical statistical data within the preset historical time period when there is a hierarchical aggregation result of the higher-level second time dimension among the two second time dimensions.

[0125] In some embodiments, when obtaining the hierarchical aggregation results for each second time dimension, the hierarchical aggregation results can be cached in a distributed dataset. This reduces access to HDFS during computation and reduces deserialization overhead.

[0126] In step S708, the server performs full aggregation processing on historical statistical data and incremental statistical data to obtain the aggregation result of the target business in the current time period.

[0127] Here, full aggregation processing means performing statistical calculations on all historical statistical data with a second time dimension and all incremental statistical data with a first time dimension.

[0128] In step S709, the server updates the aggregation results to the business report corresponding to the target business.

[0129] In this embodiment of the application, the aggregation results can be updated to the business report corresponding to the target business in an overwrite update manner, so as to overwrite the existing aggregation results in the business report.

[0130] In step S710, the server sends the business report to the terminal.

[0131] Step S711: The terminal displays the business report of the target business on the current interface.

[0132] The report data processing method provided in this application embodiment can solve the problem of data jumps and improve the data quality of report services because it performs full aggregation processing on historical statistical data and statistical data for the current time period.

[0133] In some embodiments, each newly added log information corresponds to a timestamp; correspondingly, log parsing of the newly added log information to obtain statistical data of the target business in the first time dimension can be achieved through the following steps:

[0134] Step S11: Perform log management on the newly added log information.

[0135] Step S12: When it is determined that the new log information is in a ready state, the new log information is segmented according to the time interval corresponding to the first time dimension based on the timestamp, forming multiple new log fragments.

[0136] Step S13: Perform log parsing on each newly added log segment to obtain multiple incremental statistical data of the target business in the first time dimension.

[0137] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0138] The Lambda architecture cannot well support report calculations in retrospective scenarios because the results of the real-time stream are unreliable. It must be taken over by a reliable offline stream at some point, but the timing of this takeover is difficult to control. If the coordination is not good, it will lead to data jumps. In addition, once the offline stream takes over, the data update frequency will be consistent with the offline update frequency. Customers will clearly feel that the data update rhythm has suddenly slowed down. In the business model of advertising with long retrospective cycle windows, these problems are particularly prominent, giving advertisers a very bad experience.

[0139] To address the aforementioned issues, this application provides a report data processing method. This method is implemented using a report calculation framework, specifically the Spark Streaming framework, as an example. This application presents a real-time, batch integrated report calculation architecture based on Spark Streaming, effectively solving the problems described above.

[0140] The embodiments of this application can be applied to various products that provide behavior recognition (BI) analysis capabilities, such as the delivery reports of advertising delivery systems and the diagnostic reports of advertising diagnostic analysis. Figure 9 This is a diagram of the advertising delivery report product interface implemented using the technical architecture of this application embodiment, such as... Figure 9 As shown, the campaign management platform 901 has a report page, which includes data of different types, date ranges, time dimensions, metric definitions, and selected subdivision dimensions. Figure 10 According to Figure 9 The diagram shows the data report obtained after selecting the options shown.

[0141] Figure 11 This is an architecture diagram of the report calculation framework provided in the embodiments of this application, such as... Figure 11 As shown, data in message queue 1101 can be written to HDFS 1102 on a minute-by-minute basis, and then the aggregated results can be updated to the report library 1104 in real time through a unified Spark Streaming-based computing framework 1103.

[0142] In this embodiment, the Spark Steaming computation framework is the focus of the computation, which will be further elaborated below.

[0143] Figure 12 This is a schematic diagram of the report calculation framework and some internal logic provided in the embodiments of this application. The entire framework uses minute-level HDFS logs 121 as input and Spark Streaming 122 as the report calculation engine. The calculation results can be output to various external storage 123, such as HBase (a distributed, column-oriented open source database), HDFS, PIVOT (an open source platform for building rich Internet applications and RIA applications using Java), distributed data warehouse (TDW, Tencent distributed Data Warehouse), MySQL (a relational database management system), etc.

[0144] like Figure 12 As shown, Spark Streaming 122 internally includes specific business logic 1221, specific tasks 1222, task scheduling module 1223, and peripheral modules 1224.

[0145] The specific business logic 1221 is mainly divided into three parts. The first part is log management, such as which logs are ready to participate in calculations within a minute. This part is mostly done on the driver. The second part is the aggregation management and calculation of logs. The third part is the output management part, which outputs the results. The second and third parts are basically executed on the executor (which implements the thread pool function). The specific tasks 1222 are the concrete tasks translated from the business logic that can be executed by the Spark engine. There are two main types of tasks: one type performs aggregation calculations at various granularities, and the other type outputs the results to the storage engine. The task scheduling module 1223 provides the task scheduling capabilities provided by the Spark Streaming framework itself, thus systematically ensuring the successful execution of tasks. The peripheral module 1224 monitors some external performance and business metrics.

[0146] In this embodiment, HDFS is used instead of the original tdbank solution for the input portion. The main consideration here is to ensure that the input is deterministic and replayable. Replayability brings two main benefits: first, when a computation task fails, the incomplete task can be completed by reading historical logs, ensuring that no data is lost; second, when problems are found in the original logs, historical dirty data can be surgically removed. These two points are difficult for tdbank to achieve.

[0147] For the computation part, Spark Streaming replaced the original Storm solution. Spark Streaming is based on mini-batch processing, ensuring the stability and correctness of each computation result. Furthermore, Spark workers are persistent, eliminating the overhead of JVM start-up and shutdown. Additionally, using aggregated results from a custom HDFS database as task checkpoints effectively avoids the poor checkpoint compatibility and inefficient serialization / deserialization of Spark Streaming. These checkpoints record the task's current state, serving as the basis for task recovery and restart.

[0148] The report calculation framework of this application solves the challenge of report backtracking calculation well through hierarchical aggregation, full calculation and idempotent update. In addition, through continuous optimization and refinement, the performance of this calculation framework is also very good.

[0149] Figure 13 This is a schematic diagram of the core aggregation logic of the report data calculation system provided in this application embodiment, such as... Figure 13As shown, the aggregation task is responsible for generating aggregation results for each table from the 1-minute log. These results are stored as files in the corresponding time shards on HDFS. As time progresses, the statistical results for each table are continuously generated on HDFS. When a batch of incremental data needs to update the results on external storage (e.g., HBase), there are two options: incremental update, which adds the incremental data to the previous results, but this violates idempotency, and redoing the task would lead to data duplication. Therefore, a full calculation method is chosen, scanning historical data for a period of time, performing a full calculation, and overwriting the report database with the final result. This output is idempotent, ensuring accuracy even when speculative execution is enabled. This is full calculation (i.e., full aggregation processing) and idempotent update. When performing full computation, scanning only historical data at a 1-minute granularity would result in poor performance. Therefore, it continuously calculates aggregation results at higher granularities (i.e., larger time intervals), such as 1-hour, 1-day, and 5-day aggregation results. During full computation, it prioritizes using higher-level (larger time interval) results to construct the full dataset to be scanned; this is hierarchical aggregation. Furthermore, these aggregation results at different granularities on HDFS are cached in the computation engine's (e.g., Apache Spark's Resilient Distributed Datasets cache, RDD cache) to minimize HDFS access during computation and reduce deserialization overhead. It's important to note that full computation does not output all historical results at once; instead, it selects keywords from the incremental data for filtering, meaning it only calculates data that has changed in the incremental data, thus reducing the output volume. This framework effectively addresses backtracking scenarios through hierarchical aggregation, full computation, and idempotent updates.

[0150] Please continue to refer to Figure 13 The HDFS intermediate aggregation result file in the figure shows the process of splitting into aggregation processing and full aggregation processing. Among them, the increment 131 in the first row of 1-minute (i.e. 1m) aggregation result is the incremental statistical data within the current time period. The data collection duration corresponding to the 1m aggregation result in the first row is the sum of the duration of the preset historical time period and the current time period. The first time dimension corresponding to the 1m aggregation result in the first row is the 1-minute time dimension.

[0151] The aggregation results from the second row (10 minutes, i.e., 10m) to the fourth row (1 day, i.e. 1d) are all in the second time dimension, and the second time dimension is 10 minutes (10m), 1 hour (1h), and 1 day (1d) respectively.

[0152] The data collection duration corresponding to the 10m aggregation result in the second row is the duration of the preset historical time period. The data enclosed in thick boxes represents historical statistical data when the second time dimension is 10m. Similarly, the data enclosed in thick boxes in the 1h aggregation result in the third row represents historical statistical data when the second time dimension is 1 hour. The data enclosed in thick boxes in the 1d aggregation result in the fourth row represents historical statistical data when the second time dimension is 1 day. Since the second time dimension of the 1d aggregation result in the fourth row is the time dimension of the maximum time interval in the current hierarchical aggregation process, all data in the 1d aggregation result in the fourth row constitutes historical statistical data. In this embodiment, under each time dimension, the aggregation process is performed again based on the aggregation result of the previous row.

[0153] In this embodiment, the intermediate results at each level of HDFS are the checkpoints; a reasonable RDD cache is used, which greatly reduces the overhead of accessing HDFS and deserialization; only the keywords appearing in the increment are output, reducing the amount of data.

[0154] The real-time batch integrated report calculation framework based on Spark Streaming proposed in this application solves many problems such as poor maintainability of the Lambda architecture, high resource consumption, and data jumps in backtracking scenarios. This technology has been applied to an advertising reporting system, optimizing resources by 50% and solving the data jump problem complained about by advertisers.

[0155] It is understood that in the embodiments of this application, the content involving user information, such as advertising data, log information, business reports, target business information, etc., if it involves data related to user information or enterprise information, when the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0156] The following continues to describe the exemplary structure of the report data processing device 354 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 5 As shown, the report data processing device 354 includes:

[0157] The first acquisition module is used to acquire new log information of the target business within the current time period; the log parsing module is used to parse the new log information to obtain at least one incremental statistical data of the target business in a first time dimension; the second acquisition module is used to acquire at least one historical statistical data of the target business within a preset historical time period before the current time period; wherein the historical statistical data is statistical data in a second time dimension, and the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension; the aggregation processing module is used to aggregate all historical statistical data with the second time dimension and all incremental statistical data with the first time dimension to obtain the aggregation result of the target business in the current time period; the update module is used to update the aggregation result to the business report corresponding to the target business.

[0158] In some embodiments, the second acquisition module is further configured to: provide multiple time dimensions with different time intervals; wherein, under each time dimension, the target service has aggregated statistical data corresponding to the time dimension; from the time dimensions with different time intervals, determine the time dimension with a time interval greater than that of the first time dimension as the second time dimension; wherein the number of the second time dimensions is at least one; acquire the aggregated statistical data of the target service corresponding to each second time dimension within the historical time period; perform hierarchical aggregation processing on the aggregated statistical data of each second time dimension to obtain the hierarchical aggregation result under the corresponding second time dimension; and determine each historical statistical data within the preset historical time period based on the hierarchical aggregation result under each second time dimension.

[0159] In some embodiments, the second acquisition module is further configured to: when there are multiple second time dimensions, determine the time interval corresponding to each second time dimension; sort the multiple second time dimensions in ascending order of the time intervals to form a time dimension sequence; and, according to the time dimension sequence, re-aggregate the aggregated statistical data on each previous second time dimension in each second time dimension to obtain the hierarchical aggregation result of the target business on the corresponding second time dimension.

[0160] In some embodiments, the second acquisition module is further configured to: when the aggregated statistical data is aggregated again in the Nth second time dimension of the time dimension sequence, determine the aggregated statistical data obtained in the (N-1)th second time dimension as the aggregated statistical data in the prior second time dimension; perform the re-aggregation process on the aggregated statistical data obtained in the (N-1)th second time dimension to obtain the aggregated statistical data of the target service in the Nth second time dimension; wherein, N is an integer greater than 1.

[0161] In some embodiments, the second acquisition module is further configured to: determine the starting statistical time point and the ending statistical time point corresponding to the hierarchical aggregation result on each second time dimension; determine the incremental data of the hierarchical aggregation result on the (N-1)th second time dimension relative to the hierarchical aggregation result on the Nth second time dimension based on the starting statistical time point and the ending statistical time point; determine all the incremental data on the second time dimension as historical statistical data within the preset historical time period; wherein, when the second time dimension is the last time dimension in the time dimension sequence, the incremental data on the second time dimension is all the hierarchical aggregation results on the second time dimension.

[0162] In some embodiments, the level of the Nth second time dimension is higher than the level of the (N-1)th second time dimension; the second acquisition module is further configured to: when there is a hierarchical aggregation result of the higher-level second time dimension in the two second time dimensions, use the hierarchical aggregation result of the higher-level second time dimension to determine the historical statistical data within the preset historical time period.

[0163] In some embodiments, the apparatus further includes a caching module for caching the hierarchical aggregation result to a distributed dataset when the hierarchical aggregation result is obtained in each of the second time dimensions.

[0164] In some embodiments, each of the re-aggregation processes corresponds to an aggregation task; the apparatus further includes: a processing module, configured to implement at least one of the following processing methods: when the current task state of any aggregation task changes, a preset scheduler is used to manage the task state of the aggregation task; wherein, the current task state includes: pending execution state, executing state, executed state, and task failure state; when the number of aggregation tasks in the pending execution state at the same time and in the same task execution environment is greater than a number threshold, the preset scheduler is used to schedule the aggregation tasks in the pending execution state; when the current task state of any aggregation task is the task failure state, the preset scheduler is used to perform task recovery processing on the aggregation task, and the current task state of the recovered aggregation task is adjusted to the pending execution state.

[0165] In some embodiments, each newly added log information corresponds to a timestamp; the log parsing module is further configured to: perform log management on the newly added log information; when it is determined that the newly added log information is in a ready state, based on the timestamp, divide the newly added log information into multiple newly added log segments according to the time interval corresponding to the first time dimension; and perform log parsing on each of the newly added log segments to obtain multiple incremental statistical data of the target service in the first time dimension.

[0166] In some embodiments, the first acquisition module is further configured to: acquire new log information of the target service within the current time period from a distributed file system; wherein the distributed file system is a file system that can be accessed multiple times to request the acquisition of log information of the target service, and some log information in the distributed file system can be deleted.

[0167] In some embodiments, the update module is further configured to: update the aggregation result to the business report corresponding to the target business in an overwrite update manner, so as to overwrite the existing aggregation result in the business report.

[0168] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0169] This application provides a computer program product or computer program that includes executable instructions, which are computer instructions stored in a computer-readable storage medium. When the processor of a report data processing device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the report data processing device performs the method described in this application.

[0170] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 6 The method shown.

[0171] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0172] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0173] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code segments). As an example, executable instructions may be deployed to execute on a single computing device (which may be a job runtime-determined device), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0174] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for processing report data, characterized in that, The method includes: Retrieve new log information for the target business within the current time period from the distributed file system; The newly added log information is parsed to obtain at least one incremental statistical data of the target business in the first time dimension; Obtain at least one historical statistical data point for the target service within a preset historical time period prior to the current time period; wherein the historical statistical data point is historical statistical data in a second time dimension, the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension, and the historical statistical data point in the second time dimension is determined by the following method: obtaining aggregated statistical data point for the target service corresponding to each second time dimension within the preset historical time period; forming a time dimension sequence for the second time dimension based on the time interval corresponding to the second time dimension; performing hierarchical aggregation processing on the aggregated statistical data point for each second time dimension based on the time dimension sequence to obtain hierarchical aggregation results on each second time dimension; and determining the historical statistical data point for each second time dimension based on the hierarchical aggregation results on each second time dimension. The hierarchical aggregation results on the second time dimension are used as task checkpoints in the distributed file system for task recovery in case of task failure. Aggregate all historical statistical data of the second time dimension and all incremental statistical data of the first time dimension to obtain the aggregation result of the target business in the current time period; The aggregation results are updated in the business report corresponding to the target business to cover the existing aggregation results in the business report, wherein the aggregation results in the updated business report are idempotent.

2. The method according to claim 1, characterized in that, The step of obtaining at least one historical statistical data point of the target service within a preset historical time period prior to the current time period includes: Multiple time dimensions with different time intervals are provided; wherein, under each time dimension, the target service has aggregated statistical data corresponding to the time dimension; From the time dimensions with different time intervals, the time dimension with a time interval greater than that of the first time dimension is determined as the second time dimension; wherein, the number of the second time dimensions is at least one.

3. The method according to claim 2, characterized in that, The step of forming a time dimension sequence based on the time interval corresponding to the second time dimension includes: When there are multiple second time dimensions, determine the time interval corresponding to each second time dimension; The multiple second time dimensions are sorted in ascending order of the time intervals to form a time dimension sequence; The step of performing hierarchical aggregation processing on the aggregated statistical data of each second time dimension according to the time dimension sequence to obtain the hierarchical aggregation result on each second time dimension includes: According to the time dimension sequence, the aggregated statistical data in the previous second time dimension are aggregated again in each second time dimension to obtain the hierarchical aggregation result of the target business in the corresponding second time dimension.

4. The method according to claim 3, characterized in that, The step of re-aggregating the aggregated statistical data in each of the previous second time dimensions according to the time dimension sequence to obtain the hierarchical aggregation result of the target business in the corresponding second time dimension includes: When the aggregated statistical data is aggregated again in the Nth second time dimension of the time dimension sequence, the aggregated statistical data obtained in the (N-1)th second time dimension is determined as the aggregated statistical data in the previous second time dimension. The aggregated statistical data obtained in the (N-1)th second time dimension is subjected to the re-aggregation process to obtain the aggregated statistical data of the target business in the Nth second time dimension. Where N is an integer greater than 1.

5. The method according to claim 4, characterized in that, The determination of historical statistical data for each of the second time dimensions based on the hierarchical aggregation results for each of the second time dimensions includes: Determine the starting and ending statistical time points corresponding to the hierarchical aggregation results in each of the second time dimensions; Based on the starting statistical time point and the ending statistical time point, determine the hierarchical aggregation result on the N-1th second time dimension, and the incremental data relative to the hierarchical aggregation result on the Nth second time dimension; All incremental data in the second time dimension are identified as historical statistical data in the second time dimension; Wherein, when the second time dimension is the last time dimension in the time dimension sequence, the incremental data on the second time dimension is all the hierarchical aggregation results on the second time dimension.

6. The method according to claim 5, characterized in that, The level of the Nth second time dimension is higher than the level of the (N-1)th second time dimension; The determination of historical statistical data for each of the second time dimensions based on the hierarchical aggregation results for each of the second time dimensions includes: When there is a hierarchical aggregation result of the higher-level second time dimension among the two second time dimensions, the historical statistical data of the second time dimension is determined by using the hierarchical aggregation result of the higher-level second time dimension.

7. The method according to claim 3, characterized in that, The method further includes: When obtaining the hierarchical aggregation result for each of the second time dimensions, the hierarchical aggregation result is cached in the distributed dataset.

8. The method according to claim 3, characterized in that, Each of the aforementioned re-aggregation processes corresponds to an aggregation task; the method further includes at least one of the following processing methods: When the current task state of any aggregated task changes, a preset scheduler is used to manage the task state of the aggregated task; wherein, the current task state includes: pending execution state, executing state, executed state, and task failure state; When the number of aggregated tasks in the pending execution state at the same time and under the same task execution environment exceeds the number threshold, the preset scheduling program is used to schedule the aggregated tasks in the pending execution state. When the current task status of any aggregated task is the failed task status, the preset scheduler is used to perform task recovery processing on the aggregated task, and the current task status of the recovered aggregated task is adjusted to the pending execution status.

9. The method according to claim 1, characterized in that, Each newly added log entry corresponds to a timestamp; The step of parsing the newly added log information to obtain at least one incremental statistical data point of the target service in the first time dimension includes: Perform log management on the newly added log information; When it is determined that the new log information is in a ready state, the new log information is segmented according to the time interval corresponding to the first time dimension based on the timestamp to form multiple new log fragments; For each newly added log segment, log parsing is performed to obtain multiple incremental statistical data of the target business in the first time dimension.

10. The method according to any one of claims 1 to 9, characterized in that, The distributed file system is a file system that can be accessed multiple times to request log information of the target business, and some log information in the distributed file system can be deleted.

11. A report data processing device, characterized in that, The device includes: The first acquisition module is used to obtain new log information of the target business within the current time period from the distributed file system; The log parsing module is used to parse the newly added log information to obtain at least one incremental statistical data of the target business in the first time dimension. The second acquisition module is used to acquire at least one historical statistical data of the target service within a preset historical time period prior to the current time period; wherein the historical statistical data is historical statistical data in a second time dimension, the time interval corresponding to the second time dimension is greater than the time interval corresponding to the first time dimension, and the historical statistical data of the second time dimension is determined by the following method: acquiring the aggregated statistical data of the target service corresponding to each second time dimension within the preset historical time period; forming a time dimension sequence of the second time dimension according to the time interval corresponding to the second time dimension; performing hierarchical aggregation processing on the aggregated statistical data of each second time dimension according to the time dimension sequence to obtain the hierarchical aggregation result on each second time dimension; determining the historical statistical data of each second time dimension based on the hierarchical aggregation result on each second time dimension; and using the hierarchical aggregation result on the second time dimension as a task checkpoint in the distributed file system for task recovery in case of task failure; The aggregation processing module is used to aggregate all historical statistical data with the second time dimension and all incremental statistical data with the first time dimension to obtain the aggregation result of the target business in the current time period. An update module is used to update the aggregation result to the business report corresponding to the target business, so as to overwrite the existing aggregation result in the business report, wherein the aggregation result in the updated business report is idempotent.

12. The apparatus according to claim 11, characterized in that, The second acquisition module is further configured to: Multiple time dimensions with different time intervals are provided; wherein, under each time dimension, the target service has aggregated statistical data corresponding to the time dimension; From the time dimensions with different time intervals, the time dimension with a time interval greater than that of the first time dimension is determined as the second time dimension; wherein, the number of the second time dimensions is at least one.

13. The apparatus according to claim 12, characterized in that, The second acquisition module is further configured to: When there are multiple second time dimensions, determine the time interval corresponding to each second time dimension; The multiple second time dimensions are sorted in ascending order of the time intervals to form a time dimension sequence; According to the time dimension sequence, the aggregated statistical data in the previous second time dimension are aggregated again in each second time dimension to obtain the hierarchical aggregation result of the target business in the corresponding second time dimension.

14. A report data processing device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the report data processing method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, The system stores executable instructions for causing a processor to execute the executable instructions to implement the report data processing method according to any one of claims 1 to 10.

16. A computer program product comprising executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the report data processing method according to any one of claims 1 to 10.

Images