Business data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a service data processing method and device, electronic equipment and a storage medium, which are at least applied to the field of cloud computing and the field of blockchain, and the method comprises the following steps: in response to a service rerun request, creating a repair task for a target service in a target time period; the repair task is used for re-executing the business process of the target business in the target time period, and the repair task is independent of the current execution process; executing the repair task to obtain a repair log; performing log analysis on the repair log to obtain business rerun statistical data of the target business in the target time period; and updating the service data by adopting the service rerun statistical data to obtain service update data for representing a service execution result of the target service. According to the method and the device, when systematized rerunning is carried out, full-amount redeployment rerunning can be realized, computing resources can be greatly saved, and the systematized rerunning efficiency can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and relate to but are not limited to a business data processing method, device, electronic device and storage medium. Background Art

[0002] In the commonly used architecture for report data calculation, there are usually two methods: real-time calculation and offline coverage calculation. Real-time calculation can ensure low latency of data, and offline coverage calculation can ensure data accuracy. In the report data calculation architecture in the related technology, a commonly used calculation architecture is composed of real-time links and offline links. The real-time link ensures low latency of data, and the offline link ensures the accuracy of data indicators; however, the separation of the offline link and the real-time link will lead to the problem of asynchronous log reporting and incorrect calculation of report data. Therefore, another calculation architecture that only retains the real-time path is usually used.

[0003] However, in order to deal with the problems of log reporting errors and report calculation anomalies, it is necessary to implement systematic backtracking and rerunning without affecting the normal operation of real-time tasks. For the computing architecture that only retains the real-time path in the related technology, although it reduces the cost of maintaining two sets of links, it brings certain difficulties in systematic rerunning. The usual practice for systematic rerunning is to redeploy a set of identical real-time tasks, recalculate the report by replaying the data source, and after the calculation is completed, switch the downstream application to the newly calculated report and then stop the task of the old link.

[0004] Obviously, this systematic re-run method has many computing tasks and cannot achieve full redeployment and re-run. It will cause a lot of computing resources to be wasted during the systematic re-run, and there will be compatibility issues between the new and old links. Summary of the invention

[0005] The embodiments of the present application provide a business data processing method, device, electronic device and storage medium, which can be applied at least in the field of cloud computing and blockchain. When performing a systematic rerun, full redeployment and rerun can be achieved, and computing resources can be greatly saved, thereby improving the efficiency of the systematic rerun.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] An embodiment of the present application provides a business data processing method, including: obtaining a business re-run request during the current execution of a target business or when the execution has been completed; the business re-run request is used to request re-execution of the business process of the target business within a target time period before the current moment; the target business generates business data for provision to other businesses during the current execution process; in response to the business re-run request, creating a repair task for the target business within the target time period; the repair task is used to re-execute the business process of the target business within the target time period, and the repair task is independent of the current execution process; executing the repair task to obtain a repair log; performing log parsing on the repair log to obtain business re-run statistical data of the target business within the target time period; using the business re-run statistical data to update the business data to obtain business update data for characterizing the business execution result of the target business.

[0008] An embodiment of the present application provides a business data processing device, which includes: an acquisition module, which is used to acquire a business re-run request during the current execution of a target business or when the execution has been completed; the business re-run request is used to request the re-execution of the business process of the target business within a target time period before the current moment; the target business generates business data for use by other businesses during the current execution process; a task creation module is used to create a repair task for the target business within the target time period in response to the business re-run request; the repair task is used to achieve the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process; a task execution module is used to execute the repair task and obtain a repair log; a log parsing module is used to perform log parsing on the repair log to obtain business re-run statistical data of the target business within the target time period; a data update module is used to update the business data using the business re-run statistical data to obtain business update data used to characterize the business execution result of the target business.

[0009] In some embodiments, the acquisition module is also used to: during the execution of the target business, obtain the business log of the target business execution reported by the business execution terminal; if an error is detected in the business log reporting, determine the time period of the business log reporting error, and determine the time period of the business log reporting error as the target time period; generate the business re-run request based on the target time period.

[0010] In some embodiments, the acquisition module is also used to: in the process of executing the target business, obtain the business log of the target business execution reported by the business execution terminal; when the other business performs other business processing based on the business data corresponding to the business log, if an abnormality occurs in the indicator calculation during other business processing, determine the time period corresponding to the business data with the abnormal indicator calculation in the business log, and determine the time period as the target time period; generate the business re-run request based on the target time period.

[0011] In some embodiments, the device further includes: a log reporting module, configured to report the repair log with a preset duration as a reporting period; and store the reported repair log in a preset distributed storage system.

[0012] In some embodiments, the log parsing module is also used to: read multiple repair logs with the preset time length from the distributed storage system; parse each repair log to obtain repair task statistical data about the target statistical indicator for each repair log under the preset time length dimension; aggregate multiple repair task statistical data corresponding to the multiple repair logs under different time dimensions to obtain intermediate aggregation results of the repair tasks under different time dimensions; based on the intermediate aggregation results of the repair tasks under different time dimensions, determine the business re-run statistical data of the target business within the target time period.

[0013] In some embodiments, the log parsing module is further used to: perform aggregation processing on the intermediate aggregation results under the different time dimensions to obtain business re-run statistics of the target business within the target time period.

[0014] In some embodiments, the repair log with the preset time length includes multiple log information; the multiple log information is the log information of a calculation batch; each of the calculation batches has a mapping relationship with a directory information in the log directory in the data storage unit; the log directory includes directory information of each section of the repair log; the device also includes: a completion identifier generation module, used to generate a calculation batch completion identifier for the repair log; the calculation batch completion identifier is used to identify that multiple log information in the corresponding repair log have been parsed and statistical data corresponding to the section of the repair log have been obtained; an information mapping module, used to map the calculation batch completion identifier with the target information of the corresponding repair log to obtain a mapping relationship between the calculation batch completion identifier and the corresponding repair log; a storage module, used to store the calculation batch completion identifier and the mapping relationship in the data storage unit.

[0015] In some embodiments, the log parsing module is also used to: parse the business log of the target business within the target time period to obtain at least one business statistical data of the business log about the target statistical indicator under the preset time dimension; aggregate the at least one business statistical data under different time dimensions to obtain business intermediate aggregation results under different time dimensions; under the same time dimension, use the repair task intermediate aggregation result to update the corresponding business intermediate aggregation result to obtain an updated business intermediate aggregation result; store the updated business intermediate aggregation result in the distributed storage system.

[0016] In some embodiments, the log parsing module is also used to: obtain the last business intermediate aggregation result in the updated business intermediate aggregation results under the first time dimension and the next business intermediate aggregation result to be generated connected to the last business intermediate aggregation result under the adjacent first time dimension and the second time dimension; determine the next business intermediate aggregation result to be generated under the second time dimension and connected to the last business intermediate aggregation result in the updated business intermediate aggregation results based on the last business intermediate aggregation result and the business intermediate aggregation result to be generated; the duration corresponding to the first time dimension is less than the duration corresponding to the second time dimension.

[0017] In some embodiments, the log parsing module is also used to: parse the business log when the target business is executed to obtain multiple business statistical data about the target statistical indicators in the business log under the preset time dimension; aggregate the multiple business statistical data under different time dimensions to obtain the business intermediate aggregation results under different time dimensions; obtain the partition information corresponding to each of the business intermediate aggregation results in the business log; generate a partition identifier of the corresponding business intermediate aggregation result based on the partition information; and store each of the partition identifiers as the target information in the log directory.

[0018] In some embodiments, the log parsing module is also used to: when obtaining the intermediate aggregation result of the repair task corresponding to the log information of each calculation batch, obtain the re-run partition information corresponding to the intermediate aggregation result of the repair task in the repair log; the re-run partition information is used to determine the log starting position and log ending position of the log information of the corresponding calculation batch in the entire repair log; generate a re-run partition identifier of the corresponding intermediate aggregation result of the repair task based on the re-run partition information; obtain the timestamp corresponding to the log information of each calculation batch; generate the version information of the intermediate aggregation result of the corresponding repair task based on the timestamp; map the re-run partition identifier of each intermediate aggregation result of the repair task and the version information and store them in the log directory.

[0019] In some embodiments, the device also includes: an intermediate aggregation result output module, which is used to obtain the intermediate aggregation result of the repair task corresponding to the rerun partition identifier when other businesses trigger a request to use the business intermediate aggregation result of the log information of any calculation batch in the distributed storage system. If the partition identifier and rerun partition identifier corresponding to the log information of the same calculation batch are stored in the log directory, and the timestamp in the version information corresponding to the rerun partition identifier is closer to the current time than the timestamp in the version information corresponding to the partition identifier, and the intermediate aggregation result of the repair task corresponding to the rerun partition identifier is output to the other businesses for use.

[0020] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned business data processing method when executing the executable instructions stored in the memory.

[0021] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned business data processing method when executing the executable instructions.

[0022] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned business data processing method.

[0023] The embodiments of the present application have the following beneficial effects: when it is necessary to rerun the target business, a repair task for the target business within the target time period is created; the repair task is used to realize the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process of the target business; then, the repair task is executed to obtain a repair log; and the repair log is parsed to obtain the business re-running statistics of the target business within the target time period; thereby the business data is updated using the business re-running statistics to obtain business update data for characterizing the business execution results of the target business. In this way, since the business process of the target business within the target time period is re-executed by a repair task that is independent of the current execution process of the target business, the business re-running statistics within the target time period are parsed by the repair log after re-execution, so as to realize accurate re-running of the target business, and obtain business re-running statistics that can accurately reflect the business data, not only can the full amount of redeployment and re-running be realized when performing a systematic re-run, but also computing resources can be greatly saved and the efficiency of the systematic re-run can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1It is a structural diagram of the Kappa architecture and Lambda architecture in related technologies;

[0025] Figure 2 It is a schematic diagram of the implementation process of Kappa architecture rerun in related technologies;

[0026] Figure 3 This is an optional architecture diagram of a business data processing system provided in an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0028] Figure 5 This is an optional flowchart of the business data processing method provided in the embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of the business report aggregation process provided by an embodiment of the present application;

[0030] Figure 7 This is another optional flowchart of the business data processing method provided in the embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of a report scenario with backtracking provided in an embodiment of the present application;

[0032] Fig. 9 It is a schematic diagram of the business data processing process provided by an embodiment of the present application;

[0033] Fig.10 This is a system architecture diagram of the Seagull statistical framework provided in an embodiment of the present application;

[0034] Fig.11 It is a schematic diagram of the process of generating intermediate results of different levels in the aggregation stage provided in an embodiment of the present application;

[0035] Fig.12 This is a schematic diagram of the HDFS intermediate aggregation result repair process provided by an embodiment of the present application;

[0036] Fig.13 This is a schematic diagram of a result directory for forming an intermediate aggregation result provided in an embodiment of the present application;

[0037] Fig.14 It is a specific protocol diagram about the rfile file provided in the embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0040] Kappa architecture (a computing architecture that only retains real-time channels) is a common architecture for report computing. This architecture provides a batch-stream integrated computing method, using a message queue (such as Kafka) with a replay function that can save historical data as a data source, and uses a real-time statistical framework (such as flink) that supports the Exactly Once capability to perform calculations, and writes them to a database that supports idempotent output (such as HBase). Compared with Lambda architecture (a computing architecture composed of real-time links and offline links), the core advantage is that there is no need to maintain two sets of code for real-time / offline separately, because Lambda architecture consists of real-time links and offline links, such as Figure 1 From the structural diagrams of the Kappa architecture and the Lambda architecture shown in the figure, it can be seen that in the Lambda architecture, the real-time link ensures low data latency, and the offline link ensures the accuracy of data indicators. In such an architecture, rerunning is relatively easy, and only the offline link needs to be rerun. In the Kappa architecture, only the real-time path is retained.

[0041] from Figure 1 It can be seen that the Lambda architecture has two independent paths, real-time and offline. Re-running can directly reschedule the offline path. Since the Kappa architecture only retains the real-time path, although it reduces the cost of maintaining two sets of paths, it brings certain difficulties in re-running, such as Figure 2 The figure shows a schematic diagram of the implementation process of rerunning the Kappa architecture in related technologies. The usual practice is to redeploy a set of identical real-time tasks, recalculate by replaying the data source, and after the calculation is completed, switch the application to the newly calculated table and then stop the old link.

[0042] Although the Kappa architecture only needs to maintain real-time channels, a new link needs to be built for re-running when repairing data. In real advertising business scenarios, the computing resources and time costs required for repair are extremely high, and it is difficult to truly implement them. This is reflected in the following: First, there are many computing tasks, and it is impossible to redeploy and re-run the entire system: for example, when the advertiser report on the delivery side has very rich dimensions and indicators, there are more than a dozen real-time tasks involved, and it is basically impossible to rebuild a set for so many computing tasks; second, the construction of the new link is incompatible with changes: since there are multiple downstream query business parties, it is very unfriendly to require the downstream query business party to switch to the data table of the new link every time the repair is re-run, and it takes a lot of effort to verify the correctness of the new link; third, conflicts will occur when the new and old links write to the same data table: since the data source is continuously arriving, the business cannot wait for all the data to arrive before starting the calculation, and the results of the same dimension indicators will be continuously updated. If the online task and the repair task write to the same data table at the same time, a write conflict will occur, and correct repair cannot be performed.

[0043] In response to at least one of the above problems, the embodiment of the present application proposes a data rerunning and backtracking scheme (i.e., a business data processing method) based on a real-time statistical framework of batch and stream integration of spark streaming, which realizes systematic rerunning without affecting the normal operation of real-time tasks, and copes with the problems of log reporting errors and report calculation anomalies, and is applied in the external reporting system of advertising. The core problem solved by the data rerunning and backtracking scheme is: first, data repair is performed through a bypass computing task (for example, Seagull instance, Seagull is a set of batch and stream integrated statistical framework based on Spark Streaming), and it is isolated from the online task to avoid affecting the normal online data computing service; secondly, the bypass computing task only reruns the data of the error period, and the results after rerunning are directly written to the online database table in situ. The advantage of this method is that it can quickly repair with less resources, and at the same time, downstream application queries do not need to be perceived; thirdly, the self-developed rfile file in the scheme is used to record the basic information and status of the entire repair process, and the online task and the repair task interact with each other through rfile (a data storage unit), so as to solve the write conflict problem of simultaneous calculation of two tasks.

[0044] Among them, in the business data processing method provided by the embodiment of the present application, first, during the current execution process of the target business or when the execution has been completed, a business re-run request is obtained; the business re-run request is used to request the re-execution of the business process of the target business within the target time period before the current moment; the target business generates business data for provision to other businesses during the current execution process; then, in response to the business re-run request, a repair task is created for the target business within the target time period; the repair task is used to achieve the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process; and the repair task is executed to obtain a repair log; then, the repair log is parsed to obtain business re-run statistical data of the target business within the target time period; finally, the business re-run statistical data is used to update the business data to obtain business update data used to characterize the business execution results of the target business. In this way, since the business process of the target business within the target time period is re-executed through a repair task independent of the current execution process of the target business, the business re-running statistical data within the target time period is parsed through the repair log after re-execution, thereby achieving accurate re-run of the target business and obtaining business re-running statistical data that can accurately reflect the business data. This not only enables full redeployment and re-running during systematic re-running, but also greatly saves computing resources and improves the efficiency of systematic re-running.

[0045] Here, firstly, an exemplary application of the business data processing device of the embodiment of the present application is described, and the business data processing device is an electronic device for implementing the business data processing method. In one implementation, the business data processing device (i.e., electronic device) provided by the embodiment of the present application can be implemented as a terminal or a server. In one implementation, the business data processing device provided by the embodiment of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device, etc., any terminal with business data processing function and report data statistics function; in another implementation, the business data processing device provided by the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application. Below, an exemplary application when the business data processing device is implemented as a server will be described.

[0046] See also Figure 3 , Figure 3 It is an optional architecture diagram of the business data processing system provided by the embodiment of the present application. In order to realize the support of any report processing application, the business data corresponding to the business log is accurately generated by the report processing application. The business data can be a kind of report data. At least a report processing application is installed on the terminal of the embodiment of the present application, wherein the report processing application can be any application that can generate stream data, for example, it can be an advertising application. The business data processing system 10 includes at least a terminal 100, a network 200 and a server 300, wherein the server 300 is a server of the report processing application, that is, it can be a server of the advertising application, or it can be a third-party server independent of the advertising application. The third-party server is used to perform log parsing and aggregation processing on the business log generated in the advertising application, obtain the business data of the target business in the advertising application, and generate the business report of the target business. The server 300 can constitute the business data processing device of the embodiment of the present application. The terminal 100 is connected to the server 300 through the network 200, and the network 200 can be a wide area network or a local area network, or a combination of the two.

[0047] When implementing business data processing, the terminal 100 obtains the business re-run operation input by the business personnel through the client of the report processing application during the current execution of the target business or when the execution has been completed, and generates a business re-run request in response to the business re-run operation. The business re-run request is used to request the re-execution of the business process of the target business within the target time period before the current moment, and the target business generates business data for use by other businesses during the current execution process.

[0048] The terminal can send a service rerun request to the server 300 through the network 200. The server 300 responds to the service rerun request and creates a repair task for the target service within the target time period. The repair task is used to re-execute the service process of the target service within the target time period, and the repair task is independent of the current execution process. The repair task is executed to obtain a repair log. Then, the repair log is parsed to obtain service rerun statistics of the target service within the target time period. Finally, the service data is updated using the service rerun statistics to obtain service update data for characterizing the service execution results of the target service. After obtaining the service update data, when other services running on the terminal 100 need to use the service data of the target service, the service update data can be obtained from the server 300 for corresponding service processing.

[0049] In some embodiments, the business data processing method can also be implemented by the terminal 100, that is, the terminal is used as the execution subject to implement the business data processing method of the embodiment of the present application, that is, the terminal creates a repair task for the target business within the target time period; the terminal executes the repair task to obtain a repair log; the terminal parses the repair log to obtain business re-running statistics of the target business within the target time period; the terminal uses the business re-running statistics to update the business data to obtain business update data used to characterize the business execution results of the target business.

[0050] The business data processing method provided in the embodiment of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 can be a cloud server. A repair task for a target business within a target time period is created through the cloud server, or a repair log is obtained by executing a repair task through the cloud server, or a repair log is parsed through the cloud server, or the business data is updated through the cloud server using business rerun statistical data.

[0051] In some embodiments, a cloud storage may be provided, and the business data and business update data of the target business may be stored in the cloud storage, or the business log when the target business is executed and the repair log when the repair task is executed may be stored in the cloud storage. In this way, when a use request or query request for the business data of the target business is received from other downstream businesses, the business data and business update data may be directly obtained from the cloud storage to realize fast data reading.

[0052] It should be noted here that cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can be achieved through cloud computing.

[0053] The business data processing system 10 involved in the embodiment of the present application can also be a distributed system of a blockchain system, wherein the distributed system can be a distributed node formed by multiple nodes (any form of computing devices in the access network, such as servers, user terminals) and clients, and a point-to-point network is formed between the nodes. The point-to-point network protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer. In the embodiment of the present application, the functions of each node in the blockchain system, the functions involved include: 1) routing, the basic function of the node, used to support communication between nodes. In addition to the routing function, the node can also have the following functions: 2) application, which is used to be deployed in the blockchain, implement specific services according to actual business needs, record the data related to the implementation function to form record data, carry a digital signature in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to verify the source and integrity of the record data successfully, and add the record data to the temporary block. 3) Blockchain, including a series of blocks that are connected in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the data submitted by the nodes in the blockchain system. 4) Consensus, a process in the blockchain network, is used to reach a consensus on the transactions in the block among the multiple nodes involved. The consensus blocks will be appended to the end of the blockchain. The mechanisms for achieving consensus include Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof-of-Stake (DPoS), Proof of Elapsed Time (PoET), etc.

[0054] In the embodiment of the present application, in the distributed system of the blockchain system, each node can record at least one of the following information: the business log when the target business is executed, and the business data obtained after parsing the business log; the repair log when the repair task is executed, and the business re-running statistics obtained after parsing the repair log; the business update data of the target business, etc. When updating the data recorded in the node in the blockchain system, the business re-running statistics of the target business in the target time period can be generated based on the business data stored in the previous node and the repair log obtained by executing the repair task, so as to update the business data based on the business re-running statistics, obtain the business update data used to characterize the business execution result of the target business, and store the business update data in the next node to avoid data tampering and ensure the accuracy of the data.

[0055] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 4 The electronic device shown may be a business data processing device, which includes: at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the business data processing device are coupled together via a bus system 340. It is understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 340 is not shown in FIG. Figure 4 Various buses are labeled as bus system 340 .

[0056] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The user interface 330 includes one or more output devices 331 that enable presentation of media content, and one or more input devices 332 .

[0058] The memory 350 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 350 may optionally include one or more storage devices physically away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiment of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 can store data to support various operations, and examples of these data include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0059] An operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, wireless compatibility certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.

[0060] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 4 A business data processing device 354 stored in the memory 350 is shown. The business data processing device 354 may be a business data processing device in an electronic device, which may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 3541, a task creation module 3542, a task execution module 3543, a log parsing module 3544, and a data update module 3545. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0061] In some embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the business data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0062] The business data processing methods provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the business data processing methods in each embodiment of the present application can be executed by a server, or by a terminal, or can be executed through interaction between a server and a terminal.

[0063] Figure 5 This is an optional flow chart of the business data processing method provided in the embodiment of the present application. Figure 5 The steps shown are explained as Figure 5 As shown, the execution subject of the business data processing method is a server as an example for description, and the method includes the following steps S101 to S105:

[0064] Step S101: during the current execution of the target service or when the execution is completed, a service re-running request is obtained.

[0065] In an embodiment of the present application, the terminal will generate a corresponding business log in real time during the normal execution of the target business, and the terminal will report the business log to the server. The server parses the business log to obtain the business data of the target business during the normal execution. The business data here can be report statistics, for example, it can be the number of clicks, exposures, number of interactions, number of plays, conversions, etc. for advertisements. When the parsed business data is counted to obtain report statistics, statistics can be performed according to different time dimensions, for example, once every 1 minute, once every 10 minutes, once every hour, or once every day. In an embodiment of the present application, after obtaining the report statistics, the server can provide the report statistics to other downstream businesses for use, for example, advertisers can query report statistics on the query terminal, and advertisers can calculate advertising costs based on the report statistics or calculate the next step of advertising push strategy, etc.

[0066] In some embodiments, during the process of the terminal reporting the business log to the server, if it is detected that an error occurs in the log reporting by the terminal, the business data processing method of the embodiment of the present application can be triggered, that is, the business operator can input the business rerun operation through the terminal, thereby generating a business rerun request, and the terminal sends the business rerun request to the server; or, when other downstream businesses use the generated report statistical data to perform corresponding calculations and business processing, if any indicator in the report statistical data (for example, it can be a click volume indicator or an exposure volume indicator, etc.) is calculated abnormally, the business data processing method of the embodiment of the present application can also be triggered, that is, the business operator can input the business rerun operation through the terminal, thereby generating a business rerun request, and the terminal sends the business rerun request to the server.

[0067] The service re-run request is used to request the re-execution of the service process of the target service within the target time period before the current moment; the target service generates service data during the current execution process for use by other services.

[0068] In an embodiment of the present application, the target time period may be one or more time periods before the current moment, each time period corresponding to a business log, a business log including multiple log information, the multiple log information can be referred to as a calculation batch of log information, that is, a business log can be considered as a calculation batch of log information. For example, the log information of each minute can be obtained, and the log information of one minute can be log parsed to obtain the intermediate aggregation result within this minute, and the intermediate aggregation result is the statistical result obtained after statistics of the data within this minute. After obtaining the intermediate aggregation results within each minute for a continuous period of time, these intermediate aggregation results can be aggregated again to obtain the intermediate aggregation results within one hour, and so on, the intermediate aggregation results of the target business in different time dimensions can be obtained.

[0069] Since the intermediate aggregation results in some time periods in the intermediate aggregation results under these different time dimensions may be wrong or the business logs of the time periods may have reporting errors, it is necessary to rerun the business in these time periods (i.e., target time periods), and these time periods may be determined as target time periods, and a business rerun request may be generated based on the target time period, and the business rerun request may include the time period identifier of the target time period. The time period identifier may be the time period start time and the time period end time of the target time period.

[0070] Step S102: In response to the service re-run request, a repair task for the target service within a target time period is created.

[0071] Here, the repair task is used to re-execute the business process of the target business within the target time period, and the repair task is independent of the current execution process. During the execution of the repair task, the target business will achieve the same process as the normal online task within the target time period, but the repair task will accurately output and report the business log within the target time period.

[0072] The repair task will not affect the normal execution of the target business during its execution. That is to say, the normal online tasks of the target business will continue to be executed. In this way, the normal online tasks will also output business data, and the repair tasks will also output business data. The business data output by these two tasks need to be written to the data storage unit for storage. Therefore, there will be a problem of business data write conflict.

[0073] Here, let's take an example to explain what write conflicts are: business reports are aggregated according to certain dimensions (such as ad exposure time), and log data of the same dimension is continuously arriving. The business cannot wait for all the data to arrive before starting calculations. Therefore, the indicator results will be continuously updated until all the logs of the dimension are available. Figure 6: is a schematic diagram of the business report aggregation process provided by the embodiment of the present application. The same graphics in the figure indicate that the events occur on the same day and need to be aggregated in the same dimension for display. TimeLine is the natural time, and the square is the exposure behavior that occurs between 10 and 11 o'clock during the event time. After 10 o'clock, it keeps flowing back. The advertiser report is updated every 3 minutes. Therefore, the exposure that occurs at 10 o'clock will be continuously accumulated and updated. Figure 6 In the table section, the exposure was updated three times from 10:00 to 11:00. If the data is not completely reflowed before the data is repaired, two tasks will appear to update a row of records in the database table at the same time; conversely, if the repair is performed after the data is completely reflowed, the business may even have to wait for 1 month (the conversion data is sent back by the advertiser, and the allowed delay window is 30 days). Therefore, in order to solve this problem, the embodiment of the present application designs a systematic re-running solution of the embodiment of the present application on the aggregation framework system of data report statistics to solve the above problems.

[0074] Step S103, executing the repair task and obtaining a repair log.

[0075] In the embodiment of the present application, during the execution of the repair task, a repair log within the target time period can be obtained and the repair log can be output in real time.

[0076] In some embodiments, since the target time period may include multiple time periods, the repair task may include multiple subtasks, and the multiple subtasks are arranged in sequence according to the corresponding time periods. When executing the repair task, each subtask may be executed in sequence. When executing each subtask, a corresponding repair log is obtained. After multiple subtasks are completed, a complete repair log corresponding to the entire repair task is obtained. After log parsing, multiple log segments in the complete repair log can be used to obtain multiple business rerun statistics.

[0077] Step S104: parsing the repair log to obtain business rerun statistics of the target business within the target time period.

[0078] In an embodiment of the present application, log parsing of the repair log refers to reading the content in the repair log to extract the pending statistical data about the target statistical indicator within the target time period, and then, after aggregating the pending statistical data within the target time period, the business rerun statistical data can be obtained.

[0079] Here, the target statistical indicator can be any one or more indicators related to the target business. For example, the target statistical indicator can be an indicator such as click volume, purchase volume, and pageview volume; the data to be counted refers to the original indicator data before aggregation and statistics are performed. For example, the data to be counted can be whether the target advertisement is clicked at a certain moment. In this way, if the target advertisement is clicked at a certain moment, the business re-run statistical data can be increased by one when the data to be counted is aggregated.

[0080] Step S105 : updating the service data using the service rerun statistical data to obtain service update data for representing the service execution result of the target service.

[0081] In the embodiment of the present application, the business data is updated by using business rerun statistical data, which can be to update the business data in the target time period to the business rerun statistical data, and the business data in other time periods remain unchanged, or the business rerun statistical data and the business data are stored in the data storage unit at the same time, just to distinguish the two data, and a corresponding data identifier can be generated for each data, so that the data identifier can be used to distinguish whether the data is business data output by the normal online process or business rerun statistical data output by the repair task. Of course, a log directory can also be constructed for the target business, and multiple directory information is stored in the log directory, each directory information is a directory of the business log in the time period corresponding to a calculation batch, and because the business log in the time period corresponding to this calculation batch corresponds to an intermediate aggregation result in the business data, each directory information can also be a directory of an intermediate aggregation result.

[0082] When the intermediate aggregation result of the time period corresponding to each calculation batch is obtained (that is, the business data corresponding to a time period in the entire business data), corresponding identification information can be generated for the intermediate aggregation result, and the identification information is the above-mentioned data identification. At the same time, the time when the intermediate aggregation result is generated is obtained, and the timestamp of generating the intermediate aggregation result is determined as the version information corresponding to the data identification, thereby obtaining the version information of each intermediate aggregation result. In this way, when an intermediate aggregation result is obtained by each aggregation, the data identification and version information of the intermediate aggregation result can be generated, and the data identification and version information are stored in the log directory in the data storage unit. In this way, if the business is rerun for any time period, there will be two version information in the target information of the time period, one is the version information corresponding to the intermediate aggregation result generated when the normal online task is running, and the other is the version information corresponding to the intermediate aggregation result generated when the repair task is running. At this time, if other downstream businesses need to use or query the business data of the time period (that is, the intermediate aggregation result corresponding to the time period), the intermediate aggregation result generated when the repair task with the latest version information is running can be output to other downstream businesses for use.

[0083] In an embodiment of the present application, by querying the log directory in the data storage unit, that is, querying the directory information corresponding to each time period in the log directory, it is determined whether there are two versions of information for a certain time period in the directory information. If so, it means that the time period has been rerun for business. Therefore, the intermediate aggregation result of the latest version of the time period is used for subsequent business processing. In this way, while ensuring the stable and orderly execution of normal online tasks of the target business, it can also be ensured that the intermediate aggregation results output by the repair task will not conflict with the intermediate aggregation results output by the normal online task, thereby avoiding the problem of write conflicts.

[0084] The business data processing method provided by the embodiment of the present application, when it is necessary to rerun the business of the target business, creates a repair task for the target business within the target time period; the repair task is used to realize the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process of the target business; then, the repair task is executed to obtain a repair log; and the repair log is parsed to obtain the business re-running statistical data of the target business within the target time period; thereby the business data is updated using the business re-running statistical data to obtain the business update data used to characterize the business execution result of the target business. In this way, since the business process of the target business within the target time period is re-executed by a repair task independent of the current execution process of the target business, the business re-running statistical data within the target time period is parsed by the repair log after re-execution, so as to realize the accurate re-run of the target business, and obtain the business re-running statistical data that can accurately reflect the business data, not only can the full amount of redeployment and re-running be realized when performing a systematic re-run, but also the computing resources can be greatly saved and the efficiency of the systematic re-run can be improved.

[0085] In some embodiments, the business data processing includes at least a terminal and a server, and a report processing application is running on the terminal, wherein the report processing application can be any application that can generate streaming data, and the report processing application can be an advertising application, which refers to any interactive application that can deliver advertisements, such as a short video application. Server 300 is a server for the report processing application, that is, it can be a server for the advertising application, or it can be a third-party server independent of the advertising application, and the third-party server is used to perform log parsing and aggregation processing on the business logs generated in the advertising application, obtain the business data of the target business in the advertising application, and generate a business report for the target business.

[0086] The embodiments of the present application can be applied to at least the following scenarios: After generating an advertisement, the advertiser can place the advertisement in a short video application. The short video application can request payment of the advertisement fee from the advertiser based on the number of business response behaviors such as clicks or purchases of the advertisement by the user through the short video application. Therefore, it is necessary to count the number of business response behaviors of the user. When counting the number of business response behaviors, the business log of the business of playing advertisements during operation can be obtained during the operation of the short video application, and the business data reflecting the number of business response behaviors can be obtained by parsing the business log. In the embodiments of the present application, the advertising application can be run on a terminal, which can be a business execution terminal, and the server can be a server of the advertising application or a third-party server of other downstream businesses.

[0087] In the embodiment of the present application, the target service may be any service in the advertising application, for example, it may be an advertising video of a certain product. Executing the target service may be playing the advertising video in the advertising application.

[0088] Figure 7 is another optional flow chart of the business data processing method provided in the embodiment of the present application, such as Figure 7 As shown, the method includes the following steps S201 to S218:

[0089] Step S201 , during the process of executing a target service at a terminal, a service log of the target service execution is generated.

[0090] In the embodiment of the present application, the terminal executes the target business through a normal online task, and during the execution process, a business log of the target business execution is generated in real time.

[0091] In some embodiments, the entire business log generated during the entire execution process of the target business can also be divided into multiple business logs with preset durations, each of which includes multiple log information, and the multiple log information is the log information of a calculation batch, that is, each business log with a preset duration is a calculation batch of logs.

[0092] In the embodiment of the present application, the reason why the entire business log is divided to form log information of multiple calculation batches is because: a log directory of the target business is pre-constructed in the data storage unit, and multiple directory information is written in the log directory, each directory information corresponds to the log information of a calculation batch, therefore, the log information of each calculation batch can be log parsed, and the parsing results can be aggregated to obtain intermediate aggregation results, so that each intermediate aggregation result and the status information corresponding to the intermediate aggregation result can be stored in a directory information. Here, the status information includes but is not limited to the calculation batch completion mark, partition mark, version information, etc. corresponding to the intermediate aggregation result. Among them, the calculation batch completion mark, partition mark, version information, etc. will be explained below.

[0093] The data storage unit may be a partial storage unit in a distributed storage system, and the data storage unit is used to store information such as the log directory of the target business. The distributed storage system may store the business log reported by the target business during normal online task execution and the repair log reported during repair task execution.

[0094] Step S202: The terminal reports the service log of the target service execution to the server.

[0095] Step S203: When a business log reporting error is detected, the terminal receives a business re-run operation input by the business personnel.

[0096] In the embodiment of the present application, the time period when the business log reports an error can be detected, and the time period when the business log reports an error can be determined as the target time period; then, a business re-run request is generated based on the target time period. Of course, the business personnel can also enter the target time period on the terminal and enter the business re-run operation.

[0097] In some other embodiments, there may also be a case where a service rerun operation is received in the following steps:

[0098] Step S204, when other businesses perform other business processing based on the business data corresponding to the business log of the target business, if an exception occurs in the indicator calculation during the other business processing, the terminal receives the business re-run operation input by the business personnel.

[0099] At this time, an exception occurs in the indicator calculation in other business processing processes, indicating that the business data needs to be re-extracted for business processing. Therefore, the business data within the target time period where the exception occurs needs to be re-obtained. Therefore, the business personnel can enter the business re-run operation.

[0100] During the implementation process, the time period corresponding to the business data with abnormal indicator calculation in the business log can be detected, and the time period can be determined as the target time period; then, a business re-run request is generated based on the target time period. Of course, the business personnel can also enter the target time period on the terminal and enter the business re-run operation.

[0101] Step S205: the terminal generates a service rerun request in response to the service rerun operation.

[0102] Here, the service identifier and target time period of the target service can be encapsulated into a service re-run request. The service re-run request is used to request re-execution of the service process of the target service within the target time period before the current moment; the target service generates service data for use by other services during the current execution process.

[0103] Step S206: The terminal sends a service re-run request to the server.

[0104] Step S207: The server creates a repair task for the target business within a target time period in response to the service rerun request.

[0105] Here, the repair task is used to re-execute the business process of the target business within the target time period, and the repair task is independent of the current execution process.

[0106] During the execution of the repair task, the target business will achieve the same process as the normal online task within the target time period, except that the repair task will accurately output and report the business log within the target time period. During the execution of the repair task, the normal execution process of the target business will not be affected. In other words, the normal online task of the target business will continue to execute, so there will be a normal online task that will output business data, and the repair task will also output business data. The business data output by these two tasks need to be written to the data storage unit for storage. Therefore, there will be a business data write conflict problem.

[0107] Step S208: The server executes the repair task and obtains a repair log.

[0108] Step S209: The server reports the repair log with a preset time period as the reporting period; and stores the reported repair log in a preset distributed storage system.

[0109] Here, the preset duration may be 1 minute, 10 minutes, or 1 hour.

[0110] In some embodiments, the reported repair log may be stored in a preset distributed storage system. In other embodiments, the repair log may also be stored in a local storage unit, and when a log reporting request is received, the stored repair log is reported.

[0111] In an embodiment of the present application, when reporting the repair log, the repair log of each preset time length can also be divided into a log of a calculation batch, and a corresponding log identifier can be generated for the log of each calculation batch, and the log identifier can be the partition information corresponding to the log of this calculation batch, that is, the repair log is divided into a partition according to the preset time length, and this partition has a unique partition identifier. When reporting the repair log, it can be that while reporting the repair log of each calculation batch, the partition identifier corresponding to the repair log of the calculation batch is also reported to the data storage unit in the preset distributed storage system, and a log directory is preset in the data storage unit, and the log directory is used to store the partition identifier corresponding to the repair log of each calculation batch.

[0112] Step S210: The server reads a plurality of repair logs with a preset duration from the distributed storage system.

[0113] In step S211, the server parses each repair log segment to obtain repair task statistics of each repair log segment with respect to a target statistical indicator under a preset time dimension.

[0114] Here, the target statistical index is determined according to the target business, and different businesses correspond to different statistical indicators. Here, taking the target business as the advertising business as an example, the target statistical index can be any index such as the number of clicks on the advertisement, the number of purchases, the number of advertisement broadcasts, etc.

[0115] Here, taking the target statistical indicator being the number of advertisement clicks as an example, by parsing the information in the repair log, it is determined whether the user performs a click operation on the played advertisement at each moment. If the user performs a click operation on the played advertisement at this moment, the corresponding repair task statistical data at that moment is determined to be 1; if the user does not perform a click operation on the played advertisement at this moment, the corresponding repair task statistical data at that moment is determined to be 0. In this way, by parsing each repair log, the repair task statistical data of each repair log at each moment can be obtained. In the embodiment of the present application, the preset duration dimension can be the minimum statistical dimension of the repair log, for example, it can be 1 minute.

[0116] In some embodiments, after obtaining the repair task statistics corresponding to the repair log at each moment, the repair task statistics can be further aggregated to obtain the repair task intermediate aggregation results of the repair task with respect to the target statistical index in a higher duration dimension. For example, the results of each minute can be aggregated to obtain the statistics of each 10 minutes, which is the intermediate aggregation result of the repair task in the 10-minute duration dimension.

[0117] It should be explained here that the aggregation processing indicated in the embodiments of the present application may be a data statistical processing, for example, counting the number of corresponding target statistical indicators within a preset time period.

[0118] In the embodiment of the present application, the reason why the repair log needs to be parsed is that common reporting scenarios are usually reporting scenarios with backtracking, and in reporting scenarios with backtracking, the business data of the current time period usually needs to be aggregated and calculated based on the business data of a future time period after the current time period. Obviously, in this case, report calculation errors are prone to occur, resulting in statistically erroneous business data.

[0119] Here, let's explain what a report scenario with backtracking is. First, let's look at an example. Figure 8 is a schematic diagram of a report scenario with backtracking provided in an embodiment of the present application, such as Figure 8As shown in the figure, after the advertising application pushes the advertiser's advertisement, if a user sees the advertisement on January 1 and clicks it, jumps to the landing page and adds the product to be purchased to the shopping cart, and places an order on January 29, for this order behavior on January 29, if it is counted according to the reporting time, then the order quantity on January 29 is 1, but if it is counted according to the deduction time of the advertisement corresponding to the order, then the order quantity on January 1 is 1. This is equivalent to modifying the data on January 1 on January 29, which is a report with a backtracking scenario, and both calibers are clearly needed. In particular, the second billing caliber allows users to intuitively feel how much advertising was consumed on January 1 and how many orders these advertising consumptions brought to the advertiser.

[0120] The business data processing method of the embodiment of the present application can solve Figure 8 The business data processing process of the backtracking report scenario shown is achieved by re-running the business process within the target time period where the error occurred before, that is, by creating a repair task and executing the repair task, and obtaining business re-run statistics through the repair log. In this way, the erroneous business data is updated with the business re-run statistics to ensure the accuracy of the counted business data. Fig. 9 is a schematic diagram of the business data processing process provided by the embodiment of the present application, such as Fig. 9 As shown in the figure, assuming that the system continuously receives order data with a billing time of 8 a.m., the real-time calculation channel (i.e., the current execution process of the target business) ignores the delay. Fig. 9 The horizontal time axis in the middle indicates that at different natural times, the advertiser can view the business data of the current execution process and see how many orders are placed at 8 o'clock at the billing time. For example, at 11 o'clock, the business data of the current execution process is 400 orders. At this time, if the report log of the current execution process shows that an error occurred between 11 o'clock and 12 o'clock, the method of the embodiment of the present application can be used to re-run and repair the data between 11 o'clock and 12 o'clock. At this time, you can create Fig. 9 The repair channel (i.e., repair task) in the process reruns the business process between 11 o'clock and 12 o'clock, obtains the repair log between 11 o'clock and 12 o'clock, and parses the repair log to determine that the business data between 11 o'clock and 12 o'clock is 200 orders (the business data determined in the real-time calculation channel is 100 orders). Therefore, these 200 orders are updated to the business data, and the erroneous 100 orders from 11 o'clock to 12 o'clock in the business data are modified to 200 orders, so that the business update data is 700 orders in total by 12 o'clock.

[0121] Step S212: The server generates a calculation batch completion mark for the repair log.

[0122] Here, the calculation batch completion mark is used to indicate that the multiple log information in the corresponding repair log has been parsed and the statistical data corresponding to the repair log has been obtained. Whenever each log in the repair log corresponding to a calculation batch has completed the above parsing, a calculation batch completion mark can also be generated for the repair log of the calculation batch.

[0123] Step S213: The server maps the calculation batch completion identifier with the target information of the corresponding repair log to obtain a mapping relationship between the calculation batch completion identifier and the corresponding repair log.

[0124] Step S214, the server stores the calculation batch completion identifier and the mapping relationship in the data storage unit.

[0125] In the embodiment of the present application, the calculation batch completion identifier and the mapping relationship can be stored in the log directory in the data storage unit. In the log directory in the data storage unit, the partition identifier corresponding to the repair log of each calculation batch is stored, wherein the calculation batch corresponds to a time period, and therefore, the partition identifier is used to identify the partition of the time period in the entire target business.

[0126] In some embodiments, since corresponding business logs are generated during the normal online task execution process for the target business, the business logs can also be divided according to the time period corresponding to each partition identifier to form multiple business log segments. Therefore, the partition identifier can also be used to identify the business log segments in the business log generated during the normal online task execution process, that is, the partition identifier and the business log segments in the business log generated during the normal online task execution process also have a mapping relationship. In this way, the business log segments in the business log generated during the normal online task execution process can be queried from the distributed storage system through the partition identifier in the log directory, and the corresponding repair log segments (for example, a repair log of a calculation batch) generated when the repair task is executed can also be queried. At the same time, based on the mapping relationship between the calculation batch completion identifier and the corresponding repair log, the calculation batch completion identifier of the repair log segment can also be queried, so as to determine based on the calculation batch completion identifier that multiple log information in the corresponding repair log of the time period corresponding to the calculation batch has been parsed, and the statistical data corresponding to the repair log segment has been obtained.

[0127] In step S215, the server aggregates the statistical data of multiple repair tasks corresponding to the multiple repair logs at different time dimensions to obtain intermediate aggregation results of the repair tasks at different time dimensions.

[0128] Here, the statistical data of multiple repair tasks corresponding to multiple repair logs are aggregated in different time dimensions. The aggregation processing can be performed in different time dimensions respectively, and the duration corresponding to the time dimension of each aggregation processing is less than the duration corresponding to the time dimension of the next aggregation processing. For example, the aggregation processing can be performed in the 1-minute duration dimension first to obtain the aggregation results of multiple repair tasks with a duration of 1 minute; then the aggregation results of the repair tasks with a duration of 1 minute are aggregated in the 10-minute duration dimension to obtain the aggregation results of the repair tasks with a duration of 10 minutes; then the aggregation results of the repair tasks with a duration of 10 minutes are aggregated in the 1-hour duration dimension to obtain the aggregation results of the repair tasks with a duration of 1 hour, and so on, until the aggregation processing under the preset multiple duration dimensions is completed.

[0129] Step S216: The server determines the business rerun statistics of the target business within the target time period based on the intermediate aggregation results of the repair tasks in different time dimensions.

[0130] In some embodiments, the intermediate aggregation results under different time dimensions may be aggregated to obtain business rerun statistics of the target business within the target time period.

[0131] Here, aggregating the intermediate aggregation results under different time dimensions refers to performing statistics on the intermediate aggregation results under different time dimensions to obtain statistical data within the entire target time period. The statistical data within the target time period is the business re-run statistical data.

[0132] For example, if the target time period is 10:00-11:00 a.m., the intermediate aggregation results under different time dimensions are: in the 1-minute time dimension, the intermediate aggregation results of each minute; in the 10-minute time dimension, the intermediate aggregation results of every 10 minutes from 10:10 to 10:20, 10:20 to 10:30, 10:30 to 10:40, 10:40 to 10:50, and 10:50 to 11:00; in the 30-minute time dimension, the intermediate aggregation results of 10:10 to 10:40. Then, when aggregating the intermediate aggregation results under different time dimensions, since the statistics to be counted are from 10 a.m. to 11 a.m., in order to reduce the amount of calculation, the intermediate aggregation results of each minute in the time period from 10 a.m. to 10:10 a.m., the intermediate aggregation results of the 30-minute time dimension from 10:10 a.m. to 10:40 a.m., and the intermediate aggregation results of the 10-minute time dimensions from 10:40 a.m. to 10:50 a.m. and 10:50 a.m. to 11 a.m. can be taken. Then, these intermediate aggregation results are aggregated again, that is, data statistics can be performed to obtain the business re-run statistics for the target time period from 10 a.m. to 11 a.m.

[0133] In some embodiments, the intermediate aggregation result of the repair task obtained after the repair task is executed can also be used to update the intermediate aggregation result of the target business obtained during the normal operation. During the implementation process, it can be achieved in the following way: first, the business log of the target business in the target time period is parsed to obtain at least one business statistical data about the target statistical indicator in the business log under the preset time dimension; then, at least one business statistical data is aggregated under different time dimensions to obtain the intermediate aggregation results of the business under different time dimensions; then, under the same time dimension, the intermediate aggregation result of the repair task is used to update the corresponding intermediate aggregation result of the business to obtain the updated intermediate aggregation result of the business; and the updated intermediate aggregation result of the business is stored in the distributed storage system.

[0134] In some embodiments, for the generated business intermediate aggregation results and the business intermediate aggregation results to be generated, the business intermediate aggregation results to be generated under different time dimensions can be generated in the following way: here, two adjacent time dimensions are taken as an example for explanation, and the two time dimensions are respectively named the first time dimension and the second time dimension, wherein the duration corresponding to the first time dimension is less than the duration corresponding to the second time dimension.

[0135] During the implementation process, the last business intermediate aggregation result among the updated business intermediate aggregation results under the first time dimension and the next business intermediate aggregation result to be generated connected to the last business intermediate aggregation result can be obtained; then, based on the last business intermediate aggregation result and the business intermediate aggregation result to be generated, the next business intermediate aggregation result to be generated under the second time dimension and connected to the last business intermediate aggregation result among the updated business intermediate aggregation results under the second time dimension is determined.

[0136] Step S217: The server updates the service data using the service rerun statistical data to obtain service update data for representing the service execution result of the target service.

[0137] Step S218: The server provides the service update data to other services for use.

[0138] In some embodiments, the repair log with a preset duration includes multiple log information; the multiple log information is log information of a calculation batch; each calculation batch has a mapping relationship with a directory information in the log directory in the data storage unit; the log directory includes directory information of each repair log. In order to avoid conflicts in the output of business data corresponding to the business logs generated during the normal execution process and the re-run process of the target business, that is, in order to distinguish the business intermediate aggregation results during the normal execution process and the re-run process, a partition identifier for uniquely identifying the business intermediate aggregation result can be generated for each business intermediate aggregation result during the normal execution process, and a re-run partition identifier for uniquely identifying the repair task intermediate aggregation result can also be generated for each repair task intermediate aggregation result during the re-run process.

[0139] In an embodiment of the present application, a partition identifier for uniquely identifying a business intermediate aggregation result can be generated in the following manner: first, the business log when the target business is executed is parsed to obtain a plurality of business statistical data about the target statistical indicator in the business log under a preset time dimension; then, the plurality of business statistical data are aggregated under different time dimensions to obtain the business intermediate aggregation results under different time dimensions; and, the partition information corresponding to each business intermediate aggregation result in the business log is obtained; a partition identifier of the corresponding business intermediate aggregation result is generated based on the partition information; and each partition identifier can also be stored as target information in the log directory.

[0140] In an embodiment of the present application, a rerun partition identifier for uniquely identifying the intermediate aggregation result of the repair task can be generated in the following manner: first, when the intermediate aggregation result of the repair task corresponding to the log information of each calculation batch is obtained, the rerun partition information corresponding to the intermediate aggregation result of the repair task in the repair log is obtained; the rerun partition information is used to determine the log start position and log end position of the log information of the corresponding calculation batch in the entire repair log; then, a rerun partition identifier of the intermediate aggregation result of the corresponding repair task is generated based on the rerun partition information; and the timestamp corresponding to the log information of each calculation batch is obtained; then, the version information of the intermediate aggregation result of the corresponding repair task is generated based on the timestamp; the rerun partition identifier and version information of the intermediate aggregation result of each repair task can also be mapped and stored in the log directory.

[0141] Since the data storage unit stores a log directory, which records the partition identifier of each intermediate aggregation result of the target business during normal execution, and the rerun partition identifier of each intermediate aggregation result of the repair task, when other downstream businesses need to use the business data of the target business during business operation, they can determine whether to rerun the business process within any target time period based on the partition identifier and the rerun partition identifier, thereby obtaining more accurate business data.

[0142] In an embodiment of the present application, when other downstream businesses trigger a request to use the intermediate aggregation results of the business for the log information of any calculation batch in the distributed storage system, if the partition identifier and the re-run partition identifier corresponding to the log information of the same calculation batch are stored in the log directory, and the timestamp in the version information corresponding to the re-run partition identifier is closer to the current moment than the timestamp in the version information corresponding to the partition identifier, then the intermediate aggregation result of the repair task corresponding to the re-run partition identifier can be obtained; and the intermediate aggregation result of the repair task can be output to other businesses for use.

[0143] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0144] The business data processing method provided in the embodiment of the present application can be applied to the scenario of re-running advertiser report data. There are two situations in which errors often occur and need to be re-run: the first is that an error occurs in the reported log. After the log is re-run and repaired, the advertiser report also needs to be repaired; the second is that an abnormality occurs in the indicator calculation and needs to be repaired.

[0145] The business data processing method provided in the embodiment of the present application performs data repair through a bypass computing task (i.e., the above-mentioned repair task, which may be, for example, a Seagull instance, and Seagull is a batch-stream integrated statistical framework based on the Spark Streaming framework) when data is rerun. The bypass computing task is isolated from the online task to avoid affecting the normal online data computing service.

[0146] Fig.10 : is a system architecture diagram of the Seagull statistical framework provided in the embodiment of the present application, such as Fig.10 As shown in the figure, the Seagull statistical framework is a real-time aggregation framework based on Spark Streaming batch and stream integration. Data aggregation includes two stages: aggregation stage and output stage.

[0147] Among them, in the data reporting stage, the business logs will be stored in the log directory of the distributed file system (HDFS, Hadoop Distributed File System) every minute. The aggregation stage of the Seagull statistical framework is responsible for reading these minute-level HDFS logs and generating the business intermediate aggregation results of the report increments under different time dimensions on the delivery end, and storing the business intermediate aggregation results on HDFS.

[0148] The output stage is responsible for reading multiple business intermediate aggregation results, aggregating the read business intermediate aggregation results again, and outputting the aggregated results to the HBase database table (that is, the data storage unit) for downstream application queries, such as advertising delivery, shopping platform, etc.

[0149] In the embodiment of the present application, both the aggregation stage and the output stage will generate meta information and checkpoint identification information (for example, it can be checkpoint identification information) inside the Seagull system, and both the meta information and checkpoint information will be stored in the HBase database table. The meta information here refers to the mapping relationship between the calculation batch (batch) and the log directory, and the checkpoint identification information is used to identify whether the corresponding calculation batch (batch) has completed data aggregation processing.

[0150] It should be noted that during the data rerun process, since the rerun of the business process will also generate business logs (i.e., repair logs), the Seagull statistical framework is also used for data aggregation during the rerun of the business process. In addition, the process of data aggregation is the same as the normal execution process of the target business.

[0151] In some embodiments, in order to ensure that the calculation is not repeated or lost when the task is restarted, Seagull adopts the "full calculation, incremental output" approach. The full calculation here means that the intermediate results of different levels (i.e. different time dimensions) will be generated in the aggregation stage. The high level is generated by the aggregation of the levels (such as Fig.11 Incremental output here means when there is new incremental data (i.e. intermediate aggregation results, such as Fig.11 When the black square in the figure needs to be output, all incremental results containing the newly added incremental data will be found in all historical intermediate results with black squares (such as Fig.11 There are four levels in the code, and there are five incremental results containing the new incremental data in the four levels, that is, Fig.11 Incremental results with fill patterns in the ), and for all incremental results (i.e. Fig.11The five incremental results in the task are aggregated again to obtain the business data after the task is restarted, and then the business data is updated to the HBase database table.

[0152] The design goal of such a data rerun solution is to build a systematic repair path that performs repairs as quickly and resource-efficiently as possible without affecting normal task execution. The ideas are as follows: First, in order to achieve faster and more economical results, the repair task will not rerun all historical data, but only rerun the data in the time period where the error occurred; second, in order to avoid affecting the execution of online tasks, the repair is performed through a bypass computing task. Considering the compatibility of downstream use, the repair task directly updates the correct result to the database table that is providing the service; third, if the two tasks are repaired in situ, there will be a write conflict problem. To address this problem, the embodiment of the present application solves it through the version number of the Hbase database table. For the intermediate aggregation results within the same time period, only the largest version will be visible to the business, and this version number is uniformly distributed by the online task; finally, during the conflict period (that is, when the online task and the repair task are running at the same time, the unified row in the HBase database table will be updated at the same time, and a conflict will occur at this time, which is called the conflict period here. It should be noted that the online task is the process of normal online execution of the target business, and the repair task is the process of re-executing the business process in the target time period of the target business), any output execution of the repair task is controlled and distributed by the online task, and the two interact through the HDFS task interaction file (which can be called the rfile file, which will be introduced in detail below).

[0153] In the embodiment of the present application, the repair process of the business process of the target business within the target time period includes the following three stages:

[0154] Phase 1: HDFS intermediate result repair.

[0155] Fig.12 This is a schematic diagram of the HDFS intermediate aggregation result repair process provided in an embodiment of the present application. The intermediate aggregation result here refers to the intermediate aggregation result obtained after aggregating the business data corresponding to the business log in different time dimensions. For online tasks, the intermediate aggregation result is the above-mentioned business intermediate aggregation result; for repair tasks, the intermediate aggregation result is the above-mentioned repair task intermediate aggregation result.

[0156] See also Fig.12 In the embodiment of the present application, only the currently generated erroneous intermediate result partition is re-run and repaired, and placed in an independent HDFS directory: Fig.12The middle left side shows the intermediate aggregation result partitions generated when a normal online task is running. a to h are the minute-level intermediate results generated by aggregating the business data in the read business logs. cd and ef are high-level intermediate results generated by merging the minute-level intermediate results. The black thick-lined boxes are partitions with calculation errors, and the dashed boxes are partitions that have not yet been produced. Fig.12 The middle right side shows the intermediate aggregation result generated by the repair task. The repair task will only rerun the error partitions that already exist at the current time, such as Fig.12 For the six partitions on the right, the repair task will not repeat the calculation for the correct or not generated partitions, such as f, h and gh partitions.

[0157] In some embodiments, for a normal online task, if the aggregation stage completion state (AGGREGATION_DONE, i.e., the aggregation of partitions corresponding to any calculation batch is completed) is detected, the old error partition (i.e., the error intermediate aggregation result partition generated when the normal online task is running) will not be visible, and the correct intermediate aggregation result partition generated when the normal online task is running will form an intermediate aggregation result result directory (i.e., HDFS directory, such as Fig.13 a, b, C, D, E, f, G, CD, EF), thus forming a complete intermediate result, in which, Fig.13 The middle dotted high-level partition "Gh" is generated by merging the h partition generated by the normal online task and the G partition generated by the repair task.

[0158] In some embodiments, while a normal online task is running, it will generate snapshot / version information based on the output batch information (i.e., calculation batch) in the rfile file together with the output batch of its own task being executed. Here, the snapshot information is a set of intermediate aggregation result partitions. The larger the set, the newer the data, and the larger the corresponding version number (version). This is also the core of solving the common output of multiple tasks. In order to ensure the order of snapshot / version, the process of generating snapshot / version information is synchronized. After the normal online task is completed, the mapping relationship between the snapshot / version information and the corresponding output batch is updated to the rfile file. At this time, the output status (status) of the intermediate aggregation result in the rfile file will be marked as the output response status (OUTPUT_RESPONSE).

[0159] Phase 2: HBase output results repair.

[0160] In the embodiment of the present application, the repair task can start outputting after detecting the snapshot / version information of the rfile file. At this time, the normal online task is also outputting, and the conflict between the two is resolved through the above version number (version).

[0161] Phase 3: Restoration and finalization.

[0162] In an embodiment of the present application, after the repair task is completed and output, the repaired intermediate aggregation result (i.e., the intermediate aggregation result of the repair task) will be copied to the result directory of the normal online task. It is worth mentioning that, considering that deletion is a high-risk operation, the erroneous intermediate aggregation results in the normal online task will not be physically deleted here, only a renaming operation will be performed.

[0163] The above is the complete repair process. The rfile file is the two running task interaction files designed for the repair process in the embodiment of the present application. The file organization form is proto buffer. For the specific protocol of the rfile file, please refer to Fig.14 The content of the agreement is shown.

[0164] The business data processing method provided in the embodiment of the present application performs data repair through a repair task. The repair task is isolated from the online task to avoid affecting the normal online data computing service. At the same time, the repair task only reruns the data during the error period, and the result after rerun is directly written into the online database table in situ. The advantage of this method is that it can quickly perform repairs with fewer resources, and at the same time, downstream application queries do not need to be aware of it. In addition, in the embodiment of the present application, a self-developed rfile file is used to record the basic information and status of the entire repair process, so that the online task and the repair task can exchange information through the rfile file, thereby solving the write conflict problem of simultaneous calculation of the two tasks.

[0165] It can be understood that in the embodiments of the present application, if the content involves user information, such as business data, business update data and other information, and if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the subject of personal information should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the subject of personal information.

[0166] The following is a description of an exemplary structure of the business data processing device 354 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 4As shown, the business data processing device 354 includes: an acquisition module 3541, which is used to obtain a business re-run request during the current execution process of the target business or when the execution has been completed; the business re-run request is used to request the re-execution of the business process of the target business within the target time period before the current moment; the target business generates business data for use by other businesses during the current execution process; a task creation module 3542 is used to create a repair task for the target business within the target time period in response to the business re-run request; the repair task is used to achieve the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process; a task execution module 3543 is used to execute the repair task and obtain a repair log; a log parsing module 3544 is used to perform log parsing on the repair log to obtain business re-run statistics of the target business within the target time period; a data update module 3545 is used to update the business data using the business re-run statistics to obtain business update data used to characterize the business execution result of the target business.

[0167] In some embodiments, the acquisition module is also used to: during the execution of the target business, obtain the business log of the target business execution reported by the business execution terminal; if an error is detected in the business log reporting, determine the time period of the business log reporting error, and determine the time period of the business log reporting error as the target time period; generate the business re-run request based on the target time period.

[0168] In some embodiments, the acquisition module is also used to: in the process of executing the target business, obtain the business log of the target business execution reported by the business execution terminal; when the other business performs other business processing based on the business data corresponding to the business log, if an abnormality occurs in the indicator calculation during other business processing, determine the time period corresponding to the business data with the abnormal indicator calculation in the business log, and determine the time period as the target time period; generate the business re-run request based on the target time period.

[0169] In some embodiments, the device further includes: a log reporting module, configured to report the repair log with a preset duration as a reporting period; and store the reported repair log in a preset distributed storage system.

[0170] In some embodiments, the log parsing module is also used to: read multiple repair logs with the preset time length from the distributed storage system; parse each repair log to obtain repair task statistical data about the target statistical indicator for each repair log under the preset time length dimension; aggregate multiple repair task statistical data corresponding to the multiple repair logs under different time dimensions to obtain intermediate aggregation results of the repair tasks under different time dimensions; based on the intermediate aggregation results of the repair tasks under different time dimensions, determine the business re-run statistical data of the target business within the target time period.

[0171] In some embodiments, the log parsing module is further used to: perform aggregation processing on the intermediate aggregation results under the different time dimensions to obtain business re-run statistics of the target business within the target time period.

[0172] In some embodiments, the repair log with the preset time length includes multiple log information; the multiple log information is the log information of a calculation batch; each of the calculation batches has a mapping relationship with a directory information in the log directory in the data storage unit; the log directory includes directory information of each section of the repair log; the device also includes: a completion identifier generation module, used to generate a calculation batch completion identifier for the repair log; the calculation batch completion identifier is used to identify that multiple log information in the corresponding repair log have been parsed and statistical data corresponding to the section of the repair log have been obtained; an information mapping module, used to map the calculation batch completion identifier with the target information of the corresponding repair log to obtain a mapping relationship between the calculation batch completion identifier and the corresponding repair log; a storage module, used to store the calculation batch completion identifier and the mapping relationship in the data storage unit.

[0173] In some embodiments, the log parsing module is also used to: parse the business log of the target business within the target time period to obtain at least one business statistical data of the business log about the target statistical indicator under the preset time dimension; aggregate the at least one business statistical data under different time dimensions to obtain business intermediate aggregation results under different time dimensions; under the same time dimension, use the repair task intermediate aggregation result to update the corresponding business intermediate aggregation result to obtain an updated business intermediate aggregation result; store the updated business intermediate aggregation result in the distributed storage system.

[0174] In some embodiments, the log parsing module is also used to: obtain the last business intermediate aggregation result in the updated business intermediate aggregation results under the first time dimension and the next business intermediate aggregation result to be generated connected to the last business intermediate aggregation result under the adjacent first time dimension and the second time dimension; determine the next business intermediate aggregation result to be generated under the second time dimension and connected to the last business intermediate aggregation result in the updated business intermediate aggregation results based on the last business intermediate aggregation result and the business intermediate aggregation result to be generated; the duration corresponding to the first time dimension is less than the duration corresponding to the second time dimension.

[0175] In some embodiments, the log parsing module is also used to: parse the business log when the target business is executed to obtain multiple business statistical data about the target statistical indicators in the business log under the preset time dimension; aggregate the multiple business statistical data under different time dimensions to obtain the business intermediate aggregation results under different time dimensions; obtain the partition information corresponding to each of the business intermediate aggregation results in the business log; generate a partition identifier of the corresponding business intermediate aggregation result based on the partition information; and store each of the partition identifiers as the target information in the log directory.

[0176] In some embodiments, the log parsing module is also used to: when obtaining the intermediate aggregation result of the repair task corresponding to the log information of each calculation batch, obtain the re-run partition information corresponding to the intermediate aggregation result of the repair task in the repair log; the re-run partition information is used to determine the log starting position and log ending position of the log information of the corresponding calculation batch in the entire repair log; generate a re-run partition identifier of the corresponding intermediate aggregation result of the repair task based on the re-run partition information; obtain the timestamp corresponding to the log information of each calculation batch; generate the version information of the intermediate aggregation result of the corresponding repair task based on the timestamp; map the re-run partition identifier of each intermediate aggregation result of the repair task and the version information and store them in the log directory.

[0177] In some embodiments, the device also includes: an intermediate aggregation result output module, which is used to obtain the intermediate aggregation result of the repair task corresponding to the rerun partition identifier when other businesses trigger a request to use the business intermediate aggregation result of the log information of any calculation batch in the distributed storage system. If the partition identifier and rerun partition identifier corresponding to the log information of the same calculation batch are stored in the log directory, and the timestamp in the version information corresponding to the rerun partition identifier is closer to the current time than the timestamp in the version information corresponding to the partition identifier, and the intermediate aggregation result of the repair task corresponding to the rerun partition identifier is output to the other businesses for use.

[0178] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0179] The embodiment of the present application provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the method described in the embodiment of the present application.

[0180] The present application embodiment provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the present application embodiment, for example, Figure 5 The method shown.

[0181] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0182] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0183] As an example, executable instructions may, but need not correspond to, a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0184] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A business data processing method, characterized in that: The method comprises: During the current execution of the target business or when the execution has been completed, a business re-run request is obtained; the business re-run request is used to request the re-execution of the business process of the target business within the target time period before the current moment; the target business generates business data for use by other businesses during the current execution process; In response to the service re-run request, creating a repair task for the target service within the target time period; the repair task is used to re-execute the service process of the target service within the target time period, and the repair task is independent of the current execution process; Execute the repair task and obtain a repair log; Perform log parsing on the repair log to obtain business rerun statistics of the target business within the target time period; The service data is updated by using the service rerun statistical data to obtain service update data for representing the service execution result of the target service.

2. The method according to claim 1, characterized in that The obtaining of a service re-running request during the current execution of the target service or when the execution has been completed includes: In the process of executing the target service, obtaining a service log of the target service execution reported by the service execution terminal; If it is detected that the service log reports an error, determine the time period when the service log reports the error, and determine the time period when the service log reports the error as the target time period; The service rerun request is generated based on the target time period.

3. The method according to claim 1, characterized in that: The obtaining of a service re-running request during the current execution of the target service or when the execution has been completed includes: In the process of executing the target service, obtaining a service log of the target service execution reported by the service execution terminal; When the other business performs other business processing based on the business data corresponding to the business log, if an abnormality occurs in the indicator calculation during the other business processing, determine the time period corresponding to the business data with the abnormal indicator calculation in the business log, and determine the time period as the target time period; The service rerun request is generated based on the target time period.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: The repair log is reported with a preset time period as a reporting period; and the reported repair log is stored in a preset distributed storage system.

5. The method according to claim 4, characterized in that The performing log parsing on the repair log to obtain business rerun statistics of the target business within the target time period includes: Reading a plurality of repair logs having the preset duration from the distributed storage system; Parse each repair log to obtain repair task statistics of each repair log about the target statistical indicator under the preset time dimension; Aggregate the statistical data of multiple repair tasks corresponding to multiple repair logs in different time dimensions to obtain intermediate aggregation results of the repair tasks in different time dimensions; Based on the intermediate aggregation results of the repair tasks in the different time dimensions, the business rerun statistics of the target business within the target time period are determined.

6. The method according to claim 5, characterized in that Determining the business rerun statistics of the target business within the target time period based on the intermediate aggregation results under the different time dimensions includes: Aggregation processing is performed on the intermediate aggregation results under the different time dimensions to obtain business rerun statistics of the target business within the target time period.

7. The method according to claim 5, characterized in that The repair log with the preset time length includes multiple log information; the multiple log information is log information of a calculation batch; each of the calculation batches has a mapping relationship with a directory information in the log directory in the data storage unit; the log directory includes directory information of each section of the repair log; After parsing any repair log to obtain repair task statistics of the corresponding repair log with respect to the target statistical indicator under the preset duration dimension, the method further includes: Generate a calculation batch completion mark of the repair log; the calculation batch completion mark is used to indicate that multiple log information in the corresponding repair log has been parsed and statistical data corresponding to the repair log has been obtained; Mapping the calculation batch completion identifier with the target information of the corresponding repair log to obtain a mapping relationship between the calculation batch completion identifier and the corresponding repair log; The calculation batch completion identifier and the mapping relationship are stored in the data storage unit.

8. The method according to claim 5, characterized in that The method further comprises: Parsing the business log of the target business within the target time period to obtain at least one business statistical data of the business log about the target statistical indicator in the preset time dimension; Aggregating the at least one business statistical data in different time dimensions to obtain intermediate business aggregation results in different time dimensions; In the same time dimension, the intermediate aggregation result of the repair task is used to update the corresponding intermediate aggregation result of the business to obtain an updated intermediate aggregation result of the business; The updated business intermediate aggregation result is stored in the distributed storage system.

9. The method according to claim 8, characterized in that The method further comprises: Under the adjacent first time dimension and second time dimension, obtain the last business intermediate aggregation result in the updated business intermediate aggregation results under the first time dimension, and the next business intermediate aggregation result to be generated that is connected to the last business intermediate aggregation result; Based on the last business intermediate aggregation result and the business intermediate aggregation result to be generated, determine the next business intermediate aggregation result to be generated in the second time dimension and connected to the last business intermediate aggregation result in the updated business intermediate aggregation result; The duration corresponding to the first time dimension is shorter than the duration corresponding to the second time dimension.

10. The method according to claim 7, characterized in that Before storing the calculation batch completion identifier and the mapping relationship in the data storage unit, the method further includes: Parsing the business log when the target business is executed to obtain a plurality of business statistical data about the target statistical indicator in the business log under the preset time dimension; Aggregate the multiple business statistical data in different time dimensions to obtain intermediate business aggregation results in different time dimensions; Obtaining the partition information corresponding to each of the intermediate aggregation results of the business in the business log; Generate a partition identifier of the corresponding business intermediate aggregation result based on the partition information; Each of the partition identifiers is stored in the log directory as the target information.

11. The method according to claim 10, characterized in that The method further comprises: When the intermediate aggregation result of the repair task corresponding to the log information of each calculation batch is obtained, the re-run partition information corresponding to the intermediate aggregation result of the repair task in the repair log is obtained; the re-run partition information is used to determine the log start position and log end position of the log information of the corresponding calculation batch in the entire repair log; Generate a rerun partition identifier of the intermediate aggregation result of the corresponding repair task based on the rerun partition information; Get the timestamp corresponding to the log information of each calculation batch; Generate version information of the intermediate aggregation result of the corresponding repair task based on the timestamp; The rerun partition identifier and the version information of each intermediate aggregation result of the repair task are mapped and stored in the log directory.

12. A business data processing device, characterized in that: The device comprises: An acquisition module is used to acquire a business re-run request during the current execution of the target business or when the execution has been completed; the business re-run request is used to request the re-execution of the business process of the target business within the target time period before the current moment; the target business generates business data for use by other businesses during the current execution process; A task creation module, used to create a repair task for the target business within the target time period in response to the business re-run request; the repair task is used to realize the re-execution of the business process of the target business within the target time period, and the repair task is independent of the current execution process; A task execution module, used to execute the repair task and obtain a repair log; A log parsing module, used to parse the repair log to obtain business rerun statistics of the target business within the target time period; The data updating module is used to update the business data using the business re-run statistical data to obtain business update data for characterizing the business execution result of the target business.

13. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the business data processing method described in any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the business data processing method described in any one of claims 1 to 11.

15. A computer program product or a computer program, the computer program product or the computer program comprising executable instructions, the executable instructions being stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the business data processing method described in any one of claims 1 to 11 is implemented.

Images