Index restoration method, system and device based on real-time marketing scene

Abstract

The invention discloses an index restoration method, system and device based on a real-time marketing scene, and the method comprises the steps: obtaining pipeline data based on a business request, pushing the pipeline data to a message queue for real-time index calculation, obtaining a first index intermediate state result, and distributing the data to a file system according to a time sequence; according to the method, a new online marketing index can be inquired, all the pipeline data needing to be calculated are found in a file system according to the time span of the marketing index, the calculation logic and the required pipeline data type, an index calculation task is initiated, and a second index intermediate state result is obtained; and finally, summarizing the second index intermediate state result and the first index intermediate state result to obtain a complete time sequence index intermediate state result set. According to the method, the index result can be obtained without collecting and re-treating the indexes of the historical flow and the same calculation logic, only the time spans for calculating the flow are different, and the index result can be obtained without re-calculating all the flows, so that the timeliness is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a financial real-time marketing scenario, and to an indicator repair method, system and device based on the real-time marketing scenario. Background Art

[0002] With the rapid development of Internet technology, the demand for real-time marketing is increasing, and the requirements for marketing indicator calculation performance are getting higher and higher. Because with the development of business, the demand for real-time marketing indicators is increasing, and the requirements for repairing new real-time marketing indicators are getting higher and higher. At the same time, the calculation results of indicators will directly affect the business effect of marketing, so the requirements for quickly repairing new indicators based on historical data are getting higher and higher.

[0003] When developers are repairing and developing marketing indicators, they need to reorganize the historical data of the newly added indicators, mine user behavior data through various operation points, and transmit this part of structured data to the server through the network. The data cleaning program converts the structured data collection into server-side flow, passes it to the calculation engine, calculates the indicators of the marketing scenario in real time, and stores the indicator results to provide indicator services to the outside world in real time.

[0004] In the prior art, when developing indicators for marketing scenarios, it is usually necessary to perform full-scale flow calculations on the indicators to ensure the correctness, timeliness, and expected marketing results of the indicator results. Since the calculation flow data of marketing scenario indicators has a large time span and a large number of flow levels, when developers perform indicator calculations, in addition to developing the calculation logic of the indicators, they also need to re-collect and re-calculate the historical flow data. The entire data link is highly complex, the development efficiency is low, and the development cycle is long, which makes it impossible to ensure the timeliness of the indicators during full calculations, and it is difficult to accurately achieve marketing goals.

[0005] In the existing marketing indicator repair and development process, in addition to developing the calculation logic of the indicator, it is also necessary to re-manage and collect complete historical flow data, and trigger the calculation of the newly developed indicator by re-pushing the flow data. The process is very complicated.

[0006] The existing marketing indicator development requires recalculating all the flow data for cumulative indicators with the same calculation logic but different time spans before the indicators can get results. This has low timeliness and affects marketing effectiveness. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies in the prior art and propose an indicator repair method, system and device based on real-time marketing scenarios. There is no need to collect and re-manage historical transactions. Development only needs to focus on the correctness of the calculation logic. For indicators with the same calculation logic, only the time span of the calculation transactions is different. There is no need to recalculate all transactions to obtain indicator results, thereby improving timeliness.

[0008] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides an indicator repair method based on a real-time marketing scenario, the method comprising the following steps:

[0009] (1) Obtain flow data based on business requests, push it to the message queue, and distribute the data to the file system in chronological order;

[0010] (2) The pipeline data in the message queue is used to perform real-time indicator calculations to obtain the intermediate state results of the first indicator;

[0011] (3) Query the newly launched marketing indicators, and find all the pipeline data to be calculated in the file system according to the time span, calculation logic and required pipeline data type of the marketing indicators, initiate the indicator calculation task, and obtain the intermediate state result of the second indicator;

[0012] (4) The intermediate state result of the second indicator is aggregated with the intermediate state result of the first indicator to obtain a complete set of intermediate state results of the time series indicators.

[0013] Furthermore, in step (1), the obtained business request is converted into flow data in a predetermined format, and the flow data is forwarded to the message queue, so that the message queue accumulates the flow data.

[0014] Furthermore, in step (3), the task initiating the indicator calculation obtains a batch of intermediate indicator results and pushes them to the message queue.

[0015] Furthermore, in step (3), for cumulative marketing indicators with the same calculation logic but different time spans, the results of the intermediate result set are filtered out according to the time span, and the final indicator results are summarized.

[0016] Furthermore, in step (3), the message queue includes a kafka cluster, RocketMQ or RabbitMQ.

[0017] Furthermore, in step (3), the file system is a Parquet or MPP database in a big data platform, which supports caching of massive data.

[0018] In a second aspect, the present invention further provides an indicator repair system based on a real-time marketing scenario, the system comprising:

[0019] Data distribution service, used to forward the flow data obtained by business requests to the message queue component and big data platform;

[0020] Big data platform, used to store pipeline data and perform parallel computing;

[0021] The indicator calculation repair component is used to filter out the flow data of the big data platform according to the flow type, calculation logic and time span of the newly launched marketing indicator, initiate indicator calculation, obtain the intermediate state result of the second indicator, and push it to the message queue component;

[0022] The message queue component is used to accumulate flow data and transmit it to the real-time indicator calculation component;

[0023] The real-time indicator calculation component is used to perform real-time indicator calculation on the pipeline data of the business request to obtain the first indicator intermediate state result, and to aggregate the second indicator intermediate state result with the first indicator intermediate state result.

[0024] In a third aspect, the present invention also provides an indicator repair device based on a real-time marketing scenario, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the indicator repair method based on a real-time marketing scenario.

[0025] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the indicator repair method based on a real-time marketing scenario is implemented.

[0026] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the indicator repair method based on a real-time marketing scenario.

[0027] Beneficial effects of the present invention:

[0028] 1. Introducing data distribution services not only ensures that the production chain of real-time marketing indicators is not affected, but also automatically completes the management and forwarding of historical flow data. Developers only need to focus on the correctness of the indicator logic.

[0029] 2. The present invention can ensure that the service does not stop during the repair process of all real-time marketing indicators, and there is no need to develop the logic of indicator repair separately. The real-time indicator calculation component can normally provide indicator query services to the outside world.

[0030] 3. In the entire real-time marketing indicator repair method, all technical components support horizontal expansion, which can ensure the final consistency of the data and the correctness of the calculation results of the final indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a schematic diagram of an indicator repair system based on a real-time marketing scenario provided by the present invention;

[0033] Figure 2 It is a structural diagram of an indicator repair device based on a real-time marketing scenario provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] The present invention provides an indicator repair method based on a real-time marketing scenario, comprising:

[0036] 1. Obtain the flow data received by the online software system;

[0037] 2. Convert the first business request obtained into stream data in a predetermined format, forward the stream data to the Kafka message queue, accumulate the stream data in the message queue, and forward a copy to the file system at the same time. This file system is generally a parquet file of the big data platform, and save it in a stream file.

[0038] 3. The indicator repair method based on the real-time marketing scenario of the present invention involves a data distribution service. The data distribution service uses an ultra-large memory bounded queue. The queue size can be configured according to the memory of the stress testing machine. Taking the 8G heap size as an example, a bounded queue size of at least 1 million can be provided to buffer real-time streaming data. The data distribution service pushes the streaming data to the kafka cluster and distributes the data in chronological order to the file system parquet for storage.

[0039] 4. In the indicator repair method based on the real-time marketing scenario of the present invention, the newly launched marketing indicators are queried, and according to the time span of the marketing indicators, the calculation logic, and the required flow data type, all the flow data that need to be calculated are found in the file system parquet, and the indicator calculation task is initiated. Generally, the computing resources of yarn are used for parallel calculation to obtain a batch of intermediate indicator results, which are pushed to the real-time computing component through kafka.

[0040] 5. The real-time calculation component is used to subscribe to Kafka messages for cumulative calculation of marketing indicators. The real-time calculation component stores intermediate indicator results internally, and aggregates the intermediate results of its own real-time calculation and the intermediate calculation results of the indicator repair calculation component to obtain a complete time series indicator intermediate result set. The existing marketing indicator development method, for cumulative indicators with the same calculation logic but different time spans, requires recalculation of all flow data before the indicators can obtain results. In the method of the present invention, for cumulative marketing indicators with the same calculation logic but different time spans, the results of the intermediate result set can be filtered out according to the time span to aggregate the final indicator results.

[0041] In addition to developing the calculation logic of the indicators, the existing marketing indicator development also requires re-management and collection of complete historical flow data, and the calculation of the newly developed indicators is completed by re-pushing the flow data. The process is very complicated. In the method of repairing marketing indicators of the present invention, the responsibility of the data distribution service is to automatically cache the flow, and when the new marketing scenario indicators are online, all related flow data are automatically filtered out from the cache according to the flow data type of the indicator. There is no need to collect and re-manage historical flow data and initiate the task of batch calculation of indicators. Developers only need to focus on the correctness of the calculation logic, and do not need to pay attention to the data management process behind it and ensure the timing of the flow.

[0042] At least one of the above technical solutions adopted in the embodiment of the present application can achieve the following beneficial effects: by obtaining the first business request received by the online software system, and using the time-series flow data generated according to the first business request, forwarding the request to the real-time computing component and the big data platform respectively through the data distribution service, the newly launched indicators based on the real-time marketing scenario do not need to re-govern and collect historical flow data, and the indicator calculation repair component can filter out the required flow data from the big data platform through the flow type characteristics and the indicator time span, start batch indicator calculation, and pass the calculated intermediate results to the real-time indicator calculation component through kafka. The real-time calculation component will summarize this part of the results to obtain the complete indicator results after the final indicator repair. Compared with the marketing indicator calculation scripts and flow collection tasks written by developers based on experience in the prior art, the timeliness of indicators in the marketing field is improved, because there is no need to pay attention to the recalculation of historical flow, and the development complexity is greatly reduced.

[0043] like Figure 1 As shown, the present invention provides an indicator repair system based on a real-time marketing scenario, which specifically includes:

[0044] The data distribution service is responsible for forwarding the business request pipeline to the Kafka component and importing it into the Parquet file of the big data platform.

[0045] The indicator calculation repair component will load the parquet data of the big data platform, filter out the flow data of the big data platform according to the flow type, calculation logic and time span of the newly launched marketing indicators, initiate indicator calculation, obtain the flow-level indicator results, and push these intermediate results to Kafka.

[0046] The big data platform component uses the CDH platform and its internal yarn resources to complete spark calculations and perform parallel calculations to obtain intermediate results of indicators.

[0047] The Kafka component accumulates stream data and completes data transmission through Kafka's subscription notification mechanism.

[0048] The real-time indicator calculation component will subscribe to the business request flow data generated in production, calculate the intermediate results in real time, and subscribe to the historical intermediate results generated by the historical flow. It will cache all the intermediate results in memory and summarize the indicator results in real time according to the marketing time span requested by the user when providing services externally, and expose the API for users to obtain the indicator results.

[0049] Corresponding to the aforementioned embodiment of an indicator repair method based on a real-time marketing scenario, the present invention also provides an embodiment of an indicator repair device based on a real-time marketing scenario.

[0050] See also Figure 2 An embodiment of the present invention provides an indicator repair device based on a real-time marketing scenario, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an indicator repair method based on a real-time marketing scenario in the above embodiment.

[0051] An embodiment of an indicator repair device based on a real-time marketing scenario provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 2 As shown, it is a hardware structure diagram of any device with data processing capability where an indicator repair device based on a real-time marketing scenario provided by the present invention is located, except Figure 2In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0052] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0053] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0054] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, an indicator repair method based on a real-time marketing scenario in the above embodiment is implemented.

[0055] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0056] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the indicator repair method based on a real-time marketing scenario.

[0057] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for repairing indicators based on real-time marketing scenarios, characterized in that: The method comprises the following steps: (1) Obtain flow data based on business requests, push it to the message queue, and distribute the data to the file system in chronological order; (2) The pipeline data in the message queue is used to perform real-time indicator calculations to obtain the intermediate state results of the first indicator; (3) Query the newly launched marketing indicators, and find all the pipeline data to be calculated in the file system according to the time span, calculation logic and required pipeline data type of the marketing indicators, initiate the indicator calculation task, and obtain the intermediate state result of the second indicator; (4) The intermediate state result of the second indicator is aggregated with the intermediate state result of the first indicator to obtain a complete set of intermediate state results of the time series indicators.

2. The indicator repair method based on real-time marketing scenario according to claim 1 is characterized in that: In step (1), the obtained business request is converted into flow data in a predetermined format, and the flow data is forwarded to the message queue, so that the message queue accumulates the flow data.

3. The indicator repair method based on real-time marketing scenario according to claim 1 is characterized in that: In step (3), the task that initiates the indicator calculation obtains a batch of intermediate indicator results and pushes them to the message queue.

4. The indicator repair method based on real-time marketing scenario according to claim 1 is characterized in that: In step (3), for cumulative marketing indicators with the same calculation logic but different time spans, the results of the intermediate result set are filtered out according to the time span, and the final indicator results are summarized.

5. The indicator repair method based on real-time marketing scenario according to claim 1 is characterized in that: In step (3), the message queue includes a Kafka cluster, RocketMQ or RabbitMQ.

6. The indicator repair method based on real-time marketing scenario according to claim 1 is characterized in that: In step (3), the file system is a Parquet or MPP database in the big data platform, which supports caching of massive data.

7. An indicator repair system based on a real-time marketing scenario according to any one of claims 1 to 6, characterized in that: The system includes: Data distribution service, used to forward the flow data obtained by business requests to the message queue component and big data platform; Big data platform, used to store pipeline data and perform parallel computing; The indicator calculation repair component is used to filter out the flow data of the big data platform according to the flow type, calculation logic and time span of the newly launched marketing indicator, initiate indicator calculation, obtain the intermediate state result of the second indicator, and push it to the message queue component; The message queue component is used to accumulate flow data and transmit it to the real-time indicator calculation component; The real-time indicator calculation component is used to perform real-time indicator calculation on the pipeline data of the business request to obtain the first indicator intermediate state result, and to aggregate the second indicator intermediate state result with the first indicator intermediate state result.

8. An indicator repair device based on real-time marketing scenarios, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, an indicator repair method based on a real-time marketing scenario as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an indicator repair method based on a real-time marketing scenario as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, an indicator repair method based on a real-time marketing scenario as described in any one of claims 1 to 6 is implemented.

Images