Report pushing optimization method and device, equipment and storage medium

Abstract

The invention relates to the field of data processing, and discloses a report pushing optimization method and device, equipment and a storage medium, and the method is used for optimizing report pushing operation. The method comprises the following steps: regularly collecting behavior data of a user accessing a report, and preprocessing the collected data to obtain preprocessed data; calculating the click rate of each report according to the preprocessed data, sorting each report according to the click rate to obtain the click rate ranking of each report, and generating a report click rate data table; inputting the report dependency relationship table, the report updating frequency table and the report click rate data table into a job scheduling optimization model to obtain a report pushing job scheduling plan; screening out reports with click rates lower than a preset click rate threshold according to the report click rate data table, and marking the reports as low-click-rate reports; and updating the ETL job table according to the report pushing job scheduling plan, and replacing the scheduling triggering mode of the low-click-rate report with trigger triggering.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a report push optimization method, device, equipment and storage medium. Background Art

[0002] The express data warehouse system carries a huge amount of data processing tasks, such as the efficient operation of more than 700 data push jobs, which directly support the data generation and display of hundreds of key reports on the reporting platform. This system plays an indispensable role in supporting corporate decision-making and business analysis. However, with the increase in business complexity and the growing demand for data, the system architecture and data management process face several challenges.

[0003] The primary problem is the mismatch between the frequency of data updates and the actual usage of reports. The system currently adopts a strategy of determining the update cycle of report data based on the frequency of data push operations. Although this mechanism ensures the flexibility of data updates, it also causes uneven resource allocation. Specifically, some reports with high frequency of access have received timely data update support, meeting the urgent needs of business departments; on the other hand, there are a considerable number of reports whose monthly or even annual click frequency is extremely low, not exceeding 100 times. The data push operations of these low-frequency reports not only consume precious server resources, but also indirectly affect the normal data update process of other reports due to occupying the processing queue, causing data delays and damaging the timeliness and accuracy of the data on the reporting platform.

[0004] What is more serious is that with the continuous emergence of new businesses and the continuous expansion of data reporting needs, the existing system is facing increasing pressure. The cumulative effect of low-frequency data push operations has aggravated the shortage of server resources, which may in turn trigger a chain reaction and affect the stability and response speed of the entire data ecosystem. This not only weakens the value of the reporting platform as the core of data services, but may also have an adverse impact on the decision-making efficiency and data insight of business departments, limiting the ability of enterprises to use data for refined management and market response.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The present invention provides a report push optimization method, device, equipment and storage medium for optimizing report push operations.

[0007] A first aspect of the present invention provides a report push optimization method, which includes: regularly collecting behavioral data of users accessing reports, and preprocessing the collected data to obtain preprocessed data; calculating the click rate of each report based on the preprocessed data, and sorting each report according to the click rate to obtain the click rate ranking of each report, and generating a report click rate data table; obtaining a preset report dependency table and a report update frequency table, and inputting the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan; filtering out reports with click rates lower than a preset click rate threshold based on the report click rate data table, and marking them as low click rate reports; updating the ETL job table according to the report push job scheduling plan, and replacing the scheduling triggering method of the low click rate report with trigger triggering.

[0008] Optionally, in a first implementation method of the first aspect of the present invention, the periodic collection of behavioral data on user access reports and preprocessing the collected data to obtain preprocessed data include: periodic collection of behavioral data on user access reports; cleaning the collected data to obtain cleaned data; converting the cleaned data to obtain preprocessed data, and the preprocessed data includes report ID, report access data and report click data.

[0009] Optionally, in a second implementation method of the first aspect of the present invention, the click rate of each report is calculated based on the preprocessed data, and each report is sorted according to the click rate to obtain the click rate ranking of each report, and a report click rate data table is generated, including: calculating the number of clicks and the total number of visits of each report based on the preprocessed data; calculating the ratio of the number of clicks of each report to the total number of visits to obtain the click rate of each report; sorting each report in descending order according to the click rate to obtain the click rate ranking of each report, and generating a report click rate data table.

[0010] Optionally, in a third implementation method of the first aspect of the present invention, the obtaining of a preset report dependency table and a report update frequency table, and inputting the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan, including: constructing a job scheduling optimization model based on a genetic algorithm; obtaining a preset report dependency table and a report update frequency table, and integrating the report dependency table, the report update frequency table and the report click rate data table into a comprehensive data set; inputting the comprehensive data set into the job scheduling optimization model to obtain a report push job scheduling plan, and the report push job scheduling plan includes the push order and push time of each report.

[0011] Optionally, in a fourth implementation method of the first aspect of the present invention, the job scheduling optimization model is constructed based on the genetic algorithm, including: constructing a basic framework of the scheduling optimization initial model, the basic framework including a data input module, a scheduling algorithm module, and a result output module, the data input module is used to receive a comprehensive data set and transmit it to the scheduling algorithm module, the scheduling algorithm module is used to generate a report push job scheduling plan according to the comprehensive data set, and transmit it to the result output module, the result output module is used to output the report push job scheduling plan; obtaining preset policy variables, objective functions, constraints and genetic algorithms, and integrating the preset policy variables, objective functions, constraints and genetic algorithms into the scheduling algorithm module to obtain a scheduling optimization initial model; collecting historical report scheduling data, using the historical report scheduling data to train the scheduling optimization initial model to obtain a job scheduling optimization model.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, the report click rate data table is used to filter out reports with a click rate lower than a preset click rate threshold, and mark them as low click rate reports, including: traversing the report click rate data table starting from the report with the last click rate ranking, and comparing the report click rate with the preset click rate threshold; if the report click rate is lower than the preset click rate threshold, placing the report in a low click rate report set; if the report click rate is higher than the preset click rate threshold, stopping traversing the report click rate data table; and marking the reports in the low click rate report set as low click rate reports.

[0013] Optionally, in a sixth implementation method of the first aspect of the present invention, the ETL job table is updated according to the report push job scheduling plan, and the scheduling trigger mode of the low click rate report is replaced with a trigger trigger, including: adjusting the execution order, time and priority of the jobs in the ETL job table according to the report push job scheduling plan to obtain an updated ETL job table; creating an innovative trigger for each of the low click rate reports, and configuring the trigger parameters according to the scheduling plan corresponding to the low click rate report; traversing the updated ETL job table, querying to obtain the low click rate report, and replacing the scheduling trigger mode of the low click rate report with a trigger trigger.

[0014] The second aspect of the present invention provides a report push optimization device, including: a preprocessing module, which is used to regularly collect behavioral data of users accessing reports, and preprocess the collected data to obtain preprocessed data; a sorting module, which is used to calculate the click rate of each report based on the preprocessed data, and sort each report according to the click rate to obtain the click rate ranking of each report, and generate a report click rate data table; a scheduling module, which is used to obtain a preset report dependency table and a report update frequency table, and input the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan; a screening module, which is used to screen out reports with a click rate lower than a preset click rate threshold according to the report click rate data table, and mark them as low click rate reports; an updating module, which is used to update the ETL job table according to the report push job scheduling plan, and replace the scheduling triggering method of the low click rate report with a trigger trigger.

[0015] Optionally, in a first implementation method of the second aspect of the present invention, the preprocessing module includes: a collection unit, used to periodically collect behavioral data of user access reports; a cleaning unit, used to clean the collected data to obtain cleaned data; a conversion unit, used to convert the cleaned data to obtain preprocessed data, and the preprocessed data includes a report ID, report access data and report click data.

[0016] Optionally, in a second implementation of the second aspect of the present invention, the sorting module includes: a first calculation unit, used to calculate the number of clicks and the total number of visits of each report based on the preprocessed data; a second calculation unit, used to calculate the ratio of the number of clicks of each report to the total number of visits, and obtain the click rate of each report; a sorting unit, used to sort each report in descending order according to the click rate, obtain the click rate ranking of each report, and generate a report click rate data table.

[0017] Optionally, in a third implementation method of the second aspect of the present invention, the scheduling module includes: a construction unit, used to construct a job scheduling optimization model based on a genetic algorithm; an integration unit, used to obtain a preset report dependency table and a report update frequency table, and integrate the report dependency table, the report update frequency table and the report click rate data table into a comprehensive data set; a scheduling unit, used to input the comprehensive data set into the job scheduling optimization model to obtain a report push job scheduling plan, and the report push job scheduling plan includes the push order and push time of each report.

[0018] Optionally, in a fourth implementation method of the second aspect of the present invention, the screening module includes: a comparison unit, used to traverse the report click rate data table starting from the report with the last click rate ranking, and compare the click rate of the report with a preset click rate threshold; a screening unit, used to place the report into a low click rate report set when the click rate of the report is lower than the preset click rate threshold, and to stop traversing the report click rate data table when the click rate of the report is higher than the preset click rate threshold; a marking unit, used to mark the reports in the low click rate report set as low click rate reports.

[0019] Optionally, in a fifth implementation of the second aspect of the present invention, the update module includes: an update unit, used to adjust the execution order, time and priority of jobs in the ETL job table according to the report push job scheduling plan to obtain an updated ETL job table; a configuration unit, used to innovate a trigger for each of the low click-through rate reports, and configure the trigger parameters according to the scheduling plan corresponding to the low click-through rate report; a replacement unit, used to traverse the updated ETL job table, query to obtain the low click-through rate report, and replace the scheduling trigger method of the low click-through rate report with a trigger trigger.

[0020] The third aspect of the present invention provides a report push optimization device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the computer-readable instructions in the memory so that the report push optimization device executes the various steps of the report push optimization method as described above.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes each step of the report push optimization method described above.

[0022] In the technical solution provided by the present invention, the behavioral data of users accessing reports are collected regularly, and the click rate of the reports is calculated based on these data. Then, the job scheduling optimization model is used to automatically generate an optimized report push job scheduling plan in combination with the report's dependency, update frequency and click rate, and the ETL job table is updated according to the report push job scheduling plan to ensure the timeliness of report data in high-frequency click time periods and reduce data statistical errors caused by data delays. For low-click rate reports, trigger triggering is used instead of traditional timed triggering, making report push more flexible and able to be triggered in real time according to data update conditions or business needs, thereby improving the timeliness and accuracy of reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A first flow chart of the report push optimization method provided by an embodiment of the present invention;

[0024] Figure 2 A second flow chart of the report push optimization method provided by an embodiment of the present invention;

[0025] Figure 3 A third flow chart of the report push optimization method provided by an embodiment of the present invention;

[0026] Figure 4 A fourth flow chart of the report push optimization method provided by an embodiment of the present invention;

[0027] Figure 5 A fifth flow chart of the report push optimization method provided by an embodiment of the present invention;

[0028] Figure 6 A sixth flow chart of the report push optimization method provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of the structure of a report push optimization device provided by an embodiment of the present invention;

[0030] Figure 8 A schematic diagram of the structure of a report push optimization device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The embodiment of the present invention provides a report push optimization method, device, equipment and storage medium, which is used to optimize the report push job. The method includes: regularly collecting user access report behavior data, and preprocessing the collected data to obtain preprocessed data; calculating the click rate of each report according to the preprocessed data, and sorting each report according to the click rate to obtain the click rate ranking of each report, and generating a report click rate data table; obtaining a preset report dependency table and a report update frequency table, and inputting the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan; filtering out reports with click rates lower than a preset click rate threshold according to the report click rate data table, and marking them as low click rate reports; updating the ETL job table according to the report push job scheduling plan, and replacing the scheduling triggering method of the low click rate report with trigger triggering.

[0032] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first embodiment of a report push optimization method in an embodiment of the present invention includes:

[0034] S101. Regularly collect behavioral data of user access reports, and pre-process the collected data to obtain pre-processed data.

[0035] It is understandable that the execution subject of the present invention may be a report push optimization device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0036] In this embodiment, the behavior data of user access reports is collected regularly from sources such as user access logs and databases. Then, the collected data is pre-processed by cleaning, deduplication, format conversion, etc., and key information such as report ID, report access time, and report click count are extracted to form pre-processed data.

[0037] S102: Calculate the click rate of each report according to the preprocessed data, sort each report according to the click rate, obtain the click rate ranking of each report, and generate a report click rate data table.

[0038] In this embodiment, the click rate (number of clicks / number of visits) of each report is calculated based on the number of clicks and visits of the report in the pre-processed data.

[0039] In this embodiment, the reports are sorted in descending order according to the click rate, a report click rate ranking is generated, and a report click rate data table is constructed. In other embodiments, the reports can also be sorted in ascending order according to the click rate.

[0040] S103: Obtain a preset report dependency table and a report update frequency table, and input the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan.

[0041] In this embodiment, a preset report dependency table and a report update frequency table are obtained, and these tables record the dependency relationships between reports and their respective update frequencies.

[0042] In this embodiment, the report click rate data table, report dependency table and report update frequency table are used as inputs and passed to the job scheduling optimization model. The job scheduling optimization model outputs an optimized report push job scheduling plan based on the input data, taking into account factors such as dependency constraints between reports, update frequency and click rate.

[0043] S104. Filter out reports whose click rates are lower than a preset click rate threshold according to the report click rate data table, and mark them as low click rate reports.

[0044] In this embodiment, for each report that is filtered out, if it is in Excel or a similar spreadsheet tool, directly add a column next to the report and fill in the "low click rate report" in the corresponding rows of these reports.

[0045] If it's in a database, add an extra column (if it doesn't already have one) to store this status, and update the records for these reports, setting the new column's value to "Low CTR Reports".

[0046] In this embodiment, the push frequency of low click-through rate reports can be reduced, for example, the low-frequency job of pushing every hour can be modified to pushing every day or every month.

[0047] S105: Update the ETL job table according to the report push job scheduling plan, and replace the scheduling triggering mode of the low click rate report with trigger triggering.

[0048] In this embodiment, according to the job scheduling plan, the job execution order, time and priority in the ETL job table are adjusted to obtain an updated ETL job table.

[0049] In this embodiment, the updated ETL job table is traversed to find out the job entries corresponding to the low click rate report, and the scheduling trigger mode of these jobs is changed from the original timed trigger to the trigger trigger. In specific implementation, it is necessary to create a corresponding trigger for each low click rate report, configure the trigger parameters (such as trigger conditions, operations performed after triggering, etc.), and then associate these triggers with the corresponding ETL job entries.

[0050] The present embodiment provides a report push optimization method, which periodically collects behavioral data of users accessing reports, calculates the click rate of reports based on these data, and then uses a job scheduling optimization model to automatically generate an optimized report push job scheduling plan in combination with report dependencies, update frequency and click rate, and updates the ETL job table according to the report push job scheduling plan to ensure the timeliness of report data in high-frequency click time periods and reduce data statistical errors caused by data delays. For low-click rate reports, trigger triggering is used instead of traditional timed triggering, making report push more flexible and able to be triggered in real time according to data update conditions or business needs, thereby improving the timeliness and accuracy of reports.

[0051] See also Figure 2 The second embodiment of the report push optimization method in the embodiment of the present invention includes:

[0052] S201. Regularly collect behavioral data of user access reports.

[0053] In this embodiment, all relevant behavior data of users accessing reports are collected regularly, such as access time, name of accessed report, user ID, browsing time, number of clicks, number of downloads, etc.

[0054] In this embodiment, if all relevant behavioral data of users accessing reports are scattered in multiple data sources (such as Web server logs, application server logs, database access records, logs of the reporting platform itself, etc.), then summary data is generated by combining the information of multiple data sources. When collecting the behavioral data of users accessing all reports in the reporting platform, the summary data of the reporting platform can be called using an API.

[0055] In this embodiment, the behavior data of users accessing all reports in the report platform may be collected weekly or monthly to optimize the reports.

[0056] S202: Clean the collected data to obtain cleaned data.

[0057] In this embodiment, the goal of data cleaning the collected data is to identify and correct errors, outliers, missing values, and other problems in the data to ensure the accuracy and consistency of the data.

[0058] Data cleaning of the collected data includes identifying missing values, handling outliers, and removing duplicate data.

[0059] For example, check which fields in the collected data have missing values, and decide whether to fill them (such as filling with mean, median, mode), delete them, or ignore them based on business logic.

[0060] For example, analyzing outliers in the data (such as extremely large or small values) and deciding whether to correct, delete, or retain and mark them.

[0061] For example, delete or merge duplicate records.

[0062] S203: Perform data conversion on the cleaned data to obtain pre-processed data.

[0063] In this embodiment, the purpose of performing data conversion on the cleaned data is to convert the cleaned data into a form suitable for analysis, such as generating new features, converting data types, standardizing or normalizing data, etc.

[0064] Feature extraction refers to extracting new features from raw data, such as user access frequency, report popularity, etc.

[0065] Data encoding refers to encoding categorical data into numerical values ​​(such as using one-hot encoding or label encoding).

[0066] Data standardization / normalization refers to adjusting the scale of numerical data so that they have the same scale to facilitate processing by certain algorithms (such as gradient descent algorithm).

[0067] In this embodiment, by cleaning and converting the behavioral data of users accessing all reports in the report platform, errors, inconsistencies and noise in the data can be removed or corrected, ensuring that the data used for subsequent analysis is accurate and reliable, and making the data easier to understand and use.

[0068] See also Figure 3 A third embodiment of a report push optimization method in an embodiment of the present invention includes:

[0069] S301. Calculate the number of clicks and total number of visits of each report based on the preprocessed data.

[0070] In this embodiment, when calculating the number of clicks for each report based on preprocessed data, key fields related to report clicks, such as report identifiers and timestamps, are identified in the preprocessed data, and then data containing click events are extracted from the preprocessed data, the click events are grouped according to the report identifier field, and the number of clicks for each report is calculated.

[0071] In this embodiment, the report identifier is used to uniquely identify the field of each report. The timestamp is a field that records the time when the click event occurs.

[0072] In this embodiment, the total number of visits to the report refers to the total number of times the report page is visited, that is, the total number of times it is browsed.

[0073] S302: Calculate the ratio of the number of clicks on each report to the total number of visits to obtain the click rate of each report.

[0074] In this embodiment, the click rate of the report = the number of clicks on the report / the total number of visits to the report*100%.

[0075] For each report, the corresponding click-through rate is calculated based on the number of clicks on the report and the total number of visits to the report.

[0076] S303 , sorting each report in descending order according to the click rate, obtaining the click rate ranking of each report, and generating a report click rate data table.

[0077] In this embodiment, the reports are sorted in descending order according to the click rate, which can be achieved through database query, sorting function in programming language or data analysis tool.

[0078] In this embodiment, a report click rate data table is generated according to the sorting results. The report click rate data table contains at least the following information:

[0079] Report ID: Uniquely identifies each report.

[0080] Click-through rate: The click-through rate for each report based on the previous calculation.

[0081] Rank: The position of each report in descending order of click-through rate.

[0082] In this embodiment, the click rate of each report is calculated based on the preprocessed data, and each report is sorted in descending order according to the click rate to obtain the click rate ranking of each report, and a report click rate data table is generated. The calculation and sorting of the click rate are realized through database query, sorting function in programming language or data analysis tool, which can greatly improve the processing efficiency.

[0083] See also Figure 4 A fourth embodiment of a report push optimization method in an embodiment of the present invention includes:

[0084] S401. Construct a job scheduling optimization model based on genetic algorithm.

[0085] In this embodiment, a job scheduling optimization model is constructed based on a genetic algorithm, specifically including: constructing a basic framework of an initial model of scheduling optimization, the basic framework including a data input module, a scheduling algorithm module, and a result output module, the data input module is used to receive a comprehensive data set and transmit it to the scheduling algorithm module, the scheduling algorithm module is used to generate a report push job scheduling plan according to the comprehensive data set, and transmit it to the result output module, the result output module is used to output the report push job scheduling plan; obtaining preset policy variables, objective functions, constraints and genetic algorithms, and integrating the preset policy variables, objective functions, constraints and genetic algorithms into the scheduling algorithm module to obtain an initial model of scheduling optimization; collecting historical report scheduling data, using the historical report scheduling data to train the initial model of scheduling optimization to obtain a job scheduling optimization model.

[0086] In this embodiment, the logic of the genetic algorithm is implemented in the scheduling algorithm module, the policy variables are encoded as chromosomes, and the objective function is used as the fitness function to evaluate the quality of the chromosomes.

[0087] In this embodiment, scheduling-related data is collected from historical reports or databases as historical report scheduling data, and the historical report scheduling data includes historical reports and corresponding report click rate rankings, report update frequencies, and report dependencies.

[0088] In this embodiment, the historical report scheduling data is divided into a data set test set, and during the training process, the policy variables are iteratively optimized through a genetic algorithm to find a scheduling solution that optimizes the objective function value.

[0089] In this embodiment, a genetic algorithm is used as an optimization algorithm, and its key operations need to be determined, such as initializing the population, fitness evaluation, selection, crossover, mutation, etc.

[0090] In this embodiment, the performance of the trained model is evaluated using the test set data, and indicators such as the model's solution quality, convergence speed, and stability are analyzed to ensure that the model meets the requirements of the scheduling problem.

[0091] In this embodiment, the model can also be iteratively optimized based on the performance evaluation results, including adjusting the parameters of the genetic algorithm, improving the design of the objective function, optimizing the processing of constraints, etc., and then repeating the training, evaluation and optimization process to obtain a job scheduling optimization model.

[0092] S402: Obtain a preset report dependency table and a report update frequency table, and integrate the report dependency table, the report update frequency table and the report click rate data table into a comprehensive data set.

[0093] In this embodiment, the report dependency table includes report IDs, report names, and dependencies between reports.

[0094] The report update frequency table includes the report ID and the update frequency (such as daily, weekly, monthly, etc.).

[0095] The report click rate data table includes report ID, date, click rate and click rate ranking.

[0096] In this embodiment, the comprehensive data set includes report ID, dependent report ID, update frequency, date, click rate and click rate ranking.

[0097] In this embodiment, the JOIN operation of SQL can be used to integrate the report dependency table, the report update frequency table, and the report click rate data table, or Python Pandas can be used for data processing, that is, first read each table as a DataFrame, and then use the merge or join method to integrate the report dependency table, the report update frequency table, and the report click rate data table.

[0098] S403: Input the comprehensive data set into the job scheduling optimization model to obtain a report push job scheduling plan, where the report push job scheduling plan includes a push order and a push time for each report.

[0099] In this embodiment, the optimization goal of the job scheduling optimization model is to ensure a high frequency of data push jobs for reports with high click rates.

[0100] The decision variables are the push time of each report and the push priority of the report (indirectly affecting the push order).

[0101] The objective function is to maximize the total weighted click-through rate benefit, that is, to ensure that reports with high weighted click-through rates are pushed more frequently within a given time window. This can be achieved by assigning a benefit value proportional to the weighted click-through rate of the report to each push and summing them up.

[0102] Constraints include push time window constraints, dependency constraints, update frequency constraints, and resource constraints. Push time window constraints refer to ensuring that reports are pushed within a predetermined time window. Dependency constraints refer to that if there are dependencies between reports, the dependencies must be pushed first. Update frequency constraints refer to that the push frequency of each report should not exceed its specified update frequency requirement. Resource constraints refer to factors such as network bandwidth and server load, which may affect the number of push jobs that can be performed simultaneously.

[0103] In this embodiment, in the scheduling algorithm module, the genetic algorithm is used to find the optimal report push job scheduling plan. Therefore, it is necessary to initialize the parameters of the genetic algorithm, such as population size, number of iterations, crossover probability, mutation probability, etc.

[0104] When the population is initialized, a certain number of individuals (i.e., potential scheduling solutions) are randomly generated, and each individual is encoded (such as binary encoding) to represent a different report push order and push time.

[0105] For each individual (scheduling scheme) in the population, the job scheduling optimization model calculates its fitness. The fitness function is designed based on the objective function, which usually aims to maximize the total weighted click-through rate benefit.

[0106] During this process, the job scheduling optimization model will consider all constraints (push time window constraints, dependency constraints, update frequency constraints and resource constraints) to evaluate the feasibility of each scheduling plan and calculate its corresponding benefit value.

[0107] Based on the results of fitness evaluation, the job scheduling optimization model selects individuals with high fitness for reproduction (selection operation) to produce the next generation of population. Then, through the crossover (or hybridization) operation, some genes (i.e. part of the scheduling plan) are exchanged from the selected individuals to generate new individuals. Finally, through the mutation operation, some genes of some individuals are randomly changed to increase the diversity of the population, which helps to escape the local optimal solution.

[0108] Repeat the fitness evaluation, selection, crossover and mutation operations until the predetermined number of iterations is reached or other stopping conditions are met (such as insignificant fitness improvement). In each iteration, the individuals in the population gradually approach the optimal solution, that is, find a scheduling solution that maximizes the total weighted click-through rate benefit while satisfying all constraints.

[0109] After multiple iterations, the genetic algorithm generates an optimal or near-optimal report push job scheduling plan, and outputs the report push job scheduling plan through the result output module. The report push job scheduling plan includes the push order and push time of each report.

[0110] In this embodiment, the genetic algorithm, as a heuristic search algorithm, is good at finding the global optimal solution or the approximate optimal solution in a large-scale search space. The job scheduling optimization model based on the genetic algorithm is combined with the report dependency table, the report update frequency table and the report click rate data table to generate a report push job scheduling plan to ensure the timeliness of the report data in the high-frequency click time period.

[0111] See also Figure 5 A fifth embodiment of a report push optimization method in an embodiment of the present invention includes:

[0112] S501 , traverse the report click rate data table starting from the report with the lowest click rate ranking, and compare the click rate of the report with a preset click rate threshold.

[0113] In this embodiment, the preset click rate threshold is set according to actual needs, for example, the preset click rate threshold may be 0.10 (ie 10%). The preset click rate threshold is used to determine whether the click rate of the report is low enough to be marked as a low click rate report.

[0114] S502: If the click rate of the report is lower than the preset click rate threshold, the report is placed in a low click rate report set; if the click rate of the report is higher than the preset click rate threshold, traversing the report click rate data table is stopped.

[0115] In this embodiment, since the reports are sorted from high to low according to their click-through rates, and when the click-through rates of the reports are compared with the preset click-through rate threshold, the report with the lowest click-through rate in the report click-through rate data table is selected. Therefore, when the click-through rate corresponding to a certain report is higher than the preset click-through rate threshold, it means that the click-through rates of the remaining reports are all higher than the preset click-through rate threshold. It is no longer necessary to compare the click-through rates of the remaining reports with the preset click-through rate threshold one by one, thereby improving the screening efficiency.

[0116] In this embodiment, an empty set (or list) is created to store report IDs whose click-through rates are lower than a preset threshold. When the click-through rate corresponding to a report is lower than the preset click-through rate threshold, the report is placed in a preset low-click-through rate report set. After the traversal is completed, a low-click-through rate report set is obtained.

[0117] S503: Mark the reports in the low click rate report set as low click rate reports.

[0118] In this embodiment, an SQL update statement may be written or a database management tool may be used to update the status field of the record corresponding to the report ID in the low click rate report set to a value indicating a low click rate (eg, setting the is_low_click_rate field to true or 1).

[0119] Furthermore, once the tagging operation is complete, verify that the low CTR reports have been correctly tagged. This can be done by checking the database records, viewing the display in the user interface, or inspecting the report file and its metadata.

[0120] In this embodiment, by starting from the report with the lowest click rate ranking and traversing in reverse order, all low click rate reports can be found more quickly, and the report can be stopped immediately when the first report with a value higher than the threshold is encountered, thereby saving traversal time and reducing the consumption of computing resources (such as CPU and memory), especially when processing large data sets.

[0121] See also Figure 6 A sixth embodiment of a report push optimization method in an embodiment of the present invention includes:

[0122] S601. Adjust the execution order, time and priority of jobs in the ETL job table according to the report push job scheduling plan to obtain an updated ETL job table.

[0123] In this embodiment, the report push job scheduling plan includes the push time, execution order and priority of each report.

[0124] When updating the ETL job table, first check the existing ETL job table to understand the current configuration of each job, including its execution order, time, and priority. Then compare and analyze the report push job scheduling plan with the configuration in the current ETL job table to find out what needs to be adjusted. Finally, based on the results of the comparative analysis, adjust the job configuration in the ETL job table, including changing the execution time of the job, adjusting the dependencies between jobs to change the execution order, and updating the priority of the job.

[0125] S602: Create a new trigger for each low click rate report, and configure the trigger parameters according to the scheduling plan corresponding to the low click rate report.

[0126] In this embodiment, trigger logic is designed for each low click rate report, including determining the trigger conditions (such as data update, no new data after a specific time interval, etc.) and the operations performed after the trigger (i.e., calling the corresponding ETL job and report push process).

[0127] In this embodiment, a new trigger is created in the ETL job scheduling system or the database management system, and the parameters of the trigger, such as the trigger condition, the trigger action, etc., are configured according to the designed trigger logic.

[0128] S603 , traverse the updated ETL job table, query to obtain a low click rate report, and replace the scheduling trigger mode of the low click rate report with a trigger trigger.

[0129] In this embodiment, the updated ETL job table is traversed, and job entries corresponding to all low click rate reports are queried according to information such as the report identifier or name.

[0130] For each job entry of the low click-through rate report, change its original scheduling trigger mode (such as timed trigger) to trigger trigger. For example, modify the Trigger Mode field in the job entry and specify the name or ID of the trigger to establish an association.

[0131] In this embodiment, by creating a trigger for a low click-through rate report, the report push timing can be dynamically adjusted according to actual needs, rather than fixedly executed according to a timing plan. This flexibility helps to better meet business needs; moreover, for low click-through rate reports, using triggers can reduce unnecessary waste of computing resources and storage resources. Only when specific conditions (such as data update) are met, the corresponding ETL job and report push are executed, thereby improving resource utilization efficiency.

[0132] The report push optimization method in the embodiment of the present invention is described above. The device in the embodiment of the present invention is described below. Figure 7 , the implementation method of the report push optimization device in the embodiment of the present invention includes:

[0133] The preprocessing module 701 is used to periodically collect the behavior data of the user access report, and preprocess the collected data to obtain preprocessed data;

[0134] A sorting module 702 is used to calculate the click rate of each report according to the pre-processed data, sort the reports according to the click rate, obtain the click rate ranking of each report, and generate a report click rate data table;

[0135] The scheduling module 703 is used to obtain a preset report dependency table and a report update frequency table, and input the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan;

[0136] A screening module 704 is used to screen out reports with a click rate lower than a preset click rate threshold according to the report click rate data table, and mark them as low click rate reports;

[0137] The updating module 705 is used to update the ETL job table according to the report push job scheduling plan, and replace the scheduling triggering mode of the low click rate report with trigger triggering.

[0138] In this embodiment, the preprocessing module 701 includes: a collection unit 7011, which is used to regularly collect behavioral data of user access reports; a cleaning unit 7012, which is used to clean the collected data to obtain cleaned data; a conversion unit 7013, which is used to convert the cleaned data to obtain preprocessed data, and the preprocessed data includes a report ID, report access data and report click data.

[0139] In this embodiment, the sorting module 702 includes: a first calculation unit 7021, which is used to calculate the number of clicks and the total number of visits of each report based on the preprocessed data; a second calculation unit 7022, which is used to calculate the ratio of the number of clicks of each report to the total number of visits, and obtain the click rate of each report; a sorting unit 7023, which is used to sort each report in descending order according to the click rate, obtain the click rate ranking of each report, and generate a report click rate data table.

[0140] In this embodiment, the scheduling module 703 includes: a construction unit 7031, which is used to construct a job scheduling optimization model based on a genetic algorithm; an integration unit 7032, which is used to obtain a preset report dependency table and a report update frequency table, and integrate the report dependency table, the report update frequency table and the report click rate data table into a comprehensive data set; a scheduling unit 7033, which is used to input the comprehensive data set into the job scheduling optimization model to obtain a report push job scheduling plan, and the report push job scheduling plan includes the push order and push time of each report.

[0141] In this embodiment, the screening module 704 includes: a comparison unit 7042, which is used to traverse the report click rate data table starting from the report with the lowest click rate ranking, and compare the click rate of the report with a preset click rate threshold; a screening unit 7042, which is used to place the report into a low click rate report set when the click rate of the report is lower than the preset click rate threshold, and to stop traversing the report click rate data table when the click rate of the report is higher than the preset click rate threshold; a marking unit 7043, which is used to mark the reports in the low click rate report set as low click rate reports.

[0142] In this embodiment, the update module 705 includes: an update unit 7051, which is used to adjust the execution order, time and priority of the jobs in the ETL job table according to the report push job scheduling plan to obtain an updated ETL job table; a configuration unit 7052, which is used to create an innovative trigger for each of the low click-through rate reports, and configure the trigger parameters according to the scheduling plan corresponding to the low click-through rate report; a replacement unit 7053, which is used to traverse the updated ETL job table, query and obtain the low click-through rate report, and replace the scheduling trigger method of the low click-through rate report with trigger triggering.

[0143] In this embodiment, the behavioral data of users accessing reports are collected regularly, and the click rate of the reports is calculated based on these data. Then, the job scheduling optimization model is used to automatically generate an optimized report push job scheduling plan in combination with the report's dependencies, update frequency, and click rate. The ETL job table is updated according to the report push job scheduling plan to ensure the timeliness of report data in high-frequency click time periods and reduce data statistical errors caused by data delays. For low-click rate reports, trigger triggering is used instead of traditional timed triggering, making report push more flexible and able to be triggered in real time according to data update conditions or business needs, thereby improving the timeliness and accuracy of reports.

[0144] above Figure 7 The report push optimization device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The report push optimization device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0145] Figure 8 It is a structural diagram of a report push optimization device provided by an embodiment of the present invention. The device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (for example, one or more processors) and a memory 820, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 833 or data 832. Among them, the memory 820 and the storage medium 830 can be short-term storage or persistent storage. The program stored in the storage medium 830 may include one or more modules (not shown), and each module may include a series of instruction operations in the device 800. Furthermore, the processor 810 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium on the device 800.

[0146] The device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input and output interfaces 860, and / or one or more operating systems 831, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc.

[0147] An embodiment of the present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the report push optimization method.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0150] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A report push optimization method, characterized in that: The report push optimization method comprises: Regularly collect behavioral data of user access reports, and pre-process the collected data to obtain pre-processed data; Calculating the click rate of each report according to the preprocessed data, and sorting each report according to the click rate to obtain the click rate ranking of each report, and generating a report click rate data table; Obtaining a preset report dependency table and a report update frequency table, and inputting the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan; According to the report click rate data table, the reports with click rates lower than a preset click rate threshold are screened out and marked as low click rate reports; The ETL job table is updated according to the report push job scheduling plan, and the scheduling triggering mode of the low click rate report is replaced with trigger triggering.

2. The report push optimization method according to claim 1, characterized in that: The periodic collection of user access report behavior data and preprocessing of the collected data to obtain preprocessed data include: Regularly collect behavioral data of user access reports; Perform data cleaning on the collected data to obtain cleaned data; The cleaned data is converted to obtain pre-processed data, wherein the pre-processed data includes a report ID, report access data, and report click data.

3. The report push optimization method according to claim 1, characterized in that: The click rate of each report is calculated according to the pre-processed data, and each report is sorted according to the click rate to obtain the click rate ranking of each report, and a report click rate data table is generated, including: Calculate the number of clicks and total number of visits for each report based on the pre-processed data; Calculate the ratio of clicks of each report to the total number of visits to obtain the click rate of each report; According to the click-through rate, the reports are sorted in descending order to obtain the click-through rate ranking of each report, and a report click-through rate data table is generated.

4. The report push optimization method according to claim 1, characterized in that: The step of obtaining a preset report dependency table and a report update frequency table, and inputting the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan includes: Constructing a job scheduling optimization model based on genetic algorithm; Obtaining a preset report dependency table and a report update frequency table, and integrating the report dependency table, the report update frequency table and the report click rate data table into a comprehensive data set; The comprehensive data set is input into a job scheduling optimization model to obtain a report push job scheduling plan, wherein the report push job scheduling plan includes a push order and a push time for each report.

5. The report push optimization method according to claim 4, characterized in that: The construction of the job scheduling optimization model based on the genetic algorithm includes: Constructing a basic framework of the initial model of scheduling optimization, the basic framework includes a data input module, a scheduling algorithm module, and a result output module, the data input module is used to receive a comprehensive data set and transmit it to the scheduling algorithm module, the scheduling algorithm module is used to generate a report push job scheduling plan according to the comprehensive data set, and transmit it to the result output module, the result output module is used to output the report push job scheduling plan; Obtaining preset policy variables, objective functions, constraints and genetic algorithms, and integrating the preset policy variables, objective functions, constraints and genetic algorithms into the scheduling algorithm module to obtain an initial scheduling optimization model; The historical report scheduling data is collected, and the scheduling optimization initial model is trained using the historical report scheduling data to obtain a job scheduling optimization model.

6. The report push optimization method according to claim 1, characterized in that: The step of filtering out reports with click rates lower than a preset click rate threshold according to the report click rate data table and marking them as low click rate reports includes: Traversing the report click rate data table starting from the report with the lowest click rate ranking, and comparing the click rate of the report with a preset click rate threshold; If the click rate of the report is lower than the preset click rate threshold, the report is placed in the low click rate report set; if the click rate of the report is higher than the preset click rate threshold, the traversal of the report click rate data table is stopped; The reports in the low click rate report set are marked as low click rate reports.

7. The report push optimization method according to claim 1, characterized in that: The updating of the ETL job table according to the report push job scheduling plan and replacing the scheduling triggering mode of the low click rate report with trigger triggering includes: Adjust the execution order, time and priority of jobs in the ETL job table according to the report push job scheduling plan to obtain an updated ETL job table; Creating a new trigger for each of the low click-through rate reports, and configuring the trigger parameters according to the scheduling plan corresponding to the low click-through rate report; Traverse the updated ETL job table, query to obtain the low click rate report, and replace the scheduling triggering method of the low click rate report with trigger triggering.

8. A report push optimization device, characterized in that: include: The preprocessing module is used to regularly collect the behavior data of users accessing the report, and preprocess the collected data to obtain preprocessed data; A sorting module, used to calculate the click rate of each report according to the pre-processed data, and sort each report according to the click rate to obtain the click rate ranking of each report, and generate a report click rate data table; A scheduling module, used to obtain a preset report dependency table and a report update frequency table, and input the report dependency table, the report update frequency table and the report click rate data table into a job scheduling optimization model to obtain a report push job scheduling plan; A screening module, used to screen out reports with a click rate lower than a preset click rate threshold according to the report click rate data table, and mark them as low click rate reports; The updating module is used to update the ETL job table according to the report push job scheduling plan, and replace the scheduling triggering mode of the low click rate report with trigger triggering.

9. A report push optimization device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute each step of the report push optimization method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the report push optimization method according to any one of claims 1 to 7 are implemented.

Images