Data preheating method and program product

Abstract

The invention discloses a data preheating method and a program product, and the method comprises the steps: obtaining a data preheating report from a preheating queue, and determining a data preheating task according to the data preheating report; wherein data preheating information is recorded in the data preheating report; the data preheating information comprises preheating rule parameters and a target data set needing to be preheated; determining a structured query statement of the target data set according to the preheating rule parameter corresponding to the target data set and a preset query statement generation template corresponding to the target data set; executing the structured query statement to obtain target preheating data from the target data set, and caching the target preheating data; the intelligent level of data preheating operation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data preheating method and program product. BACKGROUND

[0002] With the rapid development of big data, the data scale of various platforms is growing, and users' requirements for query response speed are also increasing. In order to improve the response speed of query operation, data preheating can be performed in advance to load the data that may be accessed into the cache, reducing the time-consuming of real-time reading.

[0003] The current data preheating method usually associates fixed data sets through hard-coded rules. When the business requirements change, the rule engine needs to be redeveloped and deployed, which has a long iteration cycle and high operation and maintenance cost. Therefore, there is an urgent need for a data preheating method to improve the efficiency and intelligent level of data preheating. SUMMARY

[0004] The present application provides a data preheating method and program product to solve the technical problems of poor flexibility, low efficiency, and high maintenance cost of the data preheating method in related technologies.

[0005] According to an aspect of the present application, a data preheating method is provided, which comprises:

[0006] obtaining a data preheating report from a preheating queue and determining a data preheating task according to the data preheating report; wherein the data preheating report records data preheating information; the data preheating information includes preheating rule parameters and a target data set that needs to be preheated;

[0007] determining a structured query statement of the target data set according to the preheating rule parameters corresponding to the target data set and a pre-set query statement generation template corresponding to the target data set;

[0008] executing the structured query statement to obtain target preheating data from the target data set and cache the target preheating data.

[0009] According to another aspect of the present application, a data preheating device is provided, which comprises:

[0010] a data preheating task determination module configured to obtain a data preheating report from a preheating queue and determine a data preheating task according to the data preheating report; wherein the data preheating report records data preheating information; the data preheating information includes preheating rule parameters and a target data set that needs to be preheated;

[0011] The query statement determination module is configured to determine a structured query statement of the target data set according to the preheat rule parameter corresponding to the target data set and a query statement generation template preset for the target data set.

[0012] The target preheat data acquisition module is configured to execute the structured query statement to obtain target preheat data from the target data set and cache the target preheat data.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data preheat method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the data preheat method according to any one of the embodiments of the present application when executed by the processor.

[0018] According to another aspect of the present application, the embodiments of the present disclosure further provide a computer program product comprising a computer program which, when executed by a processor, implements the data preheat method according to any one of the embodiments of the present disclosure.

[0019] The technical scheme of the embodiment of the present application firstly obtains a data warm-up report from a warm-up queue, determines a data warm-up task according to the data warm-up report, wherein the data warm-up report records data warm-up information; the data warm-up information includes a warm-up rule parameter and a target data set that needs to be warmed up; by analyzing the related information of the data warm-up report, the corresponding data warm-up task can be accurately generated, the flexibility and configurability of task generation are enhanced, so as to support dynamically adjusting the warm-up strategy according to different business scenarios; then, the structured query statement of the target data set is determined according to the warm-up rule parameter corresponding to the target data set and the query statement generation template corresponding to the target data set that is set in advance; by using the preset query statement generation template, the reusability and development efficiency are improved, and the structured query statement is generated in combination with the warm-up rule parameter, so as to ensure the accuracy and controllability of the warm-up content, realize the automatic conversion from the rule parameter to the structured query statement, and reduce manual intervention; finally, the target warm-up data is obtained from the target data set by executing the structured query statement, and the target warm-up data is cached; the target warm-up data can be directly obtained by using the structured query statement, the execution efficiency is high, the accuracy and integrity of data acquisition are improved, the target warm-up data is cached, the access speed of subsequent query operations can be significantly improved, the system response delay is effectively reduced, and the user experience is optimized; the method has good expansibility and maintainability, can quickly adapt to various data sources and complex business scenarios, and improves the intelligent level and execution efficiency of the data warm-up operation.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating any creative labor.

[0022] Figure 1 is a flow chart of a data warm-up method provided by the first embodiment of the present application;

[0023] Figure 2 is a flow chart of a data warm-up method provided by the second embodiment of the present application;

[0024] Figure 3 is a flow chart of a data warm-up method provided by the third embodiment of the present application;

[0025] Figure 4 is a structural schematic diagram of a data preheating device according to an embodiment four of the present application;

[0026] Figure 5 is a structural schematic diagram of an electronic device implementing a data preheating method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", "pre-set", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0031] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario and the like of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0032] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will need to acquire and use personal information of the user. Thus, the user can autonomously select whether to provide the personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solution of the present disclosure according to the prompt information.

[0033] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in a text manner in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select to "agree" or "disagree" to provide the personal information to the electronic device.

[0034] It can be understood that the above notification and acquisition of user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0035] It can be understood that the data (including but not limited to the data itself, acquisition or use of the data) involved in the technical solution should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0036] Embodiment one

[0037] Figure 1 A flowchart of a data preheating method is provided for the embodiment one of the present application. The embodiment can be applicable to a configuration, and the method can be executed by a data preheating device. The data preheating device can be realized in the form of hardware and / or software, and can be realized by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc. As shown in the figure, the method can specifically include: Figure 1

[0038] S110, acquiring a data preheating report from a preheating queue, and determining a data preheating task according to the data preheating report; wherein the data preheating report records data preheating information; the data preheating information includes preheating rule parameters and a target data set that needs to be preheated.

[0039] In the embodiment of the present application, preheating can be understood as a data query optimization mechanism of pre-executing a data query operation and persisting the query content to a cache to improve subsequent access efficiency. The preheating queue refers to an ordered queue for managing data preheating tasks to be executed. The preheating queue can be used to schedule the execution order of the data preheating tasks according to the sorting rules such as priority or time sequence. The preheating queue can include one or more data preheating reports.

[0040] ​Optionally, the data warm-up report can be obtained from the warm-up queue in a resource sufficient or low load state to avoid affecting normal business processes in a high load state. The data warm-up report refers to a report or other task description file recording data warm-up task related information. The data warm-up report can record information related to the data warm-up task, i.e., data warm-up information.

[0041] Further, the data warm-up information can include, but is not limited to, warm-up rule parameters and target data sets that need to be warmed up. The warm-up rule parameters can be understood as a set of parameters that control the data warm-up behavior. The warm-up rule parameters can include, but are not limited to, the time range and / or the subject of the target warm-up data. The warm-up rule parameters can be used to define the range and filtering conditions of the warm-up data. The subject can be understood as the data ownership subject specified in the warm-up rule parameters, for example, a certain date, a certain department, etc. The target data set refers to the data set that needs to be warmed up.

[0042] Specifically, according to the related information in the data warm-up report, a specific warm-up data loading task, i.e., a data warm-up task, can be generated, thereby realizing the automatic generation and scheduling of the warm-up task, improving the flexibility and configurability of task generation, supporting dynamic adjustment of the warm-up strategy according to different business scenarios, and enhancing the intelligent level of data warm-up.

[0043] As an optional technical solution of an embodiment of the present application, before the data warm-up report is obtained from the warm-up queue, the data warm-up report is added to the warm-up queue according to a preset time limit identifier carried by the data warm-up report.

[0044] The preset time limit identifier can be understood as an identifier information used to identify the execution priority or execution time requirement of the data warm-up report. The preset time limit identifier can be used to sort and schedule multiple data warm-up reports. The preset time limit identifier can be used to indicate at least the execution priority of the data warm-up report. For example, the preset time limit identifier can include priority labels such as high time limit and low time limit.

[0045] Specifically, according to the preset time limit identifier carried by the data warm-up report, the data warm-up reports can be prioritized, and the data warm-up report with high priority can be added to the warm-up queue first to realize the priority processing of the data warm-up report and reduce the corresponding data loading time. The warm-up queue can be understood as a bottom-level acceleration queue and can be used to carry out fast execution of high-priority tasks.

[0046] By adding the data warm-up reports to the warm-up queue according to the preset time limit identifier carried by the data warm-up reports, the hierarchical management and scheduling optimization of the warm-up tasks are realized, which can ensure that the high-priority tasks are executed first, improve the cache efficiency of critical data, and enhance the response capability to different business priorities.

[0047] Further, before adding the data warm-up report to the warm-up queue according to the preset time limit identifier carried by the data warm-up report, the method further comprises: in response to an identifier adding operation for the data warm-up report, adding a preset time limit identifier to the data warm-up report.

[0048] Optionally, in response to the identifier adding operation for the data warm-up report, the data warm-up report can be classified in advance and the corresponding preset time limit identifier is added.

[0049] Optionally, the classification and identifier adding of the data warm-up report can be manually configured, or automatically classified and added based on task attributes, historical execution effects and other information.

[0050] By configuring the preset time limit identifier for the data warm-up report before adding it to the warm-up queue according to the identifier adding operation, the flexibility of task priority setting can be realized, the controllability and configurability of task scheduling are improved, and the task priority can be dynamically adjusted according to actual business needs.

[0051] Further, the warm-up rule parameters can also be evaluated to realize the accuracy and effectiveness of the data warm-up task. Specifically, before or after obtaining the data warm-up report from the warm-up queue, the method further comprises: obtaining interaction feedback data of target warm-up data of a plurality of executed data cache tasks; wherein the interaction feedback data comprises at least one of an accessed state of historical cached warm-up data, query time information and interaction operation with a data access object; determining a task evaluation index of the data cache task according to the interaction feedback data and a task evaluation model; wherein the task evaluation model is obtained by training a machine learning model according to sample interaction feedback data of sample cache task target cache data and its corresponding expected evaluation index.

[0052] Optionally, the data cache task can be evaluated before or after obtaining the data warm-up report from the warm-up queue. Specifically, interaction feedback data of the target warm-up data of the plurality of data cache tasks that have been executed can be obtained. The interaction feedback data can be understood as feedback information generated after the user or system interacts with the data cache task. The interaction feedback data can be collected from historical cache hit logs, query cycle regularity, and user behavior data, etc. The interaction feedback data at least includes at least one of the accessed state of the historical cache warm-up data, query time information, and interaction operation with the data access object. The accessed state can be understood as state information such as whether the historical cache warm-up data is accessed and its access frequency. The query time information refers to the time information spent in querying the cache data. The data access object can be understood as a user, terminal or system module that accesses the cache data. The interaction operation refers to the behavior interaction between the data access object and the cache data.

[0053] The task evaluation model can be used to evaluate the execution effect of the data warm-up task and the rationality of the warm-up rule parameters. Optionally, the interaction feedback data can be input into the task evaluation model to determine the task evaluation indicators of each data cache task.

[0054] Optionally, the task evaluation model can be trained by a machine learning model in the following way: first, sample interaction feedback data of the target cache data of a plurality of sample cache tasks is collected, and the corresponding expected evaluation indicators are labeled. The expected evaluation indicators can be ideal performance indicators or target values of the sample cache tasks; then, the plurality of sample interaction feedback data is input into the machine learning model, and the machine learning model is iteratively trained according to the error between the output data and its corresponding expected evaluation indicators until the error is less than a preset threshold or the model converges, obtaining the trained task evaluation model. The preset threshold can be flexibly set according to the actual scene.

[0055] By obtaining the interaction feedback data of the historical cache task and evaluating the data cache task in combination with the task evaluation model, quantitative evaluation and optimization of the effect of the warm-up rule parameters are realized, which helps to continuously optimize the warm-up strategy, improve the cache hit rate, and enhance the intelligentization and self-adaptation ability of data warm-up.

[0056] Optionally, the warm-up rule parameters and cache hit state can also be captured in real time by a distributed log collector to construct a multi-dimensional log matrix including timestamp, hit situation and other information, so as to facilitate subsequent analysis or model training.

[0057] Optionally, the causal relationship between the warm-up rule parameters and the hit situation can be analyzed according to the accessed state of the historical cache warm-up data, the warm-up rule parameters with a hit rate less than the hit rate threshold are identified, and an elimination list is generated to facilitate subsequent optimization or elimination of the warm-up rule parameters.

[0058] In addition, the input query parameters in the missed cases can be analyzed to identify the parts that can be abstracted as general pre-warming rule parameters and stored and recommended. When setting new pre-warming rule parameters, more appropriate rule parameters can be intelligently recommended based on historical missed data, thereby improving the hit rate and accuracy of the pre-warming task.

[0059] S120, determining a structured query statement of the target data set according to the pre-warming rule parameters corresponding to the target data set and a pre-set query statement generation template corresponding to the target data set.

[0060] The query statement generation template refers to a structured template for generating a specific query statement. Optionally, the query statement generation template can be a structured query language (SQL) statement template including placeholders. Through the template design, the rapid generation and reuse of the query statement can be realized, and the system development efficiency and maintainability are improved. The structured query statement refers to a specific query statement generated according to the query statement generation template and the pre-warming rule parameters.

[0061] Optionally, the data pre-warming report can be used to determine one or more target data sets corresponding to the data pre-warming task, and a pre-set SQL template corresponding to the one or more target data sets, i.e., a query statement generation template corresponding to the target data set, can be obtained. The SQL templates corresponding to different numbers or different types of target data sets are the same or different, and each template includes one or more placeholders, which can be used for subsequent injection of specific pre-warming rule parameters.

[0062] Optionally, the pre-warming rule parameters corresponding to the target data set can be injected into the corresponding query statement generation template at the positions of the placeholders of the corresponding type, to generate a structured query statement applicable to the target data set. The structured query statement is an executable specific SQL statement, which can be used to extract the required data from the target data set.

[0063] By injecting the pre-warming rule parameters into the query statement generation template, the automation degree of the query statement generation is improved, and the accuracy and controllability of the pre-warming content are ensured, realizing the automatic conversion from the rule parameters to the SQL statement, reducing the manual intervention, and improving the intelligent level of the pre-warming task.

[0064] S130, executing the structured query statement to obtain target pre-warming data from the target data set, and caching the target pre-warming data.

[0065] In the embodiment of the application, the target pre-warming data can be understood as data extracted from the target data set according to the structured query statement.

[0066] Optionally, a structured query statement can be executed to query the target data set to obtain target pre-warming data corresponding to the data pre-warming task, and the target pre-warming data is cached to improve subsequent access efficiency.

[0067] As an optional technical solution of the embodiment of the application, the execution of the structured query statement includes: performing structural verification on the structured query statement by a syntax verification engine, generating a standardized query instruction set according to the structured query statement if the verification is passed, and executing the standardized query instruction set.

[0068] The syntax verification engine can be understood as a tool or module for verifying the legality of the structured query statement.

[0069] Specifically, the syntax verification engine can be used to verify the syntax structure of the structured query statement to ensure that it conforms to the database syntax rules and does not pose a tampering risk to the target data set. After the verification is passed, a standardized query instruction set that can be directly executed can be generated according to the structured query statement, and the standardized query instruction set is executed. The standardized query instruction set can be a sequence of executable instructions generated after the structured query statement is parsed.

[0070] The generated structured query statement is verified by the syntax verification engine, and the standardized query instruction set is generated and executed after the verification is passed, effectively ensuring the legality and execution stability of the query statement, avoiding task failure caused by syntax errors or illegal operations, and improving the robustness and security of the data pre-warming process.

[0071] As an optional technical solution of the embodiment of the application, the execution of the structured query statement includes: in response to reaching a pre-warming execution time of the data pre-warming task, executing the structured query statement corresponding to the data pre-warming task.

[0072] The preset execution time can be understood as a predetermined time point or event trigger window for executing the data pre-warming task. Optionally, a preset time window can be set to trigger the data pre-warming task periodically, for example, triggering the data pre-warming task at dawn every day to perform the data pre-warming operation.

[0073] Further, in response to reaching the pre-warming execution time of the data pre-warming task, the structured query statement corresponding to the data pre-warming task can be executed to obtain the target pre-warming data and cache the target pre-warming data.

[0074] By triggering the execution of the structured query statement in response to the preset preheating execution time, the timing scheduling and the orderly execution of the data preheating task are realized, which helps to complete the preheating operation in a low-load period of the system, avoids affecting the normal business process, and improves the utilization efficiency of system resources and the controllability of the preheating task.

[0075] In addition, after the data preheating task is completed, the next data preheating task can also be triggered. Specifically, after the corresponding structured query statement of the data preheating task is executed, the next data preheating task is determined in response to the event that the data preheating task is executed, the structured query statement is updated according to the determined next data preheating task, and the updated structured query statement is executed.

[0076] Optionally, in response to the event that the periodically executed data preheating task is executed, the next data preheating task can be determined, and the structured query statement is automatically updated according to the determined next data preheating task, for example, the preheating rule parameter of the structured query statement is changed to the data date of the next stage.

[0077] Further, after the data batch of the next stage is executed, the updated structured query statement corresponding to the next data preheating task can be triggered to start execution, so as to realize the continuous execution and automatic update of the data preheating task.

[0078] By automatically identifying and executing the next preheating task after the current data preheating task is executed, and dynamically updating the structured query statement, the continuous execution and automatic iteration of the preheating task are realized. The automation degree of the data preheating process is enhanced, and the support capability for periodic or continuous preheating demand is improved.

[0079] Optionally, the method further comprises: after the data batch is executed, the preset time limit identifier of the data preheating report is parsed, and then it is judged whether the data preheating task triggers the preheating enhancement according to the preset time limit identifier; if the preset time limit identifier of the data preheating report is high, the double-path data preheating enhancement process can be used to further improve the data loading efficiency. Wherein, the data batch execution completion can be understood as the data preheating task of the batch to be executed first.

[0080] Further, for the data preheating report which has not been executed, it is determined that the path is the full-path, the target data set corresponding to the data preheating report can be automatically triggered to be acquired, and the preheating rule parameter is injected into the query statement generation template corresponding to the target data set to obtain a complete preheating query instruction set, and full preheating is executed.

[0081] For the previously executed data preheating report, when the trigger condition of the data batch execution is met, the next data preheating task can be automatically determined, the structured query statement is updated and executed, so as to realize the continuous optimization of query performance and the dynamic scheduling of preheating task.

[0082] The technical scheme of the embodiment of the application first acquires a data preheating report from a preheating queue, determines a data preheating task according to the data preheating report, wherein the data preheating report records data preheating information; the data preheating information includes preheating rule parameters and a target data set that needs to be preheated; by analyzing the related information of the data preheating report, the corresponding data preheating task can be accurately generated, the flexibility and configurability of task generation are enhanced, so as to support dynamic adjustment of preheating strategies according to different business scenarios; then, a structured query statement of the target data set is determined according to the preheating rule parameters corresponding to the target data set and a query statement generation template corresponding to the target data set that is set in advance; by using the pre-set query statement generation template, the reusability and development efficiency are improved, and the structured query statement is generated in combination with the preheating rule parameters, so as to ensure the accuracy and controllability of the preheating content, realize the automatic conversion from the rule parameters to the structured query statement, and reduce manual intervention; finally, the target preheating data is obtained from the target data set by executing the structured query statement, and the target preheating data is cached; the target preheating data can be directly obtained by using the structured query statement, the execution efficiency is high, the accuracy and integrity of data acquisition are improved, the target preheating data is cached, the access speed of subsequent query operations can be significantly improved, the system response delay is effectively reduced, and the user experience is optimized; the application has good expansibility and maintainability, can quickly adapt to various data sources and complex business scenarios, and improves the intelligent level and execution efficiency of the data preheating operation.

[0083] Embodiment two

[0084] Figure 2A flowchart of a data preheating method provided for the second embodiment of the present application further describes an implementation of determining the structured query statement of the target data set according to the preheating rule parameters corresponding to the target data set and the pre-set query statement generation template corresponding to the target data set. Optionally, the determining the structured query statement of the target data set according to the preheating rule parameters corresponding to the target data set and the pre-set query statement generation template corresponding to the target data set comprises: obtaining the pre-set query statement generation template corresponding to the target data set from the data warehouse; wherein the query statement generation template comprises a plurality of preset placeholders; and replacing the plurality of preset placeholders in the query statement generation template with the preheating rule parameters corresponding to the target data set according to a preset placeholder replacement logic to obtain the structured query statement. The specific implementation can be seen from the description of the present embodiment. Wherein, the same or similar technical features as the foregoing embodiments are not described herein. As shown in Figure 2 The method can specifically comprise:

[0085] S210, obtaining a data preheating report from the preheating queue and determining a data preheating task according to the data preheating report; wherein the data preheating report records data preheating information; the data preheating information comprises preheating rule parameters and a target data set that needs to be preheated.

[0086] S220, obtaining a pre-set query statement generation template corresponding to the target data set from the data warehouse; wherein the query statement generation template comprises a plurality of preset placeholders.

[0087] Wherein, the data warehouse refers to a centralized data management system for storing structured data and related meta information, and the data warehouse can comprise a plurality of pre-set query statement generation templates corresponding to different target data sets.

[0088] Optionally, the query statement generation template is a basic template for generating a structured query statement, and the query statement generation template can comprise a plurality of preset placeholders. The preset placeholder can be understood as a variable symbol in the query statement generation template for indicating the position of the preheating rule parameter to be replaced. The preset placeholder can be used to replace the specific preheating rule parameter value when generating the structured query statement.

[0089] Specifically, the type and number of the preset placeholders can be flexibly set according to the structure of the target data set and the configuration of the preheating rule parameters, so as to ensure that the generated structured query statement can accurately match the query requirements of the target data set.

[0090] Optionally, the unique identification information of the data preheating report can be used to perform a matching search in the data warehouse to obtain the corresponding target data set and the query statement template corresponding to the target data set.

[0091] S230, according to the preset placeholder replacement logic, replacing a plurality of preset placeholders in the query statement generation template with the preheating rule parameters corresponding to the target data set to obtain a structured query statement.

[0092] The placeholder replacement logic refers to a rule set for guiding and controlling the replacement of each preset placeholder in the query statement generation template with a specific preheating rule parameter value. The placeholder replacement logic can include, but is not limited to, at least one of the preset replacement logics such as the mapping relationship between the placeholder and the preheating rule parameter, the replacement order, and the parameter format verification.

[0093] Optionally, according to the preset placeholder replacement logic, a plurality of preset placeholders in the query statement generation template can be replaced with the preheating rule parameter values corresponding to the target data set in sequence, thereby constructing a complete and executable structured query statement.

[0094] Specifically, when performing the parameter replacement operation, the preheating rule parameter can be subjected to type checking and format conversion first to ensure that it matches the semantic and format requirements of the corresponding placeholder; then, according to the preset placeholder replacement logic, each placeholder is replaced with the preheating rule parameter value in sequence to generate the final structured query statement.

[0095] In addition, the preheating rule parameter replacement can also support multi-level and multi-dimensional parameter replacement, for example, simultaneously replacing parameters of multiple types such as time dimension, subject dimension, etc. to meet the preheating requirements in complex query scenarios.

[0096] S240, executing the structured query statement to obtain target preheating data from the target data set and caching the target preheating data.

[0097] The technical scheme of the embodiment of the present application firstly obtains a query statement generation template corresponding to the target data set from the data warehouse, wherein the query statement generation template comprises a plurality of preset placeholders; the modularity and standardization of query statement construction are realized, the reusability and maintainability of the template are improved, the flexibility of the template is enhanced by setting the preset placeholders in the query statement generation template, and the template can adapt to data warm-up tasks of different structures and rules; then, the plurality of preset placeholders in the query statement generation template are replaced into the warm-up rule parameters corresponding to the target data set according to a preset placeholder replacement logic, so as to obtain a structured query statement; the automatic generation of the structured query statement is realized through the preset placeholder replacement logic, the efficiency and accuracy of the warm-up task configuration are improved, multi-dimensional parameter injection can be supported, the adaptability to complex warm-up rules is enhanced, the automation and intelligentization of the query statement generation are realized, and the development and maintenance costs of data warm-up are effectively reduced.

[0098] Embodiment three

[0099] As an optional example of the embodiment of the present application, the data warm-up method of the embodiment of the present application can comprise:

[0100] S310, data warm-up report publishing: when a tenant publishes a report in a portal, a preset time limit identifier is added to the data warm-up report, and the data warm-up identifier is used to indicate the execution priority of the data warm-up report; the report with a high time limit identifier is automatically entered into a warm-up queue;

[0101] S320, data warm-up information storage: data warm-up information recorded in the data warm-up report is automatically parsed, the data warm-up information comprises a data set that needs to be warmed up, a SQL template with placeholders is generated according to the data set (such as WHERE dte=?{biz_date}), and meanwhile, warm-up rule parameters such as institution, date and other filtering condition rule parameters are recorded;

[0102] Based on the blood relationship topology and the structured features of the data warm-up report, the warm-up rule parameters and the target data set meta information are extracted; through the mapping relationship between the report ID and the warm-up rule parameters, a reusable warm-up rule knowledge base is formed, and a warm-up rule parameter set is quickly extracted.

[0103] A query statement generation template, namely a SQL template, corresponding to the target data set is obtained from the data warehouse; according to the placeholder replacement logic stored in the warm-up rule knowledge base, time range, institution parameters and other dynamic variables are injected into the template, so as to obtain a structured query statement; the replaced SQL can also be subjected to structure verification through a syntax verification engine, so as to generate a standardized query instruction set that can be directly executed.

[0104] S330, data preheating trigger: trigger the preheating task of the previous day regularly every day; at the same time, after the completion of different data batches in the day, automatically trigger the replacement of the dynamic parameters in the SQL template to update the latest data date;

[0105] After the completion of the batch script execution, the preset time limit identifier is parsed; based on the preset high-efficiency report corresponding to the data set, the mode matching algorithm is used to judge whether the current batch task triggers the preheating enhancement process.

[0106] The preheating enhancement process adopts a double-path execution mechanism: for the full-path: the target data set is automatically triggered by the timing task scheduler, and the structured query statement of the target data set is determined according to the preheating rule parameters and the query statement template to generate a complete preheating instruction set;

[0107] For the incremental path: when the data batch is completed, the SQL logic replacement is started to determine the next data execution task and automatically update the structured query statement, so as to realize the query performance optimization.

[0108] S340, preheating structured query statement execution: execute the corresponding structured query statement, and cache the data preheating result set obtained by executing the structured query statement.

[0109] Specifically, the execution state and performance indicators of each preheating structured query statement can be tracked and recorded respectively to obtain data preheating results. In the distributed cache scenario, the preheating results can be stored in the distributed cache cluster according to the LRU strategy, and at the same time, they can be persisted to the columnar storage database. Specifically, the last access time of each preheating data can be recorded, and when the cache space is full, the data with the earliest access time (i.e. the longest unused) is selected to be removed from the cache. The least used data in the preheating data can also be removed and stored in the distributed cache cluster.

[0110] S350, post-preheating evaluation: record historical cache hit logs, query cycle regularity, and user behavior data, obtain evaluation indicators based on machine learning methods, and continuously optimize rule parameters to generate more accurate preheating rule parameters.

[0111] In the query request response process, the request metadata, preheating rule identifier, and cache hit state are captured in real time by the distributed log collector to construct a multi-dimensional log matrix containing timestamps and hit conditions. The causal relationship between the preheating rule and the hit condition is statistically analyzed to identify inefficient rules with a hit rate lower than a preset threshold within a preset time period and generate a removal list. The miss condition is statistically analyzed to identify abstractable preheating rule parameters, and new preheating rule parameters are recommended.

[0112] The technical scheme of the embodiment of the present application firstly realizes fine management of task priority by marking a preset time limit for a data warm-up report, which is helpful to rapid scheduling and execution of high-priority tasks, then parses data warm-up information in the data warm-up report, determines a data set and warm-up rule parameters corresponding to the report, generates a template according to a query statement corresponding to the data set and constructs a structured query statement according to the warm-up rule parameters, automatically generates the structured query statement in combination with the query statement generation template and the warm-up rule parameters, improves the automation degree and accuracy of warm-up task configuration, reduces manual intervention, and enhances the maintainability of the warm-up task; then, the data warm-up can be automatically started at regular intervals or when a trigger condition is met, so that the data warm-up can be efficiently completed at time periods such as low peak periods, resource utilization and response timeliness are improved, the structured query statement is executed and a structure set is cached, subsequent data access delay is significantly reduced, and overall performance is optimized; finally, the warm-up result is evaluated to continuously optimize and generate more accurate warm-up rule parameters; the feedback loop drives iterative updating of the warm-up rule parameters, improves the pertinence and hit rate of the warm-up content, and enhances the intelligence and adaptive ability of the data warm-up.

[0113] Embodiment four

[0114] Figure 4 A structural schematic diagram of a data warm-up device provided by the fourth embodiment of the present application is shown. The device is used to execute the data warm-up method provided by any of the above embodiments. The device and the data warm-up method of each of the above embodiments belong to the same inventive concept, and details not described in the embodiment of the data warm-up device can be referred to the embodiment of the data warm-up method. As shown in the figure, the device includes a data warm-up task determination module 410, a query statement determination module 420, and a target warm-up data acquisition module 430. Figure 4

[0115] The data warm-up task determination module 410 is configured to obtain a data warm-up report from a warm-up queue and determine a data warm-up task according to the data warm-up report. The data warm-up report records data warm-up information. The data warm-up information includes warm-up rule parameters and target data sets that need to be warmed up. The query statement determination module 420 is configured to determine a structured query statement of the target data set according to the warm-up rule parameters corresponding to the target data set and a query statement generation template corresponding to the target data set. The target warm-up data acquisition module 430 is configured to execute the structured query statement to obtain target warm-up data from the target data set and cache the target warm-up data.

[0116] ​The technical scheme of the embodiment of the application first acquires a data warm-up report from a warm-up queue by a data warm-up task determination module 410, and determines a data warm-up task according to the data warm-up report; wherein the data warm-up report records data warm-up information; the data warm-up information includes warm-up rule parameters and a target data set that needs to be warmed up; by analyzing the related information of the data warm-up report, the corresponding data warm-up task can be accurately generated, the flexibility and configurability of task generation are enhanced, so as to support dynamically adjusting the warm-up strategy according to different business scenarios; then, the query statement determination module 420 determines the structured query statement of the target data set according to the warm-up rule parameters corresponding to the target data set and a pre-set query statement generation template corresponding to the target data set; by using the pre-set query statement generation template, the reusability and development efficiency are improved, and the structured query statement is generated in combination with the warm-up rule parameters, so as to ensure the accuracy and controllability of the warm-up content, realize the automatic conversion from the rule parameters to the SQL statement, and reduce manual intervention; finally, the target warm-up data acquisition module 430 executes the structured query statement to obtain target warm-up data from the target data set and cache the target warm-up data; the target warm-up data can be directly obtained by using the structured query statement, the execution efficiency is high, the accuracy and integrity of data acquisition are improved, the target warm-up data is cached, the access speed of subsequent query operations can be significantly improved, the system response delay is effectively reduced, and the user experience is optimized; the system has good expansibility and maintainability, can quickly adapt to various data sources and complex business scenarios, and improves the intelligent level and execution efficiency of the data warm-up operation.

[0117] On the basis of the above scheme, optionally, the query statement determination module 420 includes a query statement generation template acquisition submodule and a query statement acquisition submodule. The query statement generation template acquisition submodule is configured to acquire a pre-set query statement generation template corresponding to the target data set from a data warehouse; wherein the query statement generation template includes a plurality of preset placeholders; the query statement acquisition submodule is configured to replace the plurality of preset placeholders in the query statement generation template with the warm-up rule parameters corresponding to the target data set according to a preset placeholder replacement logic, to obtain a structured query statement.

[0118] On the basis of the above scheme, optionally, the target warm-up data acquisition module 430 includes a first execution submodule. The first execution submodule is configured to perform structure verification on the structured query statement by using a syntax verification engine, generate a standardized query instruction set according to the structured query statement in a case where the verification is passed, and execute the standardized query instruction set.

[0119] Based on the above scheme, optionally, the target preheat data acquisition module 430 comprises a second execution submodule. Among them, the second execution submodule is used for executing the corresponding structured query statement of the data preheat task in response to reaching the preheat execution time of the data preheat task.

[0120] Based on the above scheme, optionally, the device further comprises a query statement updating module. Among them, the query statement updating module is used for determining the next data preheat task after executing the corresponding structured query statement of the data preheat task, updating the structured query statement according to the determined next data preheat task, and executing the updated structured query statement in response to the event that the data preheat task is executed.

[0121] Based on the above scheme, optionally, the device further comprises an identification execution module. Among them, the identification execution module is used for adding the data preheat report to the preheat queue according to the preset timeliness identifier carried by the data preheat report before the data preheat report is acquired from the preheat queue; wherein the preset timeliness identifier is used to indicate the execution priority of the data preheat report.

[0122] Based on the above scheme, optionally, the device further comprises an identification module. Among them, the identification module is used for adding a preset timeliness identifier to the data preheat report in response to an identification adding operation for the data preheat report before the data preheat report is added to the preheat queue according to the preset timeliness identifier carried by the data preheat report.

[0123] Based on the above scheme, optionally, the device further comprises an interactive feedback data acquisition module and a task evaluation module. Among them, the interactive feedback data acquisition module is used for acquiring interactive feedback data of the target preheat data of the executed multiple data cache tasks before or after the data preheat report is acquired from the preheat queue; wherein the interactive feedback data comprises at least one of the accessed state of the historical cached preheat data, the query time information and the interactive operation with the data access object; the task evaluation module is used for determining the task evaluation index of the data cache task according to the interactive feedback data and the task evaluation model; wherein the task evaluation model is obtained by training a machine learning model according to the sample interactive feedback data of the target cache data of the sample cache task and its corresponding expected evaluation index.

[0124] Based on the above scheme, optionally, the preheat rule parameter comprises the time range and / or the subject of the target preheat data.

[0125] The data preheating device provided by the embodiments of the present application can execute the data preheating method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0126] Embodiment five

[0127] Figure 5 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0128] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0129] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a speaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0130] The processor 11 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the data warm-up method.

[0131] In some embodiments, the data warm-up method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data warm-up method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data warm-up method by any other suitable means, such as by means of firmware.

[0132] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0133] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0134] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0136] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0138] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with embodiments of the application. For example, embodiments of the application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-described functions defined in the methods of the embodiments of the application are performed.

[0139] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions of the present application, and this is not limited herein.

[0140] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data pre-warming method, characterized by, The method comprises the following steps: obtaining a data warm-up report from a warm-up queue, and determining a data warm-up task according to the data warm-up report; wherein the data warm-up report records data warm-up information; the data warm-up information comprises warm-up rule parameters and a target data set that needs to be warmed up; determining a structured query statement of the target data set according to the warm-up rule parameters corresponding to the target data set and a query statement generation template corresponding to the target data set that is set in advance; executing the structured query statement to obtain target warm-up data from the target data set, and caching the target warm-up data.

2. The data warm-up method of claim 1, wherein, The method of determining the structured query statement of the target data set according to the warm-up rule parameters corresponding to the target data set and the query statement generation template corresponding to the target data set that is set in advance comprises: obtaining the query statement generation template corresponding to the target data set that is set in advance from a data warehouse; wherein the query statement generation template comprises a plurality of preset placeholders; replacing the plurality of preset placeholders in the query statement generation template with the warm-up rule parameters corresponding to the target data set according to preset placeholder replacement logic to obtain the structured query statement.

3. The data warm-up method of claim 1, wherein, The method of executing the structured query statement comprises: performing structural verification on the structured query statement by a syntax verification engine, and generating a standardized query instruction set according to the structured query statement and executing the standardized query instruction set if the verification is passed.

4. The data warm-up method of claim 1, wherein, The method of executing the structured query statement comprises: executing the structured query statement corresponding to the data warm-up task in response to reaching a warm-up execution time of the data warm-up task.

5. The data warm-up method of claim 4, wherein, After executing the structured query statement corresponding to the data warm-up task, the method further comprises: in response to an event that the data warm-up task is completed, determining a next data warm-up task, updating the structured query statement according to the determined next data warm-up task, and executing the updated structured query statement.

6. The data warm-up method of claim 1, wherein, Before obtaining the data warm-up report from the warm-up queue, the method further comprises: adding the data warm-up report to the warm-up queue according to a preset time limit identifier carried by the data warm-up report; wherein the preset time limit identifier is used to indicate the execution priority of the data warm-up report.

7. The data warm-up method of claim 1, wherein, Before adding the data warm-up report to the warm-up queue according to the preset time limit identifier carried by the data warm-up report, the method further comprises: in response to an identifier adding operation for the data warm-up report, adding a preset time limit identifier to the data warm-up report.

8. The data warm-up method of claim 1, wherein, Before or after obtaining the data warm-up report from the warm-up queue, the method further comprises: obtaining interactive feedback data of target warm-up data of a plurality of data caching tasks that have been executed; wherein the interactive feedback data comprises at least one of an accessed state of historical cached warm-up data, query time information, and an interactive operation with a data access object; The task evaluation index of the data caching task is determined according to the interaction feedback data and a task evaluation model, wherein the task evaluation model is obtained by training a machine learning model according to sample interaction feedback data of target cached data of a sample caching task and corresponding expected evaluation indexes.

9. The data warm-up method of claim 1, wherein, The preheating rule parameter includes a time range and / or a subject of the target preheating data.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the data preheating method according to any one of claims 1-9. The computer program, when executed by a processor, implements the data preheating method according to any one of claims 1-9.

Images