Predictive method and computing device for task execution

Abstract

This application discloses a method and computing device for predicting task execution, relating to the field of computing device technology. The method includes: receiving a target task, which is a data integration task targeting at least one first data source; executing the target task including extracting data from the at least one first data source; in response to the target task, sampling the data from the at least one first data source to obtain sampled data; acquiring the data volume and target information of the sampled data, wherein the target information includes quality information of the sampled data and / or performance indicators generated by the computing device during the sampling process; and predicting the execution result of the target task based on the data volume and target information, the execution result indicating whether the target task was successfully executed. This allows for timely and efficient determination of task execution status.

Description

Technical Field

[0001] This application relates to the field of computing device technology, and in particular to a method for predicting task execution and a computing device. Background Technology

[0002] Data integration tools (such as distributed data integration platforms) are used to execute data integration tasks (such as offline or real-time data integration tasks). During the execution of these tasks, verification is typically required to determine their success. However, current technologies require verification based on data from the task execution logs or the entire integrated data set after the task is completed. This approach is inefficient and resource-intensive. Furthermore, since the entire task has already been executed during verification, re-execution is necessary if the task fails, increasing operational costs and the risk of business interruption. Therefore, a solution that can determine task execution status promptly and efficiently is urgently needed. Summary of the Invention

[0003] This application provides a method and computing device for predicting task execution, thereby determining the task execution status in a timely and efficient manner.

[0004] In a first aspect, embodiments of this application provide a method for predicting task execution, applied to a computing device. The method includes: receiving a target task, which is a data integration task targeting at least one first data source; executing the target task including extracting data from at least one first data source; in response to the target task, sampling the data from at least one first data source to obtain sampled data; acquiring the data volume and target information of the sampled data, wherein the target information includes quality information of the sampled data and / or performance indicators generated by the computing device during the sampling process; and predicting the execution result of the target task based on the data volume and target information, the execution result indicating whether the target task was successfully executed.

[0005] Thus, since executing the target task requires extracting data from at least one primary data source, and the sampling task (such as sampling data from the primary data source) also requires extracting data from at least one primary data source, the process of executing the sampling task can be considered a representative process of executing the target task. Furthermore, since the sampled data and the full data are highly homologous, the execution result of extracting target data included in the target task can be predicted by the amount of sampled data and the target information. This enables rapid verification of the target task's execution result, allowing for early prediction of whether the target task has been successfully executed without fully executing the full data integration task, thus improving the verification efficiency of task execution. In addition, it avoids the enormous computational and storage resource consumption caused by full execution, and can identify potential problems such as data defects and / or performance bottlenecks early on. This solves the problems of ineffective execution and increased maintenance costs caused by delayed problem detection in existing technologies, enabling timely and efficient determination of task execution status, improving the success rate, reliability, and overall maintenance efficiency of data integration tasks.

[0006] In one possible implementation, predicting the execution result of the target task based on the amount of sampled data and target information includes: determining a first detection result based on the amount of sampled data; wherein the first detection result indicates whether the ratio of the amount of sampled data corresponding to each of the at least one first data source to a preset amount of data is within the range of the sampling ratio corresponding to each first data source; determining a second detection result based on the target information; wherein, if the target information includes quality information of the sampled data, the second detection result indicates whether the quality of the sampled data meets the quality requirements; if the target information includes operating performance indicators generated by the computing device during the sampling process, the second detection result indicates whether the operating performance of the computing device meets the performance requirements. Based on the first detection result and the second detection result, the execution result of the target task is predicted.

[0007] Thus, by determining the first and second detection results, accurate predictions of the target task's execution outcome can be achieved, comprehensively covering key influencing factors before the target task's execution. The reliability of the prediction results can be improved through multi-dimensional collaborative verification, and detection dimensions can be flexibly selected according to actual scenario requirements, balancing the comprehensiveness of prediction with the flexibility of application, thereby improving the success rate of target task execution and resource utilization efficiency.

[0008] In one possible implementation, obtaining the target information includes: obtaining the target information when the first detection result indicates that the ratio of the amount of sampled data corresponding to each first data source in at least one first data source to the preset amount of data is within the range of the sampling ratio corresponding to each first data source.

[0009] In this way, when the amount of sampled data meets the standard, the acquisition of target information can ensure the validity of the analysis sample through the prior data quantity verification. This not only avoids the distortion of quality assessment results caused by insufficient data quantity and ensures the accuracy of the analysis results, but also avoids invalid detection through the prior data quantity verification, thus ensuring the validity and reference value of the target information of the sampled data from the source.

[0010] In one possible implementation, the performance metrics include at least one of latency, throughput, computational resource utilization efficiency, or stability; stability is characterized by the probability of correct data in the sampled data.

[0011] In this way, by using latency, throughput, computing resource utilization efficiency, and stability as a multi-dimensional evaluation system for operational performance, a comprehensive and accurate quantification of the operational status of the sampling process is achieved.

[0012] In one possible implementation, if the target information includes operational performance indicators generated by the computing device during the sampling process, then determining a second detection result based on the target information includes: if the operational performance indicator includes the latency, then determining a score corresponding to the latency based on the deviation between the latency and a preset latency; determining a second detection result based on the score corresponding to the latency; or, if the operational performance indicator includes throughput, then determining a score corresponding to the throughput based on the ratio of the throughput to a preset throughput; determining a second detection result based on the score corresponding to the throughput; or, if the operational performance indicator includes computing resource utilization efficiency, then determining the performance consumption level of computing resource utilization efficiency, determining a score corresponding to computing resource utilization efficiency based on the deviation between the performance consumption level of computing resource utilization efficiency and a preset performance benchmark of the computing device; determining a second detection result based on the score corresponding to computing resource utilization efficiency; or, if the operational performance indicator includes stability, then determining a score corresponding to stability based on the probability of correct data in the sampled data; determining a second detection result based on the score corresponding to stability.

[0013] In this way, by quantifying one or more of the dimensions such as latency, throughput, computing resource utilization efficiency, or stability into corresponding scores, and comprehensively determining the second detection result based on the single-dimensional or multi-dimensional scores, a comprehensive and accurate quantitative evaluation of the running performance can be achieved, thereby improving the reliability of the execution results of the set target task.

[0014] In one possible implementation, obtaining the operational performance metrics generated by the computing device during the sampling process includes: determining the operational performance metrics based on the operational data in the root directory file of the task prediction tool, where the root directory file of the task prediction tool is used to store the operational data, which refers to the data generated by the computing device during the execution of the sampling task.

[0015] In one possible implementation, quality information is characterized by at least one of the following: first information, second information, or third information. The first information includes the proportion of non-empty fields and / or record completeness rate; the second information includes data type consistency and / or value range matching degree; and the third information includes business matching degree and / or data accuracy rate. Specifically, the proportion of non-empty fields indicates the percentage of non-empty fields in the sampled data; the record completeness rate indicates the proportion of sampled data that is completely recorded; the data type consistency indicates the consistency between the data type in the sampled data and the data type in the first data source; the value range matching degree indicates the degree of matching between the value range of the sampled data and the value range of the data in the first data source; the business matching degree indicates the degree of matching between the business information indicated by the sampled data and the business information indicated by the data in the first data source; and the data accuracy rate indicates the accuracy rate of the sampled data.

[0016] In this way, a precise quantitative assessment of the quality of the sampled data is achieved, providing a comprehensive and reliable basis for predicting the execution results of the target task, and improving the accuracy of the predicted execution results.

[0017] In one possible implementation, if the target information includes quality information of the sampled data, then determining a second detection result based on the target information includes: obtaining a score for each piece of information based on the scoring coefficient of each piece of information and the corresponding scoring coefficient; wherein the scoring coefficient of the first information is used to indicate the degree of influence of the first information on the integrity of the sampled data, and the score of the first information is used to indicate the integrity of the sampled data; the scoring coefficient of the second information is used to indicate the degree of influence of the second information on the consistency of the sampled data with data in at least one first data source, and the score of the second information is used to indicate the consistency of the sampled data with data in at least one first data source; the scoring coefficient of the third information is used to indicate the degree of influence of the third information on the accuracy of the sampled data, and the score of the third information is used to indicate the accuracy of the sampled data; and determining a second detection result based on the scores of each of the at least one piece of information.

[0018] In this way, by scoring the sampled data in multiple dimensions, the second test result can be determined, which can accurately reflect the quality level of the sampled data in one or more dimensions such as completeness, consistency or accuracy in a quantitative way, and achieve an objective and comprehensive evaluation of the quality information of the sampled data.

[0019] In one possible implementation, based on a first detection result and a second detection result, predicting the execution result of the target task includes: if the first detection result indicates that the ratio of the sampled data volume corresponding to each first data source to the preset data volume is within its corresponding sampling ratio, and the second indicator result indicates the first result, the predicted execution result indicates successful execution of the target task; the first result indicates that the quality of the sampled data meets the quality requirements and / or the operating performance of the computing device meets the performance requirements. If the first detection result indicates that the ratio of the sampled data volume corresponding to a first data source to the preset data volume is not within its sampling ratio, or the second indicator result indicates the second result, the predicted execution result indicates that the target task cannot be successfully executed. The second result indicates that the quality of the sampled data does not meet the quality requirements and / or the operating performance of the computing device does not meet the performance requirements.

[0020] Thus, by directly mapping the execution result of the target task to whether the sampled data volume meets the standard, a simple single-dimensional verification logic is used to quickly predict the feasibility of the task, reducing the computational overhead of the prediction process and adapting to rapid verification needs. By establishing a prediction logic that directly correlates the second detection result of the sampled data's quality dimension with the execution result of the target task, the success or failure of the target task can be accurately predicted based on whether the sampled data meets quality requirements. This allows for rapid identification of task failure risks due to sampled data quality defects before the task is formally executed, effectively avoiding the waste of computing power, time, and human resources caused by executing the target task based on unqualified sampled data. It also provides a reliable pre-judgment basis for the execution decision of the target task, improving the overall efficiency and success rate of the target task execution. And / or, by directly correlating the second detection result with the execution result of the target task, using whether the running performance meets performance requirements as a prediction basis, the risk of task failure due to insufficient performance can be accurately identified before the target task is executed, avoiding the waste of resources caused by ineffective execution, providing reliable support for task execution decisions, and thus improving the efficiency and success rate of the target task execution. In one possible implementation, in response to a target task, sampling data from at least one first data source to obtain sampled data includes: in response to the target task, sampling data from at least one first data source according to a sampling strategy to obtain sampled data, wherein the sampling strategy includes a sampling method and / or sampling ratio corresponding to at least one first data source. Specifically, if the first data source includes a relational database, the sampling method corresponding to the first data source includes index-based hierarchical sampling; if the first data source includes a streaming data source, the sampling method corresponding to the first data source includes time-window sampling; if the first data source includes an API-based data source, the sampling method corresponding to the first data source includes parameterized request sampling.

[0021] In this way, the sampling process is standardized and controllable, which not only ensures the representativeness of the sampled data, but also allows for flexible adaptation of the sampling strategy according to the characteristics of the data source.

[0022] In one possible implementation, the method further includes: when the first detection result indicates that the proportion of the amount of sampled data corresponding to the first data source in at least one first data source to the preset amount of data is not within the range of the sampling ratio corresponding to the first data source, adjusting the sampling ratio corresponding to the first data source based on preset information to obtain an updated sampling ratio; wherein, the preset information includes the amount of data in the first data source; the preset information also includes the proportion of null data in the sampled data corresponding to the first data source and / or the business priority of the sampled data; and sampling the data in the first data source based on the updated sampling ratio.

[0023] In this way, by introducing preset information to dynamically adjust the sampling ratio, the sampling ratio can be adapted to data sources with different data sizes, ensuring that the amount of sampled data meets the evaluation requirements while avoiding resource waste or insufficient sample representativeness caused by excessive or insufficient data volume.

[0024] In one possible implementation, adjusting the sampling ratio corresponding to the first data source based on preset information to obtain the updated sampling ratio includes: adjusting the sampling ratio corresponding to the first data source based on the coefficient corresponding to the data volume in the first data source and the current sampling ratio corresponding to the first data source to obtain the updated sampling ratio.

[0025] In this way, by adjusting the current sampling ratio and the amount of sampled data by merging the data volume of the data source, the amount of sampled data can be adapted to the actual data scale of the first data source.

[0026] In one possible implementation, before receiving the target task, the method further includes: obtaining an initial detection result, which indicates the availability of the target configuration file, which is used to configure the rules for executing the target task; and outputting indication information, wherein the indication information indicates that the target task should be initiated if the target configuration file is available.

[0027] Thus, if the target configuration file is available, an instruction message is output to indicate that the target task should be initiated. This helps to intercept potential risks such as configuration errors, abnormal data source connections, and insufficient resource adaptation, and avoids invalid execution, interruption, or data pollution of the full data integration task due to defects in basic conditions. The feasibility of task execution can be verified from the perspective of the configuration file without waiting for the full data integration task to start.

[0028] In one possible implementation, the target task is used to instruct the synchronization of target data from at least one first data source to at least one second data source; obtaining an initial detection result includes: detecting whether both at least one first data source and at least one second data source are accessible; detecting whether the semantic content of the target configuration file conforms to the syntax rules; if both at least one first data source and at least one second data source are accessible and the semantic content of the target configuration file conforms to the syntax rules, determining that the initial detection result indicates that the target configuration file is available; if at least one first data source is inaccessible, at least one second data source is inaccessible, or the semantic content of the target configuration file does not conform to at least one of the syntax rules, determining that the initial detection result indicates that the target configuration file is unavailable.

[0029] In this way, by conducting multi-dimensional availability testing on the target configuration file, checking the correctness (e.g., by verifying that the semantic content of the target configuration file conforms to the syntax rules) and validity (e.g., by verifying that at least one first data source and / or at least one second data source can be accessed), pre-verification of the data integration task is achieved. This avoids the problem of subsequent full task execution being invalid or interrupted due to data source connection failure or configuration syntax errors from the source. It can reduce the waste of computing and storage resources and improve the success rate of data integration task startup and overall execution efficiency.

[0030] In one possible implementation, a first configuration interface is displayed to allow the user to configure at least one first data source. The first configuration interface is used for the user to select at least one first data source type. After the user selects at least one first data source type, the first configuration interface is also used to display configuration items corresponding to the first data source type, allowing the user to fill in the parameters corresponding to the configuration items. The first configuration interface is also used to trigger a detection of whether the first data source can be accessed based on user actions.

[0031] In this way, the first configuration interface enables integrated operation of at least one first data source type selection, parameter configuration, and access detection, improving the efficiency and accuracy of data source configuration.

[0032] In one possible implementation, a second configuration interface is displayed to allow the user to configure syntax rules. This second configuration interface is used for user-configured syntax rules.

[0033] In this way, users can configure syntax rules in the second configuration interface and verify whether the syntax content of the configuration file conforms to the syntax rules.

[0034] In one possible implementation, a third configuration interface is displayed to allow the user to configure the sampling strategy. The third configuration interface is used for user configuration of the sampling strategy.

[0035] In this way, data from the first data source can be sampled based on the user's configuration, so that the sampling process and results meet the user's actual needs.

[0036] In one possible implementation, a fourth configuration interface is displayed to allow the user to configure alarm rules. Alarm rules may include alarm triggering conditions, alarm levels and / or alarm methods, and handling suggestions.

[0037] In this way, the alarm rules can be flexibly configured, which can meet the personalized alarm needs of different business scenarios.

[0038] In one possible implementation, the method further includes: outputting a diagnostic report. The diagnostic report includes: the execution results.

[0039] In this way, the results of all the above nodes can be displayed centrally, and whether the predicted execution result indicates success or failure, they are all summarized in this node. This provides technical personnel with comprehensive and intuitive task verification basis, and also facilitates the rapid location of problem nodes, improving the efficiency of problem investigation and task optimization.

[0040] In some embodiments, the method further includes: performing a target task.

[0041] In this way, by executing the target task and the sampling verification process in parallel, task execution can be started without waiting for the verification results, which can shorten the overall task execution cycle and improve the execution efficiency of data integration tasks.

[0042] In some embodiments, the method further includes: if the execution result indicates successful execution of the target task, continuing to execute the target task; or if the execution result indicates that the target task cannot be executed, interrupting the execution of the target task.

[0043] In this way, the execution and interruption of the target task are dynamically controlled based on the execution results. When the execution is predicted to fail, the execution is terminated in time, avoiding unnecessary resource consumption. At the same time, the smooth progress of successful tasks is ensured, achieving a dual optimization of resource utilization and task execution reliability.

[0044] Secondly, embodiments of this application provide a computing device, including: a memory and a processor. The memory is used to store program instructions. The processor is used to execute the program instructions, causing the computing device to perform a task execution prediction method as described in the first aspect or any possible implementation of the first aspect.

[0045] Thirdly, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed on a computing device, cause the computing device to perform a task execution prediction method as described in the first aspect or any possible implementation thereof.

[0046] Fourthly, a computer program product is provided, the computer program product including computer execution instructions, which, when executed on a computing device, cause the computing device to perform a task execution prediction method as described in the first aspect or any possible implementation of the first aspect.

[0047] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here.

[0048] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the operation of a stream processing engine provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of another computing device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of another computing device provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating a process for configuring a task prediction tool, as provided in an embodiment of this application. Figure 6 A schematic diagram of a first configuration interface provided in an embodiment of this application; Figure 7 A schematic diagram of another first configuration interface provided in an embodiment of this application; Figure 8 A schematic diagram of another first configuration interface provided in an embodiment of this application; Figure 9 A schematic diagram of another first configuration interface provided in an embodiment of this application; Figure 10 A flowchart illustrating a task execution prediction method provided in an embodiment of this application; Figure 11 A flowchart illustrating the prediction of the execution result of a target task, provided as an embodiment of this application; Figure 12 A flowchart illustrating another task execution prediction method provided in this application embodiment; Figure 13 This is a flowchart illustrating another task execution prediction method provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0051] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0052] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0053] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0054] The following provides an exemplary description of the application scenarios of the embodiments of this application.

[0055] This application primarily applies to scenarios where data integration tools execute data integration tasks. In this scenario, the data integration tool needs to handle data integration tasks from multiple heterogeneous data sources (such as relational databases, streaming data, and Hypertext Transfer Protocol Application Programming Interface (HTTPAPI) data sources), performing data integration tasks offline or in real-time. To ensure the accuracy and stability of the data integration tasks, the execution results need to be verified to confirm whether the tasks have been successfully executed as expected.

[0056] The core process of executing a data integration task includes extracting data from at least one primary data source. Taking the data integration task instructing the synchronization of target data from at least one primary data source to at least one secondary data source as an example, the primary data source is the source data source (also called the source-end data source, representing the data storage medium sending the data to be synchronized), and the secondary data source is the target data source (also called the target-end data source, representing the data storage medium receiving the data to be synchronized). The data integration tool, based on preset task execution rules, filters and extracts the target data that meets the requirements from at least one primary data source and transmits the target data to at least one secondary data source. For example, before transmitting the target data to at least one secondary data source, the target data can be standardized (e.g., format conversion, data cleaning, field mapping, etc.). Afterward, the processed target data is transmitted to at least one secondary data source, achieving data alignment between the two ends. During this process, it is necessary to ensure that the data extraction process from the source data source (such as at least one primary data source) is reliable. Therefore, based on the execution status of the data extraction process from the source data source, it is possible to predict whether the data integration task will be executed successfully as expected.

[0057] The system architecture of the embodiments of this application will be described below as an example.

[0058] like Figure 1 As shown, this application provides a computing device.

[0059] The computing device is used to receive a target task, which is a data integration task targeting at least one first data source; executing the target task includes extracting data from at least one first data source. The computing device is also used to sample the data from at least one first data source in response to the target task, obtaining sampled data. The computing device is further used to acquire the data volume and target information of the sampled data, wherein the target information includes quality information of the sampled data and / or performance metrics generated by the computing device during the sampling process. Furthermore, the computing device is used to predict the execution result of the target task based on the data volume and target information of the sampled data, the execution result indicating whether the target task was successfully executed.

[0060] In some embodiments, the computing device is deployed with data integration tools. These tools are capable of interfacing with multiple heterogeneous data sources and support various task types, such as offline batch synchronization and real-time streaming. Specifically, for example... Figure 1 As shown, the computing device executes the target task through data integration tools.

[0061] In some embodiments, such as Figure 1 As shown, the computing device is equipped with a task prediction tool. The computing device uses this tool to predict the execution result of the target task. In one implementation, the task prediction tool is deployed as a plug-in within the data integration tool.

[0062] In one implementation, such as Figure 1 As shown, the task prediction tool includes a sampling configurator, which is used to configure a sampling strategy. The computing device samples data from at least one first data source based on the sampling strategy to obtain sampled data.

[0063] In one implementation, such as Figure 1 As shown, the task prediction tool includes a result validator, and the computing device verifies a first detection result based on the result validator. The first detection result is used to indicate whether the ratio of the sampled data volume corresponding to each of the at least one first data source to a preset data volume is within the range of the sampling ratio corresponding to each first data source.

[0064] In one implementation, if the target information includes quality information of the sampled data, the second detection result is used to indicate whether the quality of the sampled data meets the quality requirements. Optionally, the computing device also verifies the second detection result based on a result verifier.

[0065] In one implementation, if the target information includes performance metrics generated by the computing device during the sampling process, the second detection result is used to indicate whether the computing device's performance meets the performance requirements. For example... Figure 1As shown, the task prediction tool includes a performance monitor, and the computing device uses the performance monitor to monitor the performance indicators generated by the computing device during the sampling process and determines the second detection result.

[0066] Understandably, if the second detection result is used to indicate whether the quality of the sampled data meets the quality requirements and whether the operating performance of the computing device meets the performance requirements, then the second detection result is jointly determined by the operating monitor and the result verifier.

[0067] Optional, such as Figure 1 As shown, the task prediction tool also includes a configuration validator. The computing device determines an initial detection result based on the configuration validator. This initial detection result indicates whether the target configuration file (the configuration file corresponding to the target task) is available. Further, if the target configuration file is available, the computing device receives the target task.

[0068] Continue as Figure 1 As shown, the task prediction tool also includes a stream processing engine. For example... Figure 2 As shown, the computing device performs one or more of the following operations based on sampled data from at least one first data source via a stream processing engine: Operation 1: Data volume statistics. Specifically, count the data volume of the sampled data corresponding to each of the at least one first data source, and output the statistical results to the result validator.

[0069] Operation 2: Target Information Acquisition. Specifically, if the target information includes quality information of the sampled data, extract the quality information from the sampled data and output it to the result validator. Details of the quality information are provided below and will not be elaborated here. If the target information includes operational performance indicators generated by the computing device during the sampling process, obtain the operational performance indicators from the operational data in the root directory file of the task prediction tool and output the operational performance indicators to the operation monitor. Details of the root directory file are provided below and will not be elaborated here.

[0070] Operation 3: Abnormal data alarm. Specifically, abnormal data includes one or more of the following: the initial detection result indicates that the target configuration file is unavailable; the first detection result indicates that at least one first data source has a data volume corresponding to the first data source whose ratio of the sampled data volume to the preset data volume is not within the range of the sampling ratio corresponding to the first data source; the second result indicates that the quality of the sampled data does not meet the instruction requirements; or the second detection result indicates that the operating performance of the computing device does not meet the performance requirements.

[0071] Operation 4: Optimize configuration information. For details on the configuration information, please refer to one or more configuration interfaces in the following section on the specific implementation of configuring the task prediction tool. These details will not be elaborated here.

[0072] In some embodiments, such as Figure 3 As shown, the computing device includes an application presentation layer, which displays one or more of the following interfaces: a front-end configuration interface, a monitoring interface, a report generation interface, or a system management interface. The front-end configuration interface is used to configure data sources, syntax rules, sampling strategies, alarm rules, etc., allowing users to configure relevant parameters and thus configure the logic for the execution results of the predicted target task based on the task prediction tool. Details will be provided later. The monitoring interface displays the operational performance indicators generated by the computing device during the sampling process using data from at least one first data source. The report generation interface displays diagnostic reports (including execution results). The system management interface displays system-level operation and management information related to the task prediction tool, enabling users to manage the task prediction tool.

[0073] In some embodiments, continue as follows Figure 3 As shown, the computing device also includes a core service layer. The core service layer can perform several functions, including sampling data from at least one primary data source to obtain sampled data (i.e., sampling management), which can be based on, for example... Figure 1 The sampling configurator shown is implemented; the second detection result (i.e., health assessment) can be verified based on, for example... Figure 1 The result validator and runtime monitor are implemented to predict the execution results of the target task (i.e., predictive analysis). Optionally, the core service layer can also implement one or more of the following functions: verifying the target configuration file (i.e., configuration verification), which can be based on, for example... Figure 1 The configuration validator shown implements: executing the target task issued by the user (i.e., task execution); monitoring the operating performance indicators of the computing device during the sampling process (i.e., indicator monitoring), which can be based on, for example... Figure 1 The operation monitor shown implements: alarms based on alarm rules (i.e., alarm management); report generation (such as generating diagnostic reports), etc.

[0074] Optionally, at least one first data source can be deployed in the aforementioned computing device, or it can be deployed in other devices connected to the computing device's network (such as other computing devices, cloud storage nodes, third-party system hosts, etc.). Optionally, at least one second data source can be deployed in the aforementioned computing device, or it can be deployed in other devices connected to the computing device's network (such as other computing devices, cloud storage nodes, third-party system hosts, etc.). At least one first data source and at least one second data source can be deployed on the same device, or they can be deployed on different devices.

[0075] In some embodiments, the target task instructs the synchronization of target data from at least one first data source to at least one second data source. In this application embodiment, the example is illustrated by deploying at least one first data source and / or at least one second data source on another device. For example, at least one first data source (e.g., first data source 1, first data source 2) and at least one second data source (e.g., second data source 1, second data source 2) may reside in an external data source layer (e.g., on another device), continuing as follows... Figure 3 As shown, the type of data source (such as the first data source and / or the second data source) may include one or more of the following: relational database, streaming data source, interface-based data source, message queue, search server data source, or data warehouse.

[0076] In some embodiments, continue as follows Figure 3 As shown, the computing device also includes a unified data access layer. The different data source types mentioned above can be connected to the computing device through different adapters within this unified data access layer. For example, adapters may include one or more of the following: a relational database adapter corresponding to a relational database; a streaming data source adapter corresponding to a streaming data source; an interface-based data source adapter corresponding to an interface-based data source; a message queue adapter corresponding to a message queue (such as a remote dictionary server, Redis); a search server data source adapter corresponding to a search server data source; or a data warehouse adapter corresponding to a data warehouse.

[0077] In other embodiments, the computing device is communicatively connected to the terminal, which is responsible for displaying the front-end configuration interface. Users complete the relevant parameter configuration through the terminal interface, and these configurations are synchronized to the computing device to realize the above-mentioned functions and control the execution of the target task.

[0078] In this embodiment, the computing device can be a server. The server can be a single physical server or logical server, or it can be composed of two or more physical servers or logical servers that share different responsibilities, working together to achieve various server functions such as data processing and service provision.

[0079] In terms of hardware form, servers can be blade servers, high-density servers, rack servers, or tower servers, which are suitable for different application scenarios such as high-density cluster deployment in data centers and small enterprise server rooms.

[0080] like Figure 4As shown, this application embodiment provides another computing device 500. The computing device 500 includes a processor 510 and a memory 520 for storing processor-executable instructions. When the processor 510 is configured to execute instructions, the computing device 500 performs the various functions described above. For example, the processor 510 executes instructions such as... Figure 1 The data integration tools and / or task prediction tools shown (including such as Figure 1 (Examples include the sampling configurator, configuration validator, runtime monitor, result validator, stream processing engine, etc.). For example, implemented by processor 510. Figure 3 The diagram shows the various functions of the application presentation layer, core service layer, and unified data access layer.

[0081] Figure 4 The computing device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0082] The computing device 500 is manifested in the form of a general-purpose computing device. The components of the computing device 500 may include, but are not limited to: one or more processors 510, memory 520, communication bus 540 connecting different system components (including memory 520 and processor 510), and communication interface 530.

[0083] Communication bus 540 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0084] Memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computing device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 4Not shown, a disk drive may be provided for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and a removable non-volatile optical disk (e.g., a compact disc read-only memory, CD). ROM, Digital Video Disc Read-Only Memory (DVD) An optical disc drive that reads and writes to ROM or other optical media. In these cases, each drive can be connected to the communication bus 540 through one or more data media interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0085] A program / utility having a set (at least one) of program modules can be stored in memory 520. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0086] The computing device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the computing device, and / or with any device that enables the computing device to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through the communication interface 530. Furthermore, the computing device 500 can also communicate through a network adapter (… Figure 4 (Not shown) communicates with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the computing device via the communication bus 540. It should be understood that, although... Figure 4 As not shown, the computing device 500 may be used with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (RAID) systems, tape drives, and data backup storage systems.

[0087] Processor 510 executes various functional applications and data processing by running programs stored in memory 520, such as implementing the functions provided in the embodiments of this application as described below. Figure 5 and / or Figures 10-13 The method for predicting task execution is shown.

[0088] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the computing device 500. In other embodiments of this application, the computing device 500 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0089] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0090] For ease of understanding, the following description, in conjunction with the aforementioned computing device and accompanying drawings, provides an exemplary description of the task execution prediction method provided in the embodiments of this application.

[0091] Part One, such as Figures 5-9 As shown, a specific implementation method for configuring a task prediction tool is provided.

[0092] Part Two, such as Figures 10-13 As shown, a specific implementation of a task execution prediction method is provided.

[0093] Part One, such as Figure 5 As shown, this application embodiment provides a specific implementation method for configuring a task prediction tool. For example, it includes the following steps: S501-S504, which can be executed by the processor of a computing device, such as a CPU.

[0094] In this embodiment of the application, the execution result of the data integration tool is predicted based on the task prediction tool. The data integration tool is described below: The tasks performed by the data integration tool may include extracting one or more source data sources. Further, the tasks performed by the data integration tool are used to instruct the synchronization of target data from at least one source data source to at least one target data source. Different tasks may correspond to different source and / or target data sources. The data integration tool is configured with configuration templates, which are used to configure rules for executing tasks (such as target tasks). For example, the rules specify the core information of the source and target data sources, as well as task resources. Task resources refer to the hardware resources (such as allocated CPU, memory, etc.) and operating parameters (such as 8-core CPU, 16GB memory, etc.) allocated for the normal operation of the data integration task. The configuration template includes configuration items; by filling in the parameters corresponding to each configuration item in the configuration template, the data integration tool can execute the tasks issued by the user. For example, the configuration items include one or more of the following: source configuration items, target configuration items, and task resource configuration items. The source-side configuration items are used to configure the core information of the source data source, which may include: source data source type, source data source Internet Protocol (IP), port number, and synchronized data range (such as data table name, data partition, and data time interval). Optionally, it may also include: username and password for accessing the source data source, connection protocol, and connection string format. The target-side configuration items are used to configure the core information of the target data source, which may include: target data source type, target data source IP, port number, etc. Optionally, it may also include username and password for accessing the target data source. The task resource configuration items are used to configure task resources, which may include allocated memory, CPU, disk space, and network bandwidth quota.

[0095] The data integration tool executes tasks based on the user's configuration of the aforementioned configuration template. Correspondingly, the task prediction tool displays multiple configuration interfaces. By configuring the task prediction tool through these interfaces, the user can sample data from the source data sources included in the tasks executed by the data integration tool. Based on the sampled data and the performance metrics generated by the computing devices during the sampling process, the task prediction tool predicts the execution results of the tasks. The following explains how to configure the task prediction tool: S501, displays the first configuration interface to allow the user to configure at least one first data source.

[0096] like Figure 6As shown, the first configuration interface is used by the user to select at least one first data source type, such as first data source 1 being a relational database, first data source 2 being a streaming data source, and first data source 3 being an API data source. For example, if the task executed by the task prediction tool only needs to use the first data source, then only at least one first data source type needs to be selected, such as if the target task includes extracting data from at least one first data source. Optionally, the first configuration interface can also be used by the user to configure a second data source. The first configuration interface is used by the user to select at least one second data source type, where second data source 1 is a relational database, second data source 2 is an API data source, and second data source 3 is a streaming data source. As another example, if the target task requires at least one first data source and at least one second data source, then at least one first data source type and at least one second data source type need to be selected, such as if the target task instructs to synchronize target data from at least one first data source to at least one second data source.

[0097] like Figure 7 As shown, the first configuration interface is also used to display at least one configuration item for the first data source type and configuration items for task resource configuration after the user selects at least one first data source type, so that the user can fill in the parameters corresponding to the configuration items (e.g., source data source IP and port number, target data source IP and port number, CPU, etc.). Optionally, the first configuration interface is also used to display at least one configuration item for the second data source type after the user selects at least one second data source type. For example, the configuration items for the first data source type and / or the second data source type are the same as the configuration items for the source configuration and / or the target configuration mentioned above. It can be understood that after the user fills in the parameters corresponding to the configuration items, the computing device can be triggered to automatically write the parameters corresponding to the configuration items filled in by the user into the configuration template to form a configuration file. The configuration file is used to configure the triggering conditions for triggering the execution of the task, the rules for executing the task, etc. When the configuration file is used to indicate the execution of the target task, the configuration file is the target configuration file.

[0098] Optionally, after receiving the configuration parameters entered by the user, the computing device identifies the data source corresponding to the configuration parameters. If the identified data source matches the data source entered by the user, the user-configured configuration parameters are written to the configuration file. If the identified data source does not match the data source entered by the user, it indicates that the syntax of the configuration file does not conform to the syntax rules. For example, the syntax rules indicate that the data source IP is represented as "data source IP: 198". However, the syntax in the configuration file is identified as "data source IP, 198". Based on this, the data source identified by the computing device may not match the data source entered by the user. In this case, the syntax of the configuration file does not conform to the syntax rules, and the user can be prompted. If no data source is identified based on the configuration parameters entered by the user, it indicates that the identified data source is an unknown data source. This data source may be an invalid data source type or data source entered arbitrarily by the user, and the configuration file can be prompted that the syntax of the configuration file does not conform to the syntax rules.

[0099] For example, when identifying the data source corresponding to the parameters of a configuration item, the computing device can use a decision tree classification algorithm to determine the data source. For instance, when the port number is 3306 and the protocol type is relational database, it is identified as a relational database; when the port number is 9092 and the protocol type is streaming data source, it is identified as a streaming data source; when the protocol type is HTTP or HTTPS, it is identified as an HTTP API data source, and so on; other cases are identified as unknown data source types.

[0100] In addition, the first configuration interface is also used to trigger a detection based on user operations to determine whether at least one first data source can be accessed. That is, whether the data integration tool can access at least one first data source, and displays the detection result indicating whether at least one first data source can be accessed. Optionally, the first configuration interface is also used to trigger a detection based on user operations to determine whether at least one second data source can be accessed. That is, whether the data integration tool can access at least one second data source, and displays the detection result indicating whether at least one second data source can be accessed. For example, as... Figure 8 As shown, the first configuration interface displays a test button. The user operation described above can be clicking this test button. When the user clicks the test button, the computing device detects whether at least one first data source and / or at least one second data source can be accessed. Figure 9 As shown, the computing device is also used to display the detection results in response to the above user operation. For example, the detection results are: the first data source 1 can be accessed, the second data source 1 can be accessed, and the semantic content of the target configuration file conforms to the syntax rules.

[0101] The above-mentioned syntax rules can be the syntax standards of this field. For example, the data source IP must conform to a regular expression, such as `^((25 [0-5]; the port number's numerical range: 1-65535; the database account's minimum length is 6 characters, and the maximum length is 32 characters, supporting combinations of letters, numbers, and underscores, etc. The syntax rules can also be user-defined. In some embodiments, users can configure the syntax rules in a second configuration interface.

[0102] Any one or more of the above syntax rules can be configured by the user in the second configuration interface, as detailed in step S502. Of course, syntax rules can also be predefined or configured in other ways.

[0103] S502 displays a second configuration interface to allow the user to configure syntax rules.

[0104] The second configuration interface is used for users to configure syntax rules. For example, user-defined rules include: the data source key is required and cannot be an empty string; the data synchronization frequency can only be a positive integer in seconds; and the data source type identifier uses an enumeration constraint, with possible values ​​such as relational database, streaming data source, API data source, file data source, etc. These can be set according to the user's actual needs.

[0105] In this way, users can configure syntax rules in the second configuration interface and verify whether the syntax content of the configuration file conforms to the syntax rules.

[0106] S503 displays a third configuration interface to allow the user to configure the sampling strategy.

[0107] The third configuration interface is used by users to configure sampling strategies. Sampling strategies include at least one sampling method and / or sampling ratio corresponding to a first data source. For specific implementation details and related descriptions of the sampling process, please refer to Part Two below.

[0108] S504 displays the fourth configuration interface, allowing users to configure alarm rules.

[0109] For example, alarm rules can include alarm triggering conditions. They can also include alarm levels and / or alarm methods and handling suggestions, etc. Details will be provided later; we will not elaborate further here.

[0110] Optionally, the execution order of steps S501-S504 is not limited, and any one or more steps in S501-S504 can be optional. In actual implementation, the above contents can also be configured in other ways, and the contents configured in any of the above steps can also be predefined.

[0111] For example, the first, second, third, and fourth configuration interfaces described above can be collectively referred to as the front-end configuration interfaces. This allows users to configure the task prediction tool to respond according to the configuration. In practice, any one of the first, second, third, and fourth configuration interfaces can be implemented by a single configuration interface or by multiple configuration interfaces working together; this application does not limit this approach.

[0112] Part Two, such as Figure 10 As shown, this application provides a method for predicting task execution. This method is applied to a computing device, specifically executed by the device's processor (such as a CPU), and may include, for example, the following steps S1001-S1006: S1001, Obtain initial detection results. The initial detection results indicate the availability of the target configuration file. The target configuration file is used to configure the rules for executing the target task. The target task is a data integration task targeting at least one first data source. Executing the target task includes extracting data from at least one first data source.

[0113] In one implementation, the availability of the target configuration file can specifically include: whether at least one first data source can be accessed; and / or whether the syntax of the target configuration file conforms to syntax rules. Whether at least one first data source can be accessed can be understood as whether the computing device can access at least one first data source, and further, whether the data integration tools on the computing device can access that at least one first data source.

[0114] For example, if the availability of a target configuration file specifically includes whether at least one first data source can be accessed, then if at least one first data source can be accessed, the initial detection result indicates that the target configuration file is available. If at least one first data source cannot be accessed, the initial detection result indicates that the target configuration file is unavailable.

[0115] As another example, if the usability of a target configuration file specifically includes whether its syntax conforms to the syntax rules, then if the target configuration file conforms to the syntax rules, the initial detection result indicates that the target configuration file is usable; if the target configuration file does not conform to the syntax rules, the initial detection result indicates that the target configuration file is unusable.

[0116] In another example, if the availability of a target configuration file specifically includes whether at least one first data source can be accessed and whether the syntax of the target configuration file conforms to the syntax rules, then if at least one first data source can be accessed and the syntax of the target configuration file conforms to the syntax rules, the initial detection result indicates that the target configuration file is available. If at least one first data source cannot be accessed or the syntax of the target configuration file does not conform to the syntax rules, the initial detection result indicates that the target configuration file is unavailable.

[0117] Furthermore, the target task is used to instruct the synchronization of target data from at least one first data source to at least one second data source. Correspondingly, in another implementation, the availability of the target configuration file may further include whether at least one second data source is accessible.

[0118] For example, if at least one first data source and at least one second data source are accessible, and the semantic content of the target configuration file conforms to the syntax rules, the initial detection result is determined to indicate that the target configuration file is available. If at least one first data source is inaccessible, at least one second data source is inaccessible, or the semantic content of the target configuration file does not conform to at least one of the syntax rules, the initial detection result is determined to indicate that the target configuration file is unavailable.

[0119] In this embodiment, by conducting multi-dimensional availability testing on the target configuration file, the correctness (e.g., the semantic content of the target configuration file conforms to the syntax rules) and validity (e.g., whether at least one first data source and / or at least one second data source can be accessed) of the configuration file are checked. This achieves pre-verification before the execution of the data integration task, avoiding the problem of subsequent full task execution being invalid or interrupted due to data source connection failure or configuration syntax errors from the source. This can reduce the waste of computing and storage resources and improve the success rate of data integration task startup and overall execution efficiency.

[0120] In one implementation, the computing device obtains the initial detection results based on the aforementioned configuration verifier.

[0121] In some embodiments, the method further includes recording the initial detection results when the target configuration file is unavailable.

[0122] In some embodiments, the method further includes: outputting an alarm message if the target configuration file is unavailable.

[0123] For example, the alarm message might look like this: Unable to access at least one primary data source and / or the target configuration file contains grammatical errors. Another example might display one or more of the following: Unable to access at least one secondary data source; the alarm message and its corresponding alarm level (e.g., emergency alarm); and suggested actions to address the alarm message, such as modifying the target configuration file or directly blocking the task startup process. This allows for precise and efficient identification of configuration-related issues before the target task starts, reducing the risk of task failure due to configuration errors. Furthermore, the tiered alarms and clear handling suggestions help users respond quickly and resolve problems.

[0124] S1002, Output instruction information. This instruction information indicates that the target task should be initiated if the target configuration file is available.

[0125] In this embodiment of the application, when the target configuration file is available, an instruction message is output to indicate the initiation of the target task. This helps to intercept potential risks such as configuration errors, abnormal data source connections, and insufficient resource adaptation, and avoids invalid execution, interruption or data pollution of the full data integration task due to defects in basic conditions. The feasibility of task execution can be verified from the perspective of the configuration file without waiting for the full data integration task to start.

[0126] Optionally, steps S1001-S1002 above are optional steps.

[0127] S1003, Receive target task.

[0128] Optionally, if the instruction information is available in the target configuration file, the user issues the target task so that the computing device receives the target task.

[0129] For example, users can issue target tasks through the visual interface of a data integration tool, script submission, or API calls, allowing the computing device to receive the target task. Alternatively, users can issue target tasks through the visual interface of a terminal connected to the computing device, allowing the computing device to receive the target task from the terminal. The target task carries execution rule information, which may include: basic task attributes (such as task name, execution type: offline synchronization or real-time synchronization), source configuration, target configuration, data processing rules (such as format conversion, field mapping, data cleaning logic), and task resource configuration (such as preset memory quota, number of CPU cores, memory requirements, and parallelism parameters). For example, the parameters for the source configuration, target configuration, and task resource configuration must be consistent with the parameters entered in the first configuration interface. After receiving the target task, the computing device will first parse the above information carried by the target task and assign a unique identifier to the target task for execution.

[0130] S1004, in response to the target task, sample data from at least one first data source to obtain sampled data.

[0131] The process of obtaining sampled data can be regarded as performing a sampling task.

[0132] In one implementation of sampling data from at least one first data source to obtain sampled data, the data from at least one first data source is sampled according to a sampling strategy to obtain sampled data. As mentioned above, the sampling strategy includes sampling method and / or sampling ratio.

[0133] Optionally, the first data source and the sampling method have a corresponding relationship, which is configured by the user in the third configuration interface. For example, if the first data source is a relational database, its corresponding sampling method could be: index-based tiered sampling. For instance, index-based tiered sampling can be understood as: using the index fields (such as time indexes, business partition indexes) in the relational database table as the tiering basis, dividing the full data into multiple logical tiers (such as time tiers divided by day, partition tiers divided by business departments), and then uniformly extracting a certain number of data from each tier as samples, so that the samples cover the core data features of different tiers.

[0134] Another example is when the first data source is a streaming data source, the corresponding sampling method is time window sampling. For example, time window sampling can be understood as: setting a fixed time window size (such as 5 minutes or 10 minutes), the streaming data flows into the window in chronological order, and the data is sampled evenly within each window to efficiently extract samples from continuously flowing data.

[0135] Another example is that if the first data source is an API-based data source, its corresponding sampling method could be parameterized request sampling. For instance, parameterized request sampling can be understood as controlling the sampling range by configuring API request parameters (such as pagination parameters, time range parameters, and data type filtering parameters), without needing to call the full API data. For example, for an HTTP API data source, users can set the pagination parameter "100 items per page, retrieve the first 5 pages" or the time range parameter "only request data from yesterday 00:00-24:00" in the third configuration interface, and send a request to the API according to the configured parameters to extract sample data that meets the conditions.

[0136] It should be noted that the above is merely an example of the corresponding relationship and is not intended to be limiting. When there are multiple primary data sources, sampling is performed on multiple data sources based on the sampling method corresponding to each primary data source.

[0137] The sampling ratio indicates the proportion of sampled data to the total data in each first data source, and can be set based on the user's actual needs. This sampling ratio can be an initial or an updated one. For example, the lower limit of the sampling ratio range is greater than or equal to 0.01%, and the upper limit is less than or equal to 10%.

[0138] For example, the computing device samples data from at least one first data source according to a sampling strategy based on the aforementioned sampling configurator to obtain sampled data.

[0139] In one implementation, the sampling strategy may further include sampling resources, which are computational resources for executing the sampling task. The computing device executes the sampling task based on the computational resources allocated by the user for the sampling task.

[0140] For example, a small amount of necessary computing resources (such as the number of CPU cores and memory quota) and storage resources are allocated to the sampling task, without occupying the large-scale hardware resources required by the target task. For instance, the target task may require 8 CPU cores and 16GB of memory, while the sampling task is only allocated 1-2 CPU cores and 2-4GB of memory, and the disk is only used to temporarily store sampling data and execution logs, without occupying a large amount of storage space.

[0141] Optionally, the computing resources allocated to the sampling task as described above can be dynamically adjusted based on the user's actual needs.

[0142] S1005, acquire the data volume and target information of the sampled data. The target information includes the quality information of the sampled data and / or the operating performance indicators generated by the computing device during the sampling process.

[0143] For example, quality information is characterized by at least one of the following: first information, second information, or third information. The first information includes the proportion of non-empty fields and / or record completeness rate; the second information includes data type consistency and / or value range matching degree; and the third information includes business matching degree and / or data accuracy rate. Specifically, the proportion of non-empty fields indicates the percentage of non-empty fields in the sampled data. The record completeness rate indicates the percentage of sampled data that was completely recorded. Data type consistency indicates the consistency between the data type in the sampled data and the data type in the first data source. The value range matching degree indicates the degree of matching between the value range of the sampled data and the value range of the data in the first data source. The business matching degree indicates the degree of matching between the business information indicated by the sampled data and the business information indicated by the data in the first data source. Data accuracy rate indicates the accuracy rate of the sampled data. For example, one or more of the above quality information can be represented numerically. The numerical value corresponding to each piece of information can be configured by the user based on the information or it can be the ratio itself. For example, if the proportion of non-empty fields in the sampled data is 92%, then the proportion of non-empty fields is 92%. Other information can be referenced in this way and will not be elaborated further here.

[0144] In the aforementioned quality information, the first piece of information can be used to represent the completeness of the sampled data. The second piece of information can be used to represent the consistency between the sampled data and the data in the first data source. The third piece of information can be used to represent the accuracy of the sampled data. Thus, when the quality information is represented by the above multiple pieces of information, it can comprehensively cover the quality assessment dimensions of the sampled data from multiple dimensions such as completeness, consistency, and accuracy.

[0145] In one implementation, obtaining quality information from the sampled data includes extracting one or more of the aforementioned information from the sampled data. This allows for the rapid acquisition of the sampled data's quality information.

[0146] For example, performance metrics include at least one of the following: latency, throughput, computational resource utilization efficiency, or stability. Stability is characterized by the probability of correct data in the sampled data. Computational resource utilization efficiency includes one or more of the following: CPU utilization, memory utilization, and input / output (IO) efficiency.

[0147] In one implementation, the aforementioned latency, throughput, computing resource utilization efficiency, or stability can be obtained from the runtime data in the root directory file of the task prediction tool. The root directory file of the task prediction tool is used to store runtime data, which refers to the data generated by the computing device during the execution of the sampling task. For example, the runtime data in the root directory file of the task prediction tool can be monitored by a runtime monitor and can also be displayed through the runtime monitor.

[0148] For example, the latency (also known as the actual latency) can be the latency corresponding to the running data in the root directory file within a preset time window, or it can be the latency corresponding to the running data in the root directory file within a preset data range.

[0149] As another example, throughput (also known as actual throughput) can be the throughput corresponding to the running data in the root directory file within a preset time window, or it can be the throughput corresponding to the running data in the root directory file within a preset data range.

[0150] As another example, computing resource utilization efficiency (also known as actual computing resource utilization efficiency) may include one or more of the CPU utilization, memory utilization, and IO efficiency corresponding to the running data in the root directory file within a preset time window, or it may include one or more of the CPU utilization, memory utilization, and IO efficiency corresponding to the running data in the root directory file within a preset data range.

[0151] As another example, stability specifically includes error rate and accuracy. Error rate represents the percentage of erroneous records out of the sampled data. Accuracy represents the percentage of successfully processed records out of the sampled data. Error rate and accuracy can be the error rate and accuracy corresponding to the running data in the root directory file within a preset time window, or the error rate and accuracy corresponding to the running data in the root directory file within a preset data range.

[0152] In one implementation of acquiring target information, target information is acquired when the first detection result indicates that the ratio of the amount of sampled data corresponding to each first data source in at least one first data source to the preset amount of data is within the range of the sampling ratio corresponding to each first data source.

[0153] In this embodiment of the application, when the amount of sampled data corresponding to each first data source meets the standard, the sampled data (such as the sampled data corresponding to all first data sources) is then analyzed to obtain quality information and operational performance indicators. This can ensure the validity of the analysis samples through the prior data quantity verification, which not only avoids the distortion of quality assessment results caused by insufficient data quantity and ensures the accuracy of the analysis results, but also avoids invalid detection through the prior data quantity compliance verification, thus ensuring the validity and reference value of the quality information and operational performance indicators of the sampled data from the source.

[0154] S1006, based on the amount of sampled data and target information, predicts the execution result of the target task. The execution result is used to indicate whether the target task was successfully executed.

[0155] Understandably, since executing the target task requires extracting data from at least one primary data source, the sampling task (such as the task of sampling data from at least one primary data source) also requires extracting data from at least one primary data source. Therefore, the process of executing the sampling task can be considered a representative process of executing the target task. Furthermore, since the sampled data and the full dataset have a high degree of homology, the execution result of extracting the full dataset included in the target task can be predicted by examining the sampled data itself and the performance indicators generated by the computing device during the sampling process.

[0156] In this embodiment, based on steps S1003-S1006, rapid verification of the execution results of the target task can be achieved. It is possible to predict in advance whether the target task has been successfully executed without fully executing the entire data integration task, thus improving the verification efficiency of task execution. Furthermore, it avoids the enormous computational and storage resource consumption caused by full execution, and can identify potential problems such as data defects and / or performance bottlenecks at an early stage. This solves the problems of ineffective execution and increased maintenance costs caused by delayed problem detection in existing technologies, enabling timely and efficient determination of task execution status, and improving the success rate, reliability, and overall maintenance efficiency of data integration tasks.

[0157] like Figure 11 As shown, this application embodiment provides a specific implementation method for predicting the execution result of a target task based on the amount of sampled data and target information, which may include the following steps S1101-S1104.

[0158] S1101, determine a first detection result based on the amount of sampled data; wherein, the first detection result is used to indicate whether the ratio of the amount of sampled data corresponding to each first data source in at least one first data source to the preset amount of data is within the range of the sampling ratio corresponding to each first data source.

[0159] For example, the preset data volume can include the total amount of data in each primary data source. It can also be a user-defined data volume.

[0160] Understandably, if the first detection result indicates that the proportion of sampled data corresponding to at least one first data source to a preset data volume is not within the sampling ratio range corresponding to that first data source, it indicates that the sampled data volume corresponding to that first data source exceeds its corresponding preset sampling ratio range. This could be due to the proportion being lower than the lower limit of the sampling ratio (e.g., only one ten-thousandth), resulting in insufficient sample representativeness and an inability to reflect the overall data distribution and characteristics of the first data source, thus affecting the accuracy of subsequent health assessments and result predictions; or it could be higher than the upper limit of the sampling ratio (e.g., exceeding ten percent), leading to a surge in resource consumption and extended execution time for the sampling task, making minute-level rapid testing impossible. Therefore, if the first detection result indicates that the proportion of sampled data corresponding to each of the at least one first data source to a preset data volume is within the sampling ratio range, it indicates that the sampled data volume meets the standard and can be used as a representative sample for extracting target data from at least one first data source. "Meets the standard" can be understood as conforming to the data volume standard.

[0161] For example, if the first data source is a relational database, the sampling task is to extract sample data corresponding to the first data source from the target table (such as the user order table (orders)). The target table contains 1 million records, and the sampling ratio is between 0.1% and 0.12%, then the expected sample data volume could be 1000 to 1200 records. However, if the actual sample data volume is 800 records, which is outside the sampling ratio range, then it can be determined that the sample data volume corresponding to the first data source is insufficient.

[0162] In this embodiment of the application, by using the ratio of the amount of sampled data corresponding to each first data source to the preset amount of data as the criterion for judgment, the problem of evaluation distortion caused by insufficient sample size can be avoided, as well as the waste of resources caused by excessive sample size, thereby improving the rationality of the sampled data.

[0163] In some embodiments, the method further includes: when a first detection result indicates that the proportion of the amount of sampled data corresponding to the first data source in at least one first data source to a preset amount of data is not within the range of the sampling ratio corresponding to the first data source, adjusting the sampling ratio corresponding to the first data source based on preset information to obtain an updated sampling ratio for the first data source whose proportion is not within its corresponding sampling ratio; and sampling the data in the first data source based on the updated sampling ratio.

[0164] The preset information includes the amount of data in the first data source. For example, the data in the first data source can be all the data in the first data source or a portion of the data. For instance, if the first data source is a relational database, the portion of data can be a table in that database (such as the target table mentioned above), and the sampled data corresponding to the first data source is sampled from the data (all or part of the data) in the first data source.

[0165] In one implementation method for adjusting the sampling ratio corresponding to a first data source based on preset information to obtain an updated sampling ratio, the sampling ratio corresponding to the first data source is adjusted based on the coefficient corresponding to the data volume in the first data source and the current sampling ratio corresponding to the first data source to obtain the updated sampling ratio. The coefficient varies depending on the data volume in the first data source.

[0166] For example, the amount of data in the first data source is negatively correlated with its corresponding coefficient. For instance, if the amount of data in the first data source is less than 10,000 records, the corresponding coefficient can be 1.0; if the amount of data in the first data source is between 10,000 and 100,000 records, the corresponding coefficient is 0.8; if the amount of data in the first data source is between 100,000 and 1,000,000 records, the corresponding coefficient is 0.5; and if the amount of data in the first data source exceeds 1,000,000 records, the corresponding coefficient is 0.3.

[0167] In one possible implementation, adjusting the sampling ratio based on the coefficient corresponding to the data volume in the first data source and the current sampling ratio specifically involves multiplying the coefficient corresponding to the data volume in the first data source by the current sampling ratio to obtain the updated sampling ratio. Continuing with the example above, if the target table contains 1 million records, its corresponding coefficient is 0.3. Based on the current sampling ratio of 0.1%-0.12%, the updated sampling ratio would range from 0.03% to 0.036%.

[0168] Optionally, the preset information also includes the proportion of null data in the sampled data corresponding to the first data source and / or the business priority of the sampled data. The business priority of the sampled data indicates its importance at the business level. There is a correspondence between the sampled data and the business priority, which is configured by the user. For example, sampled data of production process parameters that affect the core performance of the product can be configured as high priority, while auxiliary monitoring data such as workshop environmental temperature and humidity can be configured as low priority.

[0169] In one implementation, the preset information also includes the proportion of null data in the sampled data corresponding to the first data source. Based on the coefficient corresponding to the data volume in the first data source, the coefficient corresponding to the proportion of null data in the first data source, and the current sampling ratio, the sampling ratio is adjusted to obtain the updated sampling ratio.

[0170] For example, the proportion of null data in the first data source is positively correlated with its corresponding coefficient. For instance, if the proportion of null data in the first data source is 5%, its corresponding coefficient is 1.2; if the proportion of null data in the first data source is 4%, its corresponding coefficient is 0.8.

[0171] In one possible implementation, adjusting the sampling ratio based on the coefficients corresponding to the data volume in the first data source, the coefficients corresponding to the proportion of null data in the first data source, and the current sampling ratio specifically involves multiplying the coefficients corresponding to the data volume in the first data source, the coefficients corresponding to the proportion of null data in the first data source, and the current sampling ratio to obtain the updated sampling ratio.

[0172] Continuing with the example above, if the coefficient corresponding to the amount of data in the first data source is 0.3, and the coefficient corresponding to the proportion of null data in the first data source is 1.2, based on the current sampling ratio of 0.1%-0.12%, then the range of the updated sampling ratio is 0.036%-0.0432%.

[0173] In another implementation, the preset information also includes the business priority of the sampled data corresponding to the first data source. Based on the coefficient corresponding to the data volume in the first data source, the coefficient corresponding to the business priority of the sampled data, and the current sampling ratio, the sampling ratio is adjusted to obtain the updated sampling ratio.

[0174] For example, the service priority of the sampled data is positively correlated with its corresponding coefficient. For instance, if the service priority of the sampled data is the highest priority, its corresponding coefficient is 1.2; if the service priority of the sampled data is the medium priority, its corresponding coefficient is 0.8; and if the service priority of the sampled data is the lowest priority, its corresponding coefficient is 0.4.

[0175] In one possible implementation, adjusting the sampling ratio based on the coefficients corresponding to the data volume of the data in the first data source, the coefficients corresponding to the business priority of the sampled data, and the current sampling ratio specifically involves: calculating the product of the coefficients corresponding to the data volume of the data in the first data source, the coefficients corresponding to the business priority of the sampled data, and the current sampling ratio to obtain the updated sampling ratio.

[0176] Continuing with the example above, if the coefficient corresponding to the data volume in the first data source is 0.3, and the coefficient corresponding to the business priority of the sampled data is 0.8, based on the current adoption ratio of 0.1%-0.12%, then the updated sampling ratio range is 0.024%-0.0288%.

[0177] In another implementation, the preset information also includes the proportion of null data in the sampled data corresponding to the first data source and the business priority of the sampled data. Based on the coefficients corresponding to the data volume in the first data source, the coefficients corresponding to the proportion of null data in the sampled data, and the coefficients corresponding to the business priority of the sampled data, the sampling ratio is adjusted with the current sampling ratio to obtain the updated sampling ratio.

[0178] In one possible implementation, the sampling ratio is adjusted based on the coefficients corresponding to the data volume of the data in the first data source, the coefficients corresponding to the proportion of null data in the sampled data, and the coefficients corresponding to the business priority of the sampled data, and the current sampling ratio. Specifically, the updated sampling ratio is obtained by multiplying the coefficients corresponding to the data volume of the data in the first data source, the coefficients corresponding to the proportion of null data in the sampled data, and the coefficients corresponding to the business priority of the sampled data with the current sampling ratio.

[0179] Continuing with the example above, if the coefficient corresponding to the amount of data in the first data source is 0.3, the coefficient corresponding to the proportion of null data in the sampled data is 1.2, and the coefficient corresponding to the business priority of the sampled data is 0.8, based on the current adoption ratio of 0.1%-0.12%, then the updated sampling ratio range is 0.0288%-0.03456%.

[0180] In this embodiment, when the amount of sampled data corresponding to the first data source is insufficient, the sampling ratio is dynamically adjusted and resampled by combining the data volume of the first data source, the proportion of null values ​​in the corresponding sampled data, and one or more preset information of the business priority dimension through coefficient product. This ensures the rationality and representativeness of the sampled data corresponding to the first data source, avoids distortion of subsequent health assessment and task result prediction due to inappropriate sample size, and adapts to different data characteristics and business needs, further improving the accuracy and practicality of data integration task execution result prediction.

[0181] Optionally, if the lower limit of the updated sampling ratio is less than 0.01%, or the upper limit is greater than 10%, then the amount of sampled data corresponding to the first data source is determined to be insufficient. In this way, there is no need to readjust the corresponding sampling ratio; it can be directly determined that the amount of sampled data corresponding to the first data source is insufficient, reducing computational costs.

[0182] Optionally, if the sampled data from the first data source is sampled based on the updated sampling ratio, and the ratio of the sampled data volume to the preset data volume is not within the range of the corresponding updated sampling ratio, then it is determined that the sampled data volume of the first data source is insufficient. This allows for timely identification of data sampling anomalies after the sampling ratio update, preventing the use of sampled data with insufficient volume for subsequent analysis.

[0183] Following step S1101, a second detection result is determined based on the target information. If the target information includes quality information of the sampled data, the second detection result indicates whether the quality of the sampled data meets the quality requirements; if the target information includes operating performance indicators generated by the computing device during the process, the second detection result indicates whether the operating performance of the computing device meets the performance requirements. Specifically, this may include steps S1102 and / or S1103.

[0184] S1102, Determine the second detection result based on the quality information of the sampled data.

[0185] In some embodiments, determining the second detection result based on the quality information of the sampled data includes: obtaining a score for each piece of information based on the scoring coefficient of each piece of information and the corresponding scoring coefficient. The scoring coefficient of the first information indicates the degree of influence of the first information on the integrity of the sampled data, and the score of the first information indicates the integrity of the sampled data. The scoring coefficient of the second information indicates the degree of influence of the second information on the consistency of the sampled data with data from at least one first data source, and the score of the second information indicates the consistency of the sampled data with data from at least one first data source. The scoring coefficient of the third information indicates the degree of influence of the third information on the accuracy of the sampled data, and the score of the third information indicates the accuracy of the sampled data. The second detection result is determined based on the scores of each of the at least one piece of information.

[0186] In this embodiment, the second detection result is determined by the scores corresponding to one or more of the three dimensions: the completeness of the sampled data, the consistency of the sampled data with the data in the first data source, and the accuracy of the sampled data. This can accurately quantify the completeness, consistency, and accuracy of the sampled data, improve the objectivity, accuracy, and relevance of the sampled data quality detection, and provide a basis for detecting data quality.

[0187] The following explanation uses the proportion of non-empty fields and record completeness rate as the primary information to illustrate whether the quality of the sampled data meets the requirements for data completeness: In one possible implementation, when determining whether the quality of the sampled data meets the requirements of the data integrity dimension, a weighted calculation is performed on the proportion of non-empty fields and the record integrity rate to obtain a first score. This first score is used to characterize the integrity of the sampled data. The proportion of non-empty fields has a corresponding coefficient, which can be user-configured or example-based. The proportion of non-empty fields is negatively correlated with its corresponding coefficient; that is, the larger the proportion of non-empty fields, the smaller its corresponding coefficient. Similarly, the record integrity rate has a corresponding coefficient, which can be user-configured or example-based. The record integrity rate is positively correlated with its corresponding coefficient; that is, the higher the record integrity rate, the larger its corresponding coefficient.

[0188] For example, if the proportion of non-empty fields is 92%, the corresponding coefficient is 0.6. If the record completeness rate is 95%, the corresponding coefficient is 0.4. The first score is: (92%×0.6+95%×0.4)×100≈93 points.

[0189] The following explanation uses the second set of information, including data type consistency and value range matching, to illustrate whether the quality of the sampled data meets the consistency requirement between the sampled data and the data in the first data source: In one possible implementation, when determining whether the quality of the sampled data meets the requirement of consistency between the sampled data and the data in the first data source, a weighted calculation of data type consistency and value range matching degree is performed to obtain a second score. This second score characterizes the consistency between the sampled data and the data in the first data source. Specifically, data type consistency has a corresponding coefficient, which can be user-configured or example-based. Data type consistency and its corresponding coefficient are positively correlated; that is, the higher the data type consistency, the larger its corresponding coefficient. Similarly, value range matching degree has a corresponding coefficient, which can also be user-configured or example-based. Value range matching degree and its corresponding coefficient are positively correlated; that is, the higher the value range matching degree, the larger its corresponding coefficient.

[0190] For example, a data type consistency rate of 98% corresponds to a coefficient of 0.5. A value range matching degree of 95% corresponds to a coefficient of 0.5. The second score is: (98%×0.5+95%×0.5)×100≈97 points.

[0191] The following explanation uses third-party information, including business matching degree and data accuracy, to illustrate whether the quality of the sampled data meets the accuracy requirements: In one possible implementation, when determining whether the quality of the sampled data meets the requirements of the accuracy dimension, a weighted calculation is performed on business matching degree and data accuracy to obtain a third score. This third score is used to characterize the accuracy of the sampled data. Specifically, the business matching degree and its corresponding coefficient have a corresponding relationship, which can be user-configured or example-based. The business matching degree and its corresponding coefficient are positively correlated; that is, the higher the business matching degree, the larger its corresponding coefficient. Similarly, the data accuracy and its corresponding coefficient have a corresponding relationship, which can be user-configured or example-based. The data accuracy and its corresponding coefficient are positively correlated; that is, the higher the data accuracy, the larger its corresponding coefficient.

[0192] For example, if the business rule compliance rate is 85%, the corresponding coefficient is 0.6. If the data accuracy rate is 90%, the corresponding coefficient is 0.4. The third score would be: 85% × 0.6 + 90% × 0.4 × 100 ≈ 87 points.

[0193] In one implementation, a second detection result is determined based on a first score, a second score, and a third score. The second detection result can be represented numerically. For example, continuing with the previous example, the first score is 93, the second score is 97, and the third score is 87. Accordingly, the three scores are weighted and calculated to obtain a first target score, which characterizes the second detection result. The coefficients for the first, second, and third scores can be 0.4, 0.3, and 0.3, respectively. Therefore, the first target score is: 93 × 0.4 + 97 × 0.3 + 87 × 0.3 ≈ 92.

[0194] If the first target score is greater than the first preset score, it indicates that the quality of the sampled data meets the quality requirements. If the first target score is less than or equal to the first preset score, it indicates that the quality of the sampled data does not meet the quality requirements. Thus, by quantifying the quality information, the second detection result is accurately determined. For example, the first preset score can be set according to the user's actual needs; for instance, it can be greater than or equal to 80.

[0195] In this embodiment of the application, the second detection result is determined by multi-dimensional scoring corresponding to the quality information of the sampled data. This can accurately reflect the quality level of the sampled data in various dimensions such as completeness, consistency, and accuracy in a quantitative manner, thereby achieving an objective and comprehensive evaluation of the quality information of the sampled data.

[0196] S1103, determine the second test result based on the operating performance indicators generated by the calculation equipment during the sampling process.

[0197] For example, the second detection result can be represented in numerical form.

[0198] In one implementation, if the performance metrics include latency, a fourth score is determined based on the deviation between the actual latency (i.e., the P95 delay) and the preset latency; the second detection result is then determined based on the fourth score. For example, the preset latency is set based on the user's actual needs.

[0199] For example, the actual latency is 200ms. The preset latency is 150ms. Based on this, the fourth score = 100 - (actual latency - preset latency) / preset latency × 100, the calculation process is: 100 - (200 - 150) / 150 × 100 ≈ 67 points.

[0200] In one implementation, if the performance metrics include throughput, a fifth score corresponding to the latency is determined based on the ratio of throughput to a preset throughput; a second detection result is then determined based on the fifth score. For example, the preset throughput is set based on the user's actual needs.

[0201] For example, the actual throughput is 900 messages / second. The preset throughput is 1000 messages / second. Based on this, the fifth score is: (actual throughput / expected throughput) × 100, the calculation process is: (900 / 1000) × 100 = 90 points.

[0202] In one implementation, if the performance indicators include computing resource utilization efficiency, then the performance consumption level of computing resource utilization efficiency is determined, and based on the deviation between the performance consumption level of computing resource utilization efficiency and the preset performance benchmark of the computing device, a sixth score corresponding to computing resource utilization efficiency is determined; and a second detection result is determined based on the sixth score.

[0203] A preset performance benchmark can represent the performance score when the computing device is in perfect health, such as 100 points.

[0204] Taking CPU utilization, memory utilization, and I / O efficiency as examples, in one implementation of determining the sixth score, the performance consumption level of computing resource utilization efficiency is determined based on CPU utilization, memory utilization, and I / O efficiency, as well as the coefficients corresponding to CPU utilization, memory utilization, and I / O efficiency, respectively. Then, based on the deviation between the performance consumption level of computing resource utilization efficiency and the preset performance benchmark of the computing device, the sixth score corresponding to the computing resource utilization efficiency is determined.

[0205] For example, CPU utilization is 85%, memory utilization is 90%, and IO efficiency is 95%. Based on this, and using the pre-configured relationships between CPU utilization and its corresponding coefficient, memory utilization and its corresponding coefficient, and IO efficiency and its corresponding coefficient, the coefficients for CPU utilization, memory utilization, and IO efficiency are determined respectively. For example, if the coefficient for CPU utilization is 0.4, the coefficient for memory utilization is 0.3, and the coefficient for IO efficiency is 0.3, then the sixth score is: 100 - (CPU utilization × 0.4 + memory utilization × 0.3 + (100 - IO efficiency) × 0.3), the calculation process is: 100 - (85 × 0.4 + 90 × 0.3 + 5 × 0.3) ≈ 37 points.

[0206] In one implementation, if the performance metrics include stability, then a seventh score corresponding to stability is determined based on the probability of correct data in the sampled data; and a second detection result is determined based on the seventh score.

[0207] For example, in the sampled data, there are 20 erroneous records and 1000 samples. The score corresponding to the error rate is: 100 - (20 / 1000 × 100) = 100 - 2 = 98 points. There are 980 successful records and a total of 1000 records. The score corresponding to the accuracy rate is: (980 / 1000) × 100 = 98 points. The seventh score can be the average of the score corresponding to the error rate and the score corresponding to the accuracy rate, i.e., the seventh score = (98 + 98) / 2 = 98 points.

[0208] In one implementation, when the performance metrics include one or more of latency, throughput, or computing resource utilization efficiency, a weighted calculation is performed on the scores corresponding to one or more metrics of latency, throughput, or computing resource utilization efficiency to determine the second detection result.

[0209] In another implementation, where the performance metrics also include stability, the second detection result is determined by a weighted score based on one or more metrics corresponding to latency, throughput, or computational resource utilization efficiency, and a seventh score.

[0210] For example, continuing with the previous example, the fourth score is 67 points, the fifth score is 90 points, and the sixth score is 37 points. Accordingly, the fourth, fifth, and sixth scores are weighted and calculated to obtain the first score, where the coefficients for the fourth, fifth, and sixth scores can be 0.3, 0.5, and 0.2, respectively. Based on this, the first score is: 67 × 0.3 + 90 × 0.5 + 37 × 0.2 ≈ 73 points.

[0211] For example, continuing with the previous example, the seventh score is 98 points. The seventh score is adjusted using the coefficient corresponding to stability to obtain the second score. The coefficient for stability can be 0.25. The second score is 98 × 0.25 ≈ 25 points.

[0212] The second target score is used to characterize the second detection result. The second target score can be obtained by adjusting the first score using the coefficient corresponding to the first score, and then determining the sum of the adjusted first and second scores. For example, the second target score could be: 73 × 0.4 + 25 ≈ 54 points. Here, 0.4 is the coefficient corresponding to the first score.

[0213] If the second target score is greater than the second preset score, it indicates that the operating performance meets the performance requirements. If the second target score is less than or equal to the second preset score, it indicates that the operating performance does not meet the performance requirements. Thus, by quantifying the operating performance generated during the sampling process, the second detection result is accurately determined. For example, the second preset score can be set according to the user's actual needs; for instance, it can be greater than or equal to 50.

[0214] In this embodiment of the application, by scoring the operating performance indicators generated by the computing device during the sampling process in multiple dimensions, the first detection result is determined, which can accurately reflect whether the operating performance meets the performance requirements in a quantitative way, and realize an objective and comprehensive evaluation of the operating performance indicators generated by the computing device during the sampling process.

[0215] Optionally, in the embodiments of this application, steps S1102 and / or S1103 described above may be used.

[0216] S1104, based on the first detection result and the second detection result, predict the execution result of the target task.

[0217] In a method for predicting the execution result of a target task based on a first detection result and a second detection result, if the first detection result indicates that the ratio of the sampled data volume corresponding to each first data source to the preset data volume is within its corresponding sampling ratio, and the second indicator result indicates the first result, the predicted execution result indicates successful execution of the target task; the first result indicates that the quality of the sampled data meets the quality requirements and / or the operating performance of the computing device meets the performance requirements. If the first detection result indicates that the ratio of the sampled data volume corresponding to a first data source to the preset data volume is not within its sampling ratio, or the second indicator result indicates the second result, the predicted execution result indicates unsuccessful execution of the target task. The second result indicates that the quality of the sampled data does not meet the quality requirements and / or the operating performance of the computing device does not meet the performance requirements.

[0218] In this embodiment, the execution result of the target task is directly mapped to whether the sampled data volume meets the standard. This achieves rapid prediction of task feasibility with a simplified single-dimensional verification logic, reducing computational overhead and adapting to rapid verification needs. By establishing a direct correlation prediction logic between the second detection result of the sampled data's quality dimension and the target task's execution result, the success or failure of the target task can be accurately predicted based on whether the sampled data meets quality requirements. This allows for rapid identification of task failure risks due to sampled data quality defects before formal task execution, effectively avoiding the waste of computing power, time, and manpower resources caused by executing the target task based on unqualified sampled data. It also provides a reliable pre-judgment basis for the target task's execution decision, improving the overall efficiency and success rate of the target task execution. And / or, by directly correlating the second detection result with the target task's execution result, using whether the running performance meets performance requirements as a prediction basis, the risk of task failure due to insufficient performance can be accurately identified before the target task is executed, avoiding resource waste caused by ineffective execution, providing reliable support for task execution decisions, and thus improving the efficiency and success rate of the target task execution.

[0219] In cases where the second detection result includes a first target score and a second target score, in another implementation method for predicting the execution result of the target task based on the first and second detection results, if the first detection result indicates that the ratio of the sampled data volume corresponding to each first data source to the preset data volume is within the range of the sampling ratio corresponding to each first data source, a health status is determined based on the first and second target scores. The health status indicates the quality of the sampled data and the operating performance of the computing device during the sampling process. Based on the health status, the execution result of the target task is determined.

[0220] In this embodiment, the core influencing factors before task execution are fully covered, effectively avoiding the risk of misjudgment caused by neglecting single issues such as insufficient data, quality defects, or performance shortcomings. The accuracy and reliability of the prediction results are improved through quantitative evaluation, providing a comprehensive and scientific basis for the execution decision of the target task, thereby reducing the waste of resources caused by ineffective execution and significantly improving the overall success rate and resource utilization efficiency of the target task execution.

[0221] In one method for determining health status based on a first target score and a second target score, the first target score is adjusted using a coefficient corresponding to it, resulting in an adjusted first target score. The sum of the adjusted first and second target scores is then determined to obtain a third target score, which is used to characterize the health status. For example, the third target score could be: 92 × 0.35 + 54 ≈ 86 points, where 0.35 is the coefficient corresponding to the first target score.

[0222] Health status can include a first state (e.g., excellent performance), a second state (e.g., good performance), a third state (e.g., average performance), and a fourth state (e.g., poor performance). If the third target score is greater than the third preset score, the health status is determined to be the first state. If the third target score is greater than the fourth preset score but less than or equal to the third preset score, the health status is determined to be the second state. If the third target score is greater than the fifth preset score but less than or equal to the fourth preset score, the health status is determined to be the third state. If the target score is less than the fifth preset score, the health status is determined to be the fourth state. It should be noted that the classification of health statuses is not limited; there can be more or fewer states than in the example.

[0223] For example, the third preset score can be set to 90. The fourth preset score can be set to 60. The fifth preset score can be set to 30. It should be noted that these values ​​are for illustrative reference only and are not intended to be limiting. For example, in the example above, the third target score is 86, which can represent the second health state, i.e., good.

[0224] In this embodiment, health status is classified based on a third target score and a preset score threshold, enabling multi-dimensional quantitative evaluation of sampled data and the sampling process. This integrates scattered single-dimensional indicators into a unified health status judgment standard, ensuring that the evaluation results align with actual business needs. Furthermore, the clear score range division makes health status judgment more intuitive and quantifiable, accurately reflecting the potential risks of data integration tasks.

[0225] In an implementation method that predicts the execution result of a target task based on health status, if the health status is in the first or second state, the predicted execution result indicates that the target task was successfully executed; if the health status is in the third or fourth state, the predicted execution result indicates that the target task could not be successfully executed.

[0226] In this embodiment of the application, the task execution result is predicted by the health status assessment result, thereby improving the accuracy of the prediction of the target task execution result and avoiding the misjudgment problem that may be caused by single-dimensional verification.

[0227] In this embodiment, based on steps S1101-S1104 described above, by determining the first detection result and the second detection result, and based on the first detection result and the second detection result, accurate prediction of the target task execution result is achieved, which can comprehensively cover the key influencing factors before the target task is executed. The reliability of the prediction result can be improved through multi-dimensional collaborative verification, and the detection dimensions can be flexibly selected according to the actual scenario requirements, taking into account both the comprehensiveness of the prediction and the flexibility of the application, thereby improving the success rate of the target task execution and the efficiency of resource utilization.

[0228] In some embodiments, prior to step S1304, the method may further include: executing step S1304 if output indication information indicates that the target configuration file is available. Thus, the execution status of the target task is predicted from multiple dimensions based on the initial detection result, the first detection result, and the second detection result. If the output information indicates that the target configuration file is unavailable, the execution result is determined to be that the target task cannot be successfully executed.

[0229] In other embodiments, after steps S1302 and S1303, the method may further include: predicting the execution result of the target task based on the initial detection result, the first detection result, and the second detection result. Specifically, if the initial detection result indicates that the target configuration file is unavailable, the first detection result indicates that the ratio of the amount of sampled data corresponding to the first data source in at least one first data source to the preset data amount is not within the range of the sampling ratio corresponding to the first data source, the second result indicates that the quality of the sampled data does not meet the instruction requirements, or the second detection result indicates that the operating performance of the computing device does not meet the performance requirements, the execution result is determined to be that the target task cannot be successfully executed. If the initial detection result indicates that the target configuration file is available, the first detection result indicates that the ratio of the amount of sampled data corresponding to each first data source in at least one first data source to the preset data amount is within the range of the sampling ratio corresponding to the first data source, the second result indicates that the quality of the sampled data meets the instruction requirements, and the second detection result indicates that the operating performance of the computing device meets the performance requirements, the execution result is determined to be that the target task can be successfully executed. In the embodiments of this application, the execution result is determined by multiple dimensions of detection results, making the determination of the execution result more accurate and comprehensive.

[0230] In some embodiments, the method further includes generating a diagnostic report.

[0231] The diagnostic report may include the first test result and the second test result.

[0232] In this embodiment, a diagnosis can be generated from multiple detection results. Correspondingly, in a specific implementation of determining the execution result based on the first and second detection results, the execution result is determined based on the diagnostic report. This allows multiple detection results to be aggregated into a single diagnostic report, making it easier to determine the execution result.

[0233] For example, a diagnostic report may also include initial test results. Thus, the final outcome is determined based on a combination of multidimensional test results.

[0234] Optionally, the diagnostic report may also include one or more of the following: health status, first target score, second target score, first score, second score, amount of sampled data, quality information, operational performance indicators, first score, second score, third score, fourth score, fifth score, sixth score, seventh score, etc., risk warnings, and prompts for adjusting the sampling ratio.

[0235] For example, information in a diagnostic report may include: Data quality prediction: The data quality score for the complete task (such as the first target score mentioned above) is expected to be between 85 and 90 points, with a small number of data format inconsistencies possible; Performance prediction: The throughput of the complete task is approximately 95% of the sample, with an estimated processing time of about 2 hours; Risk warning: It is recommended to check the stability of the data source connection and optimize SQL query performance; Automatic adjustment mechanism: If the health score (such as the third target score) is lower than 80 points, the system will automatically: adjust the sampling strategy (increase the sampling ratio or change the sampling method).

[0236] In some embodiments, the method further includes storing the execution result in a diagnostic report.

[0237] In this embodiment, the diagnostic report records the execution results. Regardless of whether the predicted execution result indicates success or failure, all results are summarized at this node for easy viewing by the user. This achieves centralized presentation of the entire data integration task evaluation information, providing technical personnel with comprehensive and intuitive task verification basis, facilitating rapid location of problem nodes, and improving the efficiency of problem investigation and task optimization.

[0238] like Figure 12 As shown, it is based on Figure 1 A method for predicting task execution is provided, which specifically includes the following steps: S1201, the computing device uses the configuration verifier, operation monitor and result verifier in the task prediction tool to determine whether the initial detection result, the operating performance of the computing device meets the performance requirements, the first detection result and the quality of the sampled data meet the quality requirements.

[0239] S1202, before predicting the execution result of the target task, the computing device first generates a diagnostic report based on the initial detection results, whether the operating performance of the computing device meets the performance requirements, the first detection results, and whether the quality of the sampled data meets the quality requirements.

[0240] S1203, the computing device predicts the execution result of the target task based on the diagnostic report and outputs the execution result.

[0241] In some embodiments, the method further includes: performing a target task.

[0242] In this embodiment, by executing the target task and the sampling verification process in parallel, task execution can be started without waiting for the verification results, which can shorten the overall task execution cycle and improve the execution efficiency of the data integration task.

[0243] like Figure 13 As shown in the embodiments of this application, another method for predicting task execution is provided. This method is applied to a computing device, and can be executed by the processor (such as a CPU) of the computing device, for example, it may include the following steps S1301-S1314.

[0244] S1301, Load the task prediction tool.

[0245] S1302, Determine if the task prediction tool is available. If yes, proceed to S1303; otherwise, end.

[0246] In one implementation, the usability of the task prediction tool is determined by checking its version compatibility and whether it can load correctly. For example, if the task prediction tool is version compatible and can load correctly, then it is considered usable; otherwise, it is unusable.

[0247] S1303, determine whether the initial detection result indicates that the target configuration file is available by configuring the verifier. If not, proceed to S1304; if yes, proceed to S1305.

[0248] S1304 records error information in the target configuration file.

[0249] S1305 monitors the operating performance indicators generated by the computing device during the sampling process through the runtime controller.

[0250] S1306, determine whether the operating performance of the computing device meets the performance requirements based on the operating performance indicators generated by the computing device. If not, proceed to S1307; if yes, proceed to S1308.

[0251] S1307 records runtime error information.

[0252] S1308, Based on the result validator, determine whether the first detection result indicates that the data volume meets the standard and whether the quality information of the sampled data meets the quality requirements. If not, proceed to S1309; if yes, proceed to S1310.

[0253] S1309, record verification error information.

[0254] S1310, Generate a success report indicating successful detection.

[0255] S1311 generates a diagnostic report based on success reports, runtime error information, verification error information, and target configuration file error information.

[0256] S1312, Based on the diagnostic report, determine the execution result of the target task.

[0257] S1313 If the execution result indicates that the target task has been successfully executed, then continue to execute the target task until completion.

[0258] S1314 If the execution result indicates that the target task cannot be executed successfully, then stop the execution of the target task and record the task failure result.

[0259] In this embodiment, the execution result is predicted from multiple dimensions based on the initial detection result, the first detection result, and the second detection result (including whether the operating performance of the computing device meets the performance requirements and whether the quality information of the sampled data meets the quality requirements), resulting in a more comprehensive and accurate prediction. Furthermore, the execution and interruption of the target task are dynamically controlled based on the execution result. Execution is terminated promptly when the predicted task cannot succeed, avoiding unnecessary resource consumption while ensuring the smooth progress of successful tasks, thus achieving a dual optimization of resource utilization and task execution reliability.

[0260] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0261] This application also provides a storage medium storing computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method described above.

[0262] This application also provides a computer program product, which includes a computer program that, when at least one processor executes the computer program, causes the at least one processor to perform the methods described above in this application.

[0263] The computing device, storage medium, or computer program product provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0264] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0265] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting task execution, characterized in that, Applied to a computing device, the method includes: Receive a target task, which is a data integration task for at least one first data source; execute the target task, which includes extracting data from the at least one first data source. In response to the target task, data from the at least one first data source is sampled to obtain sampled data; The data volume and target information of the sampled data are obtained, wherein the target information includes the quality information of the sampled data and / or the operating performance indicators generated by the computing device during the sampling process; Based on the amount of sampled data and the target information, the execution result of the target task is predicted, and the execution result is used to indicate whether the target task has been successfully executed.

2. The method according to claim 1, characterized in that, The prediction of the execution result of the target task based on the amount of sampled data and the target information includes: A first detection result is determined based on the amount of sampled data; wherein the first detection result is used to indicate whether the ratio of the amount of sampled data corresponding to each of the at least one first data source to a preset amount of data is within the range of the sampling ratio corresponding to each of the first data sources; Based on the target information, a second detection result is determined; wherein, if the target information includes quality information of the sampled data, the second detection result is used to indicate whether the quality of the sampled data meets the quality requirements; if the target information includes operating performance indicators generated by the computing device during the sampling process, the second detection result is used to indicate whether the operating performance of the computing device meets the performance requirements. Based on the first detection result and the second detection result, the execution result of the target task is predicted.

3. The method according to claim 2, characterized in that, Obtaining the target information includes: The target information is obtained when the first detection result indicates that the ratio of the sampled data volume corresponding to each of the at least one first data source to the preset data volume is within the range of the sampling ratio corresponding to each first data source.

4. The method according to claim 2 or 3, characterized in that, The method further includes: If the first detection result indicates that the proportion of the sampled data corresponding to the first data source in the at least one first data source to the preset data volume is not within the range of the sampling ratio corresponding to the first data source, the sampling ratio corresponding to the first data source is adjusted based on preset information to obtain an updated sampling ratio; wherein, the preset information includes the data volume of the data in the first data source; the preset information also includes the proportion of null data in the sampled data corresponding to the first data source and / or the business priority of the sampled data; The data in the first data source is sampled based on the updated sampling ratio.

5. The method according to any one of claims 1-4, characterized in that, Prior to receiving the target task, the method further includes: Obtain initial detection results, which are used to indicate the availability of the target configuration file, which is used to configure the rules for executing the target task; Output instruction information; wherein the instruction information is used to indicate that the target task should be initiated if the target configuration file is available.

6. The method according to claim 5, characterized in that, The target task is used to instruct the synchronization of target data from at least one first data source to at least one second data source; obtaining the initial detection result includes: Detect whether both the at least one first data source and the at least one second data source can be accessed; Detect whether the semantic content of the target configuration file conforms to the syntax rules; If at least one first data source and at least one second data source are accessible, and the semantic content of the target configuration file conforms to the syntax rules, the initial detection result indicates that the target configuration file is available. If at least one first data source is inaccessible, at least one second data source is inaccessible, or the semantic content of the target configuration file does not conform to at least one of the syntax rules, the initial detection result is determined to indicate that the target configuration file is unavailable.

7. The method according to any one of claims 1-6, characterized in that, In response to the target task, sampling data from the at least one first data source to obtain sampled data includes: In response to the target task, data in the at least one first data source is sampled according to a sampling strategy to obtain the sampled data. The sampling strategy includes the sampling method and / or sampling ratio corresponding to the at least one first data source. Wherein, if the first data source includes a relational database, the sampling method corresponding to the first data source includes index-based hierarchical sampling; If the first data source includes a streaming data source, then the sampling method corresponding to the first data source includes time window sampling; If the first data source includes an interface-type data source, then the sampling method corresponding to the first data source includes parameterized request sampling.

8. The method according to any one of claims 2-4, characterized in that, The quality information is characterized by at least one of the following: first information, second information, or third information; wherein the first information includes the proportion of non-empty fields and / or record completeness rate, the second information includes data type consistency and / or value range matching degree, and the third information includes business matching degree and / or data accuracy rate; wherein the proportion of non-empty fields is used to indicate the proportion of non-empty fields in the sampled data, the record completeness rate is used to indicate the proportion of the sampled data that is completely recorded, the data type consistency is used to indicate the consistency between the data type in the sampled data and the data type in the first data source, the value range matching degree is used to indicate the matching degree between the value range of the sampled data and the value range of the data in the first data source, the business matching degree is used to indicate the matching degree between the business information indicated by the sampled data and the business information indicated by the data in the first data source, and the data accuracy rate is used to indicate the accuracy rate of the sampled data; If the target information includes quality information of the sampled data, then determining the second detection result based on the target information includes: Based on each piece of information and its corresponding rating coefficient, a rating is obtained for each piece of information; wherein, the rating coefficient of the first information is used to indicate the degree of influence of the first information on the integrity of the sampled data, and the rating of the first information is used to indicate the integrity of the sampled data; the rating coefficient of the second information is used to indicate the degree of influence of the second information on the consistency of the sampled data with the data in the at least one first data source, and the rating of the second information is used to indicate the consistency of the sampled data with the data in the at least one first data source; the rating coefficient of the third information is used to indicate the degree of influence of the third information on the accuracy of the sampled data, and the rating of the third information is used to indicate the accuracy of the sampled data; A second detection result is determined based on the scores of each of the at least one piece of information.

9. The method according to any one of claims 2-4, characterized in that, The operational performance indicators include at least one of latency, throughput, computing resource utilization efficiency, or stability; stability is characterized by the probability of correct data in the sampled data. If the target information includes the operating performance indicators generated by the computing device during the sampling process, then determining the second detection result based on the target information includes: If the performance metrics include the latency, then a score corresponding to the latency is determined based on the deviation between the latency and the preset latency; the second detection result is determined based on the score corresponding to the latency. Alternatively, if the performance metrics include the throughput, then a score corresponding to the throughput is determined based on the ratio of the throughput to a preset throughput; the second detection result is determined based on the score corresponding to the throughput. Alternatively, if the operating performance indicators include the computing resource utilization efficiency, then the performance consumption level of the computing resource utilization efficiency is determined, and a score corresponding to the computing resource utilization efficiency is determined based on the deviation between the performance consumption level of the computing resource utilization efficiency and the preset performance benchmark of the computing device; the second detection result is determined based on the score corresponding to the computing resource utilization efficiency. Alternatively, if the operational performance metric includes stability, then a score corresponding to stability is determined based on the probability of correct data in the sampled data; the second detection result is determined based on the score corresponding to stability.

10. A computing device, characterized in that, It includes a processor and a memory; the processor is coupled to the memory; The memory is used to store instructions; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-9.

Images