Method and system for joint optimization of data, computing power and algorithms

Abstract

The embodiment of the application provides a kind of data, computing power and algorithm combined optimization method and system, by introducing field knowledge graph driven demand structured mechanism, fuzzy, non-technical task description information is automatically converted into structured demand information including task identification, data resource demand, computing power demand and algorithm demand;At the same time, a standardized resource portrait library is constructed, and data, computing power and algorithm resources from multiple sources and heterogeneous environments are uniformly described and quantitatively modeled.On this basis, a multi-objective optimization function is designed to integrate resource usage cost, task execution time and resource risk, to minimize the function value as the goal, to determine the global optimal solution from all feasible resource combinations based on structured demand information and standardized resource portrait library.The scheme effectively avoids the local optimal problem caused by traditional step-by-step decision, significantly improves resource allocation efficiency, task success rate and comprehensive cost benefit.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for joint optimization of data, computing power and algorithms. Background Technology

[0002] In fields such as intelligent decision-making, scientific computing, and automated operations, completing a data-driven task within a specific application scenario requires the coordinated scheduling of three types of elements: data resources supporting the scenario, computing power resources required for computation, and algorithmic resources to implement specific modeling or inference logic. Each application scenario typically employs a suitable algorithm. For example, graph neural networks are commonly used for association risk identification, while long short-term memory networks are often used for time series prediction. The choice of algorithm determines the input data format, feature structure, and underlying computing power performance, such as CPU / GPU type, memory bandwidth, and parallelism capabilities. Therefore, only when data, computing power, and algorithms are coordinated and adapted can end-to-end tasks be completed efficiently. Taking a financial anti-fraud scenario as an example, it is necessary to run a graph neural network model (algorithm), call upon multi-source heterogeneous user behavior logs and relationship graphs (data), and complete fraud behavior identification within a millisecond-level response window (relying on dedicated computing power with high throughput and low latency).

[0003] A proper match between data, computing power, and algorithms is conducive to the successful execution of corresponding tasks. In related technologies, for a specific application scenario, resource allocation often employs a sequential, separate resource configuration model. Specifically, business personnel first describe the application scenario requirements using non-technical language, data scientists find and evaluate available data, and IT architects then procure or configure computing resources based on the data and the characteristics of the algorithm to be used. For example, based on the task requirements corresponding to the application scenario, data is first manually filtered through a data catalog service. After the data is selected, the process jumps to the cloud resource management platform, where a virtual machine specification is chosen based on experience to execute the task for that application scenario.

[0004] In practical applications, the above-mentioned serial and separate resource configuration mode has at least the following drawbacks: 1) Because the description of business application scenarios is often vague and non-technical, the configured computing and data resources do not match the application scenarios; 2) Because the resource configuration process of data resources and computing resources is separate, it lacks a global perspective and it is difficult to find the optimal resource combination. For example, choosing the lowest-priced data source without considering the impact of its parsing complexity on subsequent computing power consumption may ultimately lead to an increase in overall costs or task timeouts; 3) Because data resources and computing resources are widely distributed among different cloud vendors, edge nodes, or industry data spaces, their technical standards, interface protocols, security policies, and compliance requirements vary, making it difficult to achieve automatic discovery, trusted access, and collaborative scheduling of cross-domain resources.

[0005] Therefore, there is an urgent need for a solution that jointly optimizes data, computing power, and algorithms, which can quickly and accurately recommend or generate the optimal resource combination solution based on the actual application scenario requirements from a massive, heterogeneous pool of data, computing power, and algorithm resources. Summary of the Invention

[0006] This application provides a method and system for joint optimization of data, computing power, and algorithms, which can quickly and accurately recommend or generate the optimal resource combination scheme based on the actual application scenario requirements from a massive, heterogeneous pool of data, computing power, and algorithm resources.

[0007] In a first aspect, embodiments of this application provide a method for joint optimization of data, computing power, and algorithms, the method comprising: Obtain task description information of the task to be processed in the target scene, as well as the domain knowledge graph of the target domain to which the target scene belongs. The domain knowledge graph includes: scene nodes, performance index nodes, data resource nodes and the relationship between each node. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements. Based on the structured requirement information, a pre-built standardized resource profile library is queried to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed. Construct a multi-objective optimization function based on resource usage costs, task execution time, and resource risks; The combination of data, algorithm, and computing resources corresponding to the task to be processed is determined with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination consisting of candidate data resources, candidate computing power resources, and candidate algorithm resources.

[0008] Secondly, embodiments of this application provide a data, computing power and algorithm joint optimization system, the system including: a scene analysis and standardization engine, a resource encapsulation and profiling engine, a joint optimization matching engine and an automated orchestration and scheduling engine; The scenario parsing and standardization engine is used to obtain task description information of the task to be processed in the target scenario, as well as the domain knowledge graph of the target domain to which the target scenario belongs. The domain knowledge graph includes: scenario nodes, performance index nodes, data resource nodes, and the relationships between each node. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements. The resource encapsulation and profiling engine is used to generate standardized resource profiles corresponding to heterogeneous resources from different providers based on predefined standardized resource description specifications, and store them in a standardized resource profile library. The heterogeneous resources are any one of data resources, computing power resources, and algorithm resources. A joint optimization matching engine is used to query a pre-built standardized resource profile library based on the structured requirement information to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed; construct a multi-objective optimization function based on resource usage cost, task execution time, and resource risk; and determine the combination of data, algorithm, and computing power resources corresponding to the task to be processed by minimizing the multi-objective optimization function value corresponding to the resource combination composed of candidate data resources, candidate computing power resources, and candidate algorithm resources. An automated orchestration and scheduling engine is used to generate a configuration file for deploying and executing the task based on the combination of data, algorithms and computing resources corresponding to the task to be processed, and to execute the task to be processed based on the configuration file.

[0009] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores a computer program, and when the computer program is executed by the processor, the processor can at least implement the method of joint optimization of data, computing power and algorithms as described in the first aspect.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the method for joint optimization of data, computing power, and algorithms as described in the first aspect.

[0011] Fifthly, embodiments of this application provide a computer program product, including: a computer program or instructions, which, when executed by a processor of an electronic device, enable the processor to at least implement the method of joint optimization of data, computing power and algorithms as described in the first aspect.

[0012] This application proposes a scheme for joint optimization of data, computing power, and algorithms. By introducing a domain knowledge graph-driven demand structuring mechanism and a multi-objective collaborative optimization framework, it achieves end-to-end automatic transformation from fuzzy business task intent to optimal resource combination. In specific implementation, firstly, the task description information of the task to be processed in the target scenario and the domain knowledge graph of the domain to which the scenario belongs are obtained. The domain knowledge graph defines different application scenario nodes, as well as performance indicator nodes and data resource nodes that are associated with the application scenario nodes. Semantic reasoning based on the topological structure of the domain knowledge graph transforms the non-technical task requirements in the task description information into structured requirement information containing explicit task identifiers, data resource requirements, computing power requirements, and algorithm requirements. Then, based on this structured requirement information, a search is performed in a pre-built standardized resource profile library to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed. Furthermore, a multi-objective optimization function integrating resource usage costs, task execution time, and resource risks is constructed. Minimizing this function value determines the globally optimal solution from all feasible combinations of candidate data resources, candidate computing power resources, and candidate algorithm resources. This avoids the suboptimal traps of traditional step-by-step decision-making, significantly improving the rationality of resource allocation and task success rate, and reducing overall costs and execution risks. In addition, this solution transforms fuzzy, unstructured task description information into structured demand information. It unifies and stores the massive heterogeneous resources scattered across different cloud platforms, edge nodes, or industry data spaces into a standardized resource profile library, achieving standardized and quantitative descriptions of demand and supply. This lays the foundation for subsequent joint optimization of data, computing power, and algorithms, improving optimization efficiency and accuracy. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating a method for joint optimization of data, computing power, and algorithms provided in this application embodiment; Figure 2 This is a schematic diagram of structured requirements information provided in an embodiment of this application; Figure 3 A schematic diagram illustrating a standardized resource profile provided in an embodiment of this application; Figure 4 A flowchart illustrating another method for joint optimization of data, computing power, and algorithms provided in this application embodiment; Figure 5 A schematic diagram of a system for joint optimization of data, computing power and algorithms provided in an embodiment of this application; Figure 6 A schematic diagram of a data, computing power and algorithm joint optimization architecture provided for an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0017] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0018] The method for jointly optimizing data, computing power, and algorithms provided in this application can be executed by an electronic device, which can be a terminal device such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.

[0019] Figure 1 A flowchart illustrating a method for joint optimization of data, computing power, and algorithms provided in this application embodiment is shown below. Figure 1 As shown, it may include the following steps: 101. Obtain the task description information of the task to be processed in the target scenario, as well as the domain knowledge graph of the target domain to which the target scenario belongs. The domain knowledge graph includes: scenario nodes, performance indicator nodes, data resource nodes, and the relationships between each node.

[0020] In this application embodiment, the target field refers to an industry or application field with specific business logic, technical specifications and data characteristics, such as financial risk control, smart healthcare, industrial predictive maintenance or smart city management.

[0021] A target scenario refers to a specific business context or application scenario within a target domain, often with clearly defined input and outputs and execution constraints. For example, in the field of financial risk control, real-time identification of abnormal payment operations in cross-border payments can be considered a target scenario; in the field of smart healthcare, early lung cancer screening based on multimodal imaging can be considered a target scenario.

[0022] A pending task refers to a data-driven task that needs to be executed by the computing system in a target scenario. It can correspond to a complete model inference, training, or analysis process. Pending tasks rely on specific data inputs, algorithmic logic, and computing power, and must meet the timeliness, accuracy, or compliance requirements stipulated by the scenario.

[0023] Task description information refers to the raw input provided by business personnel (i.e., users) or upper-layer applications, used to express the task intent of the task to be processed. Task description information is often presented in natural language, form fields, or semi-structured templates, and may include non-technical or weakly technical descriptions such as task objectives, expected results, time windows, and data source preferences. For example, the goal might be to complete a risk assessment of a user's transaction within 50 milliseconds, using behavioral logs and relationship network data from the past 30 days. While task description information may lack directly executable technical parameters, it implies implicit requirements for data, computing power, and algorithms.

[0024] A domain knowledge graph is a structured knowledge base oriented towards a target domain. Optionally, a domain knowledge graph can be built based on industry documents, industry technical specifications, and historical task logs within the target domain. The domain knowledge graph defines multiple types of semantic nodes and the relationships between them, including: scenario nodes representing the target scenario (such as credit risk control, industrial equipment fault warning, etc.), performance indicator nodes representing task execution quality requirements, and data resource nodes representing available data resource types (such as enterprise business registration data, medical imaging data, etc.).

[0025] In one optional embodiment, the performance indicator node can be further subdivided into computing power performance indicator nodes and algorithm performance indicator nodes. The computing power performance indicator node represents the performance dimensions that need to be considered for computing resources in the associated scenario, and these performance dimensions may include computing throughput, memory bandwidth, network latency, etc. The algorithm performance indicator node represents the performance dimensions that need to be considered for algorithm resources in the associated scenario, and these performance dimensions may include model inference latency, prediction accuracy, training convergence speed, etc.

[0026] In one optional embodiment, scene nodes, computing power performance indicator nodes, algorithm performance indicator nodes, and data resource nodes are interconnected through predefined semantic relationships. For example, a relationship containing performance indicators (i.e., "has Metric") is established between scene nodes and computing power / algorithm performance indicator nodes, and a relationship requiring resource types is established between scene nodes and data resource nodes, thereby forming a topological network that reflects the inherent coupling logic of tasks, indicators, and resources within the target domain. Based on the reasoning capabilities of the domain knowledge graph, implicit technical requirements can then be accurately parsed from fuzzy task description information and mapped to a quantifiable and matchable resource constraint space.

[0027] 102. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements.

[0028] Building upon the acquired task description information and domain knowledge graph, the unstructured task description information is further transformed into structured demand information (SDM) that can be parsed and executed by machines through semantic alignment and graph reasoning. This structured demand information is not simply a disassembly of the original task description, but rather a joint mapping of three dimensions—scenario, metrics, and resources—based on a predefined semantic relationship network within the domain knowledge graph. In summary, the structuring process of task description information uses the scenario as an anchor point and the graph relationships as the reasoning path, making implicit demands explicit and systematizing discrete demands.

[0029] In an optional embodiment, the task description information is transformed into structured requirement information based on the task description information and the domain knowledge graph, including: Identify scene description information in task description information and determine the target scene node in the domain knowledge graph that matches the scene description information; In the domain knowledge graph, starting from the target scene node, the target computing power performance index node and the target algorithm performance index node associated with the target scene node are obtained by traversing along the index association relationship, so as to determine the target computing power performance index and the target algorithm performance index corresponding to the task to be processed; the target data resource node matching the target scene node is obtained by traversing along the resource demand association relationship, so as to determine the target data resource corresponding to the task to be processed. Based on the target computing power performance indicators and the target algorithm performance indicators, identify the computing power performance constraints and algorithm performance constraints in the task description information; based on the target data resources, identify the data resource constraints in the task description information; Based on the scenario description information, computing power performance constraint information, algorithm performance constraint information, and data resource constraint information, determine the structured requirement information corresponding to the task description information.

[0030] To make it easier to understand, let's take an example. Suppose that the task description information entered by the business personnel is as follows: Risk assessment task for corporate clients, which must meet financial-grade compliance requirements; the task type is real-time inference, with a latency of less than 200 milliseconds; the model performance indicator requires an AUC greater than 0.9; and the data source must cover corporate business registration information and legal litigation records.

[0031] First, semantic parsing is performed on the task description information to identify the scenario description information—enterprise customer risk assessment—to determine the target scenario. Then, a search and matching operation is performed in a pre-built domain knowledge graph related to financial risk control to determine the target scenario node matching the scenario description information—enterprise-level real-time risk control.

[0032] Next, using the scenario node of "enterprise-level real-time risk control" as the starting point for inference, a bidirectional traversal is performed in the domain knowledge graph: traversing along the indicator association relationships (i.e., relationships including performance indicators) yields all performance indicator nodes associated with the "enterprise-level real-time risk control" scenario, including: target computing power performance indicator nodes, such as inference latency; and target algorithm performance indicator nodes, such as model AUC. Traversing along the resource requirement association relationships (i.e., relationships depending on resource types) yields the target data resource nodes matching this scenario, such as enterprise business registration information and judicial litigation records. After obtaining the target computing power performance index nodes, target algorithm performance index nodes, and target data resource nodes, based on the determined target computing power performance indexes (such as inference latency) and target algorithm performance indexes (such as model AUC), the original task description information is backtracked and parsed to accurately identify the corresponding computing power performance constraints (inference latency less than 200 milliseconds) and algorithm performance constraints (AUC greater than 0.9). Similarly, based on the determined target data resources (such as enterprise business registration information), the specific data resource constraints (which need to cover enterprise business registration information and judicial litigation records) are identified from the task description.

[0033] Optionally, if the task description information does not contain computing power performance constraint information corresponding to the target computing power performance index, the default computing power performance value pre-configured in the target computing power performance index node shall be used as the computing power performance constraint information; if the task description information does not contain algorithm performance constraint information corresponding to the target algorithm performance index, the default algorithm performance value pre-configured in the target algorithm performance index node shall be used as the algorithm performance constraint information.

[0034] Finally, the extracted scenario description information (enterprise-level real-time risk control), computing power performance constraint information (inference latency less than 200 milliseconds), algorithm performance constraint information (AUC greater than 0.9), and data resource constraint information (must cover enterprise business registration information and judicial litigation records) are integrated to determine the complete structured requirement information corresponding to the task description information.

[0035] Figure 2 This is a schematic diagram illustrating structured requirements information provided in an embodiment of this application. For example... Figure 2 As shown, the structured requirements clearly and unambiguously express all the technical requirements of the task to be processed for data, computing power and algorithms, providing accurate input for subsequent joint optimization scheduling.

[0036] like Figure 2 As shown, alternatively, different tasks can be precisely distinguished through metadata fields in their structured requirements. Specifically, each time a task is submitted, a globally unique task_id (such as uuid-123, uuid-456, etc.) can be automatically generated to ensure that the task is traceable.

[0037] 103. Based on the structured requirement information, query the pre-built standardized resource profile library to obtain candidate data resources, candidate computing power resources and candidate algorithm resources corresponding to the task to be processed.

[0038] As an optional approach to building a standardized resource profile library, one can first extract metadata, functional tags, compliance attributes, cost information, and access interface information of heterogeneous resources from different providers; then, according to predefined standardized resource description specifications, the extracted information is structured into a standardized resource profile (SRP); finally, the standardized resource profile is stored in a standardized resource profile library to form a set of standardized resource profiles that can be queried and matched.

[0039] Heterogeneous resources can be any of the following: data resources, computing power resources, and algorithm resources. In other words, the standardized resource profiles of various types of resources are similar.

[0040] To facilitate understanding, we will take the process of constructing standardized resource profiles corresponding to data resources as an example.

[0041] Suppose a data service provider offers a corporate risk dataset. When constructing a standardized resource profile for this dataset, the core information (i.e., metadata) is extracted first: unique resource identifier resource_id=data_001, provider=data_vendor_A; functional tags tags=[corporate business registration, legal litigation]; compliance attribute=financial grade; cost information cost=0.01 USD / time; access interface endpoint=https: / / api / v1 / ... Simultaneously, an automated probe is used to assess the quality of the dataset, generating a quality profile, such as a field completeness rate of 98.5% and an update frequency of once daily. Then, based on predefined standardized resource description specifications, all the above information is encapsulated into a structured data resource profile and stored in a standardized resource profile library.

[0042] Figure 3 This is a schematic diagram illustrating a standardized resource profile provided in an embodiment of this application. For example... Figure 3 As shown, the standardized resource profile clearly and structurally describes the core attributes and service capabilities of the resources, providing a reliable data foundation for efficient and accurate resource matching in the future.

[0043] Similarly, when generating standardized resource profiles corresponding to computing power resources, a set of standard benchmark test programs that include tasks such as matrix operations, data reading and writing, and model inference can be run to ensure comparability across vendors and architectures.

[0044] Based on the acquisition of structured demand information and a standardized resource profile library, the system uses structured demand as a query template and resource profile as the matching basis. Through multi-dimensional constraint matching and alignment with resource profile, the system accurately connects the predefined metadata, functional tags, compliance attributes and quality indicators in the standardized resource profile with the demands of the three dimensions of data, computing power and algorithms, mapping abstract technical specifications to specific physical or logical resource instances.

[0045] In one optional embodiment, based on structured requirement information, a pre-built standardized resource profile library is queried to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed, including: The data resource requirements (including data tags and compliance requirements), computing power requirements (including task types and performance constraints), and algorithm requirements (including performance indicator constraints) in the structured requirements information are used as query conditions. In the standardized resource profile library, candidate data resources whose data tags and compliance attributes match the data resource requirements are selected. Candidate computing power resources whose task types are compatible and whose performance indicators meet the computing power requirements are selected from the computing power resource profile. Candidate algorithm resources whose performance indicators meet the algorithm requirements are selected from the algorithm resource profile.

[0046] 104. Based on resource usage costs, task execution time, and resource risks, construct a multi-objective optimization function; with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination consisting of candidate data resources, candidate computing power resources, and candidate algorithm resources, determine the combination of data, algorithm, and computing power resources corresponding to the task to be processed.

[0047] Optionally, after the initial matching of candidate resources is completed, in order to ensure the efficiency and feasibility of subsequent optimization decisions, the candidate resource set needs to be finely screened.

[0048] Because the structured requirements information explicitly includes hard constraints across three dimensions—data, computing power, and algorithms (such as compliance level, latency cap, and performance thresholds)—while candidate resources in the standardized resource profile library may initially match at the tag or type level, there may still be instances where details are not met. For example, a computing power resource may claim to support real-time inference, but its actual latency is 250 milliseconds, failing to meet the requirement of less than 200 milliseconds; or a data resource may contain a legal litigation tag, but its compliance attribute is only at the general level, failing to meet financial-grade requirements. Therefore, before constructing a multi-objective optimization function and solving for the optimal resource combination, a feasibility filtering mechanism can be used to strictly exclude all candidate resources that violate explicit constraints, thereby narrowing the optimization scope, improving optimization efficiency, and ensuring that the final selected resource combination is technically fully executable.

[0049] In the specific implementation process, the candidate data resources, candidate computing power resources and candidate algorithm resources can be filtered for feasibility based on the constraints contained in the data resource requirements, computing power requirements and algorithm requirements. Resources that do not meet the constraints are eliminated to obtain the filtered candidate data resources, candidate computing power resources and candidate algorithm resources, which are then used as optimization objects to determine the combination of data, algorithm and computing power resources corresponding to the task to be processed.

[0050] As an optional method for determining the combination of data, algorithms, and computing resources corresponding to the task to be processed, at least one resource combination corresponding to candidate data resources, candidate computing resources, and candidate algorithm resources can be generated first. Then, the resource utilization cost optimization function value, task execution time optimization function value, and resource risk optimization function value corresponding to the target resource combination are calculated, where the target resource combination is any one of the at least one resource combination. Afterward, according to a preset optimization tendency weight, the resource utilization cost optimization function value, task execution time optimization function value, and resource risk optimization function value are weighted and fused to obtain the multi-objective optimization function value corresponding to the target resource combination. Finally, based on the multi-objective optimization function values ​​corresponding to the at least one resource combination, the resource combination with the smallest multi-objective optimization function value among the at least one resource combination is determined as the data, algorithm, and computing resource combination corresponding to the task to be processed.

[0051] It should be noted that the weighted fusion method described above is suitable for situations where there is a clear priority preference for the optimization objective (such as cost-sensitive scenarios). In complex tasks where there are significant conflicts among multiple objectives and it is difficult to pre-set fixed weights, the three objectives can be kept separate, and a multi-objective optimization framework can be used for joint solution.

[0052] Specifically, assume the decision variable X = {x_ij}, where i represents the candidate resource category (such as data, computing power, algorithm), and j represents the candidate resource number under that category. x_ij is 1 to indicate selection, and 0 to indicate no selection.

[0053] The multi-objective optimization function can be expressed as: Minimize F(X)=[F_cost(X), F_time(X), F_risk(X)].

[0054] The cost optimization function is F_cost(X)=Σ(x_ij * SRP_ij.cost_model(usage_params)), where usage_params is the resource usage estimated based on the task description in the structured requirements information.

[0055] The task execution time optimization function F_time(X) = T_data_access(X) + T_compute(X) estimates the time values ​​based on the performance benchmark and data volume in SRP.

[0056] The resource risk optimization function is F_risk(X)=1 - Π(1 - SRP_ij.risk_score)^x_ij, where risk_score is a risk score quantified based on compliance level, security certification, etc.

[0057] Optionally, the NSGA-II multi-objective genetic algorithm can be used to solve the multi-objective optimization function, outputting a set of non-dominated solutions (Pareto optimal solution set). Users can then select the final resource combination from this set based on their actual business strategies (such as cost priority, time priority, or risk avoidance). This approach avoids biases caused by manually setting weights and is more suitable for high-reliability, highly constrained industry-level business scenarios.

[0058] In summary, this application proposes a scheme for joint optimization of data, computing power, and algorithms. By introducing a domain knowledge graph-driven demand structuring mechanism and a multi-objective collaborative optimization framework, it achieves end-to-end automatic transformation from fuzzy business task intent to optimal resource combination. Specifically, the implementation first obtains the task description information of the task to be processed in the target scenario, as well as the domain knowledge graph of the domain to which the scenario belongs. The domain knowledge graph defines different application scenario nodes, and performance indicator nodes and data resource nodes that are associated with the application scenario nodes. Semantic reasoning based on the topological structure of the domain knowledge graph transforms the non-technical task requirements in the task description information into structured requirement information containing explicit task identifiers, data resource requirements, computing power requirements, and algorithm requirements. Then, based on this structured requirement information, a search is performed in a pre-built standardized resource profile library to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed. Furthermore, a multi-objective optimization function integrating resource usage costs, task execution time, and resource risks is constructed. Minimizing this function value determines the globally optimal solution from all feasible combinations of candidate data resources, candidate computing power resources, and candidate algorithm resources. This avoids the suboptimal traps of traditional step-by-step decision-making, significantly improving the rationality of resource allocation and task success rate, and reducing overall costs and execution risks. In addition, this solution transforms fuzzy, unstructured task description information into structured demand information. It unifies and stores the massive heterogeneous resources scattered across different cloud platforms, edge nodes, or industry data spaces into a standardized resource profile library, achieving standardized and quantitative descriptions of demand and supply. This lays the foundation for subsequent joint optimization of data, computing power, and algorithms, improving optimization efficiency and accuracy.

[0059] In an optional embodiment, after determining the combination of data, algorithms, and computing resources corresponding to the task to be processed through a multi-objective optimization function, task orchestration can be further performed on the task to be processed.

[0060] Figure 4 A flowchart of another method for joint optimization of data, computing power, and algorithms provided in this application embodiment is shown below. Figure 4 As shown, after determining the combination of data, algorithms, and computing resources corresponding to the task to be processed, the following steps are also included: 401. Based on the combination of data, algorithms, and computing resources corresponding to the task to be processed, generate a configuration file for deploying and executing the task to be processed. The configuration file contains the data access interface, algorithm resource acquisition address, and computing resource scheduling interface corresponding to the data resources in the combination of data, algorithms, and computing resources.

[0061] 402. Apply for computing resources from external computing resources according to the computing resource scheduling interface, and build a runtime environment for isolated execution of pending tasks.

[0062] 403. Inject the data access interface and the algorithm resources obtained from the algorithm resource acquisition address into the runtime environment, and start the computation process of the task to be processed.

[0063] 404. After detecting that the calculation process of the pending task has ended or has terminated abnormally, output the calculation results to the client device.

[0064] In practice, the execution of tasks to be processed can be managed by a state machine. The state machine includes: PENDING (pending), PREFLIGHT_CHECK (preflight check), PROVISIONING (resource preparation), RUNNING (running), MONITORING (monitoring), COMPLETED (completed), and FAILED (failed).

[0065] The pre-inspection status is used to trigger calls to the health check interfaces corresponding to each resource in the combination of data, algorithms, and computing power resources to verify the availability of each resource.

[0066] Resource preparation status, used to trigger the generation and execution of configuration files.

[0067] The monitoring status is used to trigger the periodic acquisition of log information and performance indicators of the computing process, and to trigger alarms or resource adjustment operations when the performance indicators exceed preset thresholds (for example, memory utilization exceeds 95%).

[0068] In this embodiment, after determining the combination of data, algorithms, and computing resources, the system generates configuration files and calls corresponding interfaces to achieve automatic deployment of resources and construction of an isolated operating environment. Combined with a state machine, the system performs refined management of the entire task execution process. The pre-inspection state actively verifies the health of each resource interface to ensure availability. The resource preparation state triggers the generation and distribution of configuration files. The monitoring state periodically collects logs and performance indicators (such as issuing alarms or making adjustments when memory usage exceeds 95%). Ultimately, the system achieves reliable closed-loop control of the task from startup and operation to result output, effectively improving the automation level and operational stability of task scheduling.

[0069] The following examples illustrate the data, computing power, and algorithm joint optimization schemes provided in the embodiments of this application, using several specific scenarios as examples.

[0070] Scenario 1: Financial Credit Risk Control (Focusing on cost optimization under low latency and high compliance) Suppose the task to be processed is: to build a real-time anti-fraud model service for online personal consumer credit; the task description information is: we need a real-time anti-fraud model service for online personal consumer credit, which requires the return of results within 200 milliseconds, the model AUC to be no less than 0.9, and must use a data source that complies with local financial regulatory requirements, where the compliance level of local financial regulatory requirements is LEVEL_3_FINANCE.

[0071] Based on the task description information and the domain knowledge graph, the JSON representation of the structured requirement information obtained by transforming the task description information is as follows: Based on the above structured requirement information, the candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the tasks to be processed are obtained from the pre-built standardized resource profile library as follows: Candidate data resource Data_SRP_A:{ id:“ds_gov_credit”, compliance: “LEVEL_3_FINANCE”, cost_model: “0.5”, access_time_ms: 50} / / Cost unit: USD / call; Candidate data resource Data_SRP_B:{id:“ds_internet_behavior”,compliance: “LEVEL_2_GENERAL”, cost_model: “0.1”, access_time_ms: 20}; Candidate computing power resource Compute_SRP_C: { id: “gpu_T4_local”, arch: “GPU”, cost_model: “2.0”, inference_time_ms: 30} / / Cost unit: USD / hour; Candidate computing power resource Compute_SRP_D: { id: “cpu_cloud_high_freq”, arch: “CPU”, cost_model: “0.8”, inference_time_ms: 120} / / Cost unit: USD / hour; Candidate algorithm resource Algo_SRP_E: { id: “XGBoost_v1.5”, type: “binary_classification”, metric: {“AUC”: “0.92”}}.

[0072] First, a constraint check is performed. Since the compliance level of the candidate data resource Data_SRP_B is LEVEL_2_GENERAL, it does not meet LEVEL_3_FINANCE and is therefore filtered out.

[0073] Next, assuming 3600 requests per hour, for resource combination scheme 1 (A, C, E), the corresponding latency is Latency = access_time_A + inference_time_C = 50 + 30 = 80ms (< 200ms, satisfying the constraint); the cost per hour is Cost_per_hour = (cost_A * 3600) + cost_C = (0.5 * 3600) + 2.0 = $1802.0. For resource combination scheme 2 (A, D, E), the corresponding latency is Latency = access_time_A + inference_time_D = 50 + 120 = 170ms (< 200ms, satisfying the constraint); the cost per hour is Cost_per_hour = (cost_A * 3600) + cost_D = (0.5 * 3600) + 0.8 = $1800.8.

[0074] Based on the above calculation results, the Pareto optimal solution set can be obtained: resource combination scheme 1 (A,C,E) is the performance optimal solution, and resource combination scheme 2 (A,D,E) is the cost optimal solution.

[0075] Understandably, in the serial separation mode, to ensure performance, the best-performing computing power, Compute_SRP_C, would likely be selected first. Based on this selection, the only compliant data source would be Data_SRP_A. This would result in them only being able to obtain a solution with a cost of $1802.0, and failing to discover the lower-cost solution (A, D, E) found by this invention through joint optimization, which also satisfies all constraints. In this scenario, this invention achieves a cost saving of 0.067%, demonstrating the technical advantages of joint optimization.

[0076] In practice, if the user selects the most cost-effective resource combination scheme 2 (A,D,E), then a configuration file for deployment in a Kubernetes container can be generated based on the relevant information of resource combination scheme 2 (A,D,E). The configuration file (Deployment configuration) is used for declarative management, ensuring that a specified number of service instances are always maintained during operation, and providing capabilities for resource isolation, health monitoring, and sensitive information security management.

[0077] Optionally, the following types of information can be defined in the configuration file: 1) Service instance size control: For example, configure the number of service replicas to 3, and the container orchestration platform will automatically schedule and maintain this number of running instances to achieve high availability and load balancing.

[0078] 2) Container image source: Each instance is started based on a pre-built model service image, which encapsulates the trained XGBoost binary classification model and its supporting inference service logic. It belongs to the algorithm resource package (corresponding to Algo_SRP_E type resource).

[0079] 3) Computational resource configuration: Allocate computational resources to each container instance, for example, a CPU request of 4 cores and a memory request of 8 GiB; the maximum CPU usage is 8 cores and the maximum memory usage is 16 GiB. This resource configuration is determined based on the preset computational resource specification template (Compute_SRP_D) to ensure sufficient computational power for the model inference process while preventing excessive resource consumption.

[0080] 4) Sensitive Data Security Access Mechanism: Upon service startup, access credentials for the external data interface are injected via environment variables, including the data interface address (DATA_API_ENDPOINT) and authentication key (DATA_API_KEY). This sensitive information is not hard-coded into the configuration but is dynamically read from an encrypted key object (Secret) named ds-gov-credit-secret. This key object belongs to the Controlled Data Resource Package (Data_SRP_A), thereby achieving secure credential isolation and access control.

[0081] 5) Run health status monitoring: Configure a liveness probe to periodically send HTTP requests (port 8080) to the / health path of the service inside the container, with an initial delay of 15 seconds, and then every 20 seconds thereafter. If the probe fails, the container orchestration platform will automatically restart the instance to ensure continuous service availability.

[0082] Through the above deployment configuration, this invention realizes the automated deployment, elastic scaling, resource guarantee and security operation and maintenance of fraud detection services in a cloud-native environment, effectively supporting the real-time decision-making needs in financial risk control scenarios.

[0083] Scenario 2: Target screening for new drug development (focusing on large-scale batch processing and dynamic fault tolerance) Assume the task to be processed is: target screening on a large-scale gene sequence database, prioritizing the lowest cost; the task description is: we need to perform target screening calculations on a library containing 10TB of gene sequence data, running an open-source bioinformatics software called BioAlign, the task is expected to run for 48 hours, and cost is the priority.

[0084] Based on the task description information and the domain knowledge graph, the JSON representation of the structured requirement information obtained by transforming the task description information is as follows: Based on the above structured requirement information, the candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the tasks to be processed are obtained from the pre-built standardized resource profile library as follows: Candidate data resource Data_SRP_F:{ id: “ds_genbank”, storage_type: “archive”, cost_model: “5 / TB / month”, egress_cost: “0.02 / GB”}; Candidate computing resources Compute_SRP_G: { id: “vm_standard_ondemand”, arch: “CPU”, cost_model: “1.5 / hour”, availability: “high”}; Candidate computing resources Compute_SRP_H: { id: “vm_spot_instance”, arch: “CPU”, cost_model: “0.3 / hour”, availability: “low”, “features”: [“preemptible”]}; Candidate algorithm resource Algo_SRP_I: { id: “BioAlign_v3.1”, features:[“checkpointing”]}.

[0085] When determining the optimal resource combination, based on the cost priority (optimization_priority: "cost"), the cost optimization function F_cost(X) is given a higher weight in the optimization algorithm. Specifically, among the selected candidate resources, the candidate computing power resource Compute_SRP_H (bid instance) has extremely low cost but is at risk of being interrupted, while the candidate algorithm resource Algo_SRP_I supports checkpointing. Therefore, when the algorithm supports checkpointing, it can be combined with low-availability but extremely low-cost computing power resources to achieve cost optimization. Thus, resource combination scheme 3 (F, H, I) can be recommended as the optimal scheme.

[0086] After determining that resource combination scheme 3 (F, H, I) includes computing resources with preemptible characteristics and algorithm resources with checkpointing characteristics, optionally, fault tolerance and state persistence configurations can be automatically added to the generated configuration file. Specifically, an external persistent storage volume is configured, and the task container is instructed to write the computation state to this storage volume every hour during execution; simultaneously, a monitor is configured to automatically re-apply for a new bidding instance and load the most recent checkpoint from the storage volume to resume computation once an event of a bidding instance being reclaimed is detected.

[0087] Related technologies typically require manual configuration of complex fault-tolerant scripts, and it is difficult to quantify and unify decisions regarding resource costs, interruption risks, and algorithm characteristics. In practical applications, for safety reasons, the expensive Compute_SRP_G (on-demand instance) may be chosen, resulting in a computational cost that is 5 times that of the solution proposed in this invention (1.5 / 0.3 = 5). This invention achieves significant cost savings through intelligent matching and automated fault-tolerant orchestration.

[0088] Scenario 3: Quality Inspection on Smart Manufacturing Production Lines (Focusing on Complex Cloud-Edge Collaborative Topologies) Suppose the task to be processed is: to deploy AI quality inspection on a factory production line to achieve real-time, graded defect detection; the task description is: deploy a camera on the production line to detect product defects in real time, requiring detection to be completed within 50 milliseconds. Simple defects are processed directly at the edge, while complex suspected defect images are uploaded to the cloud for further evaluation using a more powerful model.

[0089] Based on the task description information, two different execution contexts, edge and cloud, are identified. Combining this with the domain knowledge graph, an independent set of constraints is generated for each context, forming a structured requirement information containing an array of computational requirements. The JSON representation of this structured requirement information is as follows: Based on the above structured requirement information, the candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the tasks to be processed are obtained from the pre-built standardized resource profile library as follows: Candidate computing power resource Compute_SRP_J: { id: “jetson_nano_edge”, location:“edge”, cost_model: “0”, inference_time_yolov5s: 40ms}; Candidate computing power resource Compute_SRP_K: { id: “gpu_A100_cloud”, location: “cloud”, cost_model: “4.0 / hour”, inference_time_yolov5x: 20ms}; Candidate algorithm resource Algo_SRP_L: {id: “YOLOv5s”, complexity: “light”}; Candidate algorithm resource Algo_SRP_M: { id: “YOLOv5x”, complexity: “heavy”}.

[0090] Additionally, the message queue Infra_SRP_N is known to have the following characteristics: { id: “mqtt_broker”, type: “messaging”}.

[0091] Based on the structured requirements information above, it can be seen that the computational requirements of the task to be processed are an array. Based on this, a distributed topology matching mode can be used.

[0092] The constraints for `location: edge` are matched with `Compute_SRP_J` and `Algo_SRP_L`, verifying that 40ms < 50ms, meeting the latency requirement. The constraints for `location: cloud` are matched with `Compute_SRP_K` and `Algo_SRP_M`. Since the `input_trigger` and `output_action` fields are identified, it's determined that a communication mechanism is needed between the edge and the cloud; therefore, `Infra_SRP_N` (message queue) is automatically matched. Finally, a combined topology scheme including the edge, cloud, and communication components is output.

[0093] This combined topology allows for multi-target deployment: An adapter integrated with the factory's IoT device management platform (such as AWS IoT Greengrass or Azure IoT Edge) distributes a container deployment manifest containing a YOLOv5s model and MQTT client configuration to the edge device jetson_nano_edge. Simultaneously, a service containing a YOLOv5x model and MQTT server is deployed in a cloud Kubernetes cluster. The edge container processes the video stream in real time; once a flaw with a confidence level within a certain ambiguity range is detected, the frame image is published to the MQTT topic suspicious_images. The cloud service subscribes to this topic, receives the images, performs high-precision re-evaluation, and then returns the result.

[0094] For the tasks to be processed in Scenario 3, the relevant technologies require at least two independent teams (the edge computing team and the cloud development team) to perform complex interface definition and joint debugging, resulting in a long development cycle and a high risk of errors. This invention abstracts the requirements and resources of the entire cloud-edge collaborative application into unified structured requirement information and standardized resource profiles, and matches and deploys them through a unified platform, improving development and deployment efficiency by at least an order of magnitude.

[0095] Figure 5 This is a schematic diagram of a system for joint optimization of data, computing power, and algorithms provided in an embodiment of this application. Figure 5 As shown, the system includes: a scene parsing and standardization engine 11, a resource encapsulation and profiling engine 12, a joint optimization and matching engine 13, and an automated orchestration and scheduling engine 14; The scenario parsing and standardization engine 11 is used to obtain task description information of the task to be processed in the target scenario, as well as the domain knowledge graph of the target domain to which the target scenario belongs. The domain knowledge graph includes: scenario nodes, performance index nodes, data resource nodes, and the relationships between each node. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements.

[0096] The resource encapsulation and profiling engine 12 is used to generate standardized resource profiles corresponding to heterogeneous resources from different providers according to predefined standardized resource description specifications, and store them in a standardized resource profile library. The heterogeneous resources are any one of data resources, computing power resources and algorithm resources.

[0097] The joint optimization matching engine 13 is used to query a pre-built standardized resource profile library based on the structured requirement information to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed; construct a multi-objective optimization function based on resource usage cost, task execution time, and resource risk; and determine the combination of data, algorithm, and computing power resources corresponding to the task to be processed with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination composed of candidate data resources, candidate computing power resources, and candidate algorithm resources.

[0098] The automated orchestration and scheduling engine 14 is used to generate a configuration file for deploying and executing the task to be processed based on the combination of data, algorithms and computing resources corresponding to the task to be processed, and to execute the task to be processed based on the configuration file.

[0099] For ease of understanding, combined with Figure 6 The collaboration process between the engines in the system that jointly optimizes data, computing power and algorithms is described. The specific implementation process can be referred to the aforementioned embodiments, and will not be repeated here.

[0100] Figure 6 This is a schematic diagram of a data, computing power, and algorithm joint optimization architecture provided in an embodiment of this application. Figure 6 As shown, users transmit task description information (i.e., business scenario description) of tasks to be processed in the business scenario to the scenario parsing and standardization engine through the corresponding platform, so that the task description information can be transformed into structured requirement information through the scenario parsing and standardization engine.

[0101] The joint optimization matching engine, based on structured demand information, retrieves candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the tasks to be processed from a standardized resource profile library. The profiles in the standardized resource profile library are generated and updated by the resource encapsulation and profile engine based on heterogeneous resources from different providers. After retrieving these resources from the standardized resource profile library, the joint optimization matching engine further determines the optimal resource combination scheme based on a preset optimization function and outputs it to the automated orchestration and scheduling engine.

[0102] The automated orchestration and scheduling engine generates a declarative deployment file based on the optimal resource combination scheme, requests resources from external resources, and builds a task execution environment; it injects the data access interface and the algorithm resources obtained from the algorithm resource acquisition address into the runtime environment and starts the computation process of the task to be processed; after detecting that the computation process of the task to be processed has ended or terminated abnormally, it outputs the computation result to the client device.

[0103] In one possible design, the above Figure 5The structure of the data, computing power, and algorithm joint optimization device shown can be implemented as an electronic device, such as... Figure 7 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores a computer program, which, when executed by the processor 22, enables the processor 22 to at least implement the data, computing power, and algorithm joint optimization method provided in the foregoing embodiments.

[0104] The aforementioned memory 21 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0105] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data, computing power, and algorithm co-optimization device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data, computing power, and algorithm co-optimization device can be implemented as a means to implement the corresponding functions in the above method embodiments.

[0106] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0109] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for joint optimization of data, computing power and algorithm, characterized in that, The term includes: Obtain task description information of the task to be processed in the target scene, as well as the domain knowledge graph of the target domain to which the target scene belongs. The domain knowledge graph includes: scene nodes, performance index nodes, data resource nodes and the relationship between each node. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements. Based on the structured requirement information, a pre-built standardized resource profile library is queried to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed. Construct a multi-objective optimization function based on resource usage costs, task execution time, and resource risks; The combination of data, algorithm, and computing resources corresponding to the task to be processed is determined with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination consisting of candidate data resources, candidate computing power resources, and candidate algorithm resources.

2. The method according to claim 1, characterized in that, The performance indicator nodes include: computing power performance indicator nodes and algorithm performance indicator nodes. The computing power performance indicator nodes represent the performance dimensions that need to be considered for computing power resources in the associated scenarios. The performance dimensions include computing throughput, memory bandwidth, and network latency. The algorithm performance indicator nodes represent the performance dimensions that need to be considered for algorithm resources in the associated scenarios. The performance dimensions include model inference latency, prediction accuracy, and training convergence speed.

3. The method of claim 2, wherein, The step of transforming the task description information into structured requirement information based on the task description information and the domain knowledge graph includes: Identify the scene description information in the task description information, and determine the target scene node in the domain knowledge graph that matches the scene description information; In the domain knowledge graph, starting from the target scene node, the target computing power performance index node and the target algorithm performance index node associated with the target scene node are obtained by traversing along the index association relationship, so as to determine the target computing power performance index and the target algorithm performance index corresponding to the task to be processed; the target data resource node matching the target scene node is obtained by traversing along the resource demand association relationship, so as to determine the target data resource corresponding to the task to be processed. Based on the target computing power performance indicators and target algorithm performance indicators, identify the computing power performance constraints and algorithm performance constraints in the task description information; based on the target data resources, identify the data resource constraints in the task description information; Based on the scenario description information, computing power performance constraint information, algorithm performance constraint information, and data resource constraint information, the structured requirement information corresponding to the task description information is determined.

4. The method of claim 3, wherein, The method further includes: If the task description information does not contain computing power performance constraint information corresponding to the target computing power performance index, then the default computing power performance value pre-configured in the target computing power performance index node will be used as the computing power performance constraint information. If the task description information does not contain algorithm performance constraint information corresponding to the target algorithm performance index, then the default algorithm performance value pre-configured in the target algorithm performance index node will be used as the algorithm performance constraint information.

5. The method of claim 1, wherein, Before determining the combination of data, algorithm, and computing resources corresponding to the task to be processed, with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination consisting of candidate data resources, candidate computing power resources, and candidate algorithm resources, the method further includes: Based on the constraints contained in the data resource requirements, computing power requirements, and algorithm requirements, the candidate data resources, candidate computing power resources, and candidate algorithm resources are subjected to feasibility filtering, and resources that do not meet the constraints are eliminated to obtain the filtered candidate data resources, candidate computing power resources, and candidate algorithm resources.

6. The method of claim 1, wherein, The goal is to determine the combination of data, algorithm, and computing resources corresponding to the task to be processed, with the objective of minimizing the multi-objective optimization function value corresponding to the resource combination consisting of candidate data resources, candidate computing power resources, and candidate algorithm resources. This includes: Generate at least one resource combination corresponding to the candidate data resources, candidate computing power resources and candidate algorithm resources; Calculate the resource utilization cost optimization function value, task execution time optimization function value, and resource risk optimization function value corresponding to the target resource combination, wherein the target resource combination is any one of the at least one resource combination; Based on preset optimization tendency weights, the resource usage cost optimization function value, task execution time optimization function value, and resource risk optimization function value are weighted and fused to obtain the multi-objective optimization function value corresponding to the target resource combination; Based on the multi-objective optimization function values ​​corresponding to the at least one resource combination, the resource combination with the smallest multi-objective optimization function value among the at least one resource combination is determined as the data, algorithm, and computing power resource combination corresponding to the task to be processed.

7. The method of claim 1, wherein, The method further includes: Based on the combination of data, algorithms, and computing resources corresponding to the task to be processed, a configuration file for deploying and executing the task to be processed is generated. The configuration file contains the data access interface, algorithm resource acquisition address, and computing resource scheduling interface corresponding to the data resources in the combination of data, algorithms, and computing resources. The computing resources are requested from external computing resources through the computing resource scheduling interface, and a running environment is constructed for isolated execution of the task to be processed. The data access interface and the algorithm resources obtained from the algorithm resource acquisition address are injected into the runtime environment, and the computing process of the task to be processed is started. After detecting the end of the computation process of the task to be processed or its abnormal termination, the computation result is output to the client device.

8. The method of claim 7, wherein, The execution process of the task to be processed is managed by a state machine, which includes a scheduling state, a pre-inspection state, a resource preparation state, a running state, a monitoring state, a completion state, and a failure state. The pre-inspection state is used to trigger the call to the health check interface corresponding to each resource in the combination of data, algorithm and computing power resources to verify the availability of each resource; the resource preparation state is used to trigger the generation and execution of the configuration file; the monitoring state is used to trigger the periodic acquisition of log information and performance indicators of the computing process, and trigger alarm or resource adjustment operation when the performance indicators exceed the preset threshold.

9. The method of claim 1, wherein, The standardized resource profile library is constructed in the following manner: Extract metadata, functional tags, compliance attributes, cost information, and access interface information of heterogeneous resources from different providers. The heterogeneous resources are any one of data resources, computing power resources, and algorithm resources. Based on predefined standardized resource description specifications, the extracted information is structured into standardized resource profiles; The standardized resource profiles are stored in a standardized resource profile library to form a set of standardized resource profiles that can be queried and matched.

10. A system for joint optimization of data, computing power and algorithms, characterized in that, The system includes: a scene parsing and standardization engine, a resource encapsulation and profiling engine, a joint optimization and matching engine, and an automated orchestration and scheduling engine; The scenario parsing and standardization engine is used to obtain task description information of the task to be processed in the target scenario, as well as the domain knowledge graph of the target domain to which the target scenario belongs. The domain knowledge graph includes: scenario nodes, performance index nodes, data resource nodes, and the relationships between each node. Based on the task description information and the domain knowledge graph, the task description information is transformed into structured requirement information, which includes task identifier, data resource requirements, computing power requirements, and algorithm requirements. The resource encapsulation and profiling engine is used to generate standardized resource profiles corresponding to heterogeneous resources from different providers based on predefined standardized resource description specifications, and store them in a standardized resource profile library. The heterogeneous resources are any one of data resources, computing power resources, and algorithm resources. A joint optimization matching engine is used to query a pre-built standardized resource profile library based on the structured requirement information to obtain candidate data resources, candidate computing power resources, and candidate algorithm resources corresponding to the task to be processed; construct a multi-objective optimization function based on resource usage cost, task execution time, and resource risk; and determine the combination of data, algorithm, and computing power resources corresponding to the task to be processed by minimizing the multi-objective optimization function value corresponding to the resource combination composed of candidate data resources, candidate computing power resources, and candidate algorithm resources. An automated orchestration and scheduling engine is used to generate a configuration file for deploying and executing the task to be processed based on the combination of data, algorithms and computing resources corresponding to the task to be processed, and to execute the task to be processed based on the configuration file.

Images