Artificial intelligence control and computing power data collaboration method and system

Abstract

The present application relates to the technical field of artificial intelligence control, and more particularly to an artificial intelligence control and computing power data collaboration method. The semantic extraction of artificial intelligence computing tasks obtains a task semantic structure, the semantic annotation of computing power resources and data resources in a distributed environment obtains a resource semantic structure, an alignment mapping relationship between the two is established, and accordingly, computing-intensive, data-intensive and communication-intensive operations are respectively mapped into computing power demand nodes, data demand nodes and transmission demand edges. The associated edges are established in combination with resource dynamic supply and data storage location to form a computing power data collaboration graph. According to the execution time sequence dependency relationship, a constraint subgraph of a data supply bottleneck is identified, a transmission path cost is evaluated, and a collaborative execution path containing a computing power node allocation sequence, a data node access sequence and a cross-node transmission path is generated. The method improves the computing power data collaboration scheduling efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence control technology, and in particular to a method and system for the collaborative operation of artificial intelligence control and computing data. Background Technology

[0002] In the field of AI computing task scheduling, the conventional approach is to use a scheduling mechanism based on resource monitoring and task queues. When a task is submitted, it is described with its computing resource requirements, data storage requirements, and priority parameters. The scheduling system then uses hardware monitoring metrics such as processor load, memory usage, and disk read / write speed of each node in the distributed environment, employing heuristic rules or constraint programming methods to allocate nodes. The computational operations, data access operations, and communication operations contained within a task are not explicitly separated in the scheduling dimension. The scheduler relies on a directed acyclic graph to represent the execution order between tasks, generating an execution plan based on objective functions such as critical path length or earliest completion time, and continuously adjusting the mapping relationship between tasks and nodes using a periodic heartbeat mechanism.

[0003] The problem with conventional approaches is that task descriptions and resource descriptions are at different levels of abstraction. Task submission parameters focus on the quantity of hardware resources, failing to reflect the differentiated consumption characteristics of computing, data, and network resources during the execution phase of a specific task. The matching accuracy between resource supply and demand remains only at the level of overall balance. As the scale of data processing increases or the computational logic becomes more complex, coarse-grained matching leads to both resource idleness and local overload, and task execution efficiency decreases significantly with the increase in concurrency.

[0004] The problem with conventional methods lies in their insufficient adaptability to dynamic changes in the operating environment. Conventional scheduling relies on periodically sampled resource status snapshots, making it difficult to incorporate the physical distribution of data resources and changes in the dynamic supply capacity of nodes into path planning in a timely manner. The selection of cross-node data transmission paths is based on network status determination at the loading moment, resulting in a lag in responding to resource configuration fluctuations during task execution. Structural changes caused by node failures or data migration require waiting for the entire scheduling cycle to be corrected, thus limiting the overall resource utilization and the ability to guarantee task completion time. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for artificial intelligence control and computing power data collaboration, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides a method for coordinating artificial intelligence control and computing power data, comprising:

[0007] Semantic extraction is performed on the execution logic of artificial intelligence computing tasks to obtain the task semantic structure;

[0008] Capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain the resource semantic structure;

[0009] Establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure;

[0010] Based on the alignment mapping relationship, the computationally intensive operations of the artificial intelligence computing task are mapped to computing power demand nodes, the data-intensive operations are mapped to data demand nodes, and the communication-intensive operations are mapped to transmission demand edges. Furthermore, according to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, computing power supply association edges are established between the computing power demand nodes and computing power resources, and data supply association edges are established between the data demand nodes and data resources, forming a computing power data collaboration graph that integrates task demand and resource supply.

[0011] Based on the execution timing dependency between computationally intensive and data-intensive operations in the task semantic structure, the constrained subgraphs with data supply bottlenecks in the computing power-data collaboration graph are identified, and the path cost of the transmission demand edges in the constrained subgraphs is evaluated to generate a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

[0012] Semantic extraction is performed on the execution logic of artificial intelligence computing tasks to obtain the task semantic structure, including:

[0013] The execution logic of the artificial intelligence computing task is parsed to identify the operation units involved in the execution logic and the calling dependencies between the operation units.

[0014] Based on the computational load characteristics and data access characteristics of the operation unit, the operation unit is classified into operation types;

[0015] Based on the relative strength relationship between the proportion of computation instructions and the proportion of memory access instructions in the computational load characteristics of the operation unit, operation units where the proportion of computation instructions is higher than the proportion of memory access instructions are marked as computation-intensive operations, and operation units where the proportion of memory access instructions is higher than the proportion of computation instructions are marked as data-intensive operations.

[0016] Based on the call dependency relationship, data transmission paths located between different operation units are identified. Then, according to the ratio of data transmission volume on the data transmission path to data access volume within the operation unit, operations corresponding to data transmission paths where the ratio exceeds a preset communication density threshold are marked as communication-intensive operations.

[0017] The computationally intensive operations, data-intensive operations, and communication-intensive operations are arranged topologically according to the call dependencies to construct the task semantic structure.

[0018] Capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain the resource semantic structure, including:

[0019] Based on the processor architecture information and parallel processing capability information of the computing resources, a computing capability feature vector is generated for the computing resources; based on the storage medium type information and data throughput rate information of the data resources, a storage capability feature vector is generated for the data resources; based on the network topology connection relationship and link bandwidth information between nodes in the distributed environment, a cross-node transmission capability feature vector is generated.

[0020] The computing power feature vector is projected onto a preset computing power semantic space, and the computing power label of the computing power resource is determined by the distribution position of the computing power feature vector in the computing power semantic space;

[0021] Based on the similarity matching relationship between the storage capacity feature vector and the preset storage capacity benchmark set, the label corresponding to the benchmark item with the highest similarity is selected from the storage capacity benchmark set as the storage capacity label of the data resource;

[0022] Based on the weighted combination of the link bandwidth component and the network delay component in the transmission capability feature vector, the transmission capability label for cross-node transmission is determined by a preset transmission capability classification rule;

[0023] The computing capability tags corresponding to the computing power resources, the storage capability tags corresponding to the data resources, and the transmission capability tags corresponding to cross-node connections are associated and organized to construct the resource semantic structure.

[0024] Establishing a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure includes:

[0025] Extract the call dependencies between the computationally intensive operations, the data-intensive operations, and the communication-intensive operations from the task semantic structure, and construct a task execution dependency graph;

[0026] Extract the topological connection relationships between the computing power resources, the data resources and the cross-node transmission paths from the resource semantic structure, and construct a resource topology connection graph;

[0027] Identify the set of operation subgraphs with strong connectivity in the task execution dependency graph, and extract the topological invariants of each operation subgraph in the set of operation subgraphs;

[0028] Identify the set of resource subgraphs with homogeneous supply capabilities in the resource topology connection graph, and extract the structural signature of each resource subgraph in the set of resource subgraphs;

[0029] Based on the graph isomorphism criterion, the structural similarity between the topological invariant and the structural signature is calculated, and a subgraph-level mapping relationship is established between the operation subgraph and the resource subgraph according to the structural similarity.

[0030] The operation nodes inside the operation subgraph and the resource nodes inside the resource subgraph in the subgraph-level mapping relationship are mapped according to the local correspondence between the topological invariant and the structural signature to generate the cross-semantic space alignment mapping relationship.

[0031] Based on the dynamic supply capacity of computing resources and the distribution of data resource storage locations in the resource panorama view, a computing power supply association edge is established between the computing power demand node and the computing power resource, including:

[0032] Obtain the load status information of the computing power resources at the current moment and the physical distribution location information of the data resources from the panoramic view of the resources;

[0033] Based on the load status information, the remaining processing capacity and allocable time window of the computing resources are calculated, and the remaining processing capacity and allocable time window are used as the dynamic supply capacity of the computing resources.

[0034] Based on the matching relationship between the computing load demand of the computing power demand node and the dynamic supply capacity of the computing power resources, under the constraint that the computing load demand is less than or equal to the remaining processing capacity, a computing power supply association edge is established between the computing power demand node and the computing power resources, and a supply capacity weight is marked on the computing power supply association edge. The supply capacity weight represents the allocatable processing capacity of the computing power resources for the computing power demand node.

[0035] Establish data supply association edges between the data demand nodes and data resources to form a computing power data collaboration graph that integrates task demand and resource supply, including:

[0036] Based on the physical distribution location information, calculate the network distance and data transmission delay between the data resource and the data demand node, and construct a data reachability evaluation index based on the network distance and data transmission delay;

[0037] Under the condition that the data accessibility evaluation index meets the preset accessibility threshold, a data supply association edge is established between the data demand node and the data resource, and an access cost weight is marked for the data supply association edge. The access cost weight represents the time cost required for the data demand node to access the data resource.

[0038] The computing power demand node, the data demand node, the transmission demand edge, the computing power supply association edge, and the data supply association edge are organized into a graph structure to construct the computing power data collaboration graph.

[0039] Based on the execution timing dependencies between computationally intensive and data-intensive operations in the task semantic structure, constrained subgraphs with data supply bottlenecks in the computing power-data collaboration graph are identified. Path cost evaluation is performed on the transmission demand edges in these constrained subgraphs, generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and cross-node transmission paths, including:

[0040] Traverse the computing power data collaboration graph, count the number of data supply association edges associated with each data demand node, mark the data demand node whose number of data supply association edges is less than the amount of data resources required by the data demand node as a bottleneck node, extract the bottleneck node and the adjacent computing power demand node and the adjacent transmission demand edge to form the constraint subgraph.

[0041] Obtain the source computing power demand node and the target computing power demand node connected by each of the transmission demand edges in the constraint subgraph. Based on the computing power supply association edges associated with the source computing power demand node and the target computing power demand node, determine the candidate computing power resource set for the source computing power demand node and the target computing power demand node. Calculate the network transmission cost between computing power resources in the candidate computing power resource set and combine it with the data transmission volume of the transmission demand edge in a weighted manner to generate the path cost evaluation value of the transmission demand edge.

[0042] Based on the predecessor-successor relationship in the execution time-series dependency, the computing power demand nodes and the data demand nodes in the constraint subgraph are arranged in ascending order according to the path cost evaluation value, and the computing power node allocation sequence and the data node access sequence are generated.

[0043] Based on the network transmission cost between adjacent nodes in the computing power node allocation sequence and the data node access sequence, the cross-node transmission path is constructed to form the collaborative execution path.

[0044] A second aspect of the present invention provides an artificial intelligence control and computing power data collaboration system, comprising:

[0045] The semantic extraction unit is used to extract the semantics of the execution logic of artificial intelligence computing tasks to obtain the semantic structure of the tasks.

[0046] The resource annotation unit is used to perform capability semantic annotation on computing resources and data resources in the panoramic view of resources in the distributed environment to obtain the resource semantic structure.

[0047] Alignment mapping unit, used to establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure;

[0048] The collaborative graph unit is used to map the computationally intensive operations of the artificial intelligence computing task to computing power demand nodes, the data-intensive operations to data demand nodes, and the communication-intensive operations to transmission demand edges based on the alignment mapping relationship. According to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, it establishes computing power supply association edges between computing power demand nodes and computing power resources, and establishes data supply association edges between data demand nodes and data resources, forming a computing power data collaborative graph that integrates task demand and resource supply.

[0049] The path generation unit is used to identify the constrained subgraphs with data supply bottlenecks in the computing power data collaboration graph based on the execution time sequence dependencies between computationally intensive operations and data-intensive operations in the task semantic structure, and to evaluate the path cost of the transmission demand edges in the constrained subgraphs, thereby generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

[0050] A third aspect of the present invention provides an electronic device, comprising:

[0051] processor;

[0052] Memory used to store processor-executable instructions;

[0053] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0055] The beneficial effects of this application are as follows:

[0056] By leveraging semantic extraction and capability semantic annotation, artificial intelligence tasks and distributed resources are unified into a comparable semantic space. The alignment mapping across semantic spaces eliminates the semantic gap between task descriptions and resource capabilities, enabling computationally intensive, data-intensive, and communication-intensive operations to be accurately identified and located. This improves the accuracy of task-resource matching from the source and lays a reliable foundation for subsequent collaborative scheduling.

[0057] A dynamic mapping is established between the task semantic structure and the resource semantic structure to form a computing power data collaboration graph. This graph integrates the dynamic supply capacity of computing power resources and the storage location distribution of data resources in real time, enabling scheduling decisions to perceive instantaneous changes in resource status, effectively avoid waiting caused by idle computing power or data hotspots, improve resource load balancing, and enhance the system's adaptability in complex distributed environments.

[0058] The process identifies and evaluates the path costs of constrained subgraphs with data supply bottlenecks in the collaborative graph, generating collaborative execution paths that include computing node allocation sequences, data node access sequences, and cross-node transmission paths. This process prioritizes eliminating data transmission bottlenecks, reducing the number of cross-node data moves, and shortening the critical path length, thereby reducing the overall task execution latency and improving computing power utilization efficiency. Attached Figure Description

[0059] Figure 1 A flowchart illustrating the collaborative method of artificial intelligence control and computing power data;

[0060] Figure 2 This is a schematic diagram of the thermal distribution of data accessibility.

[0061] Figure 3 A schematic diagram for identifying bottleneck nodes in data supply. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0064] Figure 1 This is a flowchart illustrating the artificial intelligence control and computing power data collaboration method according to an embodiment of the present invention, as shown below. Figure 1As shown, the method includes:

[0065] Semantic extraction is performed on the execution logic of artificial intelligence computing tasks to obtain the task semantic structure;

[0066] Capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain the resource semantic structure;

[0067] Establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure;

[0068] Based on the alignment mapping relationship, the computationally intensive operations of the artificial intelligence computing task are mapped to computing power demand nodes, the data-intensive operations are mapped to data demand nodes, and the communication-intensive operations are mapped to transmission demand edges. Furthermore, according to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, computing power supply association edges are established between the computing power demand nodes and computing power resources, and data supply association edges are established between the data demand nodes and data resources, forming a computing power data collaboration graph that integrates task demand and resource supply.

[0069] Based on the execution timing dependency between computationally intensive and data-intensive operations in the task semantic structure, the constrained subgraphs with data supply bottlenecks in the computing power-data collaboration graph are identified, and the path cost of the transmission demand edges in the constrained subgraphs is evaluated to generate a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

[0070] In one optional implementation, semantic extraction is performed on the execution logic of the artificial intelligence computing task to obtain the task semantic structure, including:

[0071] The execution logic of the artificial intelligence computing task is parsed to identify the operation units involved in the execution logic and the calling dependencies between the operation units.

[0072] Based on the computational load characteristics and data access characteristics of the operation unit, the operation unit is classified into operation types;

[0073] Based on the relative strength relationship between the proportion of computation instructions and the proportion of memory access instructions in the computational load characteristics of the operation unit, operation units where the proportion of computation instructions is higher than the proportion of memory access instructions are marked as computation-intensive operations, and operation units where the proportion of memory access instructions is higher than the proportion of computation instructions are marked as data-intensive operations.

[0074] Based on the call dependency relationship, data transmission paths located between different operation units are identified. Then, according to the ratio of data transmission volume on the data transmission path to data access volume within the operation unit, operations corresponding to data transmission paths where the ratio exceeds a preset communication density threshold are marked as communication-intensive operations.

[0075] The computationally intensive operations, data-intensive operations, and communication-intensive operations are arranged topologically according to the call dependencies to construct the task semantic structure.

[0076] The execution logic of artificial intelligence computing tasks is parsed into an operable sequence structure. Operation sequence parsing is performed on the task's execution logic, traversing all identifiable operation units within the execution logic and establishing call dependencies between these operation units. An operation unit refers to the smallest schedulable computational fragment with independent semantic boundaries within the execution logic, such as matrix multiplication, convolution operations, data reading, and feature aggregation. Call dependencies describe the input-output transmission relationship between operation units; that is, when the output of one operation unit serves as the input of another, a call dependency exists between them. Through static analysis or dynamic tracing of the execution logic, all operation units and the directed dependency graph structure between them can be completely extracted, providing a foundation for subsequent operation type classification.

[0077] After identifying the operation units and their invocation dependencies, the operation units are categorized by operation type based on their computational load and data access characteristics. The computational load characteristics are determined by the proportion of computation instructions. and the percentage of memory access instructions To depict, among them This indicates the proportion of floating-point instructions issued by this operation unit during execution to the total number of instructions. This represents the proportion of memory read / write instructions to the total number of instructions. Data access characteristics include dimensions such as the size of the dataset accessed within the operation unit, access frequency, and access locality. By jointly analyzing these two types of characteristics, operation units can be classified into different resource consumption patterns, thus providing a type basis for subsequent task semantic structure construction.

[0078] The classification of computationally intensive and data-intensive operations is based on... and The relative strength relationship between them is determined. When a certain operation unit's... Higher than When an operation unit is marked as a computationally intensive operation, it indicates that its execution primarily consumes processor computing resources and has relatively low dependence on memory bandwidth. Typical scenarios include large-scale matrix multiplication and forward propagation of convolutional layers in deep neural networks. Higher than When an operation unit is marked as a data-intensive operation, it indicates that its execution is primarily limited by data read / write throughput. Typical scenarios include large-scale feature lookup, embedded layer table lookup, and batch reads in data preprocessing pipelines. This relative strength comparison method based on instruction-level granularity can objectively reflect the resource consumption tendency of the operation unit during actual hardware execution, avoiding the bias caused by relying on subjective experience to manually label the operation type.

[0079] After classifying computationally intensive and data-intensive operations, the next step is to identify communication-intensive operations. Based on established call dependencies, data transfer paths between different operation units are identified. Each data transfer path involves a certain amount of data transfer, denoted as . This represents the total amount of data transferred between operation units along the path. Simultaneously, there is a certain amount of data access within the operation units connected by the path, denoted as . This indicates the total amount of data accessed from local storage by the operation unit during execution. and The ratio is defined as the communication density index, denoted as . ,Right now .when Exceeding the preset communication density threshold At this point, the operation corresponding to the data transmission path is marked as a communication-intensive operation. The significance of identifying communication-intensive operations lies in the fact that these operations often become the main source of cross-node data movement in a distributed execution environment, and their transmission cost directly affects the overall task's execution latency and resource utilization efficiency. Communication Intensity Threshold The system can be adaptively configured based on the network bandwidth conditions and task latency requirements of the specific deployment environment. In high-bandwidth interconnection environments, the threshold can be appropriately increased to reduce unnecessary labeling, while in low-bandwidth or high-latency network environments, the threshold should be decreased to more sensitively capture potential communication bottlenecks.

[0080] After labeling the three types of operations, computationally intensive, data-intensive, and communication-intensive operations are topologically arranged according to their call dependencies to construct the task semantic structure. The topology arrangement process employs a directed acyclic graph (DAG) sorting method, using operation units as nodes and call dependencies as directed edges, unfolding the execution order of operation units layer by layer according to the dependency hierarchy. During the topology arrangement process, it is necessary to handle the parallel branch structure between operation units, i.e., the situation where multiple operation units are independent in terms of dependencies and can be executed concurrently. For operation units in parallel branches, they are represented as nodes at the same level in the task semantic structure, explicitly annotating their parallel execution attributes, so that the parallel computing resources in the distributed environment can be fully utilized in the subsequent computing power data collaboration graph construction phase.

[0081] The final task semantic structure is a directed acyclic graph labeled with operation types. Each node carries a type label (computation-intensive, data-intensive, or communication-intensive) for its corresponding operation unit, along with descriptions of its computational load and data access characteristics. Each directed edge carries call dependencies and corresponding data transfer information. This structured representation transforms the task execution semantics from abstract program code into a semantic graph structure capable of resource requirement analysis. This provides clear and complete task-side semantic input for establishing cross-semantic alignment mappings with the resource semantic structure. The accuracy of the task semantic structure directly determines the quality of the subsequent computational power data collaboration graph construction and the optimization effect of the collaborative execution path. Therefore, the accuracy of operation sequence parsing and type classification is a crucial foundation for the efficient operation of the entire collaboration method.

[0082] In one optional implementation, capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain a resource semantic structure, including:

[0083] Based on the processor architecture information and parallel processing capability information of the computing resources, a computing capability feature vector of the computing resources is generated;

[0084] Based on the storage medium type information and data throughput rate information of the data resource, a storage capacity feature vector of the data resource is generated;

[0085] Based on the network topology connections and link bandwidth information between nodes in the distributed environment, a cross-node transmission capability feature vector is generated;

[0086] The computing power feature vector is projected onto a preset computing power semantic space, and the computing power label of the computing power resource is determined by the distribution position of the computing power feature vector in the computing power semantic space;

[0087] Based on the similarity matching relationship between the storage capacity feature vector and the preset storage capacity benchmark set, the label corresponding to the benchmark item with the highest similarity is selected from the storage capacity benchmark set as the storage capacity label of the data resource;

[0088] Based on the weighted combination of the link bandwidth component and the network delay component in the transmission capability feature vector, the transmission capability label for cross-node transmission is determined by a preset transmission capability classification rule;

[0089] The computing capability tags corresponding to the computing power resources, the storage capability tags corresponding to the data resources, and the transmission capability tags corresponding to cross-node connections are associated and organized to construct the resource semantic structure.

[0090] Starting from a panoramic view of resources in a distributed environment, capability features are extracted for three types of objects: computing resources, data resources, and network topology between nodes. These features are then transformed into semantically comparable annotation results and organized into a unified resource semantic structure to support the alignment and mapping between the semantic structure of subsequent tasks and the semantic structure of resources.

[0091] For feature extraction of computing resources, the processor architecture information of each computing node in the distributed environment is read, including instruction set type, number of cores, hardware acceleration unit type (such as GPU, TPU, FPGA, etc.), and supported parallel computing modes (such as data parallelism, pipelined parallelism, tensor parallelism). Simultaneously, the parallel processing capability information of the node is obtained, including peak floating-point operation rate, memory bandwidth, cache hierarchy parameters, etc. The above information is arranged according to a predefined feature dimension order to form a computing capability feature vector of the computing resources. . Each component in the algorithm corresponds to a specific processor capability metric. For example, the first component represents the peak computing power of single-precision floating-point operations, the second component represents the peak computing power of double-precision floating-point operations, the third component represents the memory bandwidth, the fourth component represents the upper limit of the number of parallel threads, and so on, to ensure that the feature vectors of different computing nodes are comparable in the same dimensional space.

[0092] For feature extraction of data resources, the storage media type information of each storage node in the distributed environment is read, including disk type (HDD, SSD, NVMe SSD, persistent memory, etc.), storage capacity, read / write latency range, and data throughput information, including sequential read / write rate and random read / write rate. This information is then arranged according to predefined feature dimensions to form a storage capacity feature vector for the data resources. . Each component corresponds to indicators such as sequential read rate, sequential write rate, random read rate, random write rate, storage medium type encoding value, and storage capacity, ensuring that the feature vectors of different storage nodes are compared in the same dimensional space.

[0093] Feature extraction of network topology between nodes is performed by analyzing the network topology connections between nodes in a distributed environment. This yields bandwidth information (including nominal bandwidth and measured available bandwidth) and link delay information (including a combined estimate of propagation delay and queuing delay) for each link. The network features of each cross-node connection are then arranged according to predefined dimensions to form a cross-node transmission capability feature vector. This includes the link bandwidth component. With network latency components And link type encoding (such as InfiniBand, Ethernet, NVLink, etc.).

[0094] In obtaining the computational capability feature vector Then, it is projected onto a predefined computational capability semantic space. This semantic space is constructed during the offline training phase through cluster analysis of a large number of historical computing power resource samples. Different regions in the space correspond to different types of computational capability semantic labels, such as "high-throughput tensor computing," "low-latency inference," and "general-purpose parallel computing." After linear or nonlinear projection transformation, the data is mapped to the semantic space. The computing power label to which the resource belongs is determined based on its landing point in the semantic space (i.e., its distance relationship with the cluster centers corresponding to each semantic label). The projection transformation matrix has been trained during the semantic space construction phase and can be directly called during the runtime phase without the need for re-optimization.

[0095] In obtaining the storage capacity feature vector Next, it is compared with a preset storage capacity benchmark set for similarity. The storage capacity benchmark set consists of several benchmark items, each containing a standard storage capacity feature vector and a corresponding storage capacity label, such as "high-speed, low-latency local storage," "large-capacity sequential access," and "distributed high-concurrency access." Calculation The cosine similarity between the data resource and the standard feature vector of each benchmark item is used to select the label corresponding to the benchmark item with the highest similarity as the storage capacity label for that data resource. When the highest similarity is below a preset threshold... If the data resource is not found, it will be marked as an "unknown type storage resource" and a manual review or dynamic benchmark expansion process will be triggered to ensure the coverage integrity of the benchmark set.

[0096] In obtaining the feature vector of cross-node transmission capability Next, the link bandwidth component is extracted. With network latency components According to the preset weighting coefficient and Calculate the weighted combination value ,Right now .in and Based on the pre-defined relative sensitivity of the target task to bandwidth and latency, for bandwidth-sensitive task scenarios, Take the larger value; for latency-sensitive task scenarios, Take the larger value. The numerical range is used to classify cross-node connections into different levels through preset transmission capability classification rules, such as "high-speed low-latency transmission", "medium-speed medium-latency transmission", and "low-speed high-latency transmission", and the corresponding level label is used as the transmission capability label for the cross-node connection. The division boundary of each level in the transmission capability classification rules is determined by historical network performance statistics and supports dynamic updates based on network status changes at runtime.

[0097] After generating the three types of tags, the computing capability tags corresponding to each computing resource node, the storage capability tags corresponding to each data resource node, and the transmission capability tags corresponding to each cross-node connection are associated and organized according to the topology of the distributed environment. Specifically, with nodes as the basic unit, computing capability tags and storage capability tags are attached to the attribute fields of the corresponding nodes; with connections as the basic unit, transmission capability tags are attached to the attribute fields of the corresponding connections. Through this joint representation of node attributes and edge attributes, a resource semantic structure is constructed. This resource semantic structure not only retains the topological connection information of the original resource panorama view, but also attaches semantic-level capability descriptions to each node and connection edge. This allows for direct matching based on semantic tags when establishing cross-semantic space alignment mappings between the task semantic structure and the resource semantic structure, without having to re-parse the original underlying hardware parameters each time, thus significantly improving the efficiency and scalability of the alignment mapping.

[0098] When nodes in a distributed environment go online, go offline, or experience performance changes, it is necessary to re-extract the feature vectors of the affected nodes, update the corresponding capability labels, and propagate the changes to the relevant node and edge attributes in the resource semantic structure. This ensures that the resource semantic structure remains consistent with the actual distributed environment and provides accurate semantic basis for the dynamic adjustment of the subsequent computing power data collaboration graph.

[0099] In one optional implementation, establishing a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure includes:

[0100] Extract the call dependencies between the computationally intensive operations, the data-intensive operations, and the communication-intensive operations from the task semantic structure, and construct a task execution dependency graph;

[0101] Extract the topological connection relationships between the computing power resources, the data resources and the cross-node transmission paths from the resource semantic structure, and construct a resource topology connection graph;

[0102] Identify the set of operation subgraphs with strong connectivity in the task execution dependency graph, and extract the topological invariants of each operation subgraph in the set of operation subgraphs;

[0103] Identify the set of resource subgraphs with homogeneous supply capabilities in the resource topology connection graph, and extract the structural signature of each resource subgraph in the set of resource subgraphs;

[0104] Based on the graph isomorphism criterion, the structural similarity between the topological invariant and the structural signature is calculated, and a subgraph-level mapping relationship is established between the operation subgraph and the resource subgraph according to the structural similarity.

[0105] The operation nodes inside the operation subgraph and the resource nodes inside the resource subgraph in the subgraph-level mapping relationship are mapped according to the local correspondence between the topological invariant and the structural signature to generate the cross-semantic space alignment mapping relationship.

[0106] The graph structure representations are extracted from the task semantic structure and the resource semantic structure respectively, and then the structural alignment between the two semantic spaces is achieved through a graph isomorphism determination mechanism. The core of this process is to unify the abstract task execution logic and the topological connection relationship of physical resources under the same structural description framework, so that the mapping between operation nodes and resource nodes has structural semantic rationality.

[0107] A task execution dependency graph is constructed using various operation units as nodes and directed edges representing the data and control flows between operations. Computationally intensive operation nodes carry descriptions of computing power requirements, data-intensive operation nodes carry descriptions of data access patterns, and communication-intensive operation nodes carry descriptions of data transfer volumes across operation units. The dependency types labeled on the edges include data dependency (the output of the preceding operation serves as the input of the subsequent operation) and control dependency (the execution state of the preceding operation determines whether the subsequent operation is triggered). These two types of dependencies collectively characterize the temporal constraints and data flow paths between operations during task execution. The resulting task execution dependency graph is a directed graph whose node set encompasses all identifiable operation units in the artificial intelligence computing task, and whose edge set fully records the dependencies between operation units.

[0108] A resource topology graph is constructed using various resource entities as nodes and physical or logical connections between resources as edges. Computing resource nodes carry descriptions of their computing capabilities, while data resource nodes carry descriptions of their storage capabilities and data distribution characteristics. Cross-node transmission paths are represented as edges, carrying transmission capability characteristics such as link bandwidth and network latency. The resource topology graph reflects the actual physical deployment relationships and interconnectivity of various resource entities in a distributed environment, and its topology directly determines the feasibility constraints of resource supply during subsequent task scheduling.

[0109] A strongly connected component decomposition algorithm (Tarjan's algorithm) is used to analyze the task execution dependency graph. Operation nodes with bidirectional reachable paths are grouped into the same strongly connected component, resulting in several operation subgraphs. Within each operation subgraph, there are tight data flow or control flow cyclic dependencies between operation nodes. During task scheduling, these nodes must be allocated resources as a whole and cannot be split into mutually unreachable resource nodes. For each operation subgraph, its topological invariants are extracted, including the number of nodes in the subgraph. Number of edges , degree sequence (i.e., the ordered sequence of in-degree and out-degree of each node in the subgraph) and the node type distribution vector within the subgraph (Record the proportion of computationally intensive, data-intensive, and communication-intensive operation nodes). Topological invariants remain unchanged under graph isomorphic transformations, and therefore can be used as structural fingerprints of the operation subgraph for subsequent structural similarity calculations with resource subgraphs.

[0110] This process identifies a set of resource subgraphs with homogeneous supply capabilities within a resource topology connection graph. By enumerating subgraphs in the resource topology connection graph that satisfy specific connectivity conditions, it extracts resource subgraphs with similar topological structures that can collaboratively provide computing, storage, and transmission capabilities. For each resource subgraph, a structural signature is extracted, which includes the number of nodes in the subgraph. Number of edges , degree sequence and the distribution vector of resource node types within the subgraph (Records the number and proportion of computing resource nodes and data resource nodes, as well as their connection relationships). The design of the structural signature maintains the same dimension and semantics as the topological invariants, ensuring a clear correspondence in the calculation of structural similarity between the two.

[0111] Based on the graph isomorphism criterion, when calculating the structural similarity between the topological invariants of the operational subgraph and the structural signature of the resource subgraph, normalized differences are calculated for each dimension component of the topological invariants and structural signatures, and a comprehensive structural similarity score is obtained by weighted summation. Specifically, the differences in the number of nodes, the differences in the number of edges, the edit distance of the degree sequence, and the cosine similarity of the node type distribution vectors together constitute the basis for calculating structural similarity. The range of values ​​is A value closer to 1 indicates a greater similarity in topology between the operation subgraph and the resource subgraph, making it more likely to establish an effective mapping relationship. A structural similarity threshold is set. ,when When a structural matching relationship exists between the corresponding operation subgraph and the resource subgraph, it is considered to be included in the subgraph-level mapping relationship set. If multiple resource subgraphs have structural similarity values ​​exceeding a threshold with the same operation subgraph, then... The highest-valued resource subgraph is used as the priority matching object for this operation subgraph to ensure the uniqueness and optimality of the mapping relationship.

[0112] A cross-semantic space alignment mapping relationship is generated, further refining the subgraph-level mapping relationship to the node level. For operation subgraphs and resource subgraphs with established subgraph-level mapping relationships, the operation nodes within the operation subgraph are mapped one by one to the resource nodes within the resource subgraph based on the local correspondence between topological invariants and structural signatures. The specific mapping rules are as follows: computationally intensive operation nodes are preferentially mapped to computing power resource nodes, data-intensive operation nodes are preferentially mapped to data resource nodes, and dependency edges corresponding to communication-intensive operations are preferentially mapped to cross-node transmission paths with high transmission capacity. During the node-level mapping process, the correspondence between the local topological position of the operation node (i.e., its degree centrality and betweenness centrality in the operation subgraph) and the local topological position of the resource node is considered simultaneously to ensure that nodes with similar topological structures are preferentially paired. Finally, the node-level mapping results in all subgraph-level mapping relationships are summarized to form a cross-semantic space alignment mapping relationship covering all operation nodes and resource nodes, providing a complete node correspondence foundation for the subsequent construction of the computing power data collaboration graph.

[0113] Through the complete process described above, from graph structure extraction, topological invariant and structural signature calculation, graph isomorphism determination to node-level mapping, the alignment mapping relationship across semantic space can accurately reflect the topological constraints of resource supply capacity while preserving the logical structural characteristics of task execution. This provides a reliable structural semantic basis for subsequent task scheduling and resource allocation decisions based on this mapping relationship.

[0114] In one optional implementation, based on the dynamic supply capacity of computing resources and the distribution of data resource storage locations in the resource panorama view, a computing power supply association edge is established between the computing power demand node and the computing power resource, including:

[0115] Obtain the load status information of the computing power resources at the current moment and the physical distribution location information of the data resources from the panoramic view of the resources;

[0116] Based on the load status information, the remaining processing capacity and allocable time window of the computing resources are calculated, and the remaining processing capacity and allocable time window are used as the dynamic supply capacity of the computing resources.

[0117] Based on the matching relationship between the computing load demand of the computing power demand node and the dynamic supply capacity of the computing power resources, under the constraint that the computing load demand is less than or equal to the remaining processing capacity, a computing power supply association edge is established between the computing power demand node and the computing power resources, and a supply capacity weight is marked on the computing power supply association edge. The supply capacity weight represents the allocatable processing capacity of the computing power resources for the computing power demand node.

[0118] Based on the alignment mapping relationship, computationally intensive operations of artificial intelligence computing tasks are mapped to computing power demand nodes, data-intensive operations are mapped to data demand nodes, and communication-intensive operations are mapped to transmission demand edges.

[0119] The system acquires the current load status information of computing resources from the resource overview, collecting real-time operational metrics for each computing node, including current CPU or GPU utilization, memory usage, task queue length, and the number of submitted but incomplete computation jobs. These metrics collectively constitute a snapshot of the current load status of the computing resource. Simultaneously, it also needs to obtain the physical distribution information of data resources, including the storage node identifiers of each data shard, the geographic location code of the data center to which they belong, and the distribution of data replicas. Integrating these two types of information forms a complete description of the current state of computing power and data resources in the distributed environment, providing an input basis for subsequent dynamic provisioning capacity calculations.

[0120] Based on load status information, the remaining processing capacity of computing resources is calculated. Using the rated processing capacity of each computing node as a baseline, the currently occupied processing capacity is subtracted to obtain the remaining processing capacity that the node can provide at the current moment. Let the rated peak processing capacity of the computing node be... The total processing capacity currently allocated to running tasks is Then the remaining processing capacity satisfies In real-world distributed computing scenarios, It can be measured in floating-point operations per second (FLOPS) or the equivalent number of computing units. This is obtained by accumulating the resource usage declarations of each task in the current task queue.

[0121] The calculation of the allocatable time window needs to be analyzed at the task scheduling granularity. For each computing node, a task scheduling timeline is maintained, recording the expected start and end times of each submitted task. By scanning this timeline, continuous idle intervals with no assigned tasks in the future are identified, and the set of these idle intervals is defined as the allocatable time window of that computing node. . It consists of several time intervals, the length of which represents the duration during which the node can accept new task scheduling within the corresponding time period. and Together, they serve as the dynamic supply capability of computing power resources, which can simultaneously reflect the availability of the resource in terms of both processing capacity and time, avoiding the problem of ignoring time scheduling conflicts when matching based solely on instantaneous load rates.

[0122] Establish a computing power supply correlation edge to match the computing load demand of each computing power demand node with the dynamic supply capacity of computing power resources. Let the processing power required for the artificial intelligence computing operation corresponding to a certain computing power demand node be denoted as . The constraint condition is: A computing power supply association edge is established between the computing power demand node and the corresponding computing power resource only if this inequality holds. If a certain computing power node's... If the constraints are not met, the node will not participate in the construction of the associated edges of the nodes with current computing power requirements, thus naturally excluding infeasible resource allocation schemes at the graph structure level.

[0123] For computing power supply-related edges that meet the constraints, it is also necessary to label them with supply capacity weights. . This represents the actual processing capacity that the computing resource can be allocated to nodes with current computing power requirements; its value is taken as... and The difference, that is . The larger the value, the more capacity the computing power resource still has after meeting the current demand nodes, and the potential to undertake other concurrent tasks. The smaller the value, the more likely the resource's processing capacity is nearing saturation, and subsequent scheduling should prioritize whether its time window is sufficient. This is achieved by carrying [the value] on the associated edges. In the information and computing power data collaboration graph, each computing power supply-related edge not only represents a feasible resource allocation option, but also quantifies the supply margin of that option, providing differentiated weighting basis for the generation of subsequent collaborative execution paths.

[0124] In actual construction, because the load status of computing resources in a distributed environment changes dynamically over time, the collection of load status information needs to maintain a certain refresh frequency to ensure... and The calculation results can reflect the current true state of the resources. When there are a large number of computing power demand nodes, a batch matching method can be used to match all computing power demand nodes. Value and all candidate computing resources The values ​​are compared in a matrix format to quickly filter out node pairs that meet the constraints. Then, the filtered results are batch-established with associated edges and assigned... Weights are used to improve the overall efficiency of graph construction.

[0125] When multiple computing resources satisfy the constraints of the same computing power demand node, all satisfying computing resources will establish a computing power supply association edge with that demand node, forming a one-to-many association structure. This design preserves the choice space in the subsequent path generation stage, allowing the optimization of collaborative execution paths to be weighed among multiple feasible computing power allocation schemes, rather than locking a single allocation result in advance during the graph construction stage. Simultaneously, The difference in weights will guide the path generation process to prioritize the use of computing resources with more supply margin, thereby improving the overall balance of resource allocation in the computing power data collaboration graph and avoiding the situation where some nodes are overloaded while other nodes are idle.

[0126] In addition, allocable time windows Information can also be recorded as additional attributes on the associated edges, so that in scenarios with strict time constraints, the path generation stage can directly read the available time intervals corresponding to each computing power supply associated edge, and determine whether it is compatible with the execution time sequence dependency of computationally intensive operations in the task semantic structure, thereby further ensuring the feasibility of the generated collaborative execution path in the time dimension.

[0127] In one optional implementation, a data supply association edge is established between the data demand node and the data resource to form a computing power data collaboration graph that integrates task demand and resource supply, including:

[0128] Based on the physical distribution location information, calculate the network distance and data transmission delay between the data resource and the data demand node, and construct a data reachability evaluation index based on the network distance and data transmission delay;

[0129] Under the condition that the data accessibility evaluation index meets the preset accessibility threshold, a data supply association edge is established between the data demand node and the data resource, and an access cost weight is marked for the data supply association edge. The access cost weight represents the time cost required for the data demand node to access the data resource.

[0130] The computing power demand node, the data demand node, the transmission demand edge, the computing power supply association edge, and the data supply association edge are organized into a graph structure to construct the computing power data collaboration graph.

[0131] A data supply association is established between data demand nodes and data resources to obtain the physical distribution location information of data resources. In a distributed environment, data resources are deployed across storage nodes, data centers, or edge devices in different geographical locations, and the storage nodes are interconnected through network links with differentiated bandwidth and latency characteristics. The physical distribution location information includes the network topology coordinates of the storage nodes, their subnet identifiers, and descriptions of link connectivity with other nodes. This information comes from the semantic annotation results of the data resources' capabilities in the resource panorama view.

[0132] Based on physical distribution location information, for each pair of data request nodes and candidate data resources, the network distance and data transmission latency between them are calculated. The network distance reflects the number of hops or the topological path length between the storage node where the data resource resides and the computation location associated with the data request node, and can be obtained by performing a shortest path algorithm on the network topology map in the resource panorama view. The data transmission latency comprehensively considers three components: link propagation latency, queuing latency, and transmission time caused by data volume. Let the network distance between the data request node and the data resource be... The propagation delay is Queue delay is The amount of data to be transmitted is The peak bandwidth of the link is Then the end-to-end data transmission delay It can be represented as .in and Positive correlation It is dynamically estimated from the current link load status.

[0133] Based on network distance With data transmission delay Construct data accessibility assessment indicators . This comprehensive assessment measures whether data resources can be accessed by data-demanding nodes with acceptable latency under the current network conditions. The calculation method involves a normalized weighted fusion of network distance and transmission latency. Let the normalized network distance value be... The normalized value of the transmission delay is The distance weighting coefficient is The delay weighting coefficient is ,but . The range of values ​​is A higher value indicates better accessibility of the data resource. The normalization process uses the maximum network distance and maximum transmission delay of all candidate data resources in the current resource panorama view as the normalization benchmark to ensure that indicators of different dimensions participate in the fusion calculation on the same scale. and The value of can be configured according to the emphasis on latency sensitivity and topological distance sensitivity in specific application scenarios. In typical scenarios, the sum of the two is 1.

[0134] exist Meet the preset accessibility threshold Under the conditions that, i.e. At that time, a data supply association edge is established between the corresponding data demand node and the data resource. The setting is based on the task's tolerance limit for data access latency. For real-time inference-type artificial intelligence computing tasks, Typically, a higher setting is used to ensure that data access latency meets inference latency constraints; for offline training tasks... The range of available data resources can be appropriately reduced. When a certain data resource... Below In this case, the data resource is not included in the supply candidate set of the corresponding data demand node, and no data supply association edge is established between the two. Thus, the pruning of unreachable resources is completed in the graph construction stage, avoiding the introduction of infeasible data access paths in the subsequent path planning stage.

[0135] After establishing the data supply association edges, assign access cost weights to each association edge. . This characterizes the time overhead required for data-demanding nodes to access corresponding data resources, directly expressed as end-to-end data transmission latency. This is based on calculations. However, it takes into account the local read latency of storage nodes during actual access. (Determined by the type of storage medium and the efficiency of data indexing). Defined as . Extracting from the storage capacity characteristics of data resources, for solid-state storage media, Typically, this process takes place on the order of microseconds to milliseconds; for mechanical hard disk storage... It can last for tens of milliseconds. The time unit (milliseconds) is marked on the data supply association edge, which can be directly used in the constraint subgraph path cost evaluation during the subsequent collaborative execution path generation stage.

[0136] After completing the construction of all computing power demand nodes, data demand nodes, transmission demand edges, computing power supply association edges, and data supply association edges, the above graph elements are organized into a unified graph structure to construct a computing power-data collaboration graph. The graph structure organization process is managed hierarchically according to node and edge types: computing power demand nodes and data demand nodes constitute the task demand layer, representing the execution logic decomposition results of artificial intelligence computing tasks; computing power resource nodes and data resource nodes constitute the resource supply layer, representing schedulable physical or virtual resource entities in a distributed environment; transmission demand edges connect task demand nodes at the same layer, reflecting the data flow transmission relationship between computing operations; computing power supply association edges and data supply association edges connect task demand nodes and resource supply nodes across layers, carrying supply capacity weights (computing power supply association edges) or access cost weights. (Data supply to related edges) forms a complete two-layer heterogeneous graph structure.

[0137] The graph structure of the computing power data collaboration graph adopts a storage method combining adjacency lists and edge attribute dictionaries. The adjacency list uses node identifiers as keys to store the identifiers of all nodes adjacent to a given node and the corresponding edge type labels. The edge attribute dictionary uses ordered node pairs as keys to store attribute fields such as edge type, supply capacity weight, or access cost weight. This storage method supports efficient retrieval and weight reading of specific edge types during subsequent constraint subgraph identification and path cost evaluation, while also facilitating local updates of edge attributes without reconstructing the entire graph when resource states dynamically change.

[0138] After the graph structure is organized, the computing power data collaboration graph retains both the structural information of the task execution logic and the quantitative information of resource supply capacity, providing a complete graph analysis foundation for subsequently identifying data supply bottleneck constraint subgraphs based on execution time dependencies. (The data supply-related edges are labeled...) In the cost evaluation of the transmission demand edge path of the constrained subgraph, it is included as a data access latency item in the comprehensive cost calculation, ensuring that the generated collaborative execution path simultaneously meets the latency constraints of task execution in both computing power allocation and data access dimensions.

[0139] Figure 2This is a heatmap diagram illustrating the data accessibility distribution, showing the distribution of accessibility assessment metrics between data resource nodes and data demand nodes in a distributed environment. The horizontal axis represents the data resource ID, and the vertical axis represents the data demand node ID, with color depth transitioning from light blue (low accessibility) to dark blue (high accessibility). The diagram clearly reflects the accessibility screening process before establishing data supply association edges. Based on preset accessibility thresholds, pruning is performed, and only node pairs that meet the conditions are associated with each other, with access cost weights labeled. This effectively filters out invalid data access paths, providing reliable data layer support for collaborative graph construction.

[0140] In one optional implementation, based on the execution timing dependencies between computationally intensive and data-intensive operations in the task semantic structure, a constrained subgraph with a data supply bottleneck in the computing power-data collaboration graph is identified. Path cost evaluation is performed on the transmission demand edges in the constrained subgraph, generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path, including:

[0141] Traverse the computing power data collaboration graph, count the number of data supply association edges associated with each data demand node, mark the data demand node whose number of data supply association edges is less than the amount of data resources required by the data demand node as a bottleneck node, extract the bottleneck node and the adjacent computing power demand node and the adjacent transmission demand edge to form the constraint subgraph.

[0142] Obtain the source computing power demand node and the target computing power demand node connected by each of the transmission demand edges in the constraint subgraph. Based on the computing power supply association edges associated with the source computing power demand node and the target computing power demand node, determine the candidate computing power resource set for the source computing power demand node and the target computing power demand node. Calculate the network transmission cost between computing power resources in the candidate computing power resource set and combine it with the data transmission volume of the transmission demand edge in a weighted manner to generate the path cost evaluation value of the transmission demand edge.

[0143] Based on the predecessor-successor relationship in the execution time-series dependency, the computing power demand nodes and the data demand nodes in the constraint subgraph are arranged in ascending order according to the path cost evaluation value, and the computing power node allocation sequence and the data node access sequence are generated.

[0144] Based on the network transmission cost between adjacent nodes in the computing power node allocation sequence and the data node access sequence, the cross-node transmission path is constructed to form the collaborative execution path.

[0145] After obtaining the computing power data collaboration graph, it is necessary to identify local structures with data supply bottlenecks so that subsequent path cost evaluations can focus on the regions that truly constrain execution efficiency. This involves traversing all data demand nodes in the computing power data collaboration graph and counting the number of data supply association edges associated with each data demand node, denoted as . Each data requirement node corresponds to a data-intensive operation in the task semantic structure. The amount of data resources required for this operation is given by the operation description in the task semantic structure, denoted as . .when When a bottleneck node is identified, it indicates that the node cannot obtain sufficient data resources from the current resource panorama view, and the node is marked as a bottleneck node. After completing the full graph traversal, all bottleneck nodes are extracted, and then the computing power demand nodes adjacent to each bottleneck node and the transmission demand edges connecting these nodes are extracted. These nodes and edges together form a constraint subgraph. The scope of the constraint subgraph is limited to the direct neighborhood of the bottleneck node, ensuring that the computational scale of subsequent evaluation is controllable, while not omitting the critical path structure that affects data supply.

[0146] For each transmission demand edge in the constraint subgraph, its path cost evaluation value needs to be calculated. Each transmission demand edge connects a source computing power demand node and a target computing power demand node. Through the computing power supply association edges associated with the source computing power demand node, the candidate computing power resource set for that node can be determined, denoted as . Similarly, by using the computing power supply association edges associated with the target computing power demand node, the set of candidate computing power resources for the target node is determined, denoted as . .right and The computing resources are paired up, and the network transmission cost between each pair of computing resources is calculated. The network transmission cost comprehensively considers the physical link bandwidth and routing path length between the two computing resource nodes. For candidate resource pairs... Its network transmission cost is denoted as The specific value is determined by the link topology attributes recorded in the resource panorama view.

[0147] After obtaining the network transmission cost of the candidate resource pair, it is weighted and combined with the data transmission volume carried by the transmission demand edge to generate the path cost evaluation value of the transmission demand edge. Let the data transmission volume of the transmission demand edge be... Candidate resources The corresponding minimum network transmission cost is Path cost evaluation value The calculation method is as follows ,in The weighting coefficient for network transmission cost. Let be the weighting coefficient for the data transmission volume, and both be preset positive real numbers that satisfy . By using a weighted combination, the path cost assessment value simultaneously reflects the physical cost of the transmission path and the load pressure brought by the data scale, avoiding resource allocation bias caused by a single-dimensional assessment. The above calculation is performed sequentially on each transmission demand edge in the constraint subgraph to obtain the set of path cost assessment values ​​for all transmission demand edges.

[0148] Based on the execution time dependencies between computationally intensive and data-intensive operations in the task semantic structure, the predecessor-successor relationships of each node in the constraint subgraph are determined. Execution time dependencies describe the directed dependencies between operations, meaning an operation can only begin execution after all its predecessor operations are completed. In the constraint subgraph, both computational power demand nodes and data demand nodes are constrained by this dependency. The computational power demand nodes in the constraint subgraph are arranged in ascending order of path cost evaluation values. While satisfying the predecessor-successor relationship, computational power demand nodes with lower path costs are prioritized at the beginning of the sequence, thus generating a computational power node allocation sequence. For nodes with the same path cost evaluation value, their order is further distinguished based on their topological ranking position in the time dependency relationship to ensure the determinism of the sequence. The generation of the data node access sequence is similar; data demand nodes in the constraint subgraph are arranged in ascending order of path cost evaluation values, and the access order is determined while satisfying the time dependency relationship. The computational power node allocation sequence and the data node access sequence together describe the optimal execution order of various operation nodes within the constraint subgraph, providing an ordered node foundation for the subsequent construction of cross-node transmission paths.

[0149] Based on the computing power node allocation sequence and data node access sequence, the network transmission cost between adjacent nodes in the sequence is calculated to construct a cross-node transmission path. For two adjacent computing power demand nodes in the computing power node allocation sequence, their corresponding candidate computing power resource sets have been determined in the path cost evaluation stage. The selected path minimizes the network transmission cost. The resource pair with the smallest cost is used as the endpoint of the transmission path segment, and the routing path and its cost are recorded. For adjacent data request nodes in the data node access sequence, the transmission cost between adjacent data resources is calculated based on the location of the data resource pointed to by the associated edge of their associated data supply, and the path with the smallest cost is selected as the cross-node transmission path segment. The computing power node allocation sequence and each transmission path segment in the data node access sequence are connected sequentially to form a complete cross-node transmission path. The cross-node transmission path records the complete execution trajectory from the first computing power request node, through the data access operations of each data request node, to the last computing power request node. The transmission cost of each segment on the path has been optimized and selected using the path cost evaluation value.

[0150] The computing node allocation sequence, data node access sequence, and cross-node transmission path are merged to form the final collaborative execution path. This collaborative execution path comprehensively describes, within the constraint subgraph, which computing resources should be allocated to computationally intensive operations of the AI ​​computing task, in what order should data-intensive operations access which data resources, and through which physical transmission path should data between adjacent operations be transferred. With bottleneck nodes as the core constraint, the collaborative execution path uses a path cost evaluation-driven sorting mechanism to ensure that the overall transmission cost of the execution path is minimized under conditions of limited data supply, thereby improving the end-to-end execution efficiency of AI computing tasks in a distributed environment.

[0151] Figure 3 This diagram illustrates the identification of bottleneck nodes in data supply. It statistically analyzes the number of data supply edges associated with each data-demanding node in the computing power data collaboration graph. Blue bars represent the actual number of data supply edges associated with each data-demanding node, and red dashed lines mark the amount of data resources required by that node. When the number of data supply edges is lower than the required amount of data resources, the node is identified as a bottleneck node. The bottleneck node, along with its adjacent computing power-demanding nodes and transmission-demanding edges, constitutes a constraint subgraph. This diagram clearly shows the distribution of data supply bottlenecks, providing a focused scope for subsequent path cost evaluation in the constraint subgraph, avoiding blind searches across the entire graph, and significantly improving the targeting and efficiency of collaborative execution path generation.

[0152] A second aspect of the present invention provides an artificial intelligence control and computing power data collaboration system, comprising:

[0153] The semantic extraction unit is used to extract the semantics of the execution logic of artificial intelligence computing tasks to obtain the semantic structure of the tasks.

[0154] The resource annotation unit is used to perform capability semantic annotation on computing resources and data resources in the panoramic view of resources in the distributed environment to obtain the resource semantic structure.

[0155] Alignment mapping unit, used to establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure;

[0156] The collaborative graph unit is used to map the computationally intensive operations of the artificial intelligence computing task to computing power demand nodes, the data-intensive operations to data demand nodes, and the communication-intensive operations to transmission demand edges based on the alignment mapping relationship. According to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, it establishes computing power supply association edges between computing power demand nodes and computing power resources, and establishes data supply association edges between data demand nodes and data resources, forming a computing power data collaborative graph that integrates task demand and resource supply.

[0157] The path generation unit is used to identify the constrained subgraphs with data supply bottlenecks in the computing power data collaboration graph based on the execution time sequence dependencies between computationally intensive operations and data-intensive operations in the task semantic structure, and to evaluate the path cost of the transmission demand edges in the constrained subgraphs, thereby generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

[0158] A third aspect of the present invention provides an electronic device, comprising:

[0159] processor;

[0160] Memory used to store processor-executable instructions;

[0161] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0162] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0163] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for coordinating artificial intelligence control with computing power and data, characterized in that, include: Semantic extraction is performed on the execution logic of artificial intelligence computing tasks to obtain the task semantic structure; Capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain the resource semantic structure; Establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure; Based on the alignment mapping relationship, the computationally intensive operations of the artificial intelligence computing task are mapped to computing power demand nodes, the data-intensive operations are mapped to data demand nodes, and the communication-intensive operations are mapped to transmission demand edges. Furthermore, according to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, computing power supply association edges are established between the computing power demand nodes and computing power resources, and data supply association edges are established between the data demand nodes and data resources, forming a computing power data collaboration graph that integrates task demand and resource supply. Based on the execution timing dependency between computationally intensive and data-intensive operations in the task semantic structure, the constrained subgraphs with data supply bottlenecks in the computing power-data collaboration graph are identified, and the path cost of the transmission demand edges in the constrained subgraphs is evaluated to generate a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

2. The method according to claim 1, characterized in that, Semantic extraction is performed on the execution logic of artificial intelligence computing tasks to obtain the task semantic structure, including: The execution logic of the artificial intelligence computing task is parsed to identify the operation units involved in the execution logic and the calling dependencies between the operation units. Based on the computational load characteristics and data access characteristics of the operation unit, the operation unit is classified into operation types; Based on the relative strength relationship between the proportion of computation instructions and the proportion of memory access instructions in the computational load characteristics of the operation unit, operation units where the proportion of computation instructions is higher than the proportion of memory access instructions are marked as computation-intensive operations, and operation units where the proportion of memory access instructions is higher than the proportion of computation instructions are marked as data-intensive operations. Based on the call dependency relationship, data transmission paths located between different operation units are identified. Then, according to the ratio of data transmission volume on the data transmission path to data access volume within the operation unit, operations corresponding to data transmission paths where the ratio exceeds a preset communication density threshold are marked as communication-intensive operations. The computationally intensive operations, data-intensive operations, and communication-intensive operations are arranged topologically according to the call dependencies to construct the task semantic structure.

3. The method according to claim 1, characterized in that, Capability semantic annotation is performed on computing resources and data resources in the resource panorama view of the distributed environment to obtain the resource semantic structure, including: Based on the processor architecture information and parallel processing capability information of the computing resources, a computing capability feature vector is generated for the computing resources; based on the storage medium type information and data throughput rate information of the data resources, a storage capability feature vector is generated for the data resources; based on the network topology connection relationship and link bandwidth information between nodes in the distributed environment, a cross-node transmission capability feature vector is generated. The computing power feature vector is projected onto a preset computing power semantic space, and the computing power label of the computing power resource is determined by the distribution position of the computing power feature vector in the computing power semantic space; Based on the similarity matching relationship between the storage capacity feature vector and the preset storage capacity benchmark set, the label corresponding to the benchmark item with the highest similarity is selected from the storage capacity benchmark set as the storage capacity label of the data resource; Based on the weighted combination of the link bandwidth component and the network delay component in the transmission capability feature vector, the transmission capability label for cross-node transmission is determined by a preset transmission capability classification rule; The computing capability tags corresponding to the computing power resources, the storage capability tags corresponding to the data resources, and the transmission capability tags corresponding to cross-node connections are associated and organized to construct the resource semantic structure.

4. The method according to claim 1, characterized in that, Establishing a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure includes: Extract the call dependencies between the computationally intensive operations, the data-intensive operations, and the communication-intensive operations from the task semantic structure, and construct a task execution dependency graph; Extract the topological connection relationships between the computing power resources, the data resources and the cross-node transmission paths from the resource semantic structure, and construct a resource topology connection graph; Identify the set of operation subgraphs with strong connectivity in the task execution dependency graph, and extract the topological invariants of each operation subgraph in the set of operation subgraphs; Identify the set of resource subgraphs with homogeneous supply capabilities in the resource topology connection graph, and extract the structural signature of each resource subgraph in the set of resource subgraphs; Based on the graph isomorphism criterion, the structural similarity between the topological invariant and the structural signature is calculated, and a subgraph-level mapping relationship is established between the operation subgraph and the resource subgraph according to the structural similarity. The operation nodes inside the operation subgraph and the resource nodes inside the resource subgraph in the subgraph-level mapping relationship are mapped according to the local correspondence between the topological invariant and the structural signature to generate the cross-semantic space alignment mapping relationship.

5. The method according to claim 1, characterized in that, Based on the dynamic supply capacity of computing resources and the distribution of data resource storage locations in the resource panorama view, a computing power supply association edge is established between the computing power demand node and the computing power resource, including: Obtain the load status information of the computing power resources at the current moment and the physical distribution location information of the data resources from the panoramic view of the resources; Based on the load status information, the remaining processing capacity and allocable time window of the computing resources are calculated, and the remaining processing capacity and allocable time window are used as the dynamic supply capacity of the computing resources. Based on the matching relationship between the computing load demand of the computing power demand node and the dynamic supply capacity of the computing power resources, under the constraint that the computing load demand is less than or equal to the remaining processing capacity, a computing power supply association edge is established between the computing power demand node and the computing power resources, and a supply capacity weight is marked on the computing power supply association edge. The supply capacity weight represents the allocatable processing capacity of the computing power resources for the computing power demand node.

6. The method according to claim 5, characterized in that... A data supply association edge is established between the data demand node and the data resource to form a computing power data collaboration graph that integrates task demand and resource supply, including: Based on the physical distribution location information, calculate the network distance and data transmission delay between the data resource and the data demand node, and construct a data reachability evaluation index based on the network distance and data transmission delay; Under the condition that the data accessibility evaluation index meets the preset accessibility threshold, a data supply association edge is established between the data demand node and the data resource, and an access cost weight is marked for the data supply association edge. The access cost weight represents the time cost required for the data demand node to access the data resource. The computing power demand node, the data demand node, the transmission demand edge, the computing power supply association edge, and the data supply association edge are organized into a graph structure to construct the computing power data collaboration graph.

7. The method according to claim 1, characterized in that... Based on the execution timing dependencies between computationally intensive and data-intensive operations in the task semantic structure, constrained subgraphs with data supply bottlenecks in the computing power-data collaboration graph are identified. Path cost evaluation is performed on the transmission demand edges in these constrained subgraphs, generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and cross-node transmission paths, including: Traverse the computing power data collaboration graph, count the number of data supply association edges associated with each data demand node, mark the data demand node whose number of data supply association edges is less than the amount of data resources required by the data demand node as a bottleneck node, extract the bottleneck node and the adjacent computing power demand node and the adjacent transmission demand edge to form the constraint subgraph. Obtain the source computing power demand node and the target computing power demand node connected by each of the transmission demand edges in the constraint subgraph. Based on the computing power supply association edges associated with the source computing power demand node and the target computing power demand node, determine the candidate computing power resource set for the source computing power demand node and the target computing power demand node. Calculate the network transmission cost between computing power resources in the candidate computing power resource set and combine it with the data transmission volume of the transmission demand edge in a weighted manner to generate the path cost evaluation value of the transmission demand edge. Based on the predecessor-successor relationship in the execution time-series dependency, the computing power demand nodes and the data demand nodes in the constraint subgraph are arranged in ascending order according to the path cost evaluation value, and the computing power node allocation sequence and the data node access sequence are generated. Based on the network transmission cost between adjacent nodes in the computing power node allocation sequence and the data node access sequence, the cross-node transmission path is constructed to form the collaborative execution path.

8. An artificial intelligence control and computing power data collaborative system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The semantic extraction unit is used to extract the semantics of the execution logic of artificial intelligence computing tasks to obtain the semantic structure of the tasks. The resource annotation unit is used to perform capability semantic annotation on computing resources and data resources in the panoramic view of resources in the distributed environment to obtain the resource semantic structure. Alignment mapping unit, used to establish a cross-semantic space alignment mapping relationship between the task semantic structure and the resource semantic structure; The collaborative graph unit is used to map the computationally intensive operations of the artificial intelligence computing task to computing power demand nodes, the data-intensive operations to data demand nodes, and the communication-intensive operations to transmission demand edges based on the alignment mapping relationship. According to the dynamic supply capacity of computing power resources and the storage location distribution of data resources in the resource panorama view, it establishes computing power supply association edges between computing power demand nodes and computing power resources, and establishes data supply association edges between data demand nodes and data resources, forming a computing power data collaborative graph that integrates task demand and resource supply. The path generation unit is used to identify the constrained subgraphs with data supply bottlenecks in the computing power data collaboration graph based on the execution time sequence dependencies between computationally intensive operations and data-intensive operations in the task semantic structure, and to evaluate the path cost of the transmission demand edges in the constrained subgraphs, thereby generating a collaborative execution path that includes a computing power node allocation sequence, a data node access sequence, and a cross-node transmission path.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images