A deep learning-based heterogeneous computing resource collaborative management system and method

Abstract

The application discloses a heterogeneous computing resource collaborative management system and method based on deep learning, and belongs to the technical field of computing resource management. The application receives a task description, extracts computing core features through semantic analysis to form a feature set, converts the feature set into a task vector to input a pre-training model, and outputs a computing requirement range; a feature library is established by extracting parameters and states of a selected host, and an adaptive list is obtained through core computing hard matching, heterogeneous scene adaptation and state filtering, and the state is updated and bound after the user selects the host; storage features are extracted from the task vector, an initial capacity is calculated, and the storage resource and requirement are abstracted into a graph structure for matching to obtain a block scheme, and redundant capacity is integrated into a storage configuration; network demand indexes are extracted from the task vector, the host and the storage configuration, a network feature library is established, a screening rule is defined, and an optimal network scheme is determined by using a path optimization algorithm after matching and adapting resources.

Description

Technical Field

[0001] This invention relates to the field of computing resource management technology, specifically a heterogeneous computing resource collaborative management system and method based on deep learning. Background Technology

[0002] With the deep integration of the digital economy with fields such as artificial intelligence and scientific computing, the demand for heterogeneous computing resources for tasks is becoming increasingly complex. It is necessary to meet the GPU computing power requirements of high computing power intensive tasks, as well as adapt to the storage I / O and network bandwidth requirements of data intensive tasks. The collaborative management of heterogeneous computing resources has become the core of ensuring the efficient operation of tasks.

[0003] Existing technologies largely rely on manual estimation of task computing power requirements or rough calculations based on simple parameters. They fail to utilize deep learning to study the mapping patterns between historical task characteristics and computing power, leading to significant errors in the estimation of computing power requirements. This results in either task delays due to insufficient computing power or resource waste due to excess computing power. Host selection often focuses only on static hard matching of core computing power parameters, ignoring heterogeneous scenario adaptation and real-time host status. This easily leads to problems such as core parameters meeting requirements but scenario incompatibility, or hosts being selected despite failure, resulting in low adaptation accuracy. Storage capacity calculations do not incorporate the task's entire lifecycle data requirements and fail to use graph structures to optimally map storage resources to task needs, resulting in low storage utilization. Redundancy capacity is not differentiated according to task reliability requirements, leading to either excessive or insufficient reliability. Summary of the Invention

[0004] The purpose of this invention is to provide a heterogeneous computing resource collaborative management system and method based on deep learning to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Firstly, this application provides a method for collaborative management of heterogeneous computing resources based on deep learning, comprising the following steps:

[0007] The system receives a task description input from the user, extracts the core features affecting computing power requirements through semantic parsing, and forms a task feature set. The task feature set is then transformed into a task vector, which is input into a pre-trained computing power requirement prediction model. The model learns the feature and computing power mapping rules based on historical task data and outputs the range of computing power requirements for the task.

[0008] Extract the core computing power parameters and status information of all candidate hosts to build a host computing power feature library; based on the host computing power feature library, perform core computing power hard matching, heterogeneous scenario adaptation and secondary status filtering to form a list of adapted hosts; convert the list of adapted hosts into a selection interface, where users select hosts and perform status update and binding operations.

[0009] Extract core features affecting storage allocation from task vectors to construct a storage demand feature set; calculate the initial storage capacity based on the storage demand features and calibrate it in conjunction with the selected host configuration; abstract storage resources and task requirements into a graph structure, and achieve the optimal mapping between task storage requirements and storage resources through a graph matching algorithm to maximize storage utilization and obtain a block scheme; calculate the redundancy capacity according to task reliability requirements and integrate the block scheme to form a storage configuration.

[0010] The network resource demand characteristics are extracted from the task vector, selected host configuration, and storage configuration to form network demand indicators; a network resource feature library is built based on the network demand indicators, and screening rules that match the demands are formulated; suitable resources are matched from the network resource feature library based on the screening rules, and the optimal network connection scheme is determined through path optimization algorithms.

[0011] In conjunction with the first aspect, in a first embodiment of the first aspect of this application, the task description received from the user is used to extract core features affecting computing power requirements through semantic parsing to form a task feature set, including:

[0012] The system receives task descriptions submitted by users in natural language, structured forms, or mixed formats and performs preprocessing. Based on a pre-trained natural language understanding model, it performs deep semantic analysis on the preprocessed text to identify explicit features affecting computing power requirements. This natural language understanding model uses a BERT fine-tuning model. Combining a domain knowledge graph and a historical task feature library, it mines implicit features that are not directly expressed but affect computing power requirements, thus completing the feature dimensions. The initially identified and completed features are standardized to eliminate expression differences and verify logical conflicts. The standardized explicit and implicit features are integrated to form a task feature set.

[0013] In conjunction with the first aspect, in the second embodiment of the first aspect of this application, the step of converting the task feature set into a task vector, inputting it into a pre-trained computing power demand prediction model, and the model outputting the computing power demand range of the task based on the feature and computing power mapping rules learned from historical task data includes:

[0014] The system classifies various types of features in the task feature set, transforming non-numerical features into computable numerical forms and integrating them into a unified-dimensional task vector. This task vector is then input into a pre-trained computing power demand prediction model. The model learns the mapping relationship between features and computing power through a multi-layer network structure. Branch networks process categorical features, numerical features, and textual features separately, while cross-attention layers capture the association weights of different feature branches. The model activates the mapping patterns learned from historical task data through hidden layers, performs similarity matching between the current task vector and historical feature vectors, and calls the computing power mapping parameters of the most similar task. The ReLU activation function and residual connections capture the non-linear relationship between features and computing power. The model's output layer generates the computing power demand range through a dual-branch structure.

[0015] In conjunction with the first aspect, in the third embodiment of the first aspect of this application, the step of performing core computing power hard matching, heterogeneous scenario adaptation, and secondary state filtering based on the host computing power feature library to form an adapted host list includes:

[0016] Determine core computing power metrics based on task type, ensuring that parameters in the host feature library match the required metrics. Iterate through the host computing power feature library, filtering hosts whose core computing power metrics fall entirely within the required range, and excluding hosts whose core computing power metrics exceed the range. When a task requires a specific hardware architecture, filter hosts in the host feature library whose hardware architecture tags match the task requirements. When a task requires distributed computing, filter hosts with distributed communication interfaces and whose resource pools can form continuous node clusters. Filter hosts with an idle status tag, excluding allocated, faulty, or under maintenance hosts, and check the temporary lock status of hosts, retaining only completely idle hosts. Based on the health score in the host feature library, set a threshold to exclude hosts with health scores below the threshold. When a task has resource pool geographical restrictions, filter hosts in the host feature library whose resource pool identifier matches the constraints; when there are no restrictions, prioritize retaining hosts within the same resource pool.

[0017] The filtering results are integrated to form a list of compatible hosts.

[0018] In conjunction with the first aspect, in the fourth embodiment of the first aspect of this application, the step of calculating the initial storage capacity based on storage demand characteristics and calibrating it in conjunction with the selected host configuration includes:

[0019] Quantitative parameters are extracted from the storage demand feature set, and the initial storage capacity is calculated by accumulating data throughout the entire life cycle. All data volume units are unified to GB and rounded up to the smallest allocation unit of the storage device.

[0020] Determine the types, number, and maximum capacity of the storage interfaces supported by the host; extract the storage protocols supported by the host and exclude interfaces that are incompatible with the task data transmission protocol; obtain the theoretical maximum IOPS and bandwidth of the host storage interfaces as a performance matching threshold; when the initial capacity exceeds the maximum capacity supported by a single interface, split the capacity into multiple interfaces; integrate the calibration results and output the final initial storage capacity that conforms to the host configuration and meets the task requirements.

[0021] In conjunction with the first aspect, in the fifth embodiment of the first aspect of this application, the step of abstracting storage resources and task requirements into a graph structure, and achieving the optimal mapping between task storage requirements and storage resources through a graph matching algorithm to maximize storage utilization and obtain a block partitioning scheme includes:

[0022] The storage resource pool and task storage requirements are modeled as directed weighted graphs, specifically: for the storage resource graph... ,node Based on independent storage units as basic nodes, including physical storage devices or logical storage pools, node attributes include capacity limit, current free capacity, read / write bandwidth, IOPS, storage type, and the resource pool to which it belongs. This represents the physical connection links between storage units, with edge weights equal to the data transmission latency between storage units; the weight is set to ∞ when there is no direct connection. (For the task storage requirement graph...) ,node Task storage requirements are broken down into sub-nodes based on the data lifecycle, including input data blocks. Intermediate result blocks and output data blocks Node attributes include required capacity, read / write frequency, data type, and reliability requirements. This represents the data dependency relationship between child nodes, and the edge weight is the amount of data transmitted between nodes.

[0023] Based on the optimal I / O unit of the storage device, the free capacity of the nodes in the storage resource graph is divided into several standard blocks to ensure that the blocks match the hardware read and write units; according to the standard block size of the storage unit, the capacity of each node in the task storage requirement graph is divided into an integer number of standard blocks, while maintaining the dependency relationship between child nodes; granularity verification is performed to ensure that the total number of task blocks is not greater than the total number of free standard blocks of the storage resource, and that the size of a single task block is not greater than the maximum supported size of a single block of the storage unit.

[0024] Task request nodes are prioritized by a weighted average of read / write frequency and data transfer volume, while storage resource nodes are prioritized by a weighted average of their idle capacity utilization potential and performance metrics. Starting with the highest priority task request node, storage resource nodes are traversed, and storage nodes with remaining capacity not less than the task block capacity, performance metrics not less than the task requirement threshold, and priority within the same resource pool are selected for mapping. For task nodes with dependencies, they are prioritized to be mapped to the same storage node. The storage utilization rate after the initial matching is calculated, and storage nodes with a storage utilization rate less than the set threshold are optimized through block migration.

[0025] In conjunction with the first aspect, in the sixth embodiment of the first aspect of this application, the step of calculating redundant capacity based on task reliability requirements and integrating the block scheme to form a storage configuration includes:

[0026] Reliability requirement features influencing redundancy strategies are extracted from the task vector, including data importance, tolerable loss rate, business continuity requirements, and sensitivity to historical failures. These reliability requirement features are categorized into high reliability, medium reliability, and low reliability, and mapped to redundancy strategies. The block partitioning scheme is matched with the redundancy strategy; for each block, redundancy capacity is calculated according to its reliability level, and the redundancy capacity of all blocks is summed to obtain the total task redundancy requirement. This total task redundancy requirement is allocated to storage resources, prioritizing storage units physically isolated from the storage nodes where the main blocks reside to carry redundant data. The block partitioning scheme and redundancy configuration are integrated to form a storage configuration.

[0027] In conjunction with the first aspect, in the seventh embodiment of the first aspect of this application, the step of constructing a network resource feature database based on network demand indicators and formulating screening rules that match the demand includes:

[0028] Network demand metrics are standardized and integrated into a network resource feature library. Mapping rules are defined between network demand metrics and fields in the network resource feature library, including: peak bandwidth demand is mapped to bandwidth limit and available bandwidth, latency tolerance is mapped to latency, cross-resource pool demand is mapped to coverage resource pool, concurrent connection demand is mapped to maximum device connection count, and reliability demand is mapped to failure rate and redundancy capability. Based on the mapping rules, multi-dimensional filtering rules are formulated according to the rigidity of demand, and the rule priorities are clearly defined. The rule priorities, from high to low, include: hard constraint rules, soft constraint rules, and optimization constraint rules. Hard constraint rules include bandwidth constraints, protocol constraints, and topology constraints; soft constraint rules include latency constraints and load constraints; and optimization constraint rules include reliability optimization and energy consumption optimization.

[0029] In conjunction with the first aspect, in the eighth embodiment of the first aspect of this application, the step of matching suitable resources from the network resource feature library based on filtering rules and determining the optimal network connection scheme through a path optimization algorithm includes:

[0030] Traverse the network resource feature library, check whether each type of resource satisfies hard constraint rules, soft constraint rules and optimization constraint rules in turn, and form a candidate resource list sorted by priority; perform availability and correlation verification on the candidate resources, update the real-time status of the candidate resources, exclude resources whose status has changed to maintenance or overload, check the connectivity between candidate resources, and when the same resource is selected by multiple task candidate sets, allocate according to task priority, and low priority tasks are automatically switched to the second best candidate resources;

[0031] Candidate resources are abstracted into a structured topology graph, and target weights are assigned according to task attributes. For each pair of host and storage block connection requirements, the A* algorithm is used to find the path with the minimum weight based on the edge weights of the topology graph. For scenarios where the same host needs to connect to multiple storage blocks, the maximum flow minimum cut algorithm is used to assign the paths of different hosts and storage blocks to different links to determine the optimal network connection scheme.

[0032] Secondly, this application provides a deep learning-based heterogeneous computing resource collaborative management system, comprising:

[0033] The computing power requirement range calculation module includes a task feature set generation unit and a computing power requirement range calculation unit. The task feature set generation unit receives the task description input by the user, extracts the core features affecting computing power requirements through semantic parsing, and forms a task feature set. The computing power requirement range calculation unit converts the task feature set into a task vector, inputs it into a pre-trained computing power requirement prediction model, and outputs the computing power requirement range of the task based on the features and computing power mapping rules learned from historical task data.

[0034] The host selection module includes a host computing power feature library construction unit, a compatible host list generation unit, and a host selection unit. The host computing power feature library construction unit extracts the core computing power parameters and status information of all candidate hosts to construct a host computing power feature library. The compatible host list generation unit, based on the host computing power feature library, performs core computing power hard matching, heterogeneous scenario adaptation, and secondary status filtering to form a compatible host list. The host selection unit converts the compatible host list into a selection interface, allowing the user to select a host and perform status update and binding operations.

[0035] The storage configuration generation module includes: a storage requirement feature acquisition unit, a host configuration calibration unit, a block scheme generation unit, and a storage configuration generation unit. Specifically, the storage requirement feature acquisition unit extracts core features affecting storage allocation from task vectors to construct a storage requirement feature set; the host configuration calibration unit calculates the initial storage capacity based on the storage requirement features and calibrates it in conjunction with the selected host configuration; the block scheme generation unit abstracts storage resources and task requirements into a graph structure, and uses a graph matching algorithm to achieve the optimal mapping between task storage requirements and storage resources, maximizing storage utilization and obtaining a block scheme; the storage configuration generation unit calculates redundancy capacity based on task reliability requirements and integrates the block schemes to form a storage configuration.

[0036] The optimal network connection scheme generation module includes: a network demand indicator acquisition unit, a filtering rule formulation unit, and an optimal network connection scheme generation unit. The network demand indicator acquisition unit extracts network resource demand features from task vectors, selected host configurations, and storage configurations to form network demand indicators. The filtering rule formulation unit constructs a network resource feature library based on the network demand indicators and formulates filtering rules that match the demands. The optimal network connection scheme generation unit matches suitable resources from the network resource feature library based on the filtering rules and determines the optimal network connection scheme through a path optimization algorithm.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] 1. This invention uses a pre-trained deep learning model to learn the mapping pattern between features and computing power from historical task data, which can accurately output the range of computing power requirements for tasks, avoid the bias of manual estimation, and lay an accurate foundation for subsequent resource matching.

[0039] 2. The host selection of the present invention adopts a multi-dimensional logic of core computing power hard matching, heterogeneous scenario adaptation and secondary status filtering, which not only ensures that the core computing power meets the standard, but also takes into account the heterogeneous scenario and the real-time status of the host, greatly improving the accuracy and availability of host adaptation.

[0040] 3. This invention uses a graph structure to abstractly model storage resources and task requirements, and combines greedy matching and global optimization algorithms to achieve optimal mapping, thereby improving storage utilization. It also calculates redundant capacity differently based on task reliability requirements, thus balancing storage efficiency and data reliability. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the steps of a heterogeneous computing resource collaborative management method based on deep learning according to the present invention.

[0042] Figure 2 This is a system architecture diagram of a heterogeneous computing resource collaborative management system based on deep learning, according to the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example: Figures 1-2 As shown, the present invention provides a technical solution:

[0045] like Figure 1 As shown, this application provides a method for collaborative management of heterogeneous computing resources based on deep learning, including the following steps:

[0046] Step S100: Receive the task description input by the user, extract the core features affecting computing power demand through semantic parsing to form a task feature set; convert the task feature set into a task vector, input it into a pre-trained computing power demand prediction model, and output the computing power demand range of the task based on the features and computing power mapping rules learned from historical task data.

[0047] Specifically, the system receives task descriptions submitted by users in natural language, structured forms, or mixed formats and performs preprocessing. Based on a pre-trained natural language understanding model, it performs deep semantic analysis on the preprocessed text to identify explicit features affecting computing power requirements. This natural language understanding model uses a BERT fine-tuning model. Combining a domain knowledge graph and a historical task feature library, it mines implicit features that are not directly expressed but affect computing power requirements, thus completing the feature dimensions. The initially identified and completed features are standardized to eliminate expression differences and verify logical conflicts. The standardized explicit and implicit features are integrated to form a task feature set.

[0048] Furthermore, the model classifies the various types of features in the task feature set, transforming non-numerical features into computable numerical forms and integrating all types of features into a unified-dimensional task vector. This task vector is then input into a pre-trained computing power demand prediction model. The model learns the mapping relationship between features and computing power through a multi-layer network structure. Branch networks process categorical features, numerical features, and text features respectively, while cross-attention layers capture the association weights of different feature branches. The model activates the mapping patterns learned from historical task data through hidden layers, performs similarity matching between the current task vector and historical feature vectors, and calls the computing power mapping parameters of the most similar task. The ReLU activation function and residual connections capture the non-linear relationship between features and computing power. The model's output layer generates the computing power demand range through a dual-branch structure.

[0049] In one specific embodiment, the user inputs a task description: "Train 1 million 256×256 pixel RGB images using a ResNet50 model, to be completed within 3 days, with an allowable accuracy error of ≤1%, and supporting 8-way parallel computing." After preprocessing and removing redundant descriptions, it is standardized as: "Task type: AI model training; Core algorithm: ResNet50; Data scale: 1 million 256×256×3 pixel images; Time constraint: 72 hours; Accuracy requirement: ≤1%; Parallelism: 8-way."

[0050] Based on BERT fine-tuning model analysis, explicit features were extracted: Task type: AI model training (labeling); Core algorithm: ResNet50 (text features); Data features: 1 million images (equivalent to 186GB, 8-bit precision), 256×256 resolution; Constraints: Time 72 hours, accuracy ≤1%, parallelism 8.

[0051] Supplemented by domain knowledge graph: Algorithm complexity: 5.3 × 10^9 FLOPs for a single sample of ResNet50; Data preprocessing requirements: Image decoding and normalization require an additional 15% computing power; Time constraint transformation: Minimum computing power density ≥ 2.1 PFlops is required to complete the transformation within 72 hours.

[0052] Standardized feature set: Numerical features: data volume 186GB (Z-score 0.82 after standardization), time 72h (0.65), parallelism 8 (0.71); Class features: ResNet50 (embedding vector dimension 32, cosine similarity with ResNet101 0.89); No conflict verification (186GB data + 8-way parallelism can be completed within 72 hours).

[0053] The standardized features are integrated into a 512-dimensional task vector: category embedding vector (32-dimensional) + numerically normalized features (8-dimensional) + text compression vector (472-dimensional); the cross-feature layer calculates the interaction term "data volume × algorithm complexity" (weight 0.35) to enhance feature correlation.

[0054] After the model inputs a 512-dimensional task vector: the cross-attention layer assigns the highest weight (0.42) to the "algorithm complexity" feature; it matches the "ResNet50+1.2 million images" task (cosine similarity 0.91) in the historical database and calls its mapping parameters; the output computing power requirement range is 22-38 PFlops of GPU computing power (95% confidence level), which meets the 72-hour completion constraint.

[0055] Step S200: Extract the core computing power parameters and status information of all candidate hosts and build a host computing power feature library; based on the host computing power feature library, perform core computing power hard matching, heterogeneous scenario adaptation and secondary status filtering to form a list of adapted hosts; convert the list of adapted hosts into a selection interface, where users select hosts and perform status update and binding operations.

[0056] Specifically, core computing power metrics are determined based on task type, ensuring that the parameters in the host feature library are consistent with the required metrics. The host computing power feature library is traversed to filter hosts whose core computing power metrics fall entirely within the computing power requirement range, excluding hosts whose core computing power metrics exceed the range. When a task requires a specific hardware architecture, hosts in the host feature library whose hardware architecture tags match the task requirements are selected. When a task requires distributed computing, hosts with distributed communication interfaces and whose resource pools can form a continuous node cluster are selected. Hosts with the status tag "idle" are selected, excluding those that are allocated, awaiting repair, or under maintenance. The temporary lock status of hosts is checked, and only completely idle hosts are retained. Based on the health score in the host feature library, a threshold is set to exclude hosts with a health score below the threshold. When a task has resource pool geographical restrictions, hosts in the host feature library whose resource pool identifier matches the constraints are selected; when there are no restrictions, hosts within the same resource pool are prioritized for retention.

[0057] The filtering results are integrated to form a list of compatible hosts.

[0058] In one specific embodiment, the core parameters and status of the four candidate hosts are extracted:

[0059] Host 1: Model NVIDIA DGX A100, GPU computing power 32 PFlops (8×A100), NVIDIA architecture, supports InfiniBand distributed interface, status "idle", health score 92, belongs to resource pool "Wuhan Smart Computing Pool";

[0060] Host 2: Model: Dell R7525 (4×V100), GPU computing power: 24 PFlops, architecture: NVIDIA, supports InfiniBand, status: "Idle", health score: 88, resource pool: "Wuhan Smart Computing Pool";

[0061] Host 3: Model: Inspur NF5288 (2×V100), GPU computing power: 18 PFlops, architecture: NVIDIA, no distributed interface, status: "under maintenance", health score: 75, resource pool: "Intelligent Computing Pool of Hong Kong Special Administrative Region of the People's Republic of China";

[0062] Host 4: Model: Huawei Atlas 900, GPU computing power: 40 PFlops, architecture: ARM, supports RoCE interface, status: "Idle", health score: 90, resource pool: "Shenzhen Smart Computing Pool".

[0063] Using the S100's output GPU computing power of 22-38 PFlops as a hard constraint:

[0064] Host 1 (32 PFlops) and Host 2 (24 PFlops) are within the range and are reserved;

[0065] Host 3 (18 PFlops < 22) and Host 4 (40 PFlops > 38) exceed the constraints and are directly excluded.

[0066] The task is to train ResNet50 AI, requiring "NVIDIA architecture + distributed communication interface":

[0067] Both Host 1 and Host 2 are based on NVIDIA architecture and support the InfiniBand distributed interface, which meets the requirements of the scenario. If there are AMD architecture hosts (although the computing power meets the requirements), they do not support NVIDIA mixed precision training, so there are no such hosts here and no need to exclude them.

[0068] Set filter criteria: Status "Idle", Health score ≥ 85:

[0069] Host 1 (idle, 92 points) and Host 2 (idle, 88 points) both meet the requirements, so they are retained;

[0070] If there is a host of the same specifications that is "faulty and awaiting repair" (e.g., GPU computing power of 28 PFlops but health score of 70), then exclude it;

[0071] There are no geographical restrictions on the resource pool; hosts within the same resource pool (Wuhan Smart Computing Pool) are given priority and no additional screening is required.

[0072] The final list of compatible hosts is as follows:

[0073] Host 1 (NVIDIA DGX A100, 32 PFlops, Wuhan Smart Computing Pool, health level 92).

[0074] Host 2 (Dell R7525, 24 PFlops, Wuhan Smart Computing Pool, health status 88);

[0075] The list is transformed into a selection interface. After the user selects host 1, its status is updated to "Pending Assignment Locked" and the task ID is bound.

[0076] Step S300: Extract the core features affecting storage allocation from the task vector and construct a storage demand feature set; calculate the initial storage capacity based on the storage demand features and calibrate it in conjunction with the selected host configuration; abstract the storage resources and task requirements into a graph structure, and realize the optimal mapping between task storage requirements and storage resources through a graph matching algorithm to maximize storage utilization and obtain a block scheme; calculate the redundancy capacity according to the task reliability requirements and integrate the block scheme to form a storage configuration.

[0077] Specifically, quantitative parameters are extracted from the storage demand characteristics set, the initial storage capacity is calculated by accumulating data throughout the entire life cycle, all data volume units are unified to GB, and rounded up to the smallest allocation unit of the storage device;

[0078] Determine the types, number, and maximum capacity of the storage interfaces supported by the host; extract the storage protocols supported by the host and exclude interfaces that are incompatible with the task data transmission protocol; obtain the theoretical maximum IOPS and bandwidth of the host storage interfaces as a performance matching threshold; when the initial capacity exceeds the maximum capacity supported by a single interface, split the capacity into multiple interfaces; integrate the calibration results and output the final initial storage capacity that conforms to the host configuration and meets the task requirements.

[0079] Furthermore, the storage resource pool and task storage requirements are modeled as directed weighted graphs, specifically: for the storage resource graph... ,node Based on independent storage units as basic nodes, including physical storage devices or logical storage pools, node attributes include capacity limit, current free capacity, read / write bandwidth, IOPS, storage type, and the resource pool to which it belongs. This represents the physical connection links between storage units, with edge weights equal to the data transmission latency between storage units; the weight is set to ∞ when there is no direct connection. (For the task storage requirement graph...) ,node Task storage requirements are broken down into sub-nodes based on the data lifecycle, including input data blocks. Intermediate result blocks and output data blocks Node attributes include required capacity, read / write frequency, data type, and reliability requirements. This represents the data dependency relationship between child nodes, and the edge weight is the amount of data transmitted between nodes.

[0080] Based on the optimal I / O unit of the storage device, the free capacity of the nodes in the storage resource graph is divided into several standard blocks to ensure that the blocks match the hardware read and write units; according to the standard block size of the storage unit, the capacity of each node in the task storage requirement graph is divided into an integer number of standard blocks, while maintaining the dependency relationship between child nodes; granularity verification is performed to ensure that the total number of task blocks is not greater than the total number of free standard blocks of the storage resource, and that the size of a single task block is not greater than the maximum supported size of a single block of the storage unit.

[0081] Task request nodes are prioritized by a weighted average of read / write frequency and data transfer volume, while storage resource nodes are prioritized by a weighted average of their idle capacity utilization potential and performance metrics. Starting with the highest priority task request node, storage resource nodes are traversed, and storage nodes with remaining capacity not less than the task block capacity, performance metrics not less than the task requirement threshold, and priority within the same resource pool are selected for mapping. For task nodes with dependencies, they are prioritized to be mapped to the same storage node. The storage utilization rate after the initial matching is calculated, and storage nodes with a storage utilization rate less than the set threshold are optimized through block migration.

[0082] Furthermore, reliability requirement features influencing redundancy strategies are extracted from the task vector, including data importance, tolerable loss rate, business continuity requirements, and sensitivity to historical failures. These reliability requirement features are categorized into high reliability, medium reliability, and low reliability, and mapped to redundancy strategies. The block partitioning scheme is matched with the redundancy strategy; for each block, redundancy capacity is calculated according to its reliability level, and the redundancy capacity of all blocks is summed to obtain the total task redundancy requirement. This total task redundancy requirement is allocated to storage resources, prioritizing storage units physically isolated from the storage nodes where the main blocks reside to carry redundant data. The block partitioning scheme and redundancy configuration are integrated to form a storage configuration.

[0083] In one specific embodiment, storage requirement features are extracted from the task vector: the input data size for the ResNet50 training task is 186GB, the intermediate result generation coefficients are 3 (requiring 558GB), the output data size is 20GB, and the temporary space coefficient is 15% (27.9GB). The initial storage capacity is calculated by accumulating the data over the entire lifecycle: 186 + 558 + 20 + 27.9 ≈ 791.9GB. Rounded up to the minimum storage allocation unit of 10GB, the initial capacity is set at 790GB.

[0084] Selected host 1 (NVIDIA DGX A100) supports two NVMe interfaces, with a maximum capacity of 2TB per interface. The protocol is compatible with NVMe over Fabrics (matching the task transmission protocol), and the interface IOPS is 200,000 with a bandwidth of 32GB / s (meeting the task performance threshold). The 790GB capacity does not exceed the maximum capacity of a single interface and does not require splitting. After calibration, the final initial capacity is 790GB (NVMe type storage).

[0085] Storage resource graph nodes: 3 NVMe units (Unit 1: Wuhan pool, 800GB idle, 3GB / s bandwidth; Unit 2: Wuhan pool, 700GB idle, 2.8GB / s bandwidth; Unit 3: Shenzhen pool, 1TB idle, 2.5GB / s bandwidth), edge weights (transmission latency): Unit 1-Unit 25ms, Unit 1-Unit 350ms. Task requirement graph nodes: Input block (186GB, read / write frequency 1), Middle block (558GB, read / write frequency 5), Output block (20GB, read / write frequency 2), edge weights (transmission volume): Input → Middle 200GB, Middle → Output 30GB.

[0086] The optimal IO unit for NVMe is 128KB. A unit of 1800GB is split into 6250 standard 128KB blocks. Task partitioning: Input block 186GB is split into 1453 blocks, intermediate block 558GB is split into 4359 blocks, and output block 20GB is split into 156 blocks (total 5968 ≤ 6250, verification passed). Priority matching: Intermediate blocks (priority 5 × 558 = 2790) are preferentially mapped to unit 1 (800GB of free space, same pool). Input blocks and output blocks are also mapped to unit 1 because they depend on intermediate blocks. The total allocation is 764GB, and the utilization rate is 764 / 800 = 95.5% (exceeding the 85% threshold, no migration required). Segmentation: Unit 1 carries the main input, intermediate, and output blocks.

[0087] The task data is of "extremely high" importance (high reliability), and the redundancy strategy is "dual replicas + 10% checksum". Redundancy is calculated as follows: intermediate block 558GB → 558 + 55.8 = 613.8GB, input block 186GB → 186 + 18.6 = 204.6GB, output block 20GB → 20 + 2 = 22GB, for a total redundancy of 840.4GB. Partial redundancy is preferentially carried in unit 2 (Wuhan pool, 700GB free space), which is physically isolated from unit 1. The remaining redundancy is allocated to unit 3 (Shenzhen pool). Final storage configuration: primary block (764GB) in unit 1, redundant block (840.4GB) in unit 2 + unit 3, for a total storage of 1604.4GB.

[0088] Step S400: Extract network resource demand features from task vectors, selected host configurations, and storage configurations to form network demand indicators; construct a network resource feature library based on network demand indicators and formulate screening rules that match the demands; match suitable resources from the network resource feature library based on the screening rules and determine the optimal network connection scheme through path optimization algorithms.

[0089] Specifically, network demand metrics are standardized and integrated into a network resource feature library. Mapping rules are defined between network demand metrics and fields in the network resource feature library, including: peak bandwidth demand is mapped to bandwidth limit and available bandwidth, latency tolerance is mapped to latency, cross-resource pool demand is mapped to coverage resource pool, concurrent connection demand is mapped to the maximum number of connections per device, and reliability demand is mapped to failure rate and redundancy capability. Based on the mapping rules, multi-dimensional filtering rules are formulated according to the rigidity of demand, and the rule priorities are clarified. The rule priorities, from high to low, include: hard constraint rules, soft constraint rules, and optimization constraint rules. Hard constraint rules include bandwidth constraints, protocol constraints, and topology constraints; soft constraint rules include latency constraints and load constraints; and optimization constraint rules include reliability optimization and energy consumption optimization.

[0090] Furthermore, the network resource feature library is traversed, and each type of resource is checked in turn to see if it meets the hard constraint rules, soft constraint rules, and optimization constraint rules, forming a candidate resource list sorted by priority; the availability and relevance of the candidate resources are checked, the real-time status of the candidate resources is updated, resources whose status has changed to maintenance or overload are excluded, the connectivity between candidate resources is checked, and when the same resource is selected by multiple task candidate sets, it is allocated according to task priority, and low-priority tasks are automatically switched to the second-best candidate resources;

[0091] Candidate resources are abstracted into a structured topology graph, and target weights are assigned according to task attributes. For each pair of host and storage block connection requirements, the A* algorithm is used to find the path with the minimum weight based on the edge weights of the topology graph. For scenarios where the same host needs to connect to multiple storage blocks, the maximum flow minimum cut algorithm is used to assign the paths of different hosts and storage blocks to different links to determine the optimal network connection scheme.

[0092] In one specific embodiment, requirements are extracted from task, host, and storage configurations:

[0093] Peak bandwidth: Total data storage volume is 1604.4GB (72 hours of transmission), the calculated peak demand is 0.18Gbps, and after reserving redundancy, it is calculated as 0.2Gbps;

[0094] Latency tolerance: AI training parameter synchronization must be ≤10ms;

[0095] Topology requirements: Coverage of Wuhan local area (host + storage 1 / 2), Wuhan-Shenzhen cross-pool (storage 3);

[0096] Concurrent connections: 8 GPUs x 3 storage units on the host machine = 24 connections;

[0097] Reliability: Link redundancy must be supported (for high-reliability tasks).

[0098] The network resource feature database contains three candidate links:

[0099] Link 1 (Wuhan): 10Gbps bandwidth, 7.5Gbps available bandwidth, 5ms latency, NVMe over Fabrics protocol support, 0.02% failure rate, 25% load;

[0100] Link 2 (Wuhan-Shenzhen): 100Gbps bandwidth, 70Gbps available bandwidth, 12ms latency, protocol supported, 0.05% failure rate, 30% load;

[0101] Link 3 (Wuhan): 1Gbps bandwidth, 0.6Gbps available bandwidth, 8ms latency, protocol support, 0.1% failure rate, 40% load. Selection criteria: Hard constraints (bandwidth ≥ 0.2Gbps, protocol matching, topology coverage) → Soft constraints (latency ≤ 8ms, load + estimated occupancy ≤ 70%) → Optimization constraints (failure rate ≤ 0.05%).

[0102] Traversing the feature library for filtering:

[0103] Link 1: Meets hard constraints (10Gbps≥0.2Gbps, covering Wuhan) and soft constraints (latency 5ms≤8ms, load 25%+0.2 / 10=2.5%=27.5%≤70%). Optimization constraints are met, and it is given priority for retention.

[0104] Link 2: Hard constraints met (covering Wuhan-Shenzhen), soft constraint latency 12ms > 8ms, marked as "to be observed";

[0105] Link 3: Hard constraints are met, but its bandwidth of 1Gbps has a lower priority than Link 1. Soft constraints are met (load 40%+2.5%=42.5%≤70%), so it is the second-best candidate. Final candidate list: Link 1 (first), Link 2 (second), Link 3 (last).

[0106] Update real-time status:

[0107] Links 1, 2, and 3 are not under maintenance or overloaded. Link 1 is connected to Wuhan switch A and B (supporting storage 1 and 2), and Link 2 is connected to Wuhan switch A and Shenzhen switch C (supporting storage 3).

[0108] There are no resource conflicts (this task has a higher priority than other tasks), so all candidate resources are retained and there is no need to switch to the suboptimal one.

[0109] Topology nodes: Host (Wuhan), Switch A / B (Wuhan), Switch C (Shenzhen), Storage 1 / 2 (Wuhan), Storage 3 (Shenzhen); Edge weight (latency 0.6 + bandwidth 0.3 + cost 0.1):

[0110] Host → A (Link 1, Weight 0.3), A → B (Link 1, 0.3), A → C (Link 2, 0.8), B → Storage 1 / 2 (0.2), C → Storage 3 (0.2).

[0111] The A* algorithm finds the optimal path: Host → Storage 1 / 2: Host → A → B → Storage 1 / 2 (weight 0.3 + 0.3 + 0.2 = 0.8); Host → Storage 3: Host → A → C → Storage 3 (weight 0.3 + 0.8 + 0.2 = 1.3). Multi-path optimization (maximum flow, minimum cut): Ensure that the load on link 1 is ≤70% (currently 27.5%, with sufficient reserves), and finally determine these two paths as the optimal network connection scheme.

[0112] like Figure 2 As shown, this application provides a deep learning-based heterogeneous computing resource collaborative management system, including:

[0113] The computing power requirement range calculation module includes a task feature set generation unit and a computing power requirement range calculation unit. The task feature set generation unit receives the task description input by the user, extracts the core features affecting computing power requirements through semantic parsing, and forms a task feature set. The computing power requirement range calculation unit converts the task feature set into a task vector, inputs it into a pre-trained computing power requirement prediction model, and outputs the computing power requirement range of the task based on the features and computing power mapping rules learned from historical task data.

[0114] The host selection module includes a host computing power feature library construction unit, a compatible host list generation unit, and a host selection unit. The host computing power feature library construction unit extracts the core computing power parameters and status information of all candidate hosts to construct a host computing power feature library. The compatible host list generation unit, based on the host computing power feature library, performs core computing power hard matching, heterogeneous scenario adaptation, and secondary status filtering to form a compatible host list. The host selection unit converts the compatible host list into a selection interface, allowing the user to select a host and perform status update and binding operations.

[0115] The storage configuration generation module includes: a storage requirement feature acquisition unit, a host configuration calibration unit, a block scheme generation unit, and a storage configuration generation unit. Specifically, the storage requirement feature acquisition unit extracts core features affecting storage allocation from task vectors to construct a storage requirement feature set; the host configuration calibration unit calculates the initial storage capacity based on the storage requirement features and calibrates it in conjunction with the selected host configuration; the block scheme generation unit abstracts storage resources and task requirements into a graph structure, and uses a graph matching algorithm to achieve the optimal mapping between task storage requirements and storage resources, maximizing storage utilization and obtaining a block scheme; the storage configuration generation unit calculates redundancy capacity based on task reliability requirements and integrates the block schemes to form a storage configuration.

[0116] The optimal network connection scheme generation module includes: a network demand indicator acquisition unit, a filtering rule formulation unit, and an optimal network connection scheme generation unit. The network demand indicator acquisition unit extracts network resource demand features from task vectors, selected host configurations, and storage configurations to form network demand indicators. The filtering rule formulation unit constructs a network resource feature library based on the network demand indicators and formulates filtering rules that match the demands. The optimal network connection scheme generation unit matches suitable resources from the network resource feature library based on the filtering rules and determines the optimal network connection scheme through a path optimization algorithm.

[0117] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for collaborative management of heterogeneous computing resources based on deep learning, characterized in that, Includes the following steps: The system receives a task description input from the user, extracts the core features affecting computing power requirements through semantic parsing, and forms a task feature set. The task feature set is then transformed into a task vector, which is input into a pre-trained computing power requirement prediction model. The model learns the feature and computing power mapping rules based on historical task data and outputs the range of computing power requirements for the task. Extract the core computing power parameters and status information of all candidate hosts to build a host computing power feature library; based on the host computing power feature library, perform hard matching of core computing power, adaptation of heterogeneous scenarios and secondary filtering of status to form a list of adapted hosts; The list of compatible hosts is transformed into a selection interface, where users select a host and perform status update and binding operations. Extract core features affecting storage allocation from task vectors to construct a storage demand feature set; calculate the initial storage capacity based on the storage demand features and calibrate it in conjunction with the selected host configuration; abstract storage resources and task requirements into a graph structure, and achieve the optimal mapping between task storage requirements and storage resources through a graph matching algorithm to maximize storage utilization and obtain a block scheme; calculate the redundancy capacity according to task reliability requirements and integrate the block scheme to form a storage configuration. The network resource demand characteristics are extracted from the task vector, selected host configuration, and storage configuration to form network demand indicators; a network resource feature library is built based on the network demand indicators, and screening rules that match the demands are formulated; suitable resources are matched from the network resource feature library based on the screening rules, and the optimal network connection scheme is determined through path optimization algorithms.

2. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The task description received from the user is used to extract core features affecting computing power requirements through semantic parsing, forming a task feature set, including: The system receives task descriptions submitted by users in natural language, structured forms, or mixed formats and performs preprocessing. Based on a pre-trained natural language understanding model, it performs deep semantic analysis on the preprocessed text to identify explicit features affecting computing power requirements. This natural language understanding model uses a BERT fine-tuning model. Combining a domain knowledge graph and a historical task feature library, it mines implicit features that are not directly expressed but affect computing power requirements, thus completing the feature dimensions. The initially identified and completed features are standardized to eliminate expression differences and verify logical conflicts. The standardized explicit and implicit features are integrated to form a task feature set.

3. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process involves converting the task feature set into a task vector, inputting it into a pre-trained computing power demand prediction model, and the model, based on features and computing power mapping patterns learned from historical task data, outputting the computing power demand range for the task, including: The system classifies various types of features in the task feature set, transforming non-numerical features into computable numerical forms and integrating them into a unified-dimensional task vector. This task vector is then input into a pre-trained computing power demand prediction model. The model learns the mapping relationship between features and computing power through a multi-layer network structure. Branch networks process categorical features, numerical features, and textual features separately, while cross-attention layers capture the association weights of different feature branches. The model activates the mapping patterns learned from historical task data through hidden layers, performs similarity matching between the current task vector and historical feature vectors, and calls the computing power mapping parameters of the most similar task. The ReLU activation function and residual connections capture the non-linear relationship between features and computing power. The model's output layer generates the computing power demand range through a dual-branch structure.

4. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process involves hard matching of core computing power, adaptation to heterogeneous scenarios, and secondary filtering based on the host computing power feature library to form a list of adapted hosts, including: Determine core computing power metrics based on task type, ensuring that parameters in the host feature library match the required metrics. Iterate through the host computing power feature library, filtering hosts whose core computing power metrics fall entirely within the required range, and excluding hosts whose core computing power metrics exceed the range. When a task requires a specific hardware architecture, filter hosts in the host feature library whose hardware architecture tags match the task requirements. When a task requires distributed computing, filter hosts with distributed communication interfaces and whose resource pools can form continuous node clusters. Filter hosts with an idle status tag, excluding allocated, faulty, or under maintenance hosts, and check the temporary lock status of hosts, retaining only completely idle hosts. Based on the health score in the host feature library, set a threshold to exclude hosts with health scores below the threshold. When a task has resource pool geographical restrictions, filter hosts in the host feature library whose resource pool identifier matches the constraints; when there are no restrictions, prioritize retaining hosts within the same resource pool. The filtering results are integrated to form a list of compatible hosts.

5. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The calculation of the initial storage capacity based on storage demand characteristics, and the calibration in conjunction with the selected host configuration, includes: Quantitative parameters are extracted from the storage demand feature set, and the initial storage capacity is calculated by accumulating data throughout the entire life cycle. All data volume units are unified to GB and rounded up to the smallest allocation unit of the storage device. Determine the types, number, and maximum capacity of the storage interfaces supported by the host; extract the storage protocols supported by the host and exclude interfaces that are incompatible with the task data transmission protocol; obtain the theoretical maximum IOPS and bandwidth of the host storage interfaces as a performance matching threshold; when the initial capacity exceeds the maximum capacity supported by a single interface, split the capacity into multiple interfaces; integrate the calibration results and output the final initial storage capacity that conforms to the host configuration and meets the task requirements.

6. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process of abstracting storage resources and task requirements into a graph structure, and using a graph matching algorithm to achieve the optimal mapping between task storage requirements and storage resources, thereby maximizing storage utilization and obtaining a block partitioning scheme, includes: The storage resource pool and task storage requirements are modeled as directed weighted graphs, specifically: for the storage resource graph... ,node Based on independent storage units as basic nodes, including physical storage devices or logical storage pools, node attributes include capacity limit, current free capacity, read / write bandwidth, IOPS, storage type, and the resource pool to which it belongs. This represents the physical connection links between storage units, with edge weights equal to the data transmission latency between storage units; the weight is set to ∞ when there is no direct connection. (For the task storage requirement graph...) ,node Task storage requirements are broken down into sub-nodes based on the data lifecycle, including input data blocks. Intermediate result blocks and output data blocks Node attributes include required capacity, read / write frequency, data type, and reliability requirements. This represents the data dependency relationship between child nodes, and the edge weight is the amount of data transmitted between nodes. Based on the optimal I / O unit of the storage device, the free capacity of the nodes in the storage resource graph is divided into several standard blocks to ensure that the blocks match the hardware read and write units; according to the standard block size of the storage unit, the capacity of each node in the task storage requirement graph is divided into an integer number of standard blocks, while maintaining the dependency relationship between child nodes; granularity verification is performed to ensure that the total number of task blocks is not greater than the total number of free standard blocks of the storage resource, and that the size of a single task block is not greater than the maximum supported size of a single block of the storage unit. Task request nodes are prioritized by a weighted average of read / write frequency and data transfer volume, while storage resource nodes are prioritized by a weighted average of their idle capacity utilization potential and performance metrics. Starting with the highest priority task request node, storage resource nodes are traversed, and storage nodes with remaining capacity not less than the task block capacity, performance metrics not less than the task requirement threshold, and priority within the same resource pool are selected for mapping. For task nodes with dependencies, they are prioritized to be mapped to the same storage node. The storage utilization rate after the initial matching is calculated, and storage nodes with a storage utilization rate less than the set threshold are optimized through block migration.

7. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process of calculating redundant capacity based on task reliability requirements and integrating the block-based scheme to form a storage configuration includes: Reliability requirement features influencing redundancy strategies are extracted from the task vector, including data importance, tolerable loss rate, business continuity requirements, and sensitivity to historical failures. These reliability requirement features are categorized into high reliability, medium reliability, and low reliability, and mapped to redundancy strategies. The block partitioning scheme is matched with the redundancy strategy; for each block, redundancy capacity is calculated according to its reliability level, and the redundancy capacity of all blocks is summed to obtain the total task redundancy requirement. This total task redundancy requirement is allocated to storage resources, prioritizing storage units physically isolated from the storage nodes where the main blocks reside to carry redundant data. The block partitioning scheme and redundancy configuration are integrated to form a storage configuration.

8. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process of constructing a network resource feature library based on network demand indicators and formulating screening rules that match demand includes: Network demand metrics are standardized and integrated into a network resource feature library. Mapping rules are defined between network demand metrics and fields in the network resource feature library, including: peak bandwidth demand is mapped to bandwidth limit and available bandwidth, latency tolerance is mapped to latency, cross-resource pool demand is mapped to coverage resource pool, concurrent connection demand is mapped to maximum device connection count, and reliability demand is mapped to failure rate and redundancy capability. Based on the mapping rules, multi-dimensional filtering rules are formulated according to the rigidity of demand, and the rule priorities are clearly defined. The rule priorities, from high to low, include: hard constraint rules, soft constraint rules, and optimization constraint rules. Hard constraint rules include bandwidth constraints, protocol constraints, and topology constraints; soft constraint rules include latency constraints and load constraints; and optimization constraint rules include reliability optimization and energy consumption optimization.

9. The method for collaborative management of heterogeneous computing resources based on deep learning according to claim 1, characterized in that, The process of matching suitable resources from a network resource feature library based on filtering rules and determining the optimal network connection scheme through a path optimization algorithm includes: Traverse the network resource feature library, check whether each type of resource satisfies hard constraint rules, soft constraint rules and optimization constraint rules in turn, and form a candidate resource list sorted by priority; perform availability and correlation verification on the candidate resources, update the real-time status of the candidate resources, exclude resources whose status has changed to maintenance or overload, check the connectivity between candidate resources, and when the same resource is selected by multiple task candidate sets, allocate according to task priority, and low priority tasks are automatically switched to the second best candidate resources; Candidate resources are abstracted into a structured topology graph, and target weights are assigned according to task attributes. For each pair of host and storage block connection requirements, the A* algorithm is used to find the path with the minimum weight based on the edge weights of the topology graph. For scenarios where the same host needs to connect to multiple storage blocks, the maximum flow minimum cut algorithm is used to assign the paths of different hosts and storage blocks to different links to determine the optimal network connection scheme.

10. A deep learning-based heterogeneous computing resource collaborative management system, using the deep learning-based heterogeneous computing resource collaborative management method according to any one of claims 1-9, characterized in that, include: The computing power requirement range calculation module includes a task feature set generation unit and a computing power requirement range calculation unit. The task feature set generation unit receives the task description input by the user, extracts the core features affecting computing power requirements through semantic parsing, and forms a task feature set. The computing power requirement range calculation unit converts the task feature set into a task vector, inputs it into a pre-trained computing power requirement prediction model, and outputs the computing power requirement range of the task based on the features and computing power mapping rules learned from historical task data. The host selection module includes a host computing power feature library construction unit, a compatible host list generation unit, and a host selection unit. The host computing power feature library construction unit extracts the core computing power parameters and status information of all candidate hosts to construct a host computing power feature library. The compatible host list generation unit, based on the host computing power feature library, performs core computing power hard matching, heterogeneous scenario adaptation, and secondary status filtering to form a compatible host list. The host selection unit converts the compatible host list into a selection interface, allowing the user to select a host and perform status update and binding operations. The storage configuration generation module includes: a storage requirement feature acquisition unit, a host configuration calibration unit, a block scheme generation unit, and a storage configuration generation unit. Specifically, the storage requirement feature acquisition unit extracts core features affecting storage allocation from task vectors to construct a storage requirement feature set; the host configuration calibration unit calculates the initial storage capacity based on the storage requirement features and calibrates it in conjunction with the selected host configuration; the block scheme generation unit abstracts storage resources and task requirements into a graph structure, and uses a graph matching algorithm to achieve the optimal mapping between task storage requirements and storage resources, maximizing storage utilization and obtaining a block scheme; the storage configuration generation unit calculates redundancy capacity based on task reliability requirements and integrates the block schemes to form a storage configuration. The optimal network connection scheme generation module includes: a network demand indicator acquisition unit, a filtering rule formulation unit, and an optimal network connection scheme generation unit. The network demand indicator acquisition unit extracts network resource demand features from task vectors, selected host configurations, and storage configurations to form network demand indicators. The filtering rule formulation unit constructs a network resource feature library based on the network demand indicators and formulates filtering rules that match the demands. The optimal network connection scheme generation unit matches suitable resources from the network resource feature library based on the filtering rules and determines the optimal network connection scheme through a path optimization algorithm.

Images