Multi-source computing power data integration and intelligent scheduling system and method
Through the integration of multi-source computing power data and intelligent scheduling system, the unified resource abstraction layer and adaptive intelligent scheduling engine are used to solve the problem of insufficient intelligent computing power resource scheduling in the existing technology, and efficient and flexible resource utilization and data are achieved. Processing capability.
Patent Information
- Application Number
- CN202411200939.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-29
AI Technical Summary
The existing computing resource scheduling methods are not intelligent and flexible enough, resulting in low resource utilization, low execution efficiency, and difficult to adapt to network fluctuations and load changes.
Design a multi-source computing power data integration and intelligent scheduling system, including central control servers, cloud platforms and edge devices, virtualized management through a unified resource abstraction layer, build dynamic data integration modules and adaptive intelligent scheduling engines, and integrate machines Learn and reinforcement learning technology, support the edge-cloud-terminal collaborative computing framework, ensuring data privacy and security.
It significantly improves the utilization rate of multi-source heterogeneous computing resources, achieves accurate matching of task requirements and resources, reduces resource idleness and waste, enhances the system's ability to adapt to network fluctuations and load changes, and improves data processing capabilities and data quality control.
Smart Images

Figure CN118916147A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent control technology, and in particular to a multi-source computing power data integration and intelligent scheduling system and method. Background Art
[0002] In today's rapidly developing information age, computing resources have become a key driving force for social progress. With the rapid development of cutting-edge technologies such as cloud computing, big data, and artificial intelligence, the scheduling and management of computing resources are facing many new challenges.
[0003] How to achieve efficient configuration and utilization of computing resources has become a focus of the industry. Computing scheduling refers to the process of allocating tasks to appropriate computing resources for processing through optimization algorithms based on task characteristics and computing requirements after receiving computing task requests. Existing computing task scheduling methods are not smart or flexible enough, and their execution efficiency is relatively low. Summary of the invention
[0004] Based on the above problems, the present invention proposes a multi-source computing power data integration and intelligent scheduling system and method. The solution of the present invention can greatly improve the utilization rate of multi-source heterogeneous computing resources, and can more intelligently, flexibly and accurately match task requirements and available resources, reducing idle resources and waste; it can improve the system's adaptability to network fluctuations and load changes and enhance the system's data processing capabilities and data quality control.
[0005] In view of this, one aspect of the present invention proposes a multi-source computing power data integration and intelligent scheduling system, including: a central control server, a cloud platform, an edge device and a local terminal; The central control server is configured as follows: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
[0006] Another aspect of the present invention provides a multi-source computing power data integration and intelligent scheduling method, comprising: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
[0007] Optionally, the step of establishing a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management includes: Identify all computing resource types in the system; Classify computing resources and establish a resource type hierarchy; Design a general computing resource description model; Define key attributes for each computing resource type; Develop automated tools to collect detailed information about each specific computing resource; Mapping the collected information into a computing resource description model; Design virtualization management interfaces to abstract underlying hardware details; Implement resource pooling mechanism to transform physical resources into manageable virtual resource units; Design a unified API for resource allocation, monitoring, and recycling; Generate cross-platform resource management protocols; Deploy distributed monitoring agents to collect resource usage in real time; Design a central monitoring panel to display the overall resource status; Build dynamic resource discovery and registration mechanisms; Build a role-based access control system; Design encrypted communication channels to protect resource information transmission; Build a cache mechanism to speed up the query of frequently accessed resource information; Optimize storage and retrieval algorithms for computing resource descriptions; Design a modular architecture to support the access of new computing resource types; Build a distributed resource management framework to support large-scale clusters.
[0008] Optionally, the step of constructing a dynamic data integration module for implementing real-time collection, cleaning and standardization of multi-source heterogeneous data includes: Identify all data source types, classify data sources, and establish a data source catalog; Define common data structures and design metadata models to describe data attributes, sources, and timestamps; Develop dedicated data collectors for each data source type and build a data access layer that supports multiple protocols; Build a stream processing framework, and design and implement streaming data processing pipelines; Design data format conversion functions, develop data validation and error handling mechanisms, and design data cleaning algorithms; Perform data field mapping and conversion, develop unit conversion and standardization function libraries, and design data semantic conversion mechanisms; Build real-time data quality detection algorithms, design data quality scoring systems, and develop abnormal data processing mechanisms; Design data fusion algorithms to handle conflicts and complementarities of multi-source data, and develop data association analysis functions and data consistency check mechanisms; Build data caching mechanisms and design parallel processing architectures to optimize data storage and indexing strategies; Adopt a microservices architecture, identify dynamic configuration management, and design a distributed processing framework; Develop data flow monitoring dashboards to visualize data processing and design alarm and notification mechanisms.
[0009] Optionally, the step of designing an adaptive intelligent scheduling engine, integrating a machine learning algorithm, and performing task allocation and resource scheduling according to historical execution task data and real-time system status data includes: Design a data collection module based on the data collector, and use the data collection module to obtain historical task execution data and real-time system status data; Perform data cleaning and feature extraction processes on historical task execution data and real-time system status data, and build a data storage system to support fast query and analysis; Define the task feature set based on the features extracted from historical task execution data, develop a task classification algorithm, identify the task type, and build a task demand prediction model; Design resource status feature sets based on features extracted from real-time system status data, develop resource performance prediction models and resource availability assessment algorithms; Select machine learning algorithms, design model training processes and model evaluation / validation mechanisms; According to the task demand prediction model, resource performance prediction model and resource availability evaluation algorithm, develop a resource scheduling decision model based on machine learning model, determine the multi-objective optimization mechanism, and design a scheduling strategy scoring system; Build a real-time reasoning module to generate scheduling decisions; Determine decision execution and feedback mechanisms; Develop anomaly detection and handling processes.
[0010] Optionally, the step of constructing an edge-cloud-terminal collaborative computing framework that can support dynamic migration and load balancing of computing tasks includes: Develop lightweight edge devices for preliminary data processing and analysis at the source of data; Design communication protocols between edge devices to support distributed collaboration; Develop dynamic resource management and task scheduling algorithms to achieve load balancing among edge devices; Establish an elastically scalable cloud platform to handle large-scale data and complex computing tasks; Develop cloud platform resource management and task scheduling algorithms based on machine learning to achieve dynamic migration and load balancing across edge devices; Provide API interface to support two-way communication and collaboration between edge devices and cloud platforms; Develop a collaboration mechanism between edge devices and cloud platforms, including task allocation strategies, data transmission protocols, and security authentication mechanisms; Develop cross-level resource monitoring and scheduling algorithms to achieve dynamic load balancing between edge, cloud and terminal; Design fault-tolerant mechanisms to ensure the continued reliable operation of computing tasks.
[0011] Optionally, the step of establishing a global performance optimization model and continuously optimizing the overall system performance and resource utilization by using reinforcement learning technology includes: Identify key metrics that describe the overall performance of the system; Establish a global performance model covering edge devices, communication networks, local terminals, and cloud platforms, and describe the relationship and influencing factors between indicators at each level; Define the system state space; Design a reinforcement learning algorithm suitable for the problem; Determine the reward function based on the overall performance index of the system; Establish a real-time monitoring mechanism between edge and cloud to collect system operation status data; Design data preprocessing and feature extraction methods to provide input data for reinforcement learning; Use the pre-processed and feature-extracted running status data to train the reinforcement learning model and learn the optimal resource scheduling and task migration strategy; Deploy the trained model to the system and make performance optimization decisions in real time; Continuously monitor the system operation status and iteratively optimize the reinforcement learning model; Integrate the reinforcement learning optimization module into the edge-cloud-terminal collaborative framework to ensure the coordinated work of various components.
[0012] Optionally, the step of integrating secure multi-party computing and differential privacy technology to ensure privacy protection and security of data during integration and processing includes: Clarify the privacy needs and compliance requirements of all parties in the system; Design a comprehensive privacy protection framework based on secure multi-party computation and differential privacy, covering all aspects of data collection, transmission, storage and processing; Select the corresponding secure multi-party computing algorithm; Design secure communication and computation protocols between multiple parties to ensure that each party can only obtain the final result and cannot spy on the intermediate process; Implement a secure multi-party computing module and integrate it with other system components; Identify privacy-sensitive data in the system and determine the corresponding differential privacy protection goals; Design a differential privacy algorithm based on noise injection to minimize privacy leakage while ensuring data availability; Integrate the differential privacy module into the data processing process to ensure the privacy protection of data at all stages; Establish a privacy risk assessment mechanism to regularly evaluate the privacy protection effectiveness of the system; Design a real-time monitoring module to detect possible privacy breaches and trigger response measures; Continuously optimize privacy protection solutions based on evaluation results and monitoring feedback; Design targeted privacy protection test scenarios, covering different attack models and data processing scenarios; Verify the effectiveness and reliability of secure multi-party computing and differential privacy technologies through testing; Optimize and improve based on test results.
[0013] Optionally, the step of determining an elastic scaling mechanism to support dynamic adjustment of system scale and seamless access to new computing resources includes: Analyze the business requirements and service characteristics of the system, and determine the goals and key indicators of elastic scaling; Design elastic scaling architecture and mechanisms based on business requirements and technical characteristics; Establish a system resource monitoring module to collect CPU, memory, and network indicator data in real time; Use machine learning methods to predict future resource demand trends based on monitoring data and historical load patterns; Formulate dynamic expansion and contraction decision strategies based on the forecast results of resource demand change trends; Implement an automated scaling execution mechanism; Design access mechanisms for new computing resources to support seamless integration of heterogeneous hardware devices and cloud services; Implement resource discovery, scheduling, and orchestration functions to ensure that new resources can be effectively utilized by the system; Establish optimization algorithms based on resource utilization and service indicators to allocate and schedule computing resources; Adopt appropriate resource scheduling strategies based on resource characteristics and application requirements; Design redundant backup and self-healing mechanisms to ensure that the system can still run smoothly when resources fail or fluctuate; Design fault detection, isolation, and automatic recovery functions to improve system reliability.
[0014] Optionally, the step of using a machine learning method to predict future resource demand change trends based on monitoring data and historical load patterns includes: Collect monitoring data from cloud platforms, edge devices, and local terminals in real time; Performing operations of cleaning, filling missing values, and detecting outliers on the collected monitoring data to obtain first monitoring data; Extracting appropriate feature variables from the first monitoring data according to business characteristics and monitoring indicators; Normalize and standardize the feature variables to ensure the consistency of dimensions between features; Select a machine learning algorithm suitable for time series prediction as the model to be trained; Obtain historical monitoring data and historical load patterns of cloud platforms, edge devices, and terminals; Divide the historical monitoring data into a training set and a validation set, train and cross-validate the training model, and obtain the first model; Evaluate the performance index of the first model and iteratively optimize the first model; By using the trained first model, historical load patterns and characteristic variables are input to predict the resource demand change trend within a certain period of time in the future for different resource types.
[0015] Adopting the technical scheme of the present invention, the multi-source computing power data integration and intelligent scheduling method includes establishing a unified resource abstraction layer, standardizing the description of multi-source heterogeneous computing resources and virtualizing management; constructing a dynamic data integration module for realizing real-time collection, cleaning and standardized processing of multi-source heterogeneous data; designing an adaptive intelligent scheduling engine, integrating machine learning algorithms, and performing task allocation and resource scheduling according to historical execution task data and real-time system status data; constructing an edge-cloud-terminal collaborative computing framework that can support dynamic migration and load balancing of computing tasks; establishing a global performance optimization model, and continuously optimizing the overall system performance and resource utilization through reinforcement learning technology; integrating secure multi-party computing and differential privacy technology to ensure privacy protection and security of data during integration and processing; determining an elastic scaling mechanism to support dynamic adjustment of system scale and seamless access to new computing resources. Through this scheme, the utilization rate of multi-source heterogeneous computing resources can be greatly improved, and task requirements and available resources can be matched more intelligently, flexibly and accurately, reducing resource idleness and waste; the system's adaptability to network fluctuations and load changes can be improved, and the system's data processing capabilities and data quality control can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a schematic block diagram of a multi-source computing power data integration and intelligent scheduling system provided by an embodiment of the present invention; Figure 2 It is a flow chart of a multi-source computing power data integration and intelligent scheduling method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0017] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0019] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0020] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0021] Refer to the following Figure 1 to Figure 2 To describe a multi-source computing power data integration and intelligent scheduling system and method provided according to some embodiments of the present invention.
[0022] like Figure 1 As shown, an embodiment of the present invention provides a multi-source computing power data integration and intelligent scheduling system, including: a central control server, a cloud platform, an edge device and a local terminal; The central control server is configured as follows: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
[0023] It should be known that Figure 1 The block diagram of the multi-source computing power data integration and intelligent scheduling system shown is for illustration only, and the number of modules shown does not limit the protection scope of the present invention. The multi-source computing power data integration and intelligent scheduling system provided in this embodiment can be used to execute the corresponding multi-source computing power data integration and intelligent scheduling method embodiments. For the specific implementation process, please refer to the description of the following method embodiments, which will not be repeated here.
[0024] See also Figure 2 Another embodiment of the present invention provides a multi-source computing power data integration and intelligent scheduling method, including: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; In this step, the unified resource abstraction layer includes: a resource description model, which is used to standardize the description of different types of computing resources; a virtualization management interface, which is used to uniformly manage physical resources and virtual resources; and a resource monitoring module, which monitors the status and performance indicators of various types of resources in real time.
[0025] Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; In this step, the dynamic data integration module includes: a distributed data acquisition subsystem that supports parallel acquisition of multi-source data; a real-time data stream processing engine that performs instant data cleaning and conversion; and a data quality assessment and control mechanism that ensures the consistency and reliability of the integrated data.
[0026] Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; In this step, the adaptive intelligent scheduling engine includes: a multi-objective optimization algorithm that comprehensively considers factors such as task completion time, resource utilization, energy consumption, etc.; a prediction model that predicts task execution time and resource requirements based on historical data; and a dynamic scheduling strategy generator that dynamically adjusts the scheduling strategy according to the system status and prediction results.
[0027] Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; In this step, the edge-cloud-terminal collaborative computing framework includes: a task decomposition module that decomposes complex tasks into subtasks that can be executed at different levels; a dynamic task migration mechanism that dynamically adjusts the task execution location according to network conditions and computing load; and a distributed state synchronization mechanism that ensures data consistency between levels.
[0028] Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; In this step, the global performance optimization model includes: a performance indicator collector, which collects performance data of various aspects of the system in real time; a reinforcement learning model, which optimizes resource allocation and scheduling decisions through continuous learning; and a simulation environment, which is used to simulate and evaluate the effects of different scheduling strategies.
[0029] Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; In this step, the integration of secure multi-party computing and differential privacy technology includes: a data encryption module to ensure the security of data during transmission and storage; a secure multi-party computing protocol to support cross-domain computing while protecting privacy; and a differential privacy processor to add noise to data analysis and output results to protect individual privacy.
[0030] Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
[0031] In this step, the elastic scaling mechanism includes: an automatic resource discovery module that can automatically identify and access newly added computing resources; a dynamic load balancer that dynamically adjusts resource allocation according to system load; and horizontal and vertical expansion support that allows the system to be scaled without interrupting service.
[0032] In the embodiments of the present invention, "system" refers to a computing system or infrastructure environment that requires resource scheduling and management, which can be the following: cloud computing environment (such as public cloud, private cloud or hybrid cloud platform, including various virtual computing resources), data center (including physical servers, storage devices, network devices and other hardware resources), edge computing environment (devices deployed near the data source and computing resources at the edge of the network), high-performance computing cluster (parallel computing system composed of multiple computing nodes, including heterogeneous resources such as CPU, GPU), embedded system (such as industrial control equipment, Internet of Things equipment, etc., which may include special hardware such as FPGA and dedicated accelerators), etc. Whether it is cloud, data center, edge or embedded system, it is necessary to identify and manage various heterogeneous computing resources in the system to provide efficient computing services for upper-level applications; this is also the basis for realizing intelligent scheduling strategies, and it is necessary to accurately grasp the types of computing resources and performance indicators available in the system. In short, "system" here refers to a computing environment that requires resource management and scheduling, which can be different forms of computing infrastructure such as cloud platform, data center, edge device or a whole composed of these facilities. Identifying various heterogeneous computing resources in the system is a prerequisite for realizing intelligent scheduling.
[0033] The technical solution of this embodiment can greatly improve the utilization rate of multi-source heterogeneous computing resources through a unified resource abstraction layer and an adaptive intelligent scheduling engine; it can more accurately match task requirements and available resources, reducing resource idleness and waste; the edge-cloud-terminal collaborative computing framework enables the system to flexibly adjust the execution location of computing tasks according to real-time conditions, which can not only optimize performance, but also improve the system's adaptability to network fluctuations and load changes; the dynamic data integration module supports large-scale, real-time heterogeneous data processing, significantly enhancing the system's data processing capabilities and data quality control; the global performance optimization model can continuously improve the overall performance of the system through continuous learning and optimization, including key indicators such as task completion time and throughput; integrated secure multi-party computing and differential computing Privacy technology ensures that while improving computing efficiency, sensitive data and user privacy can also be protected, making the system suitable for a wider range of application scenarios; the elastic scaling mechanism enables the system to easily cope with scale changes and seamlessly integrate new computing resources, improving the scalability and adaptability of the system; the unified management interface and automated scheduling mechanism can greatly reduce the management complexity of large-scale heterogeneous systems and reduce the need for human intervention; through machine learning algorithms, the system can more accurately predict task execution time and resource requirements, thereby making better scheduling decisions; intelligent scheduling and global optimization can improve energy efficiency, reduce unnecessary energy consumption, and help achieve green computing; by optimizing resource allocation and task scheduling, the system can provide end users with more stable and responsive services. In general, the solution of this embodiment has achieved significant technical effects in terms of resource utilization efficiency, system performance, data processing capability, security, scalability, and management convenience.
[0034] In some possible implementations of the present invention, the steps of establishing a unified resource abstraction layer, describing multi-source heterogeneous computing resources in a standardized manner, and performing virtualization management include: Identify all types of computing resources in the system (such as CPU, GPU, FPGA, dedicated accelerators, etc.); In this step, the hardware topology of the system can be automatically scanned through the system management tool or API to obtain information about various computing resources, including CPU model, GPU model, FPGA model, dedicated accelerator model, etc.; collect performance monitoring indicators of various computing resources, such as CPU utilization, GPU load, FPGA resource occupancy, etc.; use the monitoring tool provided by the operating system or a third-party monitoring platform to extract characteristic indicators of various computing resources based on the collected monitoring data, such as CPU instruction set support, GPU memory bandwidth, FPGA logic unit utilization, etc.; combine resource characteristics to establish an identification model for computing resource types; use machine learning algorithms to identify and classify unknown resources. In this way, it is possible to comprehensively and automatically identify various computing resources involved in the system and obtain their detailed performance characteristics. Through the solution of this step, accurate identification of various computing resources helps to optimize resource allocation and scheduling and improve utilization; for heterogeneous computing resources, dynamic scheduling can be performed according to real-time load to improve the scalability of the system; complete resource topology and performance data help to build a comprehensive resource visualization and monitoring system; rich resource feature data can be used as input to the machine learning model to achieve more accurate resource demand prediction. In short, this computing resource identification technology can provide key support for the dynamic resource management of heterogeneous systems and improve the overall utilization efficiency of computing resources.
[0035] Classify computing resources and establish a resource type hierarchy; In this step, through system scanning and monitoring, basic information of various computing resources is collected, including model, manufacturer, architecture, etc. These metadata constitute the basis for resource classification; based on the collected resource metadata, the functional characteristics, performance indicators and other attributes of various resources are analyzed, such as the instruction set of CPU, the graphics processing capability of GPU, the parallel computing characteristics of FPGA, etc.; combined with the computing resource types identified in the previous step, different types of computing resources are hierarchically classified according to similar attribute characteristics, and a tree structure or a mesh structure can be used to organize the relationship between different types of resources; standardized labels are defined for each type of resource to describe the type, performance, use, etc. of the resource. The label system is conducive to subsequent resource retrieval and scheduling; the classified resources are organized according to the system topology to form a logical topology diagram of the resources; the topology model reflects the hierarchical relationship and interconnection relationship between resources. Through the above steps, a complete computing resource type hierarchy can be established, a clear resource classification system and labels can be implemented, and resource visualization and manageability can be enhanced; based on the resource topology model, more intelligent resource discovery, selection and scheduling can be achieved; hierarchical resource classification is applicable to various heterogeneous computing environments and enhances versatility; standardized resource types and labels are conducive to cross-system resource sharing and collaboration; the resource classification system provides important input features for machine learning models and supports automated resource management. In short, establishing a computing resource type hierarchy is the key foundation for realizing dynamic and intelligent resource management, which can bring significant technical effects.
[0036] Design a general computing resource description model (e.g. based on XML or JSON); In this step, according to the aforementioned resource classification system / resource type hierarchy, determine the basic elements and attributes required to describe computing resources, such as resource types, performance indicators, functional characteristics, deployment environment, etc.; organize resource elements and attributes into a data model in XML or JSON format, and the model should have a good hierarchy and extensibility; formulate grammatical rules for resource description language, such as element name, attribute definition, value type, etc.; clarify the semantic definition of various resource attributes to ensure cross-system interoperability; write software tools for parsing, querying, and generating resource description documents, and support the verification, conversion, and management of resource description documents. This universal resource description language can achieve: standardized description methods, which help resources migrate between different systems; unified description language, which facilitates resource discovery, orchestration, and scheduling; cross-system resource description, which provides a basis for resource sharing and collaboration; structured resource information, which provides rich input for machine learning models; and standard description language, which enhances information interaction between different systems.
[0037] Define the key attributes of each computing resource type (e.g., processing power, memory size, energy consumption characteristics, etc.); In this step, various major computing resource types, such as CPU, GPU, FPGA, storage device, etc., can be identified based on the aforementioned resource classification system / resource type hierarchy; for each resource type, its key functional characteristics and performance indicators are systematically analyzed; for example, the CPU's processing power, memory size, instruction set support, etc., the GPU's floating-point computing performance, video memory capacity, memory bandwidth, etc., the storage device's capacity, IOPS, latency, etc.; for each resource type, a set of parameter indicators that can comprehensively describe its characteristics is determined; the parameters should cover various aspects of the resource's processing power, storage capacity, energy consumption characteristics, deployment environment, etc. Taking CPU resources as an example, the following key attribute parameters can be defined: processor architecture (x86, ARM, RISC-V, etc.), number of cores (number of physical cores of the CPU), number of threads (number of logical threads of the CPU), instruction set (SSE, AVX, NEON and other instruction set support), clock frequency (CPU main frequency), cache (L1 / L2 / L3 cache size), memory support (memory type, capacity, bandwidth), power consumption (rated power consumption or thermal design power consumption), performance (single-core performance indicators, multi-core performance indicators), deployment environment (operating system, virtualization support); organize the key attributes of various resources into a standardized data model to facilitate subsequent resource description and management. XML, JSON or other structured formats can be used to define the attribute model; collect and maintain the attribute data of various computing resources to build a comprehensive resource attribute database; the database should support attribute query, comparison and analysis functions. Defining the key attributes of each computing resource in this way can achieve a comprehensive description of resource attributes and enhance resource visualization and manageability; intelligent matching based on attribute models can achieve more accurate resource selection and scheduling; standardized attribute definitions are suitable for computing resources of different types and manufacturers; sharing of attribute data is conducive to cross-system resource collaboration and business collaboration; structured attribute information provides important input features for machine learning models.
[0038] Develop automated tools to collect detailed information about each specific computing resource; Mapping the collected information into a computing resource description model; Design virtualization management interfaces to abstract underlying hardware details; Implement resource pooling mechanism to transform physical resources into manageable virtual resource units; Design a unified API for resource allocation, monitoring, and recycling; Generate cross-platform resource management protocols; Deploy distributed monitoring agents to collect resource usage in real time; Design a central monitoring panel to display the overall resource status; Build dynamic resource discovery and registration mechanisms; Build a role-based access control (RBAC) system; Design encrypted communication channels to protect resource information transmission; Build a cache mechanism to speed up the query of frequently accessed resource information; Optimize storage and retrieval algorithms for computing resource descriptions; Design a modular architecture to support the access of new computing resource types; In this step, a set of standardized resource interfaces can be designed to describe and manage different types of computing resources. The interfaces should include key functions such as resource discovery, resource description, and resource allocation. The interface definition should follow the principles of openness and scalability to facilitate the access of new resource types in the future. Based on the standard interface, a unified resource management layer is built to coordinate and schedule various types of computing resources. The resource management layer should have core functions such as resource registration, resource monitoring, and resource allocation. The management layer can adopt microservice or component-based design to improve flexibility and scalability. For each type of computing resource, a corresponding resource adapter component is developed. The adapter is responsible for converting the properties and functions of a specific resource into the format defined by the standard interface. The adapter can be an independent module or integrated into the resource management layer. The resource management layer should provide a convenient extension mechanism to support dynamic access to new resource types. New resources can be seamlessly integrated into the resource management system by simply implementing the adapter of the standard interface. Above the resource management layer, a resource orchestration layer is built to be responsible for the coordination and scheduling of cross-heterogeneous resources. The orchestration layer can flexibly combine different types of computing resources according to business needs. The orchestration layer should provide a declarative resource orchestration language to facilitate users to define resource topology and constraints. Through the above design, a modular computing resource management architecture can be realized. Through standardized interface design, the access of new types of resources becomes simple and efficient; the modular architecture supports dynamic expansion and can support the development of new technologies without comprehensive reconstruction; the unified resource management layer can coordinate and schedule different types of computing resources; the resource adapter mechanism shields the underlying resource details and provides a unified abstract view; the resource orchestration layer supports declarative resource topology definition to achieve coordinated scheduling across heterogeneous resources; the orchestration layer can allocate and optimize resources according to business needs to improve resource utilization efficiency; the unified resource management interface is conducive to monitoring and visual presentation of resource status; the standardized resource attribute definition provides a basis for realizing automated resource management. In short, through modular architecture design, it is easy to support the access of new types of computing resources, greatly enhancing the openness, flexibility and automation level of the system.
[0039] Build a distributed resource management framework to support large-scale clusters.
[0040] Through the scheme of this embodiment, unified management of heterogeneous resources is achieved, which simplifies the complexity of the system; improves resource management efficiency and reduces management overhead; better allocates resources and load balances through a global view; reduces resource fragmentation and improves overall utilization efficiency; supports dynamic resource scheduling and can quickly respond to changes in computing needs; facilitates the introduction of new computing resources and improves the scalability of the system; provides detailed and standardized resource information for intelligent scheduling algorithms; achieves more accurate task-resource matching and improves computing efficiency; provides a unified resource access interface to reduce the complexity of application development; supports cross-platform deployment and migration of applications; quickly discovers and handles resource failures through real-time monitoring; supports dynamic replacement and failover of resources; optimizes energy use through better resource allocation strategies; supports intelligent scheduling based on energy consumption characteristics; a unified access control mechanism enhances system security, and fine-grained resource management supports more accurate security strategies; reduces hardware investment and operating costs through better resource planning and utilization; supports cost-based intelligent resource allocation; standardized resource descriptions help to more accurately estimate task performance and support performance modeling and optimization based on historical data. Through this unified resource abstraction layer, the system can more effectively manage and utilize multi-source heterogeneous computing resources, thereby significantly improving overall computing efficiency and system performance.
[0041] In some possible implementations of the present invention, the step of constructing a dynamic data integration module for implementing real-time collection, cleaning and standardization of multi-source heterogeneous data includes: Identify all data source types (such as databases, file systems, APIs, sensors, etc.), classify data sources, and establish a data source catalog; In this step, all data source types used inside and outside the organization are comprehensively sorted out, including databases, file systems, API interfaces, sensors, etc.; basic information of each data source, such as name, type, location, connection method, etc., is collected; data sources are classified according to their characteristics, such as storage type, access method, data format, etc. A hierarchical classification system can be used, such as structured, semi-structured, and unstructured by storage type; batch, real-time, etc. by access method; the data source information collected by the survey is organized into a comprehensive data source directory; the directory should contain detailed metadata for each data source, such as name, type, location, connection method, data format, owner, etc.; a unified data model and metadata standard is used to ensure the consistency and readability of the directory information; a management and maintenance mechanism for the data source directory is established to ensure the timely updating and accuracy of the directory information; a standardized process for adding, changing, and withdrawing data sources is formulated, and automated management is implemented; metadata management tools or data directory services can be considered to improve management efficiency; and a data source directory is established based on the directory. Registration and discovery mechanism; allows data users to find and access the required data sources through the directory; can provide search, browse, subscription and other functions, so that users can quickly locate the required data; through this step, a comprehensive and structured data source directory can be established, which centrally displays the detailed information of all data sources in the organization and provides a comprehensive perspective for data management; helps to sort out data assets, discover data islands, and improve data sharing and reuse; standardized data source registration and discovery mechanism enables data users to easily find the required data source; improves the accessibility and availability of data assets and reduces the cost of data use; unified metadata standards and management mechanisms ensure the consistency and credibility of data source directory information; provides basic support for data governance, data quality control, etc.; based on the data source directory, orchestration and integration across heterogeneous data sources can be achieved; provides standardized data access capabilities for data analysis, business applications, etc.; standardized processes and automated tools for directory management improve the efficiency and accuracy of data source information maintenance; enhance the agility and sustainability of data asset management. In short, establishing a comprehensive data source directory can bring a panoramic view of data assets to the organization, improve data discoverability, support data governance, and lay the foundation for data integration and automated management.
[0042] Define common data structures (such as based on JSON or Avro) and design metadata models to describe data attributes, sources, and timestamps; Develop dedicated data collectors for each data source type and build a data access layer that supports multiple protocols (such as MQTT, HTTP, WebSocket, etc.); Build stream processing frameworks (such as Apache Flink, Kafka Streams), and design and implement streaming data processing pipelines; Design data format conversion functions, develop data validation and error handling mechanisms, and design data cleaning algorithms (such as deduplication, completion, etc.); Perform data field mapping and conversion, develop unit conversion and standardization function libraries, and design data semantic conversion mechanisms; Build real-time data quality detection algorithms, design data quality scoring systems, and develop abnormal data processing mechanisms; Design data fusion algorithms to handle conflicts and complementarities of multi-source data, and develop data association analysis functions and data consistency check mechanisms; Build data caching mechanisms (to improve the processing speed of frequently accessed data) and design parallel processing architectures (to improve data processing throughput), and optimize data storage and indexing strategies; Adopt a microservices architecture (to support independent scaling of modules), determine dynamic configuration management (to support instant access to new data sources), and design a distributed processing framework (to support large-scale data processing); Develop data flow monitoring dashboards to visualize data processing and design alarm and notification mechanisms.
[0043] The solution of this embodiment can realize real-time data processing, reduce data delay, and improve data throughput through parallel processing; improve data consistency and reliability through automated cleaning and standardization, and reduce the impact of erroneous data through real-time data quality control; can process heterogeneous data of different formats and structures, and realize correlation analysis and integration of multi-source data; support rapid access and configuration of new data sources, and adapt to different data processing needs and scenarios; standardized data is easier to analyze and use, and provides a unified data access interface to simplify application development; support real-time data analysis and decision support, and can quickly respond to data changes and anomalies; reduce storage requirements through data cleaning and compression, and intelligently store data in a hierarchical manner to optimize cost and performance; support large-scale data processing to adapt to data volume growth; modular design facilitates functional expansion and upgrading; real-time monitoring and alarm mechanisms improve system stability, and data backup and recovery mechanisms enhance data security; provide high-quality, standardized data for machine learning and AI analysis, and support complex data mining and predictive analysis; support data traceability, facilitate auditing and compliance management, and realize data privacy protection and sensitive information processing; automated data processing reduces manual intervention; improves data utilization efficiency and reduces data management costs. Through this dynamic data integration module, the system can efficiently process multi-source heterogeneous data, provide high-quality, real-time data support, and lay a solid data foundation for subsequent intelligent scheduling and analytical decision-making.
[0044] In some possible implementations of the present invention, the steps of designing an adaptive intelligent scheduling engine, integrating a machine learning algorithm, and performing task allocation and resource scheduling according to historical execution task data and real-time system status data include: Design a data collection module based on the data collector, and use the data collection module to obtain historical task execution data and real-time system status data; Perform data cleaning and feature extraction processes on historical task execution data and real-time system status data, and build a data storage system to support fast query and analysis; Define task feature sets (such as computational complexity, memory requirements, data dependencies, etc.) based on features extracted from historical task execution data, develop task classification algorithms, identify task types, and build task demand prediction models; In this step, the historical task execution data is analyzed to identify the key features that affect task execution, such as computational complexity, memory requirements, data dependency, etc.; based on these features, a task feature set is established to provide a basis for subsequent task classification and prediction. The feature set should contain quantitative indicators and qualitative descriptions to fully reflect the characteristics of the task; based on the defined task feature set, a suitable task classification algorithm is developed, such as decision tree, K-Means clustering, etc.; the algorithm is trained and verified using historical task data to ensure classification accuracy; the classification algorithm should be able to automatically identify and classify the types of newly submitted tasks; the execution data of historical tasks is collected, including resource consumption, execution time, quality indicators, etc.; combined with the task feature set, a task demand prediction model is established, such as a regression model, a neural network, etc.; the prediction model can estimate the required resources, execution time, quality indicators, etc. based on the task characteristics; the trained task classification algorithm and demand prediction model are deployed to the actual task management system; when a new task is submitted, type identification and demand prediction are automatically performed to provide a decision basis for task scheduling, resource allocation, etc.; as new task data continues to accumulate, the model parameters are regularly updated to improve the prediction accuracy. Through this step, the key features that affect task execution can be fully extracted, laying the foundation for subsequent classification and prediction; the definition of feature sets helps to deeply understand the intrinsic characteristics of tasks; the task classification algorithm based on machine learning can accurately identify the type of new tasks; automated task classification improves the efficiency and accuracy of task management; the prediction model established using historical data can estimate the required resources, time, etc. for new tasks; accurate demand prediction helps to reasonably schedule tasks and allocate resources, improving overall operating efficiency; deployed in the actual task management system, it can continuously collect new data and update model parameters; it has the ability of self-learning and optimization, and continuously improves prediction accuracy as business develops; it provides quantitative basis and intelligent suggestions for key decisions such as task scheduling and resource allocation; it helps to improve the scientificity and rationality of task management and reduce operating costs. In short, through the definition of task features, the development of classification algorithms and the construction of demand prediction models, automated task type identification and demand prediction can be achieved, providing strong support for enterprise task management and resource optimization.
[0045] Design resource status feature sets (such as CPU usage, memory usage, network bandwidth, etc.) based on features extracted from real-time system status data, and develop resource performance prediction models and resource availability assessment algorithms; In this step, the real-time system monitoring data is analyzed to identify key features that reflect the resource status, such as CPU usage, memory usage, network bandwidth, etc.; based on these features, a comprehensive resource status feature set is established to provide a basis for subsequent performance prediction and availability evaluation; the feature set should contain quantitative indicators and dynamic change trends to fully reflect the real-time status of resources; based on the defined resource status feature set, a suitable performance prediction model is developed, such as time series analysis, neural network, etc.; historical monitoring data is used to train and verify the model to ensure prediction accuracy; the prediction model can predict the performance change trend in the future period based on the current status of the resource; a resource availability evaluation algorithm is developed by combining the resource status feature set and the performance prediction model; the algorithm can comprehensively consider the current status and future prediction of the resource to evaluate whether it can meet business needs; availability evaluation indicators can include resource usage thresholds, performance indicator warnings, etc.; the trained performance prediction model and availability evaluation algorithm are deployed to the actual resource management system; when new resource status data is collected, performance prediction and availability evaluation are automatically performed to provide decision-making basis for resource scheduling and capacity expansion, etc.; as new monitoring data continues to accumulate, model parameters are regularly updated to improve prediction accuracy and evaluation results. Through this step, the key features that reflect the real-time status of resources can be fully extracted, laying the foundation for subsequent prediction and evaluation; the definition of feature sets helps to deeply understand the operating status of resources; the prediction model based on machine learning can accurately predict the performance change trend of resources in the future; accurate performance prediction provides a basis for resource scheduling and expansion, and improves resource utilization efficiency; using performance prediction results, develop algorithms for comprehensive evaluation of resource availability; availability evaluation results help to timely discover resource bottlenecks and take corresponding countermeasures; deployed in the actual resource management system, it can continuously collect new monitoring data and update model parameters; it has the ability of self-learning and optimization, and continuously improves prediction accuracy and evaluation effects as business develops; it provides quantitative basis and intelligent suggestions for key decisions such as resource scheduling and expansion; it helps to improve the scientificity and rationality of resource management and reduce operating costs. In short, through the definition of resource status features, the development of performance prediction models and the construction of availability evaluation algorithms, it is possible to perceive and predict the real-time status of resources, providing strong support for the resource management and business assurance of enterprises.
[0046] Select machine learning algorithms (such as random forests, deep neural networks, reinforcement learning, etc.), design model training processes (including feature selection, model parameter tuning, etc.) and model evaluation / validation mechanisms; According to the task demand prediction model, resource performance prediction model and resource availability evaluation algorithm, develop a resource scheduling decision model based on machine learning model, determine the multi-objective optimization mechanism (to balance performance, energy consumption, cost and other factors), and design a scheduling strategy scoring system; In this step, historical monitoring data can be used to predict the number and type of tasks and their resource requirements in the future by using methods such as time series analysis and deep learning. The prediction results provide a basis for subsequent resource scheduling. According to business goals, multiple indicators that need to be optimized are determined, such as performance, energy consumption, cost, etc. A multi-objective optimization model is established to weigh the conflicts and trade-offs between different goals and seek the optimal balance point. The optimization goal can be to maximize system throughput, minimize energy consumption cost, etc. A resource scheduling decision model is constructed using the results of the task demand prediction model, resource performance prediction model, and resource availability evaluation algorithm. Reinforcement learning, genetic algorithms, etc. are used. Machine learning methods are used to automatically generate scheduling strategies that meet multi-objective optimization. The generation process of scheduling strategies can take into account factors such as task priority, resource utilization, and energy consumption cost. A scoring mechanism for scheduling strategies is established, which comprehensively considers multiple dimensions such as performance, energy consumption, and cost. The execution effects of various scheduling schemes are simulated based on predicted task requirements and resource availability. The scheduling strategy with the highest score is selected as the final execution plan. The generated scheduling strategy is deployed to the actual resource management system. New task requirements and resource status data are continuously collected, and the prediction model and scheduling strategy are regularly updated. According to the actual execution effect, the multi-objective optimization weights are adjusted to continuously optimize the scheduling strategy. Through this step, the task demand prediction model based on machine learning can more accurately predict future workloads; provide a reliable basis for resource scheduling and improve the pertinence of scheduling solutions; comprehensively consider multiple business goals such as performance, energy consumption, and cost to seek the optimal balance; meet specific needs in different scenarios and improve the flexibility of resource management; use machine learning models to automatically generate scheduling strategies that meet multi-objective optimization; the generation process of scheduling strategies is more intelligent and automated, reducing the cost of manual intervention; establish a comprehensive scheduling strategy scoring mechanism, comprehensively consider multiple key indicators; select the optimal scheduling strategy by simulating execution effects, and improve the accuracy of decision-making; deploy the scheduling system to the actual environment, continuously collect new monitoring data and update the model; have the ability of self-learning and optimization, and continuously improve the scheduling effect as the business develops. In short, the resource scheduling decision model based on the machine learning model can achieve more intelligent and automated resource scheduling, provide enterprises with excellent performance, low energy consumption and reasonable cost resource management solutions, and significantly improve the operating efficiency of the entire system.
[0047] Build a real-time reasoning module to generate scheduling decisions; Determine decision execution and feedback mechanisms; Develop anomaly detection and handling processes.
[0048] In this embodiment, it can also include: designing online learning algorithms to continuously optimize models; implementing model updates and version control mechanisms; developing A / B testing frameworks to evaluate the performance of new and old models; developing interfaces with underlying resource management systems to implement the execution logic of task allocation and resource allocation; designing rollback and fault recovery mechanisms; building a comprehensive performance indicator monitoring system, developing scheduling effect analysis tools, implementing visual dashboards, and displaying key performance indicators; designing modular architectures to support loosely coupled integration of components; implementing distributed computing frameworks to improve large-scale scheduling capabilities; optimizing system response time and throughput; implementing data encryption and access control mechanisms; designing privacy protection algorithms to protect sensitive information; and developing audit log systems to track scheduling decision-making processes.
[0049] The solution of this embodiment uses machine learning algorithms to achieve more accurate task-resource matching, reduce human intervention, and improve scheduling speed and accuracy; based on historical data and real-time status, it realizes dynamic allocation of resources, reduces resource waste, and improves overall utilization; the adaptive learning mechanism enables the system to cope with changing workloads and environments, and supports the rapid integration of new tasks and resources; through machine learning models, it improves the prediction accuracy of task execution time and resource requirements; supports better load balancing and capacity planning; balances multiple goals such as performance, energy consumption, and cost to achieve global optimization; supports dynamic adjustment of optimization goals according to different scenarios; improves system stability through anomaly detection and processing mechanisms; real-time monitoring and Automatic adjustment reduces system failures; intelligent scheduling takes energy consumption factors into account and optimizes overall energy use; supports green computing and sustainable development goals; provides interpretability of scheduling decisions and increases system credibility; supports auditing and backtracking of decision-making processes; distributed architecture supports efficient scheduling of large-scale clusters and improves system scalability and throughput; reduces task waiting time through better resource allocation; improves system response speed and service quality; reduces hardware and electricity costs by optimizing resource utilization and energy efficiency; reduces manual intervention and reduces operation and maintenance costs; provides rich data and insights for system performance analysis and optimization, supports predictive maintenance and proactive optimization; ensures the security of sensitive data through privacy protection mechanisms. Through this adaptive intelligent scheduling engine, the system can achieve more efficient, flexible, and intelligent task allocation and resource scheduling, significantly improve overall system performance and resource utilization efficiency, and provide users with better service quality.
[0050] In some possible implementations of the present invention, the step of constructing an edge-cloud-terminal collaborative computing framework that can support dynamic migration and load balancing of computing tasks includes: Develop lightweight edge devices (such as embedded devices, IoT devices, etc.) for preliminary data processing and analysis at the source of data; Design communication protocols between edge devices to support distributed collaboration; Develop dynamic resource management and task scheduling algorithms to achieve load balancing among edge devices; Establish an elastically scalable cloud platform to handle large-scale data and complex computing tasks; Develop cloud platform resource management and task scheduling algorithms based on machine learning to achieve dynamic migration and load balancing across edge devices; Provide API interface to support two-way communication and collaboration between edge devices and cloud platforms; Develop a collaboration mechanism between edge devices and cloud platforms, including task allocation strategies, data transmission protocols, and security authentication mechanisms; Develop cross-level resource monitoring and scheduling algorithms to achieve dynamic load balancing between edge, cloud and terminal; Design fault-tolerant mechanisms to ensure the continued reliable operation of computing tasks.
[0051] Through the scheme of this embodiment, a collaborative computing framework between edge devices, cloud platforms, and local terminals that supports dynamic migration and load balancing can be constructed. The edge devices can process data quickly and reduce latency; the cloud can handle complex computing tasks and improve overall computing capabilities; dynamic resource management and task scheduling can be used to cope with load fluctuations; fault-tolerant mechanisms ensure the continuity of critical tasks; cross-level load balancing can be used to improve resource utilization; edge devices and cloud resources can be flexibly expanded according to demand, and support a variety of application scenarios. In general, this edge-cloud-terminal collaborative computing framework can make full use of edge devices and cloud resources to achieve dynamic migration and load balancing of computing tasks, thereby improving the performance, reliability, and flexibility of the overall system.
[0052] In this embodiment, it also includes: implementing an end-to-end encryption mechanism to protect data transmission and storage security; developing a distributed identity authentication system to ensure cross-layer access control; designing a privacy-preserving computing framework to support the secure processing of sensitive data; implementing a task checkpoint mechanism to support rapid recovery of tasks; developing a fault detection and isolation system to improve system reliability; designing a task rescheduling strategy to deal with node failures; implementing a preloading and predictive execution mechanism to reduce task startup time; developing a resource reservation strategy to optimize the execution efficiency of key tasks; designing an adaptive computing offloading strategy to balance local execution and remote execution; designing a unified task submission and management interface, and developing visual monitoring and debugging tools; providing SDK and documentation to facilitate third-party application integration; implementing integration and end-to-end testing of each module; developing a performance benchmark suite to evaluate the overall system performance; conducting large-scale simulation tests to verify the performance of the system in complex scenarios.
[0053] The solution of this embodiment fully utilizes the computing resources of the edge, cloud, and end layers through dynamic task allocation; reduces idle resources and improves overall computing efficiency; selects the optimal execution location according to task characteristics and resource conditions; reduces task waiting time and improves response speed through load balancing; supports dynamic resource expansion and contraction to adapt to load changes; realizes self-regulation and optimization of the system through task migration; reduces the computing burden of end devices and extends battery life; reduces network latency and improves interactive performance through local computing; reduces unnecessary data transmission through intelligent data transmission strategies; uses edge nodes for data preprocessing to reduce cloud network pressure; sensitive data can be processed locally or at edge nodes to reduce data exposure risks; distributed security mechanisms improve the overall security of the system; and edge nodes can be in the network It provides basic services during off-time, realizes local caching and asynchronous synchronization of data; improves the robustness of the system through multi-level fault tolerance mechanism; enhances the availability of the system through fault isolation and rapid recovery capabilities; supports the collaborative work of different hardware platforms through unified task description and scheduling mechanism; makes full use of the computing power of dedicated hardware (such as GPU, NPU); reduces unnecessary data center computing and energy consumption through intelligent task allocation; supports green computing and optimizes overall energy utilization; a unified development framework simplifies the application development and deployment process, supports rapid prototype verification and iterative optimization; supports distributed data processing and analysis, and improves big data processing capabilities; edge intelligence provides support for real-time decision-making; reduces hardware investment and operating costs through optimized resource allocation; and supports on-demand payment and optimizes cost structure through flexible computing models. Through this edge-cloud-terminal collaborative computing framework, the system can achieve efficient utilization of computing resources, intelligent allocation and dynamic migration of tasks, and effective balancing of global loads; this not only improves the overall system performance and resource utilization, but also provides users with better service quality and experience, and also provides strong support for the development of new applications and services.
[0054] In some possible implementations of the present invention, the step of establishing a global performance optimization model and continuously optimizing the overall system performance and resource utilization by using reinforcement learning technology includes: Determine the key indicators that describe the overall performance of the system (such as response latency, throughput, energy consumption, etc.); Establish a global performance model covering edge devices, communication networks, local terminals, and cloud platforms, and describe the relationship and influencing factors between indicators at each level; Define the system state space (including key factors such as resource allocation and task load); Design a reinforcement learning algorithm suitable for the problem (such as Q-learning, policy gradient, etc.); Determine the reward function based on the overall performance index of the system; Establish a real-time monitoring mechanism between edge and cloud to collect system operation status data; Design data preprocessing and feature extraction methods to provide input data for reinforcement learning; Use the pre-processed and feature-extracted running status data to train the reinforcement learning model and learn the optimal resource scheduling and task migration strategy; In this step, various operating status indicators such as CPU / GPU utilization, memory usage, and network load are collected through system monitoring; the collected raw data is preprocessed by cleaning and normalization to ensure data quality; effective features related to resource scheduling are extracted in combination with business knowledge to prepare for subsequent reinforcement learning modeling; resource scheduling and task migration problems are modeled as Markov decision processes (MDPs); state space (system operating status), action space (scheduling strategy and task migration plan), reward function (optimization goal), etc. are defined; deep reinforcement learning algorithms such as DQN and PPO are used to train the intelligent agent to learn the optimal scheduling and migration strategy; the reinforcement learning model is repeatedly trained iteratively using the preprocessed training data; the reward function, neural network structure, hyperparameters, etc. are adjusted to optimize the model performance; a combination of offline training and online fine-tuning is used to improve the model generalization ability; the trained reinforcement learning model is deployed to the actual resource management system; the scheduling and migration effects of the model are tested in the production environment, and feedback data is collected for further optimization; a monitoring indicator system is established to regularly evaluate the operating status and optimization potential of the model. Through this step, the reinforcement learning model can automatically learn the optimal resource scheduling strategy according to the system operation status; the scheduling strategy can dynamically adapt to changes in business needs and system status to improve resource utilization efficiency; the reinforcement learning model can learn when and where to migrate tasks to maximize system performance and minimize migration overhead; the task migration strategy can make intelligent decisions based on multiple factors such as computing load, energy consumption, and cost; by designing a reasonable reward function, the reinforcement learning model can weigh multiple optimization goals such as performance, energy consumption, and cost; find the best balance between various goals and improve the overall efficiency of resource management; after the model is deployed to the actual system, it can continuously collect new monitoring data and continuously optimize its decision-making strategy; it shows the ability of self-learning and self-adaptation, and continues to improve as business needs change. In short, using reinforcement learning technology to achieve intelligent resource scheduling and task migration can greatly improve the resource utilization efficiency and overall performance of the computing system, and provide enterprises with more flexible and reliable computing services.
[0055] Deploy the trained model to the system and make performance optimization decisions in real time; Continuously monitor the system operation status and iteratively optimize the reinforcement learning model; Integrate the reinforcement learning optimization module into the edge-cloud-terminal collaborative framework to ensure the coordinated work of various components.
[0056] Through the solution of this embodiment, a global performance optimization model based on reinforcement learning can be established and continuous optimization can be achieved. Resource allocation and task deployment can be adjusted dynamically to optimize key performance indicators; adaptively respond to complex operating environment changes and maintain high performance; the optimal resource scheduling strategy can be learned to make full use of edge devices and cloud resources; dynamic load balancing across levels can be achieved to eliminate resource bottlenecks; reinforcement learning models can be continuously optimized without human intervention; they can be flexibly configured and expanded for different application scenarios; performance optimization decisions can be automatically executed to reduce manual operation and maintenance expenses; system reliability can be improved, and failure rates and maintenance costs can be reduced. In general, the global performance optimization model based on reinforcement learning can enable the edge-cloud collaborative computing system to achieve autonomous, intelligent and efficient resource management, thereby greatly improving the overall system performance and resource utilization efficiency, and providing excellent computing services for various application scenarios.
[0057] Through the solution of this embodiment, the system can continuously learn and adapt to new workloads and environmental changes; through experience accumulation, the quality of decision-making is gradually improved; the overall status of the system is considered to avoid global suboptimality caused by local optimization; short-term and long-term goals are balanced to achieve better resource allocation; the system can automatically adjust strategies to adapt to different workloads and hardware configurations; reduce manual intervention and reduce the complexity of operation and maintenance; achieve forward-looking resource scheduling through prediction of future states; respond to possible performance bottlenecks and resource competition in advance; consider multiple performance indicators at the same time to find the best balance point; support dynamic adjustment of optimization goals to adapt to changes in business needs; learn strategies for handling various abnormal and extreme situations to enhance the robustness and fault tolerance of the system; more refined and dynamic resource allocation to reduce resource waste; make full use of system potential to improve High overall efficiency; Learn energy-saving strategies to reduce energy consumption while ensuring performance; Support the realization of green computing goals; Provide customized performance optimization according to the characteristics of different users or applications; Improve user satisfaction and service quality; The system can quickly learn and utilize the characteristics of new hardware or software components to accelerate the implementation of innovative technologies and the realization of benefits; Provide explainable optimization suggestions by analyzing the decision-making process of the model to assist managers in understanding system behavior and performance bottlenecks; Reduce the need for manual configuration and tuning, and realize automated operation and maintenance; Free up the time of operation and maintenance personnel to focus on higher-level optimization tasks; The model can adapt to the dynamic changes in system scale; Support the rapid integration and optimization of new services and applications; Reduce hardware investment and operating costs through more efficient resource utilization; Optimize the use of cloud resources to achieve better cost control. Through this global performance optimization model based on reinforcement learning, the system can achieve continuous self-optimization and evolution, continuously improve performance and resource utilization, and have strong adaptability and scalability; This can not only improve the overall effectiveness of the system, but also bring significant economic benefits and competitive advantages to enterprises.
[0058] In some possible implementations of the present invention, the step of integrating secure multi-party computing and differential privacy technology to ensure privacy protection and security of data during integration and processing includes: Clarify the privacy needs and compliance requirements of all parties in the system (such as users, enterprises, and third parties); Design a comprehensive privacy protection framework based on secure multi-party computation and differential privacy, covering all aspects of data collection, transmission, storage and processing; Select the corresponding secure multi-party computing algorithm (such as Yao's Garbled Circuit, Secret Sharing, etc.); Design secure communication and computation protocols between multiple parties to ensure that each party can only obtain the final result and cannot spy on the intermediate process; Implement a secure multi-party computing module and integrate it with other system components; Identify privacy-sensitive data in the system and determine the corresponding differential privacy protection goals; Design a differential privacy algorithm based on noise injection to minimize privacy leakage while ensuring data availability; Integrate the differential privacy module into the data processing process to ensure the privacy protection of data at all stages; Establish a privacy risk assessment mechanism to regularly evaluate the privacy protection effectiveness of the system; Design a real-time monitoring module to detect possible privacy breaches and trigger response measures; Continuously optimize privacy protection solutions based on evaluation results and monitoring feedback; Design targeted privacy protection test scenarios, covering different attack models and data processing scenarios; Verify the effectiveness and reliability of secure multi-party computing and differential privacy technologies through testing; Optimize and improve based on test results.
[0059] Through the scheme of this embodiment, secure multi-party computing and differential privacy technology can be integrated into the system to achieve comprehensive data privacy protection. Secure multi-party computing ensures that all parties can only obtain the final result and cannot spy on the intermediate process; differential privacy technology injects noise interference to weaken the attacker's ability to infer sensitive data; differential privacy algorithms minimize privacy leakage while ensuring the availability of data analysis and applications; the system can safely integrate and process data without reducing the value of data; the privacy protection scheme complies with the requirements of relevant laws, regulations and industry standards; the real-time monitoring mechanism can quickly discover and respond to privacy leakage incidents and improve compliance; privacy protection technology is decoupled from other modules of the system and has good scalability; privacy protection strategies can be flexibly configured and optimized according to different application scenarios and privacy requirements. In general, by integrating secure multi-party computing and differential privacy technology, the privacy security of data in the system can be effectively protected, and the availability and value of data can be maximized while meeting compliance requirements.
[0060] The solution of this embodiment protects individual privacy during data processing and prevents the leakage of identity and sensitive information. Even in the event of data leakage, it can ensure that individual privacy cannot be easily inferred; supports calculations on encrypted data without decrypting the original data; allows multiple parties to perform joint calculations while protecting the privacy of their respective data; reduces the risk of single-point attacks through distributed storage and computing, and will not cause global data leakage even if some participants are breached; can perform statistical analysis and machine learning while protecting privacy; provides query results with differential privacy protection to prevent inference of individual information through multiple queries; meets the requirements of strict data protection regulations such as GDPR and CCPA, provides enterprises with technical guarantees for privacy compliance, and reduces legal risks; supports cross-organizational data sharing and analysis on the premise of protecting privacy; promotes data-driven innovation while protecting the privacy rights and interests of individuals and organizations; enhances user trust in the system through a transparent privacy protection mechanism; gives users more control over their data, Improve user participation; limit the scope of data use through technical means to prevent data from being used for unauthorized purposes; realize the "use and burn" of data to reduce the risks brought by long-term data storage; realize the training of machine learning models while protecting the privacy of original data to prevent privacy leakage caused by model reverse engineering; encourage users to provide more real and complete data through privacy protection technology; reduce data loss or distortion caused by privacy concerns; provide complete audit logs to facilitate supervision and internal control; support quantitative evaluation and continuous improvement of privacy protection effects; maintain system performance while protecting privacy through algorithm optimization and hardware acceleration; support dynamic adjustment of privacy protection level according to scenarios to balance availability and privacy; defend against advanced privacy attacks such as inference attacks and reconstruction attacks; provide long-term privacy protection, which can remain effective even in the face of future computing power improvements; realize complex analysis tasks with privacy protection on large-scale data sets, and provide privacy protection solutions for data-intensive applications. By integrating these technologies, the system can ensure the privacy and security of data while providing efficient data processing and analysis capabilities; this not only meets increasingly stringent regulatory requirements, but also provides a credible foundation for data-driven innovation and promotes the healthy development of the data economy.
[0061] In some possible implementations of the present invention, the step of determining an elastic scaling mechanism to support dynamic adjustment of system scale and seamless access to new computing resources includes: Analyze the business requirements and service characteristics of the system, determine the goals and key indicators of elastic scaling (such as response time, throughput, etc.); Design elastic scaling architecture and mechanisms based on business requirements and technical characteristics (how to monitor resource usage, when to trigger expansion / contraction, etc.); Establish a system resource monitoring module to collect CPU, memory, and network indicator data in real time; Use machine learning methods to predict future resource demand trends based on monitoring data and historical load patterns; Develop dynamic scaling decision strategies based on the forecast results of resource demand change trends (how to determine when to increase / decrease resources); In this step, the current resource supply situation is evaluated and compared with the predicted demand changes; when the predicted demand exceeds the current supply capacity, the expansion operation is automatically triggered, otherwise the contraction is triggered; the formulated expansion and contraction decisions are automatically converted into specific resource configuration change operations; the expansion and contraction process is continuously monitored to ensure that the target service quality indicators are within an acceptable range. In this embodiment, this dynamic expansion and contraction strategy based on the prediction of resource demand change trends can accurately expand and contract according to the actual demand change prediction results to avoid excess or insufficient resources; maximize the use of existing resources and reduce overall resource costs; perceive the trend of resource demand changes in advance, adjust resource supply in time, and ensure the stability of business service indicators; automate the decision-making and execution process of resource expansion and contraction, reduce manual operation and maintenance costs and response time; quickly adapt to changes in resource demand, and improve the overall elasticity and reliability of IT systems. In short, this dynamic expansion and contraction strategy can effectively improve the utilization efficiency of cloud computing resources and the quality of business services.
[0062] Implement an automated scaling execution mechanism (able to quickly add or release computing resources); Design access mechanisms for new computing resources to support seamless integration of heterogeneous hardware devices and cloud services; Implement resource discovery, scheduling, and orchestration functions to ensure that new resources can be effectively utilized by the system; Establish optimization algorithms based on resource utilization and service indicators to allocate and schedule computing resources; Adopt appropriate resource scheduling strategies (such as load balancing, resource affinity, etc.) based on resource characteristics and application requirements; Design redundant backup and self-healing mechanisms to ensure that the system can still run smoothly when resources fail or fluctuate; Design fault detection, isolation, and automatic recovery functions to improve system reliability.
[0063] Through the scheme of this embodiment, a complete set of elastic scaling mechanism can be established to realize dynamic adjustment of system scale and seamless access of new computing resources, automatically allocate computing resources according to actual needs, avoid excess or insufficient resources; quickly access new resources to improve overall computing power and resource utilization efficiency; adjust resource configuration in real time according to business needs to ensure that service indicators such as response time and throughput meet standards; automatic fault detection and self-healing mechanism to improve service reliability and sustainability; dynamic scaling of resources and access to new resources can be completed without manual intervention; automated monitoring, decision-making and execution mechanism to reduce the workload of operation and maintenance personnel; support rapid access to new computing resources such as heterogeneous hardware devices and cloud services; elastic scaling capability is decoupled from other system modules to improve the flexibility of the overall architecture. In general, by determining the elastic scaling mechanism, the system can dynamically adjust the resource scale according to actual needs and quickly access new computing devices, thereby improving resource utilization efficiency, ensuring service quality, reducing operation and maintenance costs, and enhancing the flexibility of the overall architecture.
[0064] Through the solution of this embodiment, the system can automatically adjust its scale according to load changes and seamlessly transition from small scale to large scale; support rapid response to sudden traffic to ensure service quality; dynamically adjust resource configuration to avoid resource waste; minimize resource costs while ensuring performance; cope with high loads through automatic capacity expansion to prevent system crashes; quickly isolate and replace faulty nodes to improve system fault tolerance; maintain stable response time and throughput even when load changes; reduce human intervention and reduce the risk of operation and maintenance errors; allocate resources on demand to avoid over-configuration; support cost-based intelligent decision-making to optimize operating expenses; easily integrate new computing resources, such as dedicated hardware accelerators; support rapid deployment and testing of new services to accelerate the innovation cycle; achieve unified resource management in multi-cloud and hybrid cloud environments to improve system flexibility; support dynamic resource management across regions Scheduling, optimizing global user experience; achieving disaster recovery and business continuity assurance; automatically adjusting system configuration according to different workload characteristics; supporting dynamic switching of different environments such as development, testing, and production; automated scaling operations reduce the need for manual intervention and simplify the management complexity of large-scale systems; supporting independent scaling in a multi-tenant environment to ensure that security isolation is maintained during the scaling process; supporting fine-grained service-level scaling and optimizing specific functional modules; achieving application-aware resource allocation and improving application performance; optimizing energy consumption by dynamically adjusting computing resources; supporting green computing goals and reducing carbon footprint; ensuring data consistency and integrity during dynamic scaling; supporting smooth expansion of stateless and stateful services; reducing service delays and interruptions through dynamic resource adjustment; supporting personalized performance guarantees to meet the needs of different user groups. Through this elastic scaling mechanism, the system can achieve efficient resource utilization, dynamic performance optimization, and effective cost control; it not only improves the reliability and scalability of the system, but also provides enterprises with a technical foundation for coping with complex and changing market environments and supports the rapid development and innovation of businesses.
[0065] In some possible implementations of the present invention, the step of using a machine learning method to predict future resource demand change trends based on monitoring data and historical load patterns includes: Real-time collection of monitoring data from cloud platforms, edge devices, and local terminals (such as CPU utilization, memory usage, network bandwidth, etc.); Performing operations of cleaning, filling missing values, and detecting outliers on the collected monitoring data to obtain first monitoring data; Extracting appropriate characteristic variables (such as average values and peak values calculated by time window) from the first monitoring data according to business characteristics and monitoring indicators; Normalize and standardize the feature variables to ensure the consistency of dimensions between features; Select a machine learning algorithm suitable for time series prediction (such as time series analysis, neural network, random forest, etc.) as the model to be trained; Obtain historical monitoring data and historical load patterns of cloud platforms, edge devices, and terminals; Divide the historical monitoring data into a training set and a validation set, train and cross-validate the training model, and obtain the first model; Evaluate the performance indicators of the first model (such as MSE, RMSE, etc.) and iteratively optimize the first model; Using the trained first model, historical load patterns and characteristic variables are input to predict the resource demand change trend in the future for different resource types (such as CPU, memory, network bandwidth, etc.).
[0066] In this embodiment, the historical load pattern refers to the regular pattern of analyzing resource usage in the past period of time, which specifically includes the following aspects: periodic change pattern (analyzing the periodic change law of resource usage on different time scales such as one day, one week, and one month, for example, some business systems will have obvious peak and valley differences between weekdays and weekends), seasonal change pattern (analyzing the change law of resource usage between different seasons of the year, and some industries may show obvious seasonal changes due to factors such as product sales peak season), event triggering pattern (analyzing the impact of some special events on resource usage patterns, such as new product releases, marketing activities, etc., which will cause a surge in short-term resource demand), abnormal fluctuation pattern (analyzing occasional abnormal fluctuations in resource usage and their causes, such as sudden business peaks or resource exhaustion caused by system failures), etc. By analyzing these historical load patterns, we can better understand the inherent laws of resource usage and provide a basis for subsequent demand forecasting modeling; by using these historical load features as input features, combined with real-time monitoring data, and using machine learning models for prediction, we can more accurately predict future resource demand change trends, which is of great significance for achieving dynamic expansion and contraction, optimizing resource utilization, etc.
[0067] In this embodiment, the machine learning method can deeply mine the complex patterns in the monitoring data to improve the accuracy and reliability of the prediction; automate the entire process of resource demand prediction and capacity expansion and contraction decision-making, greatly reducing the cost of manual operation and maintenance; timely perceive and respond to changes in resource demand, and improve the overall elasticity and reliability of the IT system; perform refined resource scheduling and capacity expansion and contraction based on accurate demand prediction results to improve resource utilization efficiency; predict resource demand changes in advance, allocate resources in a timely manner, and ensure the stability of business service quality indicators. In short, this dynamic resource demand prediction method based on machine learning is one of the key technologies for realizing intelligent and adaptive resource management, which can significantly improve the agility and operational efficiency of IT systems.
[0068] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0070] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0071] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0072] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0073] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0074] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.
[0075] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0076] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.
Claims
1. A multi-source computing power data integration and intelligent scheduling system, characterized in that: include: Central control server, cloud platform, edge devices and local terminals; The central control server is configured as follows: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
2. A multi-source computing power data integration and intelligent scheduling method, characterized in that: include: Establish a unified resource abstraction layer to standardize the description of multi-source heterogeneous computing resources and perform virtualization management; Build a dynamic data integration module for real-time collection, cleaning and standardization of multi-source heterogeneous data; Design an adaptive intelligent scheduling engine that integrates machine learning algorithms to perform task allocation and resource scheduling based on historical execution task data and real-time system status data; Build an edge-cloud-terminal collaborative computing framework that supports dynamic migration and load balancing of computing tasks; Establish a global performance optimization model and continuously optimize the overall system performance and resource utilization through reinforcement learning technology; Integrate secure multi-party computing and differential privacy technologies to ensure privacy protection and security of data during integration and processing; Determine elastic scaling mechanisms to support dynamic adjustment of system scale and seamless access to new computing resources.
3. The multi-source computing power data integration and intelligent scheduling method according to claim 2 is characterized in that: The steps of establishing a unified resource abstraction layer, describing multi-source heterogeneous computing resources in a standardized manner, and performing virtualization management include: Identify all computing resource types in the system; Classify computing resources and establish a resource type hierarchy; Design a general computing resource description model; Define key attributes for each computing resource type; Develop automated tools to collect detailed information about each specific computing resource; Mapping the collected information into a computing resource description model; Design virtualization management interfaces to abstract underlying hardware details; Implement resource pooling mechanism to transform physical resources into manageable virtual resource units; Design a unified API for resource allocation, monitoring, and recycling; Generate cross-platform resource management protocols; Deploy distributed monitoring agents to collect resource usage in real time; Design a central monitoring panel to display the overall resource status; Build dynamic resource discovery and registration mechanisms; Build a role-based access control system; Design encrypted communication channels to protect resource information transmission; Build a cache mechanism to speed up the query of frequently accessed resource information; Optimize storage and retrieval algorithms for computing resource descriptions; Design a modular architecture to support the access of new computing resource types; Build a distributed resource management framework to support large-scale clusters.
4. The multi-source computing power data integration and intelligent scheduling method according to claim 3 is characterized in that: The steps of constructing a dynamic data integration module for realizing real-time collection, cleaning and standardization of multi-source heterogeneous data include: Identify all data source types, classify data sources, and establish a data source catalog; Define common data structures and design metadata models to describe data attributes, sources, and timestamps; Develop dedicated data collectors for each data source type and build a data access layer that supports multiple protocols; Build a stream processing framework, and design and implement streaming data processing pipelines; Design data format conversion functions, develop data validation and error handling mechanisms, and design data cleaning algorithms; Perform data field mapping and conversion, develop unit conversion and standardization function libraries, and design data semantic conversion mechanisms; Build real-time data quality detection algorithms, design data quality scoring systems, and develop abnormal data processing mechanisms; Design data fusion algorithms to handle conflicts and complementarities of multi-source data, and develop data association analysis functions and data consistency check mechanisms; Build data caching mechanisms and design parallel processing architectures to optimize data storage and indexing strategies; Adopt a microservices architecture, identify dynamic configuration management, and design a distributed processing framework; Develop data flow monitoring dashboards to visualize data processing and design alarm and notification mechanisms.
5. The multi-source computing power data integration and intelligent scheduling method according to claim 4 is characterized in that: The steps of designing an adaptive intelligent scheduling engine, integrating a machine learning algorithm, and performing task allocation and resource scheduling based on historical execution task data and real-time system status data include: Design a data collection module based on the data collector, and use the data collection module to obtain historical task execution data and real-time system status data; Perform data cleaning and feature extraction processes on historical task execution data and real-time system status data, and build a data storage system to support fast query and analysis; Define the task feature set based on the features extracted from historical task execution data, develop a task classification algorithm, identify the task type, and build a task demand prediction model; Design resource status feature sets based on features extracted from real-time system status data, develop resource performance prediction models and resource availability assessment algorithms; Select machine learning algorithms, design model training processes and model evaluation / validation mechanisms; According to the task demand prediction model, resource performance prediction model and resource availability evaluation algorithm, develop a resource scheduling decision model based on machine learning model, determine the multi-objective optimization mechanism, and design a scheduling strategy scoring system; Build a real-time reasoning module to generate scheduling decisions; Determine decision execution and feedback mechanisms; Develop anomaly detection and handling processes.
6. The multi-source computing power data integration and intelligent scheduling method according to claim 5 is characterized in that: The steps of constructing an edge-cloud-terminal collaborative computing framework that can support dynamic migration and load balancing of computing tasks include: Develop lightweight edge devices for preliminary data processing and analysis at the source of data; Design communication protocols between edge devices to support distributed collaboration; Develop dynamic resource management and task scheduling algorithms to achieve load balancing among edge devices; Establish an elastically scalable cloud platform to handle large-scale data and complex computing tasks; Develop cloud platform resource management and task scheduling algorithms based on machine learning to achieve dynamic migration and load balancing across edge devices; Provide API interface to support two-way communication and collaboration between edge devices and cloud platforms; Develop a collaboration mechanism between edge devices and cloud platforms, including task allocation strategies, data transmission protocols, and security authentication mechanisms; Develop cross-level resource monitoring and scheduling algorithms to achieve dynamic load balancing between edge, cloud and terminal; Design fault-tolerant mechanisms to ensure the continued reliable operation of computing tasks.
7. The multi-source computing power data integration and intelligent scheduling method according to claim 6 is characterized in that: The steps of establishing a global performance optimization model and continuously optimizing the overall system performance and resource utilization through reinforcement learning technology include: Identify key metrics that describe the overall performance of the system; Establish a global performance model covering edge devices, communication networks, local terminals, and cloud platforms, and describe the relationship and influencing factors between indicators at each level; Define the system state space; Design a reinforcement learning algorithm suitable for the problem; Determine the reward function based on the overall performance index of the system; Establish a real-time monitoring mechanism between edge and cloud to collect system operation status data; Design data preprocessing and feature extraction methods to provide input data for reinforcement learning; Use the pre-processed and feature-extracted running status data to train the reinforcement learning model and learn the optimal resource scheduling and task migration strategy; Deploy the trained model to the system and make performance optimization decisions in real time; Continuously monitor the system operation status and iteratively optimize the reinforcement learning model; Integrate the reinforcement learning optimization module into the edge-cloud-terminal collaborative framework to ensure the coordinated work of various components.
8. The multi-source computing power data integration and intelligent scheduling method according to claim 7 is characterized in that: The steps of integrating secure multi-party computing and differential privacy technology to ensure privacy protection and security of data during integration and processing include: Clarify the privacy needs and compliance requirements of all parties in the system; Design a comprehensive privacy protection framework based on secure multi-party computation and differential privacy, covering all aspects of data collection, transmission, storage and processing; Select the corresponding secure multi-party computing algorithm; Design secure communication and computation protocols between multiple parties to ensure that each party can only obtain the final result and cannot spy on the intermediate process; Implement a secure multi-party computing module and integrate it with other system components; Identify privacy-sensitive data in the system and determine the corresponding differential privacy protection goals; Design a differential privacy algorithm based on noise injection to minimize privacy leakage while ensuring data availability; Integrate the differential privacy module into the data processing process to ensure the privacy protection of data at all stages; Establish a privacy risk assessment mechanism to regularly evaluate the privacy protection effectiveness of the system; Design a real-time monitoring module to detect possible privacy breaches and trigger response measures; Continuously optimize privacy protection solutions based on evaluation results and monitoring feedback; Design targeted privacy protection test scenarios, covering different attack models and data processing scenarios; Verify the effectiveness and reliability of secure multi-party computing and differential privacy technologies through testing; Optimize and improve based on test results.
9. The multi-source computing power data integration and intelligent scheduling method according to claim 8 is characterized in that: The step of determining an elastic scaling mechanism to support dynamic adjustment of system scale and seamless access of new computing resources includes: Analyze the business requirements and service characteristics of the system, and determine the goals and key indicators of elastic scaling; Design elastic scaling architecture and mechanisms based on business requirements and technical characteristics; Establish a system resource monitoring module to collect CPU, memory, and network indicator data in real time; Use machine learning methods to predict future resource demand trends based on monitoring data and historical load patterns; Formulate dynamic expansion and contraction decision strategies based on the forecast results of resource demand change trends; Implement an automated scaling execution mechanism; Design access mechanisms for new computing resources to support seamless integration of heterogeneous hardware devices and cloud services; Implement resource discovery, scheduling, and orchestration functions to ensure that new resources can be effectively utilized by the system; Establish optimization algorithms based on resource utilization and service indicators to allocate and schedule computing resources; Adopt appropriate resource scheduling strategies based on resource characteristics and application requirements; Design redundant backup and self-healing mechanisms to ensure that the system can still run smoothly when resources fail or fluctuate; Design fault detection, isolation, and automatic recovery functions to improve system reliability.
10. The multi-source computing power data integration and intelligent scheduling method according to claim 9 is characterized in that: The step of using a machine learning method to predict future resource demand change trends based on monitoring data and historical load patterns includes: Collect monitoring data from cloud platforms, edge devices, and local terminals in real time; Performing operations of cleaning, filling missing values, and detecting outliers on the collected monitoring data to obtain first monitoring data; Extracting appropriate feature variables from the first monitoring data according to business characteristics and monitoring indicators; Normalize and standardize the feature variables to ensure the consistency of dimensions between features; Select a machine learning algorithm suitable for time series prediction as the model to be trained; Obtain historical monitoring data and historical load patterns of cloud platforms, edge devices, and terminals; Divide the historical monitoring data into a training set and a validation set, train and cross-validate the training model, and obtain the first model; Evaluate the performance index of the first model and iteratively optimize the first model; By using the trained first model, historical load patterns and characteristic variables are input to predict the resource demand change trend within a certain period of time in the future for different resource types.
Citation Information
Patent Citations
Distributed cloud management system, and distributed cloud management method, computer readable storage medium and application
CN113867965A
Cloud management system of hybrid architecture and management method thereof
CN116248678A
Data computing resource configuration method considering privacy protection under edge cloud collaboration framework
CN116614502A
Intelligent power grid system based on cloud edge fusion architecture and scheduling method
CN116739236A
Edge computing performance optimization system for data-intensive AI computing
CN117453411A
Cited By
Chip platform hardware abstraction layer construction method
CN119312219A
Lightweight asset risk scanning method and device
CN119312332A
Computing power arrangement method and device, equipment and storage medium
CN119356828A
Dynamic CPU scheduling method and system for RISC-V platform
CN119356897A
A RISC-V platform dynamic CPU scheduling method and system
CN119356897B
