Data integration and elastic scheduling method suitable for fragmented computing power pool

Abstract

The invention relates to the field of data integration scheduling, and discloses a data integration and elastic scheduling method suitable for a fragmented computing power pool, which comprises the following steps of: constructing the fragmented computing power pool in combination with different distributed resources, and constructing a uniform resource description model in combination with the fragmented computing power pool, and data integration and elastic partition scheduling processing of distributed resources are realized. The resource utilization rate is improved, and the calculation cost is reduced; the purposes of enhancing the elasticity of the system and conveniently coping with various complex scenes are achieved; and meanwhile, data-driven intelligent scheduling is realized, and more flexible and efficient resource utilization is realized.

Description

Technical Field

[0001] The present invention relates to the field of data integration and scheduling, and in particular to a data integration and elastic scheduling method suitable for a fragmented computing power pool. Background Art

[0002] A fragmented computing pool is a collection of distributed, heterogeneous, and dynamically changing computing resources. These resources may be distributed across different geographic locations, hardware architectures, or administrative domains. This fragmentation is caused by discontinuous resource allocation, widely varying specifications, or fluctuating utilization. Its core characteristic is that it achieves more flexible and efficient resource utilization by integrating discontinuous and diverse computing resources (such as cloud servers, edge devices, and idle GPUs).

[0003] By integrating distributed, heterogeneous computing resources and combining them with dynamic scheduling strategies, fragmented computing pools offer significant advantages and far-reaching implications for data integration and flexible scheduling. Their core advantages include improving resource utilization and reducing computing costs; enhancing system resilience and facilitating the handling of various complex scenarios; and enabling data-driven intelligent scheduling. Fragmented computing pools promote the evolution of distributed computing, supporting collaborative computing across cloud, edge, and device, and providing the computing power foundation for scenarios such as the metaverse and autonomous driving. They also enable low-cost utilization of fragmented resources for non-real-time tasks (such as log analysis and gene sequencing), expanding business boundaries and reducing carbon emissions. Through resource integration and flexible scheduling, fragmented computing pools achieve a paradigm shift from "hard allocation of fixed resources" to "soft matching of dynamic demand." Their value lies not only in improved technical efficiency but also in restructuring the computing power supply model—from centralized monopoly to open sharing, laying the foundation for the democratization of computing power in the digital economy era. Therefore, we propose a data integration and flexible scheduling method for fragmented computing pools. Summary of the Invention

[0004] The present invention overcomes the deficiencies of the prior art and provides a data integration and elastic scheduling method suitable for fragmented computing power pools.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: A first aspect of the present invention provides a data integration and flexible scheduling method applicable to a fragmented computing power pool, comprising the following steps: Based on different distributed resources, a fragmented computing power pool is integrated, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool; Combined with the real-time fragmented computing power pool, a unified resource description model is constructed to achieve standardized processing of the data integration and scheduling environment; Combined with the target unified resource description model, a target multi-dimensional resource feature matrix is constructed to perform data integration partitioning and elastic scheduling partitioning on the target distributed resources.

[0006] Furthermore, in a preferred embodiment of the present invention, the fragmented computing power pool is obtained by integrating different distributed resources, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool, specifically: Identify the distributed resources that require data integration and flexible scheduling, mark them as target distributed resources, and place information collection sensors on all target distributed resources; Obtain the cloud platform API, control the cloud platform API to connect signals to the information collection sensors of all target distributed resources, and ensure that the cloud platform API can automatically scan the information base of all target distributed resources and register computing power nodes; Among them, one target distributed resource corresponds to registering one computing power node; On the cloud platform API, a fragmented computing power pool is constructed based on computing power nodes of all registered target distributed resources, wherein the fragmented computing power pool integrates all target distributed resources; In the fragmented computing power pool, information collection sensors on different target distributed resources are used to collect individual resource information of different target distributed resources. The individual resource information of the target distributed resources includes the computing power type, memory capacity, storage performance, network bandwidth, and geographical location of the target distributed resources. Based on the individual resource information of the target distributed resources, the fragmented computing power pool is collected and updated in real time to obtain a real-time fragmented computing power pool.

[0007] Furthermore, in a preferred embodiment of the present invention, the real-time fragmented computing power pool is combined to construct a unified resource description model to achieve standardized processing of the data integration scheduling environment, specifically: Through the real-time fragmented computing power pool and the resource individual information of different target distributed resources, all target distributed resources are classified by attributes, and the attribute standards of the resource individual information of all target distributed resources are defined; Combine the attribute standards of the individual resource information of all target distributed resources to build a unified resource description model. The unified resource description model is to convert the individual resource information of different target distributed resources into comparable numerical indicators, and then map all comparable numerical indicators to the same description model through computing power normalization. The unified resource description model can perform comparative analysis on the resource individual information of different target distributed resources in the real-time fragmented computing power pool under different attribute standards; Introducing container technology to encapsulate the runtime environment of the unified resource description model, and building a dynamic metadata management system in the unified resource description model after the runtime environment is encapsulated to obtain the target unified resource description model; Among them, the dynamic metadata management system is used to perform real-time storage and update monitoring of individual resource information in the unified resource description model encapsulated by the operating environment.

[0008] Furthermore, in a preferred embodiment of the present invention, the target unified resource description model is combined to construct a target multi-dimensional resource feature matrix, and data integration partitioning and elastic scheduling partitioning are performed on the target distributed resources, specifically: In the target unified resource description model, static dimension features and dynamic dimension features are extracted, and numerical normalization processing and weight dynamic allocation processing are performed on the static dimension features and dynamic dimension features to construct a multidimensional feature matrix of the target unified resource description model, which is calibrated as the target multidimensional feature matrix; Before dynamically allocating weights to the static and dynamic dimensional features, a historical data network is obtained, historical weights corresponding to the static and dynamic dimensional features are retrieved from the historical data network, and dynamic weight allocation is performed based on the historical weights corresponding to the static and dynamic dimensional features. In combination with the target multi-dimensional feature matrix, a dynamic resource profile of different target distributed resources is constructed and generated. The method for generating the dynamic resource profile of the target distributed resource is as follows: in the target multi-dimensional feature matrix, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is calculated, a data volatility scoring mechanism is preset, and according to the data volatility scoring mechanism, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is output; In the target unified resource description model, a profile update time is preset. Within the profile update time interval, the data volatility of the target multi-dimensional feature matrix is calculated in real time to update the dynamic resource profiles of different target distributed resources. Combined with the dynamic resource profiles of different target distributed resources, data integration partitioning and elastic scheduling partitioning are performed on different target distributed resources.

[0009] Furthermore, in a preferred embodiment of the present invention, it is characterized in that the dynamic resource profiles of different target distributed resources are combined to perform data integration partitioning and elastic scheduling partitioning on different target distributed resources, specifically: In the target unified resource description model, a distributed resource area is constructed and resource partitioning rules are designed, wherein the distributed resource area includes a hot pool area, a warm pool area, and a cold pool area; Based on the distributed resource regions and resource partitioning rules, combined with the dynamic resource profile of the target distributed resource, initial partitioning processing of the target distributed resource is implemented in the target unified resource description model; After the target distributed resources are initially partitioned, a k-value clustering algorithm is introduced to cluster the features of different target distributed resources. During the feature clustering process, the dynamic boundaries of the dynamic resource portraits of the target distributed resources are updated in real time, thus implementing partition update processing of the target distributed resources in the target unified resource description model. After the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled for partitions within the target unified resource description model.

[0010] Furthermore, in a preferred embodiment of the present invention, after the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled within the target unified resource description model, specifically: In the distributed resource area of the target unified resource description model, standard node load thresholds of the hot pool area, the warm pool area, and the cold pool area are obtained, and the real-time node load values of the hot pool area, the warm pool area, and the cold pool area are monitored in real time; If the real-time node load values in the hot pool, warm pool, and cold pool areas do not remain within the standard node load threshold, the target distributed resources in the hot pool, warm pool, and cold pool areas are elastically scheduled to ensure that the real-time node load values in the hot pool, warm pool, and cold pool areas remain within the standard node load threshold. If the real-time node load value in the warm pool or cold pool does not remain within the standard node load threshold, the target distributed resource will be elastically scheduled to the hot pool first. If the real-time node load value in the hot pool area does not remain within the standard node load threshold, and after the target distributed resources are partitioned and elastically scheduled, the real-time node load value in both the warm pool area and the cold pool area does not remain within the standard node load threshold, the preset overload standard time is set; When the real-time node load value does not remain within the standard node load threshold for a duration greater than the overload standard time, an expansion warning is triggered in the hot pool area, and the hot pool area is controlled to perform capacity expansion until the real-time node load values in the hot pool area, warm pool area, and cold pool area are all maintained within the standard node load threshold.

[0011] A second aspect of the present invention further provides a data integration and elastic scheduling system applicable to a fragmented computing power pool. The data integration and elastic scheduling system includes a memory and a processor. The memory stores a data integration and elastic scheduling method. When the data integration and elastic scheduling method is executed by the processor, the following steps are implemented: Based on different distributed resources, a fragmented computing power pool is integrated, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool; Combined with the real-time fragmented computing power pool, a unified resource description model is constructed to achieve standardized processing of the data integration and scheduling environment; Combined with the target unified resource description model, a target multi-dimensional resource feature matrix is constructed to perform data integration partitioning and elastic scheduling partitioning on the target distributed resources.

[0012] This invention addresses the technical deficiencies in the background technology and has the following beneficial effects: it combines different distributed resources to construct a fragmented computing power pool, and then builds a unified resource description model based on the fragmented computing power pool, enabling data integration and flexible partition scheduling of distributed resources. This invention improves resource utilization and reduces computing costs; it also enhances system flexibility and facilitates handling of various complex scenarios; and it also implements data-driven intelligent scheduling, achieving more flexible and efficient resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.

[0014] Figure 1 A flow chart showing a data integration and elastic scheduling method applicable to a fragmented computing power pool is shown; Figure 2 A flow chart showing a method for data integration partitioning and elastic scheduling partitioning of target distributed resources is shown; Figure 3 A program view of a data integration and elastic scheduling system suitable for fragmented computing power pools is shown. DETAILED DESCRIPTION

[0015] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0016] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0017] Figure 1 A flowchart of a data integration and elastic scheduling method applicable to a fragmented computing power pool is shown, including the following steps: S102: Based on different distributed resources, a fragmented computing power pool is obtained by integrating and collecting information on the different distributed resources to obtain a real-time fragmented computing power pool; S104: Combine the real-time fragmented computing power pool to build a unified resource description model to achieve standardized processing of the data integration and scheduling environment; S106: Combine the target unified resource description model to build a target multi-dimensional resource feature matrix, and perform data integration partitioning and elastic scheduling partitioning on the target distributed resources.

[0018] Furthermore, in a preferred embodiment of the present invention, the fragmented computing power pool is obtained by integrating different distributed resources, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool, specifically: Identify the distributed resources that require data integration and flexible scheduling, mark them as target distributed resources, and place information collection sensors on all target distributed resources; Obtain the cloud platform API, control the cloud platform API to connect signals to the information collection sensors of all target distributed resources, and ensure that the cloud platform API can automatically scan the information base of all target distributed resources and register computing power nodes; Among them, one target distributed resource corresponds to registering one computing power node; On the cloud platform API, a fragmented computing power pool is constructed based on computing power nodes of all registered target distributed resources, wherein the fragmented computing power pool integrates all target distributed resources; In the fragmented computing power pool, information collection sensors on different target distributed resources are used to collect individual resource information of different target distributed resources. The individual resource information of the target distributed resources includes the computing power type, memory capacity, storage performance, network bandwidth, and geographical location of the target distributed resources. Based on the individual resource information of the target distributed resources, the fragmented computing power pool is collected and updated in real time to obtain a real-time fragmented computing power pool.

[0019] It should be noted that a fragmented computing power pool achieves more flexible and efficient resource utilization by integrating discontinuous and diverse computing resources, namely distributed resources. Distributed resources include, but are not limited to, cloud servers, edge devices, and idle CPUs. Different distributed resources are distributed across multiple physical locations and may be intermittently online or have no idle periods. It is necessary to integrate all target distributed resources to ensure efficient resource utilization. After identifying the target distributed resources, individual resource information must be collected, including their computing power type, memory capacity, storage performance, network bandwidth, and geographic location. This information provides conditional information and scheduling targets during the integration process. The cloud platform API is used for devices that perform preliminary integration of sensor data and register computing nodes. Registered computing nodes are used to build the fragmented computing power pool. Because a fragmented computing power pool is composed of different nodes, the computing power type, memory capacity, storage performance, network bandwidth, and geographic location of the target distributed resources correspond to different computing nodes. All computing nodes are topologically constructed to achieve the construction and processing of the fragmented computing power pool.

[0020] Furthermore, in a preferred embodiment of the present invention, the real-time fragmented computing power pool is combined to construct a unified resource description model to achieve standardized processing of the data integration scheduling environment, specifically: Through the real-time fragmented computing power pool and the resource individual information of different target distributed resources, all target distributed resources are classified by attributes, and the attribute standards of the resource individual information of all target distributed resources are defined; Combine the attribute standards of the individual resource information of all target distributed resources to build a unified resource description model. The unified resource description model is to convert the individual resource information of different target distributed resources into comparable numerical indicators, and then map all comparable numerical indicators to the same description model through computing power normalization. The unified resource description model can perform comparative analysis on the resource individual information of different target distributed resources in the real-time fragmented computing power pool under different attribute standards; Introducing container technology to encapsulate the runtime environment of the unified resource description model, and building a dynamic metadata management system in the unified resource description model after the runtime environment is encapsulated to obtain the target unified resource description model; Among them, the dynamic metadata management system is used to perform real-time storage and update monitoring of individual resource information in the unified resource description model encapsulated by the operating environment.

[0021] It should be noted that the real-time fragmented computing power pool is used to build a unified resource description model. This model uniformly describes the static properties of heterogeneous resources and eliminates the interference of hardware differences on upper-level scheduling. The real-time fragmented computing power pool is a prerequisite for building the unified resource description model. The unified resource description model classifies distributed resources by their basic attributes, converting their capabilities into comparable numerical metrics to facilitate the scheduling system's matching of task requirements. These metrics include computing power type, computing capacity, storage performance, and network topology. Computing power normalization involves converting the hardware instances of different distributed resources into a single value, achieving resource standardization. Runtime environment standardization aims to mask the underlying hardware differences of distributed resources and ensure consistent operation of different resources during task execution. Container technology is used to encapsulate the runtime environment, such as pre-installing dependency libraries and toolchains, to ensure a consistent execution environment for applications across different nodes, preventing resource contention caused by inconsistent resources. Finally, a dynamic metadata management system is constructed to track resource status in real time, providing accurate input for subsequent scheduling decisions.

[0022] Figure 2 The flowchart of the method for data integration partitioning and elastic scheduling partitioning of target distributed resources is shown, including the following steps: S202: Based on the target unified resource description model, a target multi-dimensional resource feature matrix is constructed, and data integration and partitioning as well as flexible scheduling partitioning are performed on the target distributed resources; S204: Based on the dynamic resource profiles of different target distributed resources, data integration and partitioning as well as elastic scheduling partitioning are performed on the different target distributed resources; S206: After the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled for partitions within the target unified resource description model.

[0023] Furthermore, in a preferred embodiment of the present invention, the target unified resource description model is combined to construct a target multi-dimensional resource feature matrix, and data integration partitioning and elastic scheduling partitioning are performed on the target distributed resources, specifically: In the target unified resource description model, static dimension features and dynamic dimension features are extracted, and numerical normalization processing and weight dynamic allocation processing are performed on the static dimension features and dynamic dimension features to construct a multidimensional feature matrix of the target unified resource description model, which is calibrated as the target multidimensional feature matrix; Before dynamically allocating weights to the static and dynamic dimensional features, a historical data network is obtained, historical weights corresponding to the static and dynamic dimensional features are retrieved from the historical data network, and dynamic weight allocation is performed based on the historical weights corresponding to the static and dynamic dimensional features. In combination with the target multi-dimensional feature matrix, a dynamic resource profile of different target distributed resources is constructed and generated. The method for generating the dynamic resource profile of the target distributed resource is as follows: in the target multi-dimensional feature matrix, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is calculated, a data volatility scoring mechanism is preset, and according to the data volatility scoring mechanism, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is output; In the target unified resource description model, a profile update time is preset. Within the profile update time interval, the data volatility of the target multi-dimensional feature matrix is calculated in real time to update the dynamic resource profiles of different target distributed resources. Combined with the dynamic resource profiles of different target distributed resources, data integration partitioning and elastic scheduling partitioning are performed on different target distributed resources.

[0024] It should be noted that within the target unified resource description model, the dynamic metadata management system can collect data from both static and dynamic dimensions. Static data includes data on distributed resources' computing power, storage performance, network characteristics, and hardware configuration; dynamic data includes data on distributed resources' real-time load, availability, and economic indicators. After collecting these data, a feature matrix is constructed to generate dynamic resource profiles for different target distributed resources. Dynamic resource profiles intuitively summarize individual information about distributed resources. During feature matrix construction, numerical normalization is required, including normalization of continuous and categorical data to ensure uniformity and standardization during the computational process. Weighting the static and dynamic data is also necessary, a necessary step in matrix construction. A method for generating dynamic resource profiles for target distributed resources is presented, enabling profile generation based on different scoring methods. Profiles require periodic updates because distributed resources may experience sudden changes in resource status, such as a sudden and severe change in GPU temperature, or the sudden addition of new nodes or the offline of old ones. Therefore, a portrait update time is preset. Within the portrait update time interval, the data volatility of the target multi-dimensional feature matrix is calculated in real time, and the dynamic resource portraits of different target distributed resources are updated to ensure that the dynamic resource portraits of the target distributed resources are always kept in an updated state, thereby improving the accuracy of data integration and scheduling.

[0025] Furthermore, in a preferred embodiment of the present invention, it is characterized in that the dynamic resource profiles of different target distributed resources are combined to perform data integration partitioning and elastic scheduling partitioning on different target distributed resources, specifically: In the target unified resource description model, a distributed resource area is constructed and resource partitioning rules are designed, wherein the distributed resource area includes a hot pool area, a warm pool area, and a cold pool area; Based on the distributed resource regions and resource partitioning rules, combined with the dynamic resource profile of the target distributed resource, initial partitioning processing of the target distributed resource is implemented in the target unified resource description model; After the target distributed resources are initially partitioned, a k-value clustering algorithm is introduced to cluster the features of different target distributed resources. During the feature clustering process, the dynamic boundaries of the dynamic resource portraits of the target distributed resources are updated in real time, thus implementing partition update processing of the target distributed resources in the target unified resource description model. After the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled for partitions within the target unified resource description model.

[0026] It should be noted that in the target unified resource description model, the purpose of constructing distributed resource areas is to partition the data in distributed resources and achieve data integration and flexible scheduling. Among them, the resource partitioning rule is to partition the data information of distributed resources into hot pool areas, warm pool areas and cold pool areas. The hot pool area is used to deploy delay-sensitive real-time tasks, such as video stream processing; the warm pool area is used to run regular batch tasks, such as log analysis and other processing; the cold pool area is used to process non-urgent tasks, such as scientific research calculations and other processing. According to different data characteristics, different target distributed resources are clustered by feature using the k-value clustering algorithm to achieve data classification processing and determine which distributed resource area the data should be stored in. The purpose of real-time updating of the dynamic boundaries of the dynamic resource portrait of the target distributed resources is to achieve real-time dynamic balance of the partitions and keep different areas from being overloaded.

[0027] Furthermore, in a preferred embodiment of the present invention, after the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled within the target unified resource description model, specifically: In the distributed resource area of the target unified resource description model, standard node load thresholds of the hot pool area, the warm pool area, and the cold pool area are obtained, and the real-time node load values of the hot pool area, the warm pool area, and the cold pool area are monitored in real time; If the real-time node load values in the hot pool, warm pool, and cold pool areas do not remain within the standard node load threshold, the target distributed resources in the hot pool, warm pool, and cold pool areas are elastically scheduled to ensure that the real-time node load values in the hot pool, warm pool, and cold pool areas remain within the standard node load threshold. If the real-time node load value in the warm pool or cold pool does not remain within the standard node load threshold, the target distributed resource will be elastically scheduled to the hot pool first. If the real-time node load value in the hot pool area does not remain within the standard node load threshold, and after the target distributed resources are partitioned and elastically scheduled, the real-time node load value in both the warm pool area and the cold pool area does not remain within the standard node load threshold, the preset overload standard time is set; When the real-time node load value does not remain within the standard node load threshold for a duration greater than the overload standard time, an expansion warning is triggered in the hot pool area, and the hot pool area is controlled to perform capacity expansion until the real-time node load values in the hot pool area, warm pool area, and cold pool area are all maintained within the standard node load threshold.

[0028] It should be noted that, since the load values that different distributed resource areas can bear are certain, data is elastically scheduled based on the real-time load values of the distributed resource areas, combined with the corresponding thresholds, in order to ensure that the distributed resource areas are not overloaded. The hot pool is the area with the highest priority, so when the real-time node load value in the warm pool area or the cold pool area does not remain within the standard node load threshold, the target distributed resource is elastically scheduled to the hot pool area first to maintain dynamic balance. If the load value of the data of the target distributed resource in the hot pool area exceeds the threshold, it can be scheduled to other areas, and other areas cannot be expanded. Therefore, when the real-time node load value in the hot pool area does not remain within the standard node load threshold, and after the target distributed resource is elastically scheduled for partitioning, the real-time node load value in both the warm pool area and the cold pool area does not remain within the standard node load threshold, in order to maintain the integrity of the resources, the hot pool area needs to be expanded. The present invention improves resource utilization, realizes staggered resource use, ensures service stability, and also optimizes comprehensive costs.

[0029] like Figure 3 As shown, the second aspect of the present invention further provides a data integration and elastic scheduling system applicable to a fragmented computing power pool, the data integration and elastic scheduling system comprising a memory 31 and a processor 32, the memory 31 storing a data integration and elastic scheduling method, and when the data integration and elastic scheduling method is executed by the processor 32, the following steps are implemented: Based on different distributed resources, a fragmented computing power pool is integrated, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool; Combined with the real-time fragmented computing power pool, a unified resource description model is constructed to achieve standardized processing of the data integration and scheduling environment; Combined with the target unified resource description model, a target multi-dimensional resource feature matrix is constructed to perform data integration partitioning and elastic scheduling partitioning on the target distributed resources.

[0030] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data integration and elastic scheduling method suitable for fragmented computing power pools, characterized in that: The following steps are involved: Based on different distributed resources, a fragmented computing power pool is integrated, and information is collected from different distributed resources to obtain a real-time fragmented computing power pool; Combined with the real-time fragmented computing power pool, a unified resource description model is constructed to achieve standardized processing of the data integration and scheduling environment; Combined with the target unified resource description model, a target multi-dimensional resource feature matrix is constructed to perform data integration partitioning and elastic scheduling partitioning on the target distributed resources.

2. A data integration and flexible scheduling method for a fragmented computing power pool according to claim 1, characterized in that: Based on different distributed resources, the fragmented computing power pool is integrated and information is collected from different distributed resources to obtain a real-time fragmented computing power pool. Specifically: Identify the distributed resources that require data integration and flexible scheduling, mark them as target distributed resources, and place information collection sensors on all target distributed resources; Obtain the cloud platform API, control the cloud platform API to connect signals to the information collection sensors of all target distributed resources, and ensure that the cloud platform API can automatically scan the information base of all target distributed resources and register computing power nodes; Among them, one target distributed resource corresponds to registering one computing power node; On the cloud platform API, a fragmented computing power pool is constructed based on computing power nodes of all registered target distributed resources, wherein the fragmented computing power pool integrates all target distributed resources; In the fragmented computing power pool, information collection sensors on different target distributed resources are used to collect individual resource information of different target distributed resources. The individual resource information of the target distributed resources includes the computing power type, memory capacity, storage performance, network bandwidth, and geographical location of the target distributed resources. Based on the individual resource information of the target distributed resources, the fragmented computing power pool is collected and updated in real time to obtain a real-time fragmented computing power pool.

3. A data integration and flexible scheduling method for a fragmented computing power pool according to claim 1, characterized in that: The above mentioned method combines the real-time fragmented computing power pool to build a unified resource description model and realize the standardized processing of the data integration and scheduling environment. Specifically: Through the real-time fragmented computing power pool and the resource individual information of different target distributed resources, all target distributed resources are classified by attributes, and the attribute standards of the resource individual information of all target distributed resources are defined; Combine the attribute standards of the individual resource information of all target distributed resources to build a unified resource description model. The unified resource description model is to convert the individual resource information of different target distributed resources into comparable numerical indicators, and then map all comparable numerical indicators to the same description model through computing power normalization. The unified resource description model can perform comparative analysis on the resource individual information of different target distributed resources in the real-time fragmented computing power pool under different attribute standards; Introducing container technology to encapsulate the runtime environment of the unified resource description model, and building a dynamic metadata management system in the unified resource description model after the runtime environment is encapsulated to obtain the target unified resource description model; Among them, the dynamic metadata management system is used to perform real-time storage and update monitoring of individual resource information in the unified resource description model encapsulated by the operating environment.

4. A data integration and flexible scheduling method for a fragmented computing power pool according to claim 1, characterized in that: The target unified resource description model is combined to construct a target multi-dimensional resource feature matrix, and data integration partitioning and elastic scheduling partitioning are performed on the target distributed resources. Specifically: In the target unified resource description model, static dimension features and dynamic dimension features are extracted, and numerical normalization processing and weight dynamic allocation processing are performed on the static dimension features and dynamic dimension features to construct a multidimensional feature matrix of the target unified resource description model, which is calibrated as the target multidimensional feature matrix; Before dynamically allocating weights to the static and dynamic dimensional features, a historical data network is obtained, historical weights corresponding to the static and dynamic dimensional features are retrieved from the historical data network, and dynamic weight allocation is performed based on the historical weights corresponding to the static and dynamic dimensional features. In combination with the target multi-dimensional feature matrix, a dynamic resource profile of different target distributed resources is constructed and generated. The method for generating the dynamic resource profile of the target distributed resource is as follows: in the target multi-dimensional feature matrix, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is calculated, a data volatility scoring mechanism is preset, and according to the data volatility scoring mechanism, the data volatility corresponding to the static dimension features and the dynamic dimension features of the target distributed resource is output; In the target unified resource description model, a profile update time is preset. Within the profile update time interval, the data volatility of the target multi-dimensional feature matrix is calculated in real time to update the dynamic resource profiles of different target distributed resources. Combined with the dynamic resource profiles of different target distributed resources, data integration partitioning and elastic scheduling partitioning are performed on different target distributed resources.

5. A data integration and flexible scheduling method for a fragmented computing power pool according to claim 4, characterized in that: The dynamic resource profiles of different target distributed resources are combined to perform data integration partitioning and elastic scheduling partitioning on different target distributed resources, specifically: In the target unified resource description model, a distributed resource area is constructed and resource partitioning rules are designed, wherein the distributed resource area includes a hot pool area, a warm pool area, and a cold pool area; Based on the distributed resource regions and resource partitioning rules, combined with the dynamic resource profile of the target distributed resource, initial partitioning processing of the target distributed resource is implemented in the target unified resource description model; After the target distributed resources are initially partitioned, a k-value clustering algorithm is introduced to cluster the features of different target distributed resources. During the feature clustering process, the dynamic boundaries of the dynamic resource portraits of the target distributed resources are updated in real time, thus implementing partition update processing of the target distributed resources in the target unified resource description model. After the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled for partitions within the target unified resource description model.

6. A data integration and flexible scheduling method for fragmented computing power pools according to claim 1, characterized in that: After the target distributed resource is partitioned and updated, the target distributed resource is elastically scheduled within the target unified resource description model, specifically: In the distributed resource area of the target unified resource description model, standard node load thresholds of the hot pool area, the warm pool area, and the cold pool area are obtained, and the real-time node load values of the hot pool area, the warm pool area, and the cold pool area are monitored in real time; If the real-time node load values in the hot pool, warm pool, and cold pool areas do not remain within the standard node load threshold, the target distributed resources in the hot pool, warm pool, and cold pool areas are elastically scheduled to ensure that the real-time node load values in the hot pool, warm pool, and cold pool areas remain within the standard node load threshold. If the real-time node load value in the warm pool or cold pool does not remain within the standard node load threshold, the target distributed resource will be elastically scheduled to the hot pool first. If the real-time node load value in the hot pool area does not remain within the standard node load threshold, and after the target distributed resources are partitioned and elastically scheduled, the real-time node load value in both the warm pool area and the cold pool area does not remain within the standard node load threshold, the preset overload standard time is set; When the real-time node load value does not remain within the standard node load threshold for a duration greater than the overload standard time, an expansion warning is triggered in the hot pool area, and the hot pool area is controlled to perform capacity expansion until the real-time node load values in the hot pool area, warm pool area, and cold pool area are all maintained within the standard node load threshold.

7. A data integration and flexible scheduling system suitable for fragmented computing power pools, characterized by: The data integration and flexible scheduling system includes a memory and a processor, and the memory stores a data integration and flexible scheduling method program. When the data integration and flexible scheduling method program is executed by the processor, the data integration and flexible scheduling method steps described in any one of claims 1-6 are implemented.

Images