Data processing method and device, equipment, medium and product

Abstract

The invention discloses a data processing method and device, equipment, a medium and a product. The method comprises the steps that related parameters of resources of a target task are obtained, and the resources comprise at least one of a CPU, a disk and a network bandwidth; the resource related parameters of the target task are input into a target task type prediction model, the task types of the target task are obtained, the task types comprise a computing power type, a storage type, a network type and a balance type, and the target task type prediction model is obtained by iteratively training an initial task prediction model through a target sample set; the target sample set comprises resource related parameters of historical tasks and task types of the historical tasks; according to the task type of the target task, resource scheduling is carried out, through the technical scheme, resource dynamic scheduling can be effectively carried out according to different types of tasks, and the resource utilization rate is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data processing method, apparatus, device, medium and product. Background Technology

[0002] With the advent of the internet age, big data has become one of the core technologies for internet development. Through the collection, storage, mining, and analysis of massive amounts of data, it produces valuable information that can be widely applied in various fields, bringing enormous convenience and benefits to society. However, as the scale of big data continues to expand, the demand for computing power, storage, and network resources is constantly increasing, making resource management increasingly important. Big data platform resource management involves the effective management and allocation of computing, storage, and network resources in a big data computing environment, encompassing resource scheduling, allocation, monitoring, and optimization. Effective big data resource management can ensure the full utilization of resources, reduce unnecessary waste, and improve system performance and reliability.

[0003] Therefore, how to effectively schedule resources has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, medium, and product to achieve effective dynamic resource scheduling based on different types of tasks.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising:

[0006] Obtain resource-related parameters of the target task, wherein the resources include at least one of CPU, disk and network bandwidth;

[0007] The resource-related parameters of the target task are input into the target task type prediction model to obtain the task type of the target task. The task type includes: computing power type, storage type, network type and balanced type. The target task type prediction model is obtained by iteratively training an initial task prediction model through a target sample set. The target sample set includes: resource-related parameters of historical tasks and task types of historical tasks.

[0008] Resource scheduling is performed based on the task type of the target task.

[0009] According to another aspect of the present invention, a data processing apparatus is provided, the data processing apparatus comprising:

[0010] The acquisition module is used to acquire resource-related parameters of the target task, wherein the resources include at least one of CPU, disk and network bandwidth;

[0011] The task type determination module is used to input the resource-related parameters of the target task into the target task type prediction model to obtain the task type of the target task. The task type includes: computing power type, storage type, network type and balanced type. The target task type prediction model is obtained by iteratively training an initial task prediction model through a target sample set. The target sample set includes: resource-related parameters of historical tasks and task types of historical tasks.

[0012] The resource scheduling module is used to schedule resources according to the task type of the target task.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the data processing method as described in any of the embodiments of the present invention.

[0019] This invention provides an embodiment of the invention that obtains resource-related parameters of a target task, wherein the resources include at least one of CPU, disk, and network bandwidth; inputs the resource-related parameters of the target task into a target task type prediction model to obtain the task type of the target task, wherein the task type includes computing power type, storage type, network type, and balanced type; the target task type prediction model is obtained by iteratively training an initial task prediction model using a target sample set, wherein the target sample set includes resource-related parameters of historical tasks and the task types of historical tasks; and performs resource scheduling based on the task type of the target task, thereby achieving effective dynamic resource scheduling based on different task types and improving resource utilization.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] Example 1

[0029] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. This embodiment is applicable to resource scheduling situations. The method can be executed by the data processing device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0030] S110, Obtain resource-related parameters for the target task.

[0031] The resources include at least one of CPU, disk, and network bandwidth.

[0032] In this embodiment, the target task can be a task to be executed. The resource-related parameters include: resource utilization rate and resource usage amount, or resource utilization rate.

[0033] S120, input the resource-related parameters of the target task into the target task type prediction model to obtain the task type of the target task.

[0034] The task types include: computing power type, storage type, network type, and balanced type. The target task type prediction model is obtained by iteratively training the initial task prediction model through a target sample set. The target sample set includes: resource-related parameters of historical tasks and task types of historical tasks.

[0035] In this embodiment, the initial task prediction model is iteratively trained using the target sample set, including:

[0036] Obtain the target sample set, which includes: resource-related parameters of historical tasks, and task types of historical tasks;

[0037] Input the resource-related parameters of the historical tasks into the initial task prediction model to obtain the predicted task type;

[0038] The parameters of the initial task prediction model are trained based on the differences between the predicted task type and the historical task type.

[0039] In this embodiment, the task type of the historical task can be determined based on task evaluation rules.

[0040] In a specific example, to train the initial task prediction model, the collected data samples (resource-related parameters of historical tasks) need to be labeled: computing power tasks are labeled as 1, storage tasks as 2, network tasks as 3, and balanced tasks as 4.

[0041] Considering the mapping relationship between computing power-intensive, storage-intensive, network-intensive, and balanced tasks and CPU, disk, and network bandwidth in real-world application scenarios, the labeling principle is as follows to improve prediction accuracy:

[0042] The initial evaluation principle for computing power type can be as follows: CPU is a rigid resource for computing power, the average CPU utilization rate is more than twice the average of all samples, and the disk utilization rate and bandwidth utilization rate are less than 2 / 3 of the average. The following conditions must be met simultaneously: average CPU utilization rate > twice the average of all samples; average disk utilization rate < 2 / 3 of the average of all samples; average network bandwidth utilization rate < 2 / 3 of the average of all samples.

[0043] The initial evaluation principle for computing power can also be as follows: CPU is a rigid resource for computing power, the average CPU utilization and average memory utilization are more than twice the average of all samples, and the disk utilization and bandwidth utilization are less than 2 / 3 of the average. The following conditions must be met simultaneously: average CPU utilization > twice the average of all samples; average memory utilization > twice the average of all samples; average disk utilization < 2 / 3 of the average of all samples; average network bandwidth utilization < 2 / 3 of the average of all samples.

[0044] The initial evaluation principle for network-type systems can be as follows: network bandwidth is a rigid resource for the network type, the network bandwidth utilization rate is more than twice the average of all samples, and the CPU utilization rate and disk utilization rate are less than 2 / 3 of the average. The following conditions must be met simultaneously: the average network bandwidth utilization rate is greater than twice the average of all samples; the average CPU utilization rate is less than 2 / 3 of the average of all samples; and the average disk utilization rate is less than 2 / 3 of the average of all samples.

[0045] The initial evaluation principle for network-type systems can also be as follows: network bandwidth is a rigid resource for the network type, the network bandwidth utilization rate is more than twice the average of all samples, and the CPU utilization rate, disk utilization rate, and memory utilization rate are less than 2 / 3 of the average. The following conditions must be met simultaneously: the average network bandwidth utilization rate is greater than twice the average of all samples; the average CPU utilization rate is less than 2 / 3 of the average of all samples; the average disk utilization rate is less than 2 / 3 of the average of all samples; and the average memory utilization rate is less than 2 / 3 of the average of all samples.

[0046] The initial evaluation principle for storage-type systems can be as follows: the disk is a rigid storage resource, the disk utilization rate is more than twice the average of all samples, and the CPU utilization rate and network bandwidth utilization rate are less than 2 / 3 of the average. The following conditions must be met simultaneously: the average disk utilization rate is greater than twice the average of all samples; the average CPU utilization rate is less than 2 / 3 of the average of all samples; and the average network bandwidth utilization rate is less than 2 / 3 of the average of all samples.

[0047] The initial assessment principle for storage can also be as follows: the disk is a rigid storage resource, the disk utilization rate is more than twice the average of all samples, and the CPU utilization rate, network bandwidth utilization rate, and memory utilization rate are less than 2 / 3 of the average. The following conditions must be met simultaneously: the average disk utilization rate is greater than twice the average of all samples; the average CPU utilization rate is less than 2 / 3 of the average of all samples; the average network bandwidth utilization rate is less than 2 / 3 of the average of all samples; and the average memory utilization rate is less than 2 / 3 of the average of all samples.

[0048] Balanced type: Apart from computing power type, storage type, and network type, the remaining task types are balanced type.

[0049] In this embodiment, tasks are categorized into computing power, storage, network, and balanced types using a target task type prediction model. Computing power tasks are primarily computational, requiring high CPU resources; CPU is a rigid resource for computing power, such as big data distributed computing tasks. Storage tasks primarily involve writing data and storing it locally, requiring high disk resources; disk is a rigid resource for storage, such as database data writing scenarios. Network tasks primarily require network transmission; network bandwidth is a rigid resource for network, such as the unified data interface service in a middleware platform, responsible for data interface forwarding. Balanced tasks are those outside the computing power, storage, and network types, where the needs of each type are relatively balanced.

[0050] In this embodiment, the initial task prediction model includes a hidden layer comprising six neurons.

[0051] For any hidden layer neuron The parameter can be expressed as:

[0052] ;

[0053] in, , ,…, As input to the model, For the CPU size of the task container, CPU utilization of the task container; This refers to the hard drive size of the task container. For task container disk usage. For the size of the task container memory, For task container memory usage, For the network bandwidth of the task container, For task container network usage. The weights are the weights between the first feature of the input layer and the i-th hidden layer neuron. The weights between the 8th feature of the input layer and the i-th hidden layer neuron are... is the bias of the i-th hidden layer neuron.

[0054] Hidden layer activation function Select The function, the output of the hidden layer is ,in, .

[0055] Construct a loss function, assuming we have For each sample, the loss function is... Represented as:

[0056] ;

[0057] in, For loss function, To predict task type, For historical missions, only when... The model is most accurate when the value is at its minimum.

[0058] S130, perform resource scheduling according to the task type of the target task.

[0059] In this embodiment, the target task can be any one of computing power type, storage type, network type, and balanced type.

[0060] In this embodiment, if the target task is of any one of computing power type, storage type, and network type, resource scheduling is performed according to the target task's task type; if the target task is of balanced type, the target task is run in the container corresponding to the target task.

[0061] Optionally, resource scheduling is performed based on the task type of the target task, including:

[0062] If the target task is of any one of computing power type, storage type, and network type, then resource scheduling is performed according to the target task's task type.

[0063] In this embodiment, if the target task is of any one of computing power type, storage type, or network type, then resource scheduling is required.

[0064] In this embodiment, the resource scheduling method based on the task type of the target task can be as follows: determine the target container and the resource container to be dispatched based on the resource utilization rate of the container; determine the resource scheduling value based on the resource utilization rate of the dispatched resource container, the resource utilization rate decay factor, the average resource utilization rate of the container, and the resource popularity coefficient; dispatch the resources corresponding to the resource scheduling value of the dispatched resource container to the target container; and run the target task in the target container.

[0065] If the target task is of a balanced type, then the target task is run in the container corresponding to the target task.

[0066] In this embodiment, if the target task is of a balanced type, no resource scheduling is required, and the target task can be run directly in the container corresponding to the target task.

[0067] Optionally, the resource-related parameters include resource utilization rate;

[0068] Resource scheduling is performed based on the task type of the target task, including:

[0069] The target container and the container to be allocated resources are determined based on the container's resource utilization rate.

[0070] In this embodiment, the method for determining the target container and the resource container to be called out based on the resource utilization rate of the container can be as follows: obtain the mean and variance of the resource utilization rate of the container; determine the discrete value of the resource utilization rate of the container based on the mean and variance of the resource utilization rate of the container; take the container with the highest resource utilization rate as the target container and the container with the lowest discrete value of resource utilization rate as the resource container to be called out.

[0071] The resource scheduling value is determined based on the resource utilization rate of the resource container being dispatched, the resource utilization rate decay factor, the average resource utilization rate of the container, and the resource heat coefficient.

[0072] In this embodiment, the resource heat coefficient is used to reflect the degree of CPU resource stress. The higher the heat, the larger the coefficient may be, thus enhancing the sensitivity of resource scheduling.

[0073] In this embodiment, the resource scheduling value is determined based on the resource utilization rate of the resource container being called up, the resource utilization rate decay factor, the average resource utilization rate of the container, and the heat coefficient of the resource. The method is as follows: the difference between the resource utilization rate of the resource container being called up and the average resource utilization rate of the container is taken as the first difference; the product of the first difference, the resource utilization rate decay factor, and the heat coefficient of the resource is taken as the resource scheduling value.

[0074] The resources corresponding to the resource scheduling value of the resource container to be called out are scheduled to the target container.

[0075] In this embodiment, the target container receives the resources dispatched by the dispatched resource container based on the resource scheduling value.

[0076] The target task is run within the target container.

[0077] Optionally, the target container and the container to be allocated resources can be determined based on the container's resource utilization, including:

[0078] Obtain the mean and variance of the container's resource utilization.

[0079] In this embodiment, the average resource utilization rate of the container is determined based on the following formula:

[0080] ;

[0081] Where m is the total number of containers, This represents the CPU utilization of the k-th container. This represents the average CPU utilization across all containers.

[0082] In this embodiment, the variance of the container's resource utilization rate is determined based on the following formula:

[0083] ;

[0084] in, This represents the variance of the container's resource utilization.

[0085] The discrete values ​​of container resource utilization are determined based on the mean and variance of container resource utilization.

[0086] In this embodiment, the discrete value of the container's resource utilization rate is determined based on the following formula:

[0087] ;

[0088] in, This represents the discrete value of the resource utilization rate of the k-th container.

[0089] The container with the highest resource utilization rate is selected as the target container, and the container with the lowest resource utilization rate dispersion value is selected as the resource to be swapped out container.

[0090] In this embodiment, if the target task is a computing power type, the container with the highest CPU utilization is selected as the target container, and the container with the lowest CPU utilization dispersion value is selected as the resource container to be called up; if the target task is a storage type, the container with the highest disk utilization is selected as the target container, and the container with the lowest disk utilization dispersion value is selected as the resource container to be called up; if the target task is a network type, the container with the highest network bandwidth utilization is selected as the target container, and the container with the lowest network bandwidth utilization dispersion value is selected as the resource container to be called up.

[0091] Optionally, the container with the highest resource utilization rate can be used as the target container, and the container with the lowest resource utilization rate dispersion value can be used as the resource outgoing container, including:

[0092] If the target task is a computing power type, then the container with the highest CPU utilization will be used as the target container, and the container with the lowest CPU utilization dispersion value will be used as the resource container to be dispatched.

[0093] In this embodiment, the target container is the container used to execute the target task. The resource container is the container that provides resources.

[0094] If the target task is of storage type, then the container with the highest disk usage will be the target container, and the container with the lowest disk usage dispersion value will be the resource container to be swapped out.

[0095] If the target task is of the network type, then the container with the highest network bandwidth utilization will be used as the target container, and the container with the lowest network bandwidth utilization dispersion value will be used as the resource to be dispatched container.

[0096] Optionally, the resource scheduling value is determined based on the resource utilization rate of the resource container being called out, the resource utilization rate decay factor, the average resource utilization rate of the container, and the resource heat coefficient, including:

[0097] The difference between the resource utilization rate of the resource container being retrieved and the average resource utilization rate of the container is taken as the first difference.

[0098] In this embodiment, the first difference is determined based on the following formula:

[0099] First difference = ;

[0100] in, This represents the CPU utilization of the k-th container. This represents the average CPU utilization across all containers.

[0101] The product of the first difference, the resource utilization decay factor, and the resource heat coefficient is used as the resource scheduling value.

[0102] In this embodiment, the resource scheduling value is determined based on the following formula:

[0103] ;

[0104] in, This is the CPU scheduling value. This is the decay factor of the CPU in the k-th container. The initial value is 1. After a resource is swapped out once, the decay factor becomes 80% of the original value. The more times the resource is swapped out, the less resources are swapped out each time. This is the CPU heat index (used to reflect the degree of CPU resource strain; the higher the heat, the larger the index may be, enhancing the sensitivity of resource scheduling).

[0105] For different task types, resource scheduling is performed accordingly. The specific resource scheduling strategies are as follows:

[0106] (2-1) Adaptive scheduling of computing power-intensive tasks:

[0107] (1) Computational tasks schedule CPU and memory from storage and network tasks. The container with the highest CPU utilization among all computational tasks is selected as the receiving container. The container with the lowest CPU utilization dispersion value among the predicted storage and network tasks is selected as the releasing container. Assume there are m containers executing tasks in parallel in the system. This represents the CPU utilization of the k-th container.

[0108] (2) Calculate the average CPU utilization of m containers, as shown in the following formula:

[0109] ;

[0110] (3) Calculate the variance of CPU utilization for m containers, as shown in the following formula:

[0111] ;

[0112] (4) Select the container with the lowest discrete value as the resource outflow container, and calculate the discrete value of CPU utilization of the k-th container. The calculation formula is as follows:

[0113] ;

[0114] (5) Resource scheduling value The calculation method is as follows:

[0115] ;

[0116] in, This is the decay factor of the CPU in the k-th container. The initial value is 1. After a resource is swapped out once, the decay factor becomes 80% of the original value. The more times the resource is swapped out, the less resources are swapped out each time.

[0117] (6) When the current CPU utilization of the container corresponding to the storage or network task is higher than the average CPU utilization of all tasks, the computing power resource scheduling is paused. When the CPU utilization of the storage or network task is lower than the average, dynamic adjustment is reactivated.

[0118] (7) Similarly, according to the above steps (1)-(6), if the CPU resources are the same, the memory resources are mobilized.

[0119] Considering the characteristics of computing power tasks—high CPU and memory usage, low disk and network requirements—the system designates the container with the highest rigid resource utilization rate among computing power tasks as the resource receiver, and the container with the lowest discrete value of utilization rate among other types as the resource provider. By calculating the mean and variance of core resource utilization rates and utilizing attenuation factors and resource scheduling values, dynamic resource scheduling can be achieved. This approach can maximize the execution efficiency of computing power tasks, optimize resource allocation, and improve the resource utilization efficiency of big data platforms.

[0120] (2-2) Adaptive Scheduling of Storage-Type Tasks: Similar to the scheduling method for computing power-type tasks described in (2-1), storage-type tasks adaptively schedule disk size from computing power-type and network-type tasks. First, disks are scheduled from computing power-type and network-type tasks, and the container with the highest disk utilization rate among all storage-type tasks is selected as the container to receive resources. The container with the lowest disk utilization dispersion value among the predicted task types (computing power-type and network-type) is selected as the container to allocate resources. Second, the mean and variance of disk utilization are calculated, and the resource scheduling value is calculated. Finally, the disk decay factor is dynamically adjusted; when the disk utilization rate of computing power-type and network-type tasks exceeds the mean disk utilization rate of all tasks, resource scheduling is paused.

[0121] (2-3) Adaptive Scheduling of Network-Based Tasks: Similar to the scheduling method for computing power-based tasks described in (2-1), network-based tasks adaptively schedule network resources from computing power-based and storage-based tasks. First, network resources are scheduled from computing power-based and storage-based tasks. The container with the highest network bandwidth utilization among all storage-based tasks is selected as the container receiving resources. Then, the container with the lowest network utilization dispersion among the predicted task types (computing power-based and storage-based) is selected as the container releasing resources. Second, the mean and variance of network utilization are calculated, and the resource scheduling value is calculated. Finally, the network attenuation factor is dynamically adjusted. When the network bandwidth utilization of computing power-based and storage-based tasks exceeds the mean network bandwidth utilization of all tasks, resource scheduling is paused.

[0122] Repeat the above steps until the conditions for pausing resource scheduling are met.

[0123] In this embodiment, the big data platform tests task types based on a target task type prediction model. Next, adaptive resource scheduling is performed for computing power-intensive, network-intensive, and storage-intensive tasks respectively. Finally, based on continuous iteration according to task types, the system achieves the highest task execution efficiency and optimal resource allocation within the big data platform, thereby improving resource utilization efficiency.

[0124] The technical solution of this embodiment obtains the resource utilization rate of a target task, wherein the resources include at least one of CPU, disk, and network bandwidth; inputs the resource utilization rate of the target task into a target task type prediction model to obtain the task type of the target task, wherein the task type includes computing power type, storage type, network type, and balanced type, and the target task type prediction model is obtained by iteratively training an initial task prediction model through a target sample set, wherein the target sample set includes the resource utilization rate of historical tasks and the task type of historical tasks; resource scheduling is performed according to the task type of the target task, which can dynamically adjust the resource scheduling strategy for the resource requirements of different types of tasks and improve resource utilization.

[0125] Example 2

[0126] Figure 2 This is a schematic diagram of a data processing device provided in an embodiment of the present invention. This embodiment is applicable to resource scheduling situations. The device can be implemented using software and / or hardware methods and can be integrated into any device that provides data processing functions, such as… Figure 2 As shown, the data processing device specifically includes: an acquisition module 210, a task type determination module 220, and a resource scheduling module 230.

[0127] The acquisition module is used to acquire resource-related parameters of the target task, wherein the resources include at least one of CPU, disk and network bandwidth.

[0128] The task type determination module is used to input the resource-related parameters of the target task into the target task type prediction model to obtain the task type of the target task. The task type includes: computing power type, storage type, network type and balanced type. The target task type prediction model is obtained by iteratively training an initial task prediction model through a target sample set. The target sample set includes: resource-related parameters of historical tasks and task types of historical tasks.

[0129] The resource scheduling module is used to schedule resources according to the task type of the target task.

[0130] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0131] Example 3

[0132] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0133] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0136] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0143] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0144] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method according to any embodiment of the invention.

[0145] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized by, The method comprises: obtaining resource-related parameters of a target task, wherein the resources comprise at least one of CPU, disk and network bandwidth; inputting the resource-related parameters of the target task into a target task type prediction model to obtain a task type of the target task, wherein the task type comprises computing power type, storage type, network type and balanced type, the target task type prediction model is obtained by iteratively training an initial task prediction model based on a target sample set, and the target sample set comprises resource-related parameters of historical tasks and task types of the historical tasks; performing resource scheduling according to the task type of the target task.

2. The method of claim 1, wherein, The resource scheduling according to the task type of the target task comprises: if the task type of the target task is any one of computing power type, storage type and network type, performing resource scheduling according to the task type of the target task; if the task type of the target task is balanced type, running the target task in a container corresponding to the target task.

3. The method of claim 2, wherein, The resource-related parameters comprise resource usage rate. The resource scheduling according to the task type of the target task comprises: determining a target container and a resource container to be removed according to resource usage rates of the containers; determining a resource scheduling value according to a resource usage rate of the resource container to be removed, a resource usage rate decay factor, a mean value of resource usage rates of the containers and a heat coefficient of the resources; scheduling resources corresponding to the resource scheduling value of the resource container to be removed to the target container; and running the target task in the target container.

4. The method of claim 3, wherein, The determination of the target container and the resource container to be removed according to the resource usage rates of the containers comprises: obtaining a mean value and a variance of the resource usage rates of the containers; determining a resource usage rate dispersion value of the containers according to the mean value and the variance of the resource usage rates of the containers; determining a container with the highest resource usage rate as the target container and a container with the lowest resource usage rate dispersion value as the resource container to be removed.

5. The method of claim 4, wherein, The determination of the target container and the resource container to be removed according to the resource usage rates of the containers comprises: if the task type of the target task is computing power type, determining a container with the highest CPU usage rate as the target container and a container with the lowest CPU usage rate dispersion value as the resource container to be removed; if the task type of the target task is storage type, determining a container with the highest disk usage rate as the target container and a container with the lowest disk usage rate dispersion value as the resource container to be removed; if the task type of the target task is network type, determining a container with the highest network bandwidth usage rate as the target container and a container with the lowest network bandwidth usage rate dispersion value as the resource container to be removed.

6. The method of claim 3, wherein, The determination of the resource scheduling value according to the resource usage rate of the resource container to be removed, the resource usage rate decay factor, the mean value of the resource usage rates of the containers and the heat coefficient of the resources comprises: determining a difference between the resource usage rate of the resource container to be removed and the mean value of the resource usage rates of the containers as a first difference value; determining a product of the first difference value, the resource usage rate decay factor and the heat coefficient of the resources as the resource scheduling value.

7. A data processing apparatus, characterized by, The method comprises: An acquisition module is configured to acquire resource-related parameters of a target task, wherein the resources include at least one of a CPU, a disk, and network bandwidth. A task type determination module is configured to input the resource-related parameters of the target task into a target task type prediction model to obtain a task type of the target task, wherein the task type includes a computing power type, a storage type, a network type, and a balanced type, the target task type prediction model is obtained by iteratively training an initial task prediction model based on a target sample set, and the target sample set includes resource-related parameters of historical tasks and task types of the historical tasks. A resource scheduling module is configured to perform resource scheduling according to the task type of the target task.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute when the computer instructions are executed.

10. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-6. The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-6.

Images