Data processing method, device, equipment, medium and program product

Abstract

This application discloses a data processing method, apparatus, device, medium, and program product in the field of computer technology. This application focuses on resource modules required for target task execution, first determining an initial core allocation plan for these resource modules based on historical core allocation information, then optimizing the initial core allocation plan based on simulated operation data of the initial core allocation plan to obtain an available core allocation plan, and finally executing the target task in the target system according to the available core allocation plan. This allows the optimal core allocation plan to be determined by combining task-related information and the actual system operation status, thereby maximizing processor utilization and task processing efficiency during operation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, medium, and program product. Background Art

[0002] Currently, each core of a multi-core processor can be bound to a fixed task. This binding method easily causes: unbalanced loads on different processor cores, with some processor cores having high loads while others are relatively idle, resulting in relatively low processor utilization.

[0003] Therefore, how to improve the utilization rate of the processor during operation is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data processing method, apparatus, device, medium and program product to improve the utilization rate of the processor during operation.

[0005] In a first aspect, the present application provides a data processing method, including: selecting resource modules required for running a target task in a target system; determining an initial core allocation plan for the resource modules based on historical core allocation information; obtaining simulated operation data of the initial core allocation plan, optimizing the initial core allocation plan according to the simulated operation data, and obtaining an available core allocation plan; and executing the target task in the target system according to the available core allocation plan.

[0006] In the second aspect, the present application provides a data processing device, including: a selection module, used to select the resource modules required for the operation of the target task in the target system; a determination module, used to determine the initial core allocation plan of the resource module based on historical core allocation information; an optimization module, used to obtain simulation operation data of the initial core allocation plan, optimize the initial core allocation plan according to the simulation operation data, and obtain an available core allocation plan; an execution module, used to execute the target task in the target system according to the available core allocation plan.

[0007] In a third aspect, the present application provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the aforementioned disclosed data processing method.

[0008] In a fourth aspect, the present application provides a non-volatile storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed data processing method when executed by a processor.

[0009] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0010] From the above scheme, it can be seen that the present application provides a data processing method, including: selecting the resource modules required for the operation of the target task in the target system; determining the initial core allocation scheme of the resource module based on historical core allocation information; obtaining the simulation operation data of the initial core allocation scheme, optimizing the initial core allocation scheme according to the simulation operation data, and obtaining an available core allocation scheme; executing the target task in the target system according to the available core allocation scheme.

[0011] It can be seen that the beneficial effects of the present application are: for the resource modules required for the operation of the target task, the initial core allocation plan of these resource modules is first basically determined based on the historical core allocation information, and then the initial core allocation plan is optimized according to the simulated operation data of the initial core allocation plan to obtain the available core allocation plan, and finally the target task is executed in the target system according to the available core allocation plan. In this way, the optimal core allocation plan can be determined by combining the task-related information and the actual operation status of the system, thereby maximizing the utilization rate and task processing efficiency of the processor during operation.

[0012] Correspondingly, the data processing device, equipment, medium and program product provided by this application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0014] Figure 1 A flow chart of a data processing method disclosed in this application;

[0015] Figure 2 This is a flow chart of a resource module selection method disclosed in this application;

[0016] Figure 3 This is a flow chart of an initial core separation method disclosed in this application;

[0017] Figure 4 This is a flow chart of a core-dividing optimization method disclosed in this application;

[0018] Figure 5 A flowchart of a task execution method disclosed in this application;

[0019] Figure 6 A schematic diagram of a data processing device disclosed in this application;

[0020] Figure 7A server structure diagram provided for this application;

[0021] Figure 8 This is a terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] Currently, each core in a multi-core processor can be bound to a fixed task. This binding method can easily lead to uneven loads on different processor cores, with some processor cores being highly loaded while others are relatively idle, resulting in relatively low processor utilization. To this end, this application provides a data processing solution that can combine task-related information and the actual system operation status to determine the optimal core allocation solution, thereby maximizing processor utilization and task processing efficiency during operation.

[0026] See also Figure 1 As shown, the embodiment of the present application discloses a data processing method, including:

[0027] S101. Select resource modules required for running a target task in a target system.

[0028] In this embodiment, the target system includes at least one server that can monitor in real time the operating status of each network card, storage device, processor, and other devices in the target system. For example, it can monitor in real time the IOPS of the network card, the IOPS (Input / Output Operations Per Second) of the storage disk, the operating frequency, power consumption, and energy consumption of each core of the processor. In one example, this information is monitored and summarized by a BMC (Baseboard Management Controller). The storage device can specifically include: a storage disk, a CPLD (Complex Programmable Logic Device) logic control unit, etc.; in addition, a non-transparent bridge is provided to connect different processors in the system.

[0029] In one embodiment, selecting resource modules required for running a target task in a target system includes: determining network card status information (such as network card IOPS), storage end status information (such as storage disk IOPS), and processor status information (such as operating frequency, power consumption, and energy consumption parameters of each processor core) in the target system; selecting network card resources, storage end resources, and processor resources required for running the target task in the target system based on the network card status information, storage end status information, and processor status information, as well as the task type (such as storage-type tasks, query-type tasks, computing-type tasks, etc.), task data volume, and processing time of the target task; and using the network card resources, storage end resources, and processor resources as resource modules. Among them, according to the network card status information, storage end status information and processor status information, as well as the task type, task data volume and processing time of the target task, the network card resources, storage end resources and processor resources required for the operation of the target task are selected in the target system, including: sorting the network cards in the target system according to the network card status information to obtain a network card sequence; sorting the storage ends in the target system according to the storage end status information to obtain a storage end sequence; sorting the processors in the target system according to the processor status information to obtain a processor sequence; and selecting the network card resources, storage end resources and processor resources from the network card sequence, storage end sequence and processor sequence respectively according to the task type, task data volume and processing time of the target task.

[0030] S102: Determine an initial core allocation plan for resource modules based on historical core allocation information.

[0031] It should be noted that historical core allocation information refers to the core allocation plans for other tasks previously processed by the target system. In one embodiment, determining an initial core allocation plan for a resource module based on the historical core allocation information includes: allocating processor cores to the resource module based on multiple dimensions based on the historical core allocation information to obtain multiple optional allocation plans; the multiple dimensions include at least two of instruction set streamlining, queue optimization, interrupt balancing, and kernel parameter tuning; and selecting the most efficient of the multiple optional allocation plans as the initial core allocation plan.

[0032] In one example, based on historical core allocation information, the number of processor cores is allocated to the resource module from the instruction set simplification dimension, including: based on historical core allocation information, optimizing the number of instructions and instruction content executed by a single core from the instruction set simplification dimension, and determining the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0033] In one example, based on historical core allocation information, the number of processor cores is allocated to the resource module from the queue optimization dimension, including: based on the historical core allocation information, optimizing the queue depth of single core execution from the queue optimization dimension, and determining the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0034] In one example, based on historical core allocation information, the number of processor cores is allocated to the resource module from the interrupt balance dimension, including: based on historical core allocation information, optimizing the interrupt time of a single core from the interrupt balance dimension, and determining the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0035] In one example, based on historical core allocation information, the number of processor cores is allocated to the resource module from the kernel parameter tuning dimension, including: based on historical core allocation information, optimizing the frequency and / or energy consumption of a single core from the kernel parameter tuning dimension, and determining the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0036] S103 : Acquire simulation operation data of the initial core allocation plan, optimize the initial core allocation plan according to the simulation operation data, and obtain an available core allocation plan.

[0037] In one embodiment, optimizing an initial core allocation plan based on simulated operation data includes: simulating the initial core allocation plan in a target system to obtain simulated operation data; and adjusting the number of processor cores corresponding to network card resources, storage resources, and processor resources in the initial core allocation plan based on the simulated operation data to optimize the initial core allocation plan. The simulated operation data includes at least a first amount of data transmitted by the network card resources and a second amount of data processed by the storage resources. The second amount of data processed by the storage resources specifically includes the sum of data flushed to the storage disk and data transmitted by a non-transparent bridge.

[0038] Among them, according to the simulation operation data, the number of processor cores corresponding to the network card resources, storage resources and processor resources in the initial core allocation plan are adjusted, including: according to the simulation operation data, determining the first data volume processed by the network card resources and the second data volume processed by the storage resources in the initial core allocation plan; if the first data volume is greater than the second data volume, and the difference between the first data volume and the second data volume is greater than the preset difference threshold, it is considered that the first data volume is much greater than the second data volume, and then the number of processor cores corresponding to the network card resources in the initial core allocation plan is made less than the number of processor cores corresponding to the storage resources, for example: the number of processor cores corresponding to the network card resources is reduced. Furthermore, after making the number of processor cores corresponding to the network card resources in the initial core allocation plan less than the number of processor cores corresponding to the storage resources, it also includes: replacing the initial core allocation plan with the adjusted initial core allocation plan, and executing the step of simulating the initial core allocation plan in the target system to obtain simulation operation data, so as to optimize the core allocation plan again based on the new simulation operation data until the first data volume and the second data volume are relatively balanced.

[0039] When the difference between the first data volume and the second data volume is not greater than a preset difference threshold, it is considered that the first data volume and the second data volume are relatively balanced, and then the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the current core allocation scheme is increased respectively, and the task processing time of the target task after each increase is calculated; the core allocation scheme corresponding to the shortest task processing time is used as the available core allocation scheme. Wherein, the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the current core allocation scheme is increased respectively, including: setting the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the current core allocation scheme to: A, B, C, and the increment value is set to X. Then, after the first increment, the number of processor cores corresponding to the network card resources, storage resources, and processor resources are: A+X, B+X, C+X; after the next increment, the number of processor cores corresponding to the network card resources, storage resources, and processor resources are: A+2X, B+2X, C+2X; X can be flexibly determined according to actual conditions.

[0040] S104: Execute the target task in the target system according to the available core allocation plan.

[0041] Taking into account the limited number of available processor cores in the target system, when executing S104, it is possible to first determine whether the number of available processor cores in the target system satisfies the available core allocation plan, that is, determine whether the number of available processor cores in the target system is not less than the total number of cores required by the available core allocation plan. If so, it is considered that the number of available processor cores satisfies the available core allocation plan, so the current available core allocation plan can be directly applied, and then the target task is executed in the target system according to the available core allocation plan until the target task is completed; if not, it is considered that the number of available processor cores does not satisfy the available core allocation plan, so the target task is first executed in the target system (that is, the target task is executed, but the current available core allocation plan is not applied), and after dynamically adjusting the available core allocation plan according to the execution situation, the adjusted core allocation plan is applied (that is, the target task continues to be executed in the target system according to the adjusted available core allocation plan).

[0042] In one embodiment, a determination is made as to whether the number of available processor cores in the target system satisfies an available core allocation plan. If so, the step of executing the target task in the target system according to the available core allocation plan is performed until the target task is completed. If the number of available processor cores does not meet the available core allocation plan, the target task is executed in the target system, the available core allocation plan is dynamically adjusted based on the execution status, and the adjusted core allocation plan is applied.

[0043] In one embodiment, dynamically adjusting the available core allocation plan based on execution status includes: determining the utilization of the processor cores occupied by the target task's IO stack; if the utilization is above a preset upper limit, adjusting some of the processor cores corresponding to the storage resources in the current core allocation plan to the current IO stack. If the utilization is below a preset lower limit, adjusting the processor cores occupied by the current IO stack to the storage resources in the current core allocation plan. The processor cores occupied by the target task's IO stack are the processor cores corresponding to the processor resources in the available core allocation plan.

[0044] It can be seen that in this embodiment, for the resource modules required for the operation of the target task, the initial core allocation plan of these resource modules is first basically determined based on the historical core allocation information, and then the initial core allocation plan is optimized according to the simulated operation data of the initial core allocation plan to obtain the available core allocation plan. Finally, the target task is executed in the target system according to the available core allocation plan. In this way, the optimal core allocation plan can be determined by combining the task-related information and the actual operation status of the system, thereby maximizing the utilization rate and task processing efficiency of the processor during operation.

[0045] In one example, the target system may include a front-end (including network cards, etc.), a back-end (including storage disks, CPLDs, and BMCs), and a CPU (including non-transparent bridges and processors). The front-end devices primarily include various plug-in cards for data reading and transmission, while the back-end primarily includes storage devices such as hard drives. The dual processors are interconnected using a non-transparent PCI bridge (NTB), enabling data backup and storage using different processors.

[0046] See Figure 2 After the CPU issues a command to read service data, the BMC reads the health status of each module on the front-end, back-end, and CPU, including fault information and operating status information (such as voltage and current). The CPLD analyzes this information and performs corresponding logical processing to select available resource modules for the currently read service data. Specifically, the BMC reads the voltage and current information of each module and confirms whether the status of each module is correct. For normal modules, the information is passed to the CPLD for logical selection. Based on parameters such as service type, total amount of service data, continuous execution time of service processing, and required storage device processing method, the CPLD selects available network cards, hard drives, processors, and other related resource modules from a large number of normal modules and sends them enable signals.

[0047] See Figure 3 Based on the previously acquired business data (such as business type, total business data volume, continuous execution time required for business processing, and required storage device processing methods), simulations are performed along various optimization dimensions to optimize execution. Optimization methods can include instruction set streamlining, multi-queue optimization, interrupt balancing, and kernel parameter tuning. After simulating corresponding core allocation solutions for each optimization dimension, a final core allocation solution is output to improve performance.

[0048] Specifically, the optimization methods are divided into two categories: instruction simplification and performance improvement. The different optimization methods included in the instruction simplification category are coupled, specifically including: based on the historical core allocation plan, calculating the percentage of the different methods included in the instruction simplification category in the total instruction set and the number of cores required, removing the methods that occupy a large number of cores and are repeated, and performing XOR processing on the retained methods, and finally obtaining the core allocation plan corresponding to the instruction simplification class. Multi-queue optimization, interrupt balancing, kernel parameter tuning and other methods are also coupled. For example: while optimizing the queue depth, optimize the interrupt time consumption, and tune the kernel parameters, so as to obtain a core allocation plan. For example, from the three aspects of queue depth, total processing time and number of occupied cores, select the one with the shortest processing time and the largest queue depth. Then, choose the most efficient one among the two core allocation plans. This can ensure the number of instructions executed by the CPU and maximize the data processing volume.

[0049] See Figure 4 Based on the business data information obtained above and the preliminarily determined core allocation plan, a plan simulation is performed, and the core allocation is adjusted according to the module health status until the optimal core allocation plan is obtained, so that the amount of data stored on the back-end hard disk of the incoming data core reaches the maximum.

[0050] Specifically, adjusting the core distribution with reference to the module health status includes: first balancing the data processing volume of the selected front-end and back-end modules, and then optimizing with the shortest total processing time. Balancing the data processing volume of the selected front-end and back-end modules includes: simulating the current core distribution plan, and calculating whether the data transmitted by the current front-end module is higher than the total amount of the current back-end disk data and NTB transmission data. If it is higher, the number of CPU cores allocated to the current front-end module is reduced, and then a second simulation is performed until the data processing volume of the front-end and back-end modules is balanced. Optimizing with the shortest total processing time includes: taking the processing time as the optimization point, referring to the current number of available cores, and allocating one more core to each part of the current allocation plan in turn. After the core distribution is completed, a simulation is performed to calculate and record the total processing time, and the time is repeatedly shortened accordingly. After the shortest time is reached, the current core distribution plan is output as the top-level available core allocation plan.

[0051] See Figure 5 , providing the core allocation plan (i.e., the available core allocation plan) to the CPU side so that the CPU side can determine whether the current number of CPU available cores can meet its execution. If it does, the corresponding task will be executed according to the current core allocation plan, and the core allocation plan will not be adjusted during the execution process. If it does not meet the requirements, the current core allocation plan will be executed, and the core allocation plan will be adjusted during the execution process.

[0052] The core allocation scheme can be adjusted by referring to the following process: determining the utilization of the processor cores occupied by each underlying IO stack during task execution, and based on this utilization, transferring the processor cores occupied by the IO stack to the backend storage, or transferring the processor cores occupied by the backend storage to the IO stack. The optimization of the IO stack is based on the available space of the CPU memory unit. The processor divides the incoming data into data segments 1, 2…n. Each data segment corresponds to an IO stack. The IO stack stores the corresponding data segments in the corresponding memory space, performs the corresponding data processing, and then completes the disk storage. The core allocation strategy corresponding to the IO stack can be called the underlying core allocation strategy.

[0053] The utilization-based core adjustment process involves: When the I / O stack's core utilization exceeds 90%, the CPU allocates backend cores to the I / O stack. When a core's utilization drops below 40%, the CPU evenly distributes the memory space corresponding to that core to the remaining memory space and allocates that core to the backend. This real-time adjustment ensures optimal utilization of each CPU core.

[0054] This embodiment takes into account the health status and data processing tasks of each unit module in the system, dynamically adjusts the number of processor cores available to each module, and can also adjust the number of cores required for the IO stack in the CPU according to the actual processing situation, so that data transmission, data storage and data processing are coordinated with each other, the CPU core can be more efficiently utilized in the storage device, and processing efficiency and performance can be maximized.

[0055] A data processing device provided in an embodiment of the present application is introduced below. The data processing device described below can be referenced with other embodiments described herein.

[0056] See also Figure 6 As shown, the embodiment of the present application discloses that the present application provides a data processing device, including: a selection module, used to select the resource module required for running a target task in a target system; a determination module, used to determine the initial core allocation plan of the resource module based on historical core allocation information; an optimization module, used to obtain simulation operation data of the initial core allocation plan, optimize the initial core allocation plan according to the simulation operation data, and obtain an available core allocation plan; an execution module, used to execute the target task in the target system according to the available core allocation plan.

[0057] In one embodiment, the selection module is used to: determine the network card status information, storage end status information and processor status information in the target system; select the network card resources, storage end resources and processor resources required for the target task to run in the target system based on the network card status information, storage end status information and processor status information, as well as the task type, task data volume and processing time of the target task; and use the network card resources, storage end resources and processor resources as resource modules.

[0058] In one embodiment, the selection module is used to: sort the network cards in the target system according to the network card status information to obtain a network card sequence; sort the storage terminals in the target system according to the storage terminal status information to obtain a storage terminal sequence; sort the processors in the target system according to the processor status information to obtain a processor sequence; and select network card resources, storage terminal resources, and processor resources from the network card sequence, storage terminal sequence, and processor sequence, respectively, based on the task type, task data volume, and processing time of the target task.

[0059] In one embodiment, the determination module is used to: allocate the number of processor cores to the resource module from multiple dimensions based on historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions include: at least two of: instruction set simplification, queue optimization, interrupt balancing and kernel parameter tuning; among the multiple optional allocation schemes, select the one with the highest processing efficiency as the initial core allocation scheme.

[0060] In one embodiment, the determination module is used to optimize the number and content of instructions executed by a single core from the instruction set simplification dimension based on historical core allocation information, and determine the number of processor cores that can be allocated by the resource module based on the corresponding optimization results.

[0061] In one embodiment, the determination module is configured to optimize the queue depth of a single core execution from a queue optimization dimension based on historical core allocation information, and determine the number of processor cores that can be allocated by the resource module based on the corresponding optimization results.

[0062] In one embodiment, the determination module is configured to optimize the interrupt time consumption of a single core from the perspective of interrupt balance according to historical core allocation information, and determine the number of processor cores that can be allocated by the resource module according to the corresponding optimization results.

[0063] In one embodiment, the determination module is used to optimize the frequency and / or energy consumption of a single core from the kernel parameter tuning dimension based on historical core allocation information, and determine the number of processor cores that can be allocated by the resource module based on the corresponding optimization results.

[0064] In one embodiment, the optimization module is used to: simulate and run the initial core allocation plan in the target system to obtain simulated operation data; and adjust the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the initial core allocation plan based on the simulated operation data to optimize the initial core allocation plan.

[0065] In one embodiment, the optimization module is used to: determine the first data volume processed by the network card resources and the second data volume processed by the storage-end resources in the initial core allocation scheme based on the simulation operation data; if the first data volume is greater than the second data volume, and the difference between the first data volume and the second data volume is greater than a preset difference threshold, then the number of processor cores corresponding to the network card resources in the initial core allocation scheme is made less than the number of processor cores corresponding to the storage-end resources.

[0066] In one embodiment, the optimization module is used to replace the initial core allocation plan with the adjusted initial core allocation plan, and perform a step of simulating the initial core allocation plan in the target system to obtain simulation operation data, so as to optimize the core allocation plan again based on the new simulation operation data.

[0067] In one embodiment, the optimization module is used to: when the difference between the first data volume and the second data volume is not greater than a preset difference threshold, increase the number of processor cores corresponding to the network card resources, storage resources and processor resources in the current core allocation scheme respectively, and calculate the task processing time of the target task after each increase; and use the core allocation scheme corresponding to the shortest task processing time as the available core allocation scheme.

[0068] In one embodiment, the execution module is used to: determine whether the number of available processor cores in the target system meets the available core allocation plan; if the number of available processor cores meets the available core allocation plan, execute the steps of executing the target task in the target system according to the available core allocation plan until the target task is completed.

[0069] In one embodiment, the execution module is configured to: if the number of available processor cores does not satisfy the available core allocation plan, execute the target task in the target system, dynamically adjust the available core allocation plan according to the execution status, and apply the adjusted core allocation plan.

[0070] In one embodiment, the execution module is used to: determine the utilization rate of the processor cores occupied by the IO stack of the target task; if the utilization rate is higher than a preset upper limit, adjust some processor cores corresponding to the storage resources in the current core allocation plan to the IO stack.

[0071] In one embodiment, the execution module is configured to adjust the processor cores occupied by the IO stack to storage resources in the current core allocation solution if the utilization rate is lower than a preset lower limit.

[0072] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0073] It can be seen that this embodiment provides a data processing device that can determine the optimal core allocation solution based on task-related information and the actual operation status of the system, thereby maximizing the utilization rate and task processing efficiency of the processor during operation.

[0074] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below can be referenced with other embodiments described herein.

[0075] An embodiment of the present application discloses an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method disclosed in any of the above embodiments.

[0076] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: select the resource modules required for running the target task in the target system; determine the initial core allocation plan of the resource module based on historical core allocation information; optimize the initial core allocation plan according to the simulated operation data of the initial core allocation plan to obtain an available core allocation plan; and execute the target task in the target system according to the available core allocation plan.

[0077] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: determine the network card status information, storage end status information and processor status information in the target system; select the network card resources, storage end resources and processor resources required for the target task to run in the target system based on the network card status information, storage end status information and processor status information, as well as the task type, task data volume and processing time of the target task; and use the network card resources, storage end resources and processor resources as resource modules.

[0078] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: sorting the network cards in the target system according to the network card status information to obtain a network card sequence; sorting the storage terminals in the target system according to the storage terminal status information to obtain a storage terminal sequence; sorting the processors in the target system according to the processor status information to obtain a processor sequence; and selecting network card resources, storage terminal resources, and processor resources from the network card sequence, storage terminal sequence, and processor sequence, respectively, based on the task type, task data volume, and processing time of the target task.

[0079] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: according to historical core allocation information, allocate the number of processor cores to the resource module from multiple dimensions to obtain multiple optional allocation schemes; the multiple dimensions include: instruction set simplification, queue optimization, interrupt balancing and kernel parameter tuning at least two; among the multiple optional allocation schemes, select the one with the highest processing efficiency as the initial core allocation scheme.

[0080] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: based on historical core allocation information, optimize the number of instructions and instruction content executed by a single core from the instruction set simplification dimension, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0081] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: optimize the queue depth of single core execution from the queue optimization dimension based on historical core allocation information, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0082] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: optimize the interrupt time of a single core from the interrupt balance dimension based on historical core allocation information, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0083] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: optimize the frequency and / or energy consumption of a single core from the kernel parameter tuning dimension based on historical core allocation information, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0084] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: simulate and run the initial core allocation plan in the target system to obtain simulated operation data; according to the simulated operation data, adjust the number of processor cores corresponding to the network card resources, storage resources and processor resources in the initial core allocation plan to optimize the initial core allocation plan.

[0085] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: determine the first data volume processed by the network card resources and the second data volume processed by the storage-end resources in the initial core allocation plan based on the simulation operation data; if the first data volume is greater than the second data volume, and the difference between the first data volume and the second data volume is greater than a preset difference threshold, then make the number of processor cores corresponding to the network card resources in the initial core allocation plan less than the number of processor cores corresponding to the storage-end resources.

[0086] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: replacing the initial core allocation plan with the adjusted initial core allocation plan, and performing a step of simulating the initial core allocation plan in the target system to obtain simulation operation data, so as to optimize the core allocation plan again based on the new simulation operation data.

[0087] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: when the difference between the first data volume and the second data volume is not greater than a preset difference threshold, the number of processor cores corresponding to the network card resources, storage resources and processor resources in the current core allocation scheme are respectively increased, and the task processing time of the target task after each increase is calculated; the core allocation scheme corresponding to the shortest task processing time is used as the available core allocation scheme.

[0088] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: determine whether the number of available processor cores in the target system meets the available core allocation plan; if the number of available processor cores meets the available core allocation plan, execute the step of executing the target task in the target system according to the available core allocation plan until the target task is completed.

[0089] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the number of available processor cores does not meet the available core allocation plan, the target task is executed in the target system, and the available core allocation plan is dynamically adjusted according to the execution situation, and the adjusted core allocation plan is applied.

[0090] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: determine the utilization rate of the processor core occupied by the IO stack of the target task; if the utilization rate is higher than the preset upper limit value, adjust part of the processor cores corresponding to the storage-side resources in the current core allocation plan to the IO stack.

[0091] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the utilization rate is lower than the preset lower limit, the processor core occupied by the IO stack is adjusted to the storage end resource in the current core allocation scheme.

[0092] Furthermore, the embodiment of the present application also provides an electronic device. The electronic device can be Figure 7 The server shown can also be Figure 8 Terminal shown. Figure 7 and Figure 8 Each of the diagrams is a structural diagram of an electronic device according to an exemplary embodiment, and the contents in the diagrams cannot be considered as any limitation on the scope of use of the present application.

[0093] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store a computer program, which is loaded and executed by the processor to implement the relevant steps of the data processing disclosed in any of the aforementioned embodiments.

[0094] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface is used to obtain external input data or output data to the outside world. The specific interface type can be selected according to specific application needs and is not specifically limited here.

[0095] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer programs and data, etc. The storage method can be temporary storage or permanent storage.

[0096] The operating system is used to manage and control the hardware devices and computer programs on the server, enabling the processor to operate and process data in the memory. It can be Windows Server, NetWare, Unix, Linux, etc. In addition to computer programs capable of performing the data processing methods disclosed in any of the aforementioned embodiments, computer programs can also include computer programs capable of performing other specific tasks. Data can include data such as application update information and other data such as application developer information.

[0097] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may specifically include but is not limited to a smartphone, tablet computer, laptop computer or desktop computer.

[0098] Generally, the terminal in this embodiment includes: a processor and a memory.

[0099] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content required to be displayed on the display. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0100] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is used to store at least the following computer program, wherein, after the computer program is loaded and executed by the processor, it can implement the relevant steps in the data processing method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include but is not limited to update information of the application.

[0101] In some embodiments, the terminal may further include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0102] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure.

[0103] A non-volatile storage medium provided in an embodiment of the present application is introduced below. The non-volatile storage medium described below can be referenced with other embodiments described herein.

[0104] A non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the aforementioned embodiment. The non-volatile storage medium is a computer-readable non-volatile storage medium that, as a carrier for resource storage, may be a read-only memory, random access memory, a magnetic disk, or an optical disk. The resources stored thereon include an operating system, a computer program, and data, and the storage method may be either temporary or permanent.

[0105] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: select the resource modules required for running the target task in the target system; determine the initial core allocation plan of the resource module based on historical core allocation information; optimize the initial core allocation plan according to the simulated operation data of the initial core allocation plan to obtain an available core allocation plan; and execute the target task in the target system according to the available core allocation plan.

[0106] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: determine the network card status information, storage end status information and processor status information in the target system; select the network card resources, storage end resources and processor resources required for the target task to run in the target system based on the network card status information, storage end status information and processor status information, as well as the task type, task data volume and processing time of the target task; and use the network card resources, storage end resources and processor resources as resource modules.

[0107] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: sorting the network cards in the target system according to the network card status information to obtain a network card sequence; sorting the storage terminals in the target system according to the storage terminal status information to obtain a storage terminal sequence; sorting the processors in the target system according to the processor status information to obtain a processor sequence; and selecting network card resources, storage terminal resources, and processor resources from the network card sequence, storage terminal sequence, and processor sequence, respectively, based on the task type, task data volume, and processing time of the target task.

[0108] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: according to historical core allocation information, allocate the number of processor cores to the resource module from multiple dimensions to obtain multiple optional allocation schemes; the multiple dimensions include: at least two of: instruction set simplification, queue optimization, interrupt balancing and kernel parameter tuning; among multiple optional allocation schemes, select the one with the highest processing efficiency as the initial core allocation scheme.

[0109] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: based on historical core allocation information, optimize the number of instructions and instruction content executed by a single core from the instruction set simplification dimension, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0110] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: based on historical core allocation information, optimize the queue depth of single core execution from the queue optimization dimension, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0111] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: based on historical core allocation information, optimize the interrupt time of a single core from the interrupt balance dimension, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0112] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: optimize the frequency and / or energy consumption of a single core from the kernel parameter tuning dimension based on historical core allocation information, and determine the number of processor cores that can be allocated to the resource module based on the corresponding optimization results.

[0113] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: simulating the operation of the initial core allocation plan in the target system to obtain simulated operation data; based on the simulated operation data, adjusting the number of processor cores corresponding to the network card resources, storage resources and processor resources in the initial core allocation plan to optimize the initial core allocation plan.

[0114] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: determine the first data volume processed by the network card resources and the second data volume processed by the storage-end resources in the initial core allocation plan based on the simulation operation data; if the first data volume is greater than the second data volume, and the difference between the first data volume and the second data volume is greater than the preset difference threshold, then make the number of processor cores corresponding to the network card resources in the initial core allocation plan less than the number of processor cores corresponding to the storage-end resources.

[0115] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: replacing the initial core allocation plan with the adjusted initial core allocation plan, and performing a step of simulating the initial core allocation plan in the target system to obtain simulation operation data, so as to optimize the core allocation plan again based on the new simulation operation data.

[0116] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: when the difference between the first data volume and the second data volume is not greater than the preset difference threshold, the number of processor cores corresponding to the network card resources, storage resources and processor resources in the current core allocation scheme are respectively increased, and the task processing time of the target task after each increase is calculated; the core allocation scheme corresponding to the shortest task processing time is used as the available core allocation scheme.

[0117] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: determine whether the number of available processor cores in the target system meets the available core allocation plan; if the number of available processor cores meets the available core allocation plan, execute the steps of executing the target task in the target system according to the available core allocation plan until the target task is completed.

[0118] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: if the number of available processor cores does not meet the available core allocation plan, the target task is executed in the target system, and the available core allocation plan is dynamically adjusted according to the execution situation, and the adjusted core allocation plan is applied.

[0119] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: determining the utilization rate of the processor core occupied by the IO stack of the target task; if the utilization rate is higher than the preset upper limit value, adjusting part of the processor cores corresponding to the storage-side resources in the current core allocation scheme to the IO stack.

[0120] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: if the utilization rate is lower than the preset lower limit, the processor core occupied by the IO stack is adjusted to the storage end resources in the current core allocation scheme.

[0121] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0122] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned data processing method when executed by a processor.

[0123] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0124] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0125] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-volatile storage medium known in the art.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data processing method, characterized in that: include: Selecting resource modules required for the target task to run in the target system; wherein the network card resources, storage resources and processor resources required for the target task to run are used as the resource modules; determining an initial core allocation scheme for the resource module based on historical core allocation information; Acquiring simulation operation data of the initial core allocation plan, optimizing the initial core allocation plan according to the simulation operation data, and obtaining an available core allocation plan; executing the target task in the target system according to the available core allocation plan; determining whether the number of available processor cores in the target system satisfies the available core allocation plan; if the number of available processor cores does not satisfy the available core allocation plan, executing the target task in the target system, dynamically adjusting the available core allocation plan according to the execution status, and applying the adjusted core allocation plan; Determining the initial core allocation scheme of the resource module based on historical core allocation information includes: Allocating the number of processor cores to the resource module from multiple dimensions based on the historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions including at least two of instruction set streamlining, queue optimization, interrupt balancing, and kernel parameter tuning; Among the multiple optional allocation schemes, the one with the highest processing efficiency is selected as the initial core allocation scheme.

2. The method according to claim 1, characterized in that The step of selecting a resource module required for running the target task in the target system includes: Determining network card status information, storage terminal status information, and processor status information in the target system; Selecting network card resources, storage end resources, and processor resources required for running the target task in the target system based on the network card status information, the storage end status information, and the processor status information, as well as the task type, task data volume, and processing duration of the target task; The network card resources, the storage end resources and the processor resources are used as the resource module.

3. The method according to claim 2, characterized in that Selecting network card resources, storage end resources, and processor resources required for running the target task in the target system based on the network card status information, the storage end status information, and the processor status information, as well as the task type, task data volume, and processing duration of the target task, including: sorting the network cards in the target system according to the network card status information to obtain a network card sequence; sorting the storage terminals in the target system according to the storage terminal status information to obtain a storage terminal sequence; sorting the processors in the target system according to the processor state information to obtain a processor sequence; According to the task type, task data volume and processing time of the target task, the network card resources, the storage end resources and the processor resources are selected from the network card sequence, the storage end sequence and the processor sequence respectively.

4. The method according to claim 1, wherein Allocating the number of processor cores for the resource module based on the historical core allocation information from the instruction set reduction dimension includes: According to the historical core allocation information, the number of instructions and instruction content executed by a single core are optimized from the instruction set simplification dimension, and the number of processor cores that can be allocated to the resource module is determined according to the corresponding optimization results.

5. The method according to claim 1, wherein Allocating the number of processor cores to the resource module from a queue optimization dimension according to the historical core allocation information includes: According to the historical core allocation information, the queue depth of the single core execution is optimized from the queue optimization dimension, and the number of processor cores that can be allocated to the resource module is determined according to the corresponding optimization result.

6. The method according to claim 1, characterized in that Allocating the number of processor cores to the resource module from an interrupt balance dimension according to the historical core allocation information includes: According to the historical core allocation information, the interruption time consumption of a single core is optimized from the interruption balance dimension, and the number of processor cores that can be allocated to the resource module is determined according to the corresponding optimization result.

7. The method according to claim 1, characterized in that Allocating the number of processor cores for the resource module from a kernel parameter tuning dimension according to the historical core allocation information includes: According to the historical core allocation information, the frequency and / or energy consumption of a single core is optimized from the kernel parameter tuning dimension, and the number of processor cores that can be allocated to the resource module is determined according to the corresponding optimization results.

8. The method according to claim 1, characterized in that Optimizing the initial core allocation scheme according to the simulation operation data includes: Simulating and running the initial core allocation scheme in the target system to obtain the simulation operation data; According to the simulation operation data, the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the initial core allocation scheme is adjusted to optimize the initial core allocation scheme.

9. The method according to claim 8, characterized in that Adjusting the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the initial core allocation scheme according to the simulation operation data includes: Determining, based on the simulation operation data, a first data volume processed by the network card resources and a second data volume processed by the storage end resources in the initial core allocation scheme; If the first data volume is greater than the second data volume, and the difference between the first data volume and the second data volume is greater than a preset difference threshold, the number of processor cores corresponding to the network card resources in the initial core allocation scheme is made less than the number of processor cores corresponding to the storage end resources.

10. The method according to claim 9, characterized in that After the number of processor cores corresponding to the network card resources in the initial core allocation scheme is made smaller than the number of processor cores corresponding to the storage end resources, the method further includes: The initial core allocation plan is replaced with the adjusted initial core allocation plan, and the initial core allocation plan is simulated and run in the target system to obtain the simulation data, so as to optimize the core allocation plan again based on the new simulation data.

11. The method according to claim 10, characterized in that Also includes: When the difference between the first data volume and the second data volume is not greater than a preset difference threshold, the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the current core allocation scheme is increased respectively, and the task processing duration of the target task after each increase is calculated; The core allocation scheme corresponding to the shortest task processing duration is used as the available core allocation scheme.

12. The method according to any one of claims 1 to 11, characterized in that Also includes: If the number of available processor cores satisfies the available core allocation plan, the step of executing the target task in the target system according to the available core allocation plan is performed until the target task is completed.

13. The method according to claim 12, characterized in that Dynamically adjusting the available core allocation scheme according to the execution status includes: Determine the utilization rate of the processor core occupied by the IO stack of the target task; If the utilization rate is higher than a preset upper limit, some processor cores corresponding to the storage-side resources in the current core allocation scheme are adjusted to the IO stack.

14. The method according to claim 13, characterized in that If the utilization rate is lower than a preset lower limit, the processor cores occupied by the IO stack are adjusted to storage resources in the current core allocation scheme.

15. A data processing device, characterized in that: include: A selection module is used to select resource modules required for the target task to run in the target system; wherein the network card resources, storage resources and processor resources required for the target task to run are used as the resource modules; a determination module, configured to determine an initial core allocation scheme for the resource module based on historical core allocation information; an optimization module, configured to obtain simulation operation data of the initial core allocation scheme, optimize the initial core allocation scheme according to the simulation operation data, and obtain an available core allocation scheme; an execution module, configured to execute the target task in the target system according to the available core allocation plan; determine whether the number of available processor cores in the target system satisfies the available core allocation plan; if the number of available processor cores does not satisfy the available core allocation plan, execute the target task in the target system, dynamically adjust the available core allocation plan based on the execution status, and apply the adjusted core allocation plan; Among them, the determination module is used to: allocate the number of processor cores to the resource module from multiple dimensions based on the historical core allocation information, and obtain multiple optional allocation schemes; the multiple dimensions include: at least two of: instruction set simplification, queue optimization, interrupt balancing and kernel parameter tuning; among the multiple optional allocation schemes, select the one with the highest processing efficiency as the initial core allocation scheme.

16. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 14.

17. A non-volatile storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images