Data processing method and device, equipment, medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a medium and a program product in the technical field of computers. According to the method, for resource modules required by target task operation, initial core allocation schemes of the resource modules are basically determined on the basis of historical core allocation information, then the initial core allocation schemes are optimized according to simulation operation data of the initial core allocation schemes, and available core allocation schemes are obtained; and finally, executing the target task in the target system according to the available core allocation scheme, so that the optimal core allocation scheme can be determined in combination with the task related information and the actual operation condition of the system, and the utilization rate and the task processing efficiency in the operation process of the processor are maximized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a data processing method, apparatus, device, medium, and program product. Background Art

[0002] Currently, each core of a multi-core processor can be fixedly bound to a task. This binding method is likely to cause unbalanced loads on different processor cores, with some processor cores having a relatively high load while some are relatively idle, resulting in a relatively low utilization rate of the processor.

[0003] Therefore, how to improve the utilization rate during the operation of the processor is a problem that those skilled in the art need to solve. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data processing method, apparatus, device, medium, and program product to improve the utilization rate during the operation of the processor.

[0005] In the first aspect, this application provides a data processing method, including: selecting a resource module required for the operation of a target task in a target system; determining an initial core allocation plan for the resource module based on historical core allocation information; obtaining simulation operation data of the initial core allocation plan, and optimizing the initial core allocation plan according to the simulation operation data to obtain an available core allocation plan; and executing the target task in the target system according to the available core allocation plan.

[0006] In the second aspect, this application provides a data processing apparatus, including: a selection module for selecting a resource module required for the operation of a target task in a target system; a determination module for determining an initial core allocation plan for the resource module based on historical core allocation information; an optimization module for obtaining simulation operation data of the initial core allocation plan and optimizing the initial core allocation plan according to the simulation operation data to obtain an available core allocation plan; and an execution module for executing the target task in the target system according to the available core allocation plan.

[0007] In the third aspect, this application provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program to implement the data processing method disclosed above.

[0008] In the fourth aspect, this application provides a non-volatile storage medium for storing a computer program, where the computer program, when executed by a processor, implements the data processing method disclosed above.

[0009] In the fifth aspect, this application provides a computer program product, including computer program / instructions, where the computer program / instructions, when executed by a processor, implement the steps of the data processing method disclosed above.

[0010] As can be seen from the above solution, the present application provides a data processing method, including: selecting resource modules required for running a target task in a target system; determining an initial core allocation plan for the resource modules based on historical core allocation information; obtaining simulation operation data of the initial core allocation plan, and optimizing the initial core allocation plan according to the simulation operation data to obtain an available core allocation plan; and executing the target task in the target system according to the available core allocation plan.

[0011] It can be seen that the beneficial effects of the present application are as follows: for the resource modules required for running the target task, first basically determine the initial core allocation plan for these resource modules based on historical core allocation information, then optimize the initial core allocation plan according to the simulation operation data of the initial core allocation plan to obtain an available core allocation plan, and finally execute the target task in the target system according to the available core allocation plan. Thus, the optimal core allocation plan can be determined by combining task-related information and the actual operation situation of the system, thereby maximizing the utilization rate and task processing efficiency during the operation of the processor.

[0012] Correspondingly, a data processing device, equipment, medium, and program product provided by the present application also have the above technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0014] Figure 1 It is a flowchart of a data processing method disclosed in the present application; Figure 2 It is a flowchart of a resource module selection method disclosed in the present application; Figure 3 It is a flowchart of an initial core distribution method disclosed in the present application; Figure 4 It is a flowchart of a core distribution optimization method disclosed in the present application; Figure 5 It is a flowchart of a task execution method disclosed in the present application; Figure 6 It is a schematic diagram of a data processing device disclosed in the present application; Figure 7 It is a server structure diagram provided by the present application; Figure 8 It is a terminal structure diagram provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0016] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0017] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0018] Currently, each core of a multi-core processor can be correspondingly bound to a fixed task. This binding method is likely to cause: uneven load on different processor cores, with some processor cores having a high load and some processor cores being relatively idle, resulting in a relatively low utilization rate of the processor. To this end, the present application provides a data processing solution that can determine the optimal core allocation scheme by combining task-related information and the actual operating conditions of the system, thereby maximizing the utilization rate and task processing efficiency during the operation of the processor.

[0019] See Figure 1 As shown, an embodiment of the present application discloses a data processing method, including: S101. Select a resource module required for the target task to run in the target system.

[0020] In this embodiment, the target system includes at least one server, which can monitor the operating status information of various devices in the target system in real time, such as the IOPS of network cards, the IOPS of storage devices (Input / Output Operations Per Second, the number of read and write operations per second), the working frequency, power consumption, and energy consumption of each core of the processor, etc. In one example, this information is monitored and summarized by the BMC (Baseboard Management Controller), and the storage device can specifically include: storage disks, CPLD (Complex Programmable Logic Device) logic control units, etc.; in addition, a non-transparent bridge is set up to connect different processors in the system.

[0021] In one implementation, select the resource modules required for the target task to run in the target system, including: determining the network card status information (such as network card IOPS), storage device status information (such as storage disk IOPS), and processor status information (such as the working frequency, power consumption, and energy consumption of each processor core, etc.) in the target system; according to the network card status information, storage device status information, and processor status information, as well as the task type (such as storage tasks, query tasks, computing tasks, etc.), task data volume, and processing duration of the target task, select the network card resources, storage device resources, and processor resources required for the target task to run in the target system; use the network card resources, storage device resources, and processor resources as resource modules. Among them, according to the network card status information, storage device status information, and processor status information, as well as the task type, task data volume, and processing duration of the target task, select the network card resources, storage device resources, and processor resources required for the target task to run in the target system, including: sort the network cards in the target system according to the network card status information to obtain a network card sequence; sort the storage devices in the target system according to the storage device status information to obtain a storage device sequence; sort the processors in the target system according to the processor status information to obtain a processor sequence; according to the task type, task data volume, and processing duration of the target task, select the network card resources, storage device resources, and processor resources in the network card sequence, storage device sequence, and processor sequence respectively.

[0022] S102. Determine the initial core allocation plan of the resource module based on the historical core allocation information.

[0023] It should be noted that the historical core allocation information is the core allocation scheme of other tasks processed by the target system. In one implementation, determining the initial core allocation scheme of the resource module based on the historical core allocation information includes: allocating the number of processor cores to the resource module from multiple dimensions according to the historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions include at least two of instruction set simplification, queue optimization, interrupt balance, and kernel parameter tuning; among the multiple optional allocation schemes, select the one with the highest processing efficiency as the initial core allocation scheme.

[0024] In one example, allocating the number of processor cores to the resource module from the dimension of instruction set simplification according to the historical core allocation information includes: optimizing the number and content of instructions executed by a single core from the dimension of instruction set simplification according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0025] In one example, allocating the number of processor cores to the resource module from the dimension of queue optimization according to the historical core allocation information includes: optimizing the queue depth executed by a single core from the dimension of queue optimization according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0026] In one example, allocating the number of processor cores to the resource module from the dimension of interrupt balance according to the historical core allocation information includes: optimizing the interrupt time consumption of a single core from the dimension of interrupt balance according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0027] In one example, allocating the number of processor cores to the resource module from the dimension of kernel parameter tuning according to the historical core allocation information includes: optimizing the frequency and / or power consumption of a single core from the dimension of kernel parameter tuning according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0028] S103. Obtain the simulation operation data of the initial core allocation scheme, and optimize the initial core allocation scheme according to the simulation operation data to obtain an available core allocation scheme.

[0029] In one implementation, optimizing the initial core allocation scheme according to the simulation operation data includes: simulating the operation of the initial core allocation scheme in the target system to obtain simulation operation data; according to the simulation operation data, adjusting the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the initial core allocation scheme to optimize the initial core allocation scheme. The simulation operation data at least includes: the first data volume transmitted by the network card resources and the second data volume processed by the storage end resources. The second data volume processed by the storage end resources specifically includes: the sum of the data written to the storage disk and the data volume transmitted by the non-transparent bridge.

[0030] Among them, according to the simulation operation data, the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the initial core allocation scheme is adjusted, including: determining the first data volume processed by the network card resources and the second data volume processed by the storage - side resources in the initial core allocation scheme according to the simulation operation data; if the first data volume is greater than the second data volume and the difference between the first data volume and the second data volume is greater than a preset difference threshold, it is considered that the first data volume is much greater than the second data volume. Then, make the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage - side resources. For example, reduce the number of processor cores corresponding to the network card resources. Further, after making the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage - side resources, it also includes: replacing the initial core allocation scheme with the adjusted initial core allocation scheme, and performing the step of simulating the operation of the initial core allocation scheme in the target system to obtain the simulation operation data, so as to optimize the core allocation scheme again based on the new simulation operation data until the first data volume and the second data volume are relatively balanced.

[0031] In the case where the difference between the first data volume and the second data volume is not greater than the preset difference threshold, it is considered that the first data volume and the second data volume are relatively balanced. Then, increase the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the current core allocation scheme respectively, and calculate the task processing duration of the target task after each increase; take the core allocation scheme corresponding to the shortest task processing duration as the available core allocation scheme. Among them, increasing the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the current core allocation scheme respectively includes: setting the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the current core allocation scheme as A, B, and C respectively, and setting the increment value as X. Then, after the first increase, the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources are A + X, B + X, and C + X respectively; after the next increase, the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources are A + 2X, B + 2X, and C + 2X respectively; X can be flexibly valued according to the actual situation.

[0032] S104. Execute the target task in the target system according to the available core allocation scheme.

[0033] Considering that the number of available processor cores in the target system is limited, when executing S104, it can first be determined whether the number of available processor cores in the target system meets the available core allocation scheme, that is: determine whether the number of available processor cores in the target system is not less than the total number of cores required by the available core allocation scheme. If so, it is considered that the number of available processor cores meets the available core allocation scheme, so the current available core allocation scheme can be directly applied, and then the target task is executed in the target system according to the available core allocation scheme until the target task is completed. If not, it is considered that the number of available processor cores does not meet the available core allocation scheme. Therefore, the target task is first executed in the target system (that is, the target task is executed, but the current available core allocation scheme is not applied). After dynamically adjusting the available core allocation scheme according to the execution situation, the adjusted core allocation scheme is then applied (that is, the target task is continued to be executed in the target system according to the adjusted available core allocation scheme).

[0034] In one implementation, it is determined whether the number of available processor cores in the target system meets the available core allocation scheme; if the number of available processor cores meets the available core allocation scheme, the step of executing the target task in the target system according to the available core allocation scheme is performed until the target task is completed. If the number of available processor cores does not meet the available core allocation scheme, the target task is executed in the target system, the available core allocation scheme is dynamically adjusted according to the execution situation, and the adjusted core allocation scheme is applied.

[0035] In one implementation, dynamically adjusting the available core allocation scheme according to the execution situation includes: determining the utilization rate of the processor cores occupied by the IO stack of the target task; if the utilization rate is higher than the preset upper limit value, some of the processor cores corresponding to the storage - side resources in the current core allocation scheme are adjusted to the current IO stack. If the utilization rate is lower than the preset lower limit value, the processor cores occupied by the current IO stack are adjusted to the storage - side resources in the current core allocation scheme. Among them, the processor cores occupied by the IO stack of the target task are the processor cores corresponding to the processor resources in the available core allocation scheme.

[0036] It can be seen that in this embodiment, for the resource modules required for the operation of the target task, the initial core allocation scheme for these resource modules is first basically determined based on the historical core allocation information, and then the initial core allocation scheme is optimized according to the simulation operation data of the initial core allocation scheme to obtain the available core allocation scheme. Finally, the target task is executed in the target system according to the available core allocation scheme. Thus, the optimal core allocation scheme can be determined by combining the task - related information and the actual operation situation of the system, so as to maximize the utilization rate and task - processing efficiency during the operation of the processor.

[0037] In one example, the target system may include: a front end (including a network card, etc.), a back end (including a storage disk, a CPLD, and a BMC), and a CPU end (including a non-transparent bridge, various processors, etc.). Among them, the front-end devices mainly include various external plug-in cards for data reading and transmission, and the back end mainly includes storage devices such as hard disks. The two processors are interconnected using an NTB (Non-Transparent PCI Bridge), and data backup storage can be specifically implemented using different processors.

[0038] Please refer to Figure 2 , after the CPU end issues a service data reading instruction, the BMC reads the health status of each module in the front end, back end, and CPU end, including whether there is a fault message, operating status information (such as voltage and current information), etc. The CPLD analyzes this information and performs corresponding logical processing to select an available resource module for the currently read service data. Specifically, the BMC end reads the voltage and current information of each module and confirms whether the status of each module is correct based on this. For normal modules, it is passed to the CPLD for logical selection. The CPLD selects relevant resource modules such as network cards, hard disks, and processors that are available from among many normal modules according to parameters such as the service type, the total amount of service data, the time for continuous execution of service processing, and the processing method required by the storage device, and sends an enabling signal to them.

[0039] Please refer to Figure 3 , for the aforementioned service data information obtained (service type, total amount of service data, time for continuous execution of service processing, and processing method required by the storage device, etc.), simulate from each optimization dimension and perform execution optimization. The optimization dimensions can be selected in ways such as instruction set reduction, multi-queue optimization, interrupt balancing, and kernel parameter tuning. After simulating corresponding core allocation schemes for various optimization dimensions, from the perspective of improving performance, finally output a core allocation scheme.

[0040] Specifically, the optimization methods are divided into two categories: instruction reduction and performance improvement. Couple the different optimization methods included in the instruction reduction category. Specifically, based on the historical core allocation scheme, calculate the percentage of the different methods included in the instruction reduction category in the total instruction set and the number of cores required, remove the methods that require a large number of cores and are repetitive, perform an exclusive OR operation on the remaining methods, and finally obtain the core allocation scheme corresponding to the instruction reduction category. Also couple the methods such as multi-queue optimization, interrupt balancing, and kernel parameter tuning. For example: while optimizing the queue depth, optimize the interrupt time consumption and perform kernel parameter tuning, thereby also obtaining a core allocation scheme. For example, select the one with the shortest processing time and the largest queue depth from the three aspects of queue depth, total processing duration, and number of cores occupied. Then select the most efficient one from the two core allocation schemes. This can ensure the number of instructions executed by the CPU and maximize the data processing volume.

[0041] Please refer to Figure 4 , based on the aforementioned business data information and the initially determined core division plan, conduct plan simulation, and adjust the core division according to the module health status until the optimal core division plan is obtained, so that the data volume of the incoming data and the data stored on the backend hard disk reaches the maximum.

[0042] Specifically, adjusting the core division according to the module health status includes: first balancing the data processing volume of the selected front-end and back-end modules, and then optimizing with the shortest total processing duration. Balancing the data processing volume of the selected front-end and back-end modules includes: simulating the current core division plan and calculating whether the data transferred by the current front-end module is higher than the total amount of data written to the current backend disk and the NTB transmission data. If it is higher, reduce the number of CPU cores allocated to the current front-end module, and then conduct a second simulation until the data processing volumes of the front-end and back-end modules are balanced. Optimizing with the shortest total processing duration includes: taking the processing time as the optimization point, referring to the currently available number of cores, allocating one more core to each part in the current allocation plan in turn. After the core division is completed, conduct a simulation to calculate and record the total processing duration, and repeat accordingly to obtain shorter and shorter times. After reaching the shortest time, output the current core division plan as the top-level available core allocation plan.

[0043] Please refer to Figure 5 , provide the core division plan (i.e., the available core allocation plan) to the CPU side, so that the CPU side can judge whether the currently available number of CPU cores can meet its execution. If it meets, execute the corresponding tasks according to the current core division plan, and no adjustment of the core division plan is made during the execution process. If it does not meet, execute the current core division plan and adjust the core division plan during the execution process.

[0044] Among them, the adjustment of the core division plan can be realized according to the following process: determine the utilization rate of the processor cores occupied by each underlying IO stack during the task execution process, and based on this utilization rate, transfer the processor cores occupied by the IO stack to the backend storage, or transfer the processor cores occupied by the backend storage to the IO stack. Among them, the optimization of the IO stack is based on the available space of the CPU memory unit. The processor divides the incoming data into data segments 1, 2...n, and one data segment corresponds to one IO stack. The IO stack stores the corresponding data segment into the corresponding memory space and completes the disk storage after corresponding data processing. The core allocation strategy corresponding to the IO stack can be called: the underlying core division strategy.

[0045] Among them, the process of adjusting the core based on the utilization rate includes: when the core utilization rate of the IO stack is higher than 90%, the CPU allocates the cores of the backend to the IO stack. When the utilization rate of a core is lower than 40%, the CPU side distributes the data in the memory space corresponding to this core to the remaining spaces, and this core is configured to the backend. Through this real-time adjustment method, ensure that the utilization rate of each core in the CPU reaches the optimal.

[0046] After considering the health status of each unit module in the system and the data processing tasks in this embodiment, the number of processor cores that each module can use is dynamically adjusted. The number of cores required for the IO stack in the CPU can also be adjusted according to the actual processing situation, so that data transmission, data storage, and data processing are coordinated with each other. The CPU cores can achieve a higher utilization rate in the storage device, and the processing efficiency and performance can also be maximized.

[0047] Next, a data processing device provided by an embodiment of the present application will be introduced. A data processing device described below can be referred to each other with other embodiments described in this article.

[0048] See Figure 6 As shown, an embodiment of the present application discloses a data processing device provided by the present application, including: a selection module, configured to select a resource module required for running a target task in a target system; a determination module, configured to determine an initial core allocation scheme for the resource module based on historical core allocation information; an optimization module, configured to obtain simulation operation data of the initial core allocation scheme, and optimize the initial core allocation scheme according to the simulation operation data to obtain an available core allocation scheme; an execution module, configured to execute the target task in the target system according to the available core allocation scheme.

[0049] In one implementation manner, the selection module is configured to: determine network card status information, storage end status information, and processor status information in the target system; according to the network card status information, storage end status information, and processor status information, as well as the task type, task data volume, and processing duration of the target task, select network card resources, storage end resources, and processor resources required for running the target task in the target system; and use the network card resources, storage end resources, and processor resources as resource modules.

[0050] In one implementation manner, the selection module is configured to: sort each network card in the target system according to the network card status information to obtain a network card sequence; sort each storage end in the target system according to the storage end status information to obtain a storage end sequence; sort each processor in the target system according to the processor status information to obtain a processor sequence; and select network card resources, storage end resources, and processor resources in the network card sequence, storage end sequence, and processor sequence respectively according to the task type, task data volume, and processing duration of the target task.

[0051] In one implementation manner, the determination module is configured to: allocate processor cores to the resource module from multiple dimensions according to historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions include at least two of instruction set simplification, queue optimization, interrupt balance, and kernel parameter tuning; and select the one with the highest processing efficiency as the initial core allocation scheme among the multiple optional allocation schemes.

[0052] In one embodiment, the determining module is configured to: optimize the number and content of instructions executed by a single core from the dimension of instruction set reduction according to historical core allocation information, and determine the number of processor cores allocable to the resource module according to the corresponding optimization result.

[0053] In one embodiment, the determining module is configured to: optimize the queue depth of a single core from the dimension of queue optimization according to historical core allocation information, and determine the number of processor cores allocable to the resource module according to the corresponding optimization result.

[0054] In one embodiment, the determining module is configured to: optimize the interrupt time of a single core from the dimension of interrupt balance according to historical core allocation information, and determine the number of processor cores allocable to the resource module according to the corresponding optimization result.

[0055] In one embodiment, the determining module is configured to: optimize the frequency and / or power consumption of a single core from the dimension of kernel parameter tuning according to historical core allocation information, and determine the number of processor cores allocable to the resource module according to the corresponding optimization result.

[0056] In one embodiment, the optimizing module is configured to: simulate and run an initial core allocation scheme in a target system to obtain simulation operation data; according to the simulation operation data, adjust the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the initial core allocation scheme respectively to optimize the initial core allocation scheme.

[0057] In one embodiment, the optimizing module is configured to: determine a first data volume processed by the network card resources and a second data volume processed by the storage resources in the initial core allocation scheme according to the simulation operation data; if the first data volume is greater than the second data volume and the difference between the first data volume and the second data volume is greater than a preset difference threshold, make the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage resources.

[0058] In one embodiment, the optimizing module is configured to: replace the initial core allocation scheme with the adjusted initial core allocation scheme, and execute the step of simulating and running the initial core allocation scheme in the target system to obtain simulation operation data, so as to optimize the core allocation scheme again based on the new simulation operation data.

[0059] In one embodiment, the optimizing module is configured to: when the difference between the first data volume and the second data volume is not greater than the preset difference threshold, increase the number of processor cores corresponding to the network card resources, storage resources, and processor resources in the current core allocation scheme respectively, and calculate the task processing duration of the target task after each increase; use the core allocation scheme corresponding to the shortest task processing duration as the available core allocation scheme.

[0060] In one embodiment, the execution module is configured to: determine whether the number of available processor cores in the target system meets the available core allocation scheme; if the number of available processor cores meets the available core allocation scheme, then perform the step of executing the target task in the target system according to the available core allocation scheme until the target task is completed.

[0061] In one embodiment, the execution module is configured to: if the number of available processor cores does not meet the available core allocation scheme, then execute the target task in the target system, dynamically adjust the available core allocation scheme according to the execution situation, and apply the adjusted core allocation scheme.

[0062] In one embodiment, the execution module is configured to: determine the utilization rate of the processor cores occupied by the IO stack of the target task; if the utilization rate is higher than the preset upper limit value, then adjust some of the processor cores corresponding to the storage - side resources in the current core allocation scheme to the IO stack.

[0063] In one embodiment, the execution module is configured to: if the utilization rate is lower than the preset lower limit value, then adjust the processor cores occupied by the IO stack to the storage - side resources in the current core allocation scheme.

[0064] Among them, for the more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.

[0065] It can be seen that this embodiment provides a data processing device, which can determine the optimal core allocation scheme by combining task - related information and the actual operation situation of the system, so as to maximize the utilization rate and task processing efficiency during the operation of the processor.

[0066] Next, an electronic device provided by an embodiment of the present application will be introduced. The electronic device described below can be mutually referred to with other embodiments described in this article.

[0067] An embodiment of the present application discloses an electronic device, including: a memory for storing a computer program; a processor for executing the computer program to implement the method disclosed in any of the foregoing embodiments.

[0068] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: select a resource module required for running the target task in the target system; determine an initial core allocation scheme for the resource module based on historical core allocation information; optimize the initial core allocation scheme according to the simulation operation data of the initial core allocation scheme to obtain an available core allocation scheme; execute the target task in the target system according to the available core allocation scheme.

[0069] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: determining the network card status information, storage end status information, and processor status information in the target system; selecting the network card resources, storage end resources, and processor resources required for running the target task in the target system according to the network card status information, storage end status information, processor status information, as well as the task type, task data volume, and processing duration of the target task; using the network card resources, storage end resources, and processor resources as a resource module.

[0070] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: sorting each network card in the target system according to the network card status information to obtain a network card sequence; sorting each storage end in the target system according to the storage end status information to obtain a storage end sequence; sorting each processor in the target system according to the processor status information to obtain a processor sequence; selecting the network card resources, storage end resources, and processor resources in the network card sequence, storage end sequence, and processor sequence respectively according to the task type, task data volume, and processing duration of the target task.

[0071] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: allocating the number of processor cores to the resource module from multiple dimensions according to the historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions include at least two of instruction set simplification, queue optimization, interrupt balance, and kernel parameter tuning; selecting the one with the highest processing efficiency as the initial core allocation scheme among the multiple optional allocation schemes.

[0072] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: optimizing the number and content of instructions executed by a single core from the dimension of instruction set simplification according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0073] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: optimizing the queue depth of a single core from the dimension of queue optimization according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0074] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: optimizing the interrupt time consumption of a single core from the dimension of interrupt balance according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0075] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: According to the historical core allocation information, optimize the frequency and / or power consumption of a single core from the dimension of kernel parameter tuning, and determine the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0076] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Simulate and run the initial core allocation scheme in the target system to obtain simulation operation data; According to the simulation operation data, adjust the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the initial core allocation scheme to optimize the initial core allocation scheme.

[0077] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: According to the simulation operation data, determine the first data volume processed by the network card resources and the second data volume processed by the storage - side resources in the initial core allocation scheme; If the first data volume is greater than the second data volume and the difference between the first data volume and the second data volume is greater than the preset difference threshold, then make the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage - side resources.

[0078] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Replace the initial core allocation scheme with the adjusted initial core allocation scheme, and execute the step of simulating and running the initial core allocation scheme in the target system to obtain simulation operation data, so as to optimize the core allocation scheme again based on the new simulation operation data.

[0079] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: In the case where the difference between the first data volume and the second data volume is not greater than the preset difference threshold, increment the number of processor cores corresponding to the network card resources, storage - side resources, and processor resources in the current core allocation scheme respectively, and calculate the task processing duration of the target task after each increment; Use the core allocation scheme corresponding to the shortest task processing duration as the available core allocation scheme.

[0080] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Determine whether the available processor cores in the target system meet the available core allocation scheme; If the available processor cores meet the available core allocation scheme, then execute the step of executing the target task in the target system according to the available core allocation scheme until the target task is completed.

[0081] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: If the number of available processor cores does not meet the available core allocation scheme, execute the target task in the target system, dynamically adjust the available core allocation scheme according to the execution situation, and apply the adjusted core allocation scheme.

[0082] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Determine the utilization rate of the processor cores occupied by the IO stack of the target task; If the utilization rate is higher than the preset upper limit value, adjust the part of the processor cores corresponding to the storage - side resources in the current core allocation scheme to the IO stack.

[0083] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: If the utilization rate is lower than the preset lower limit value, adjust the processor cores occupied by the IO stack to the storage - side resources in the current core allocation scheme.

[0084] Furthermore, an embodiment of the present application also provides an electronic device. Among them, the above - mentioned electronic device can be either Figure 7 the server shown as Figure 8 or the terminal shown as Figure 7 and Figure 8 are both structural diagrams of electronic devices shown according to an exemplary embodiment. The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

[0085] Figure 7 This is a schematic structural diagram of a server provided by an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input - output interface, and a communication bus. Among them, the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the relevant steps in the data processing disclosed in any of the foregoing embodiments.

[0086] In this embodiment, the power supply is used to provide working voltage for each hardware device on the server; The communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed here; The input - output interface is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0087] In addition, as a carrier for resource storage, the memory can be a read - only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon include an operating system, a computer program, and data, etc., and the storage method can be short - term storage or permanent storage.

[0088] Among them, the operating system is used to manage and control each hardware device and computer program on the server, so as to realize the operation and processing of data in the memory by the processor. It can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the data processing method disclosed in any of the foregoing embodiments, the computer program can further include a computer program that can be used to complete other specific tasks. In addition to data such as update information of the application program, the data can also include data such as developer information of the application program.

[0089] Figure 8 It is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal may specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.

[0090] Generally, the terminal in this embodiment includes: a processor and a memory.

[0091] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0092] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory is at least used to store the following computer programs. After the computer programs are loaded and executed by the processor, the relevant steps in the data processing method executed by the terminal side disclosed in any of the foregoing embodiments can be implemented. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, update information of the application program.

[0093] In some embodiments, the terminal may further include a display screen, an input / output interface, a communication interface, sensors, a power supply, and a communication bus.

[0094] Those skilled in the art can understand that Figure 8 the structure shown in does not constitute a limitation on the terminal, and it may include more or fewer components than those shown in the figure.

[0095] Next, a non-volatile storage medium provided by an embodiment of the present application will be introduced. The non-volatile storage medium described below can be referred to each other with other embodiments described in this article.

[0096] A non-volatile storage medium is used to store a computer program. When the computer program is executed by a processor, the data processing method disclosed in the foregoing embodiment is implemented. Among them, the non-volatile storage medium is a computer-readable non-volatile storage medium. As a carrier for storing resources, it may be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon include an operating system, a computer program, and data, etc., and the storage method may be temporary storage or permanent storage.

[0097] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: select a resource module required for running a target task in a target system; determine an initial core allocation plan for the resource module based on historical core allocation information; optimize the initial core allocation plan according to the simulation operation data of the initial core allocation plan to obtain an available core allocation plan; execute the target task in the target system according to the available core allocation plan.

[0098] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: determining the network card status information, storage end status information, and processor status information in the target system; selecting the network card resources, storage end resources, and processor resources required for running the target task in the target system according to the network card status information, storage end status information, processor status information, as well as the task type, task data volume, and processing duration of the target task; and using the network card resources, storage end resources, and processor resources as a resource module.

[0099] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: sorting each network card in the target system according to the network card status information to obtain a network card sequence; sorting each storage end in the target system according to the storage end status information to obtain a storage end sequence; sorting each processor in the target system according to the processor status information to obtain a processor sequence; and selecting the network card resources, storage end resources, and processor resources respectively in the network card sequence, storage end sequence, and processor sequence according to the task type, task data volume, and processing duration of the target task.

[0100] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: allocating the number of processor cores to the resource module from multiple dimensions according to the historical core allocation information to obtain multiple optional allocation schemes; the multiple dimensions include at least two of instruction set simplification, queue optimization, interrupt balance, and kernel parameter tuning; and selecting the one with the highest processing efficiency as the initial core allocation scheme among the multiple optional allocation schemes.

[0101] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: optimizing the number and content of instructions executed by a single core from the dimension of instruction set simplification according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0102] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: optimizing the queue depth of a single core from the dimension of queue optimization according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0103] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps can be specifically implemented: optimizing the interrupt time consumption of a single core from the dimension of interrupt balance according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

[0104] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: According to the historical core allocation information, optimize the frequency and / or power consumption of a single core from the dimension of kernel parameter tuning, and determine the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

[0105] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: Simulate and run the initial core allocation scheme in the target system to obtain simulation operation data; According to the simulation operation data, adjust the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the initial core allocation scheme to optimize the initial core allocation scheme.

[0106] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: Determine the first data volume processed by the network card resources and the second data volume processed by the storage end resources in the initial core allocation scheme according to the simulation operation data; If the first data volume is greater than the second data volume and the difference between the first data volume and the second data volume is greater than the preset difference threshold, then make the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage end resources.

[0107] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: Replace the initial core allocation scheme with the adjusted initial core allocation scheme, and execute the step of simulating and running the initial core allocation scheme in the target system to obtain simulation operation data, so as to optimize the core allocation scheme again based on the new simulation operation data.

[0108] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: In the case where the difference between the first data volume and the second data volume is not greater than the preset difference threshold, increase the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the current core allocation scheme respectively, and calculate the task processing duration of the target task after each increase; Use the core allocation scheme corresponding to the shortest task processing duration as the available core allocation scheme.

[0109] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: Determine whether the available processor cores in the target system meet the available core allocation scheme; If the available processor cores meet the available core allocation scheme, then execute the step of executing the target task in the target system according to the available core allocation scheme until the target task is completed.

[0110] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: If the number of available processor cores does not meet the available core allocation scheme, execute the target task in the target system, dynamically adjust the available core allocation scheme according to the execution situation, and apply the adjusted core allocation scheme.

[0111] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: Determine the utilization rate of the processor cores occupied by the IO stack of the target task; If the utilization rate is higher than the preset upper limit value, adjust the part of the processor cores corresponding to the storage-side resources in the current core allocation scheme to the IO stack.

[0112] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: If the utilization rate is lower than the preset lower limit value, adjust the processor cores occupied by the IO stack to the storage-side resources in the current core allocation scheme.

[0113] Next, a computer program product provided by an embodiment of the present application will be introduced. A computer program product described below may be referred to each other with other embodiments described herein.

[0114] A computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the data processing method disclosed above are implemented.

[0115] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium for storing a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0116] In this specification, each embodiment is described in a progressive manner. The key points of each embodiment are the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other.

[0117] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of non-volatile storage medium well-known in the technical field.

[0118] In this text, specific examples are used to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A data processing method, characterized in that, Including: Selecting resource modules required for running a target task in a target system; Determining an initial core allocation plan for the resource module based on historical core allocation information; Obtaining simulation operation data of the initial core allocation plan, and optimizing the initial core allocation plan according to the simulation operation data to obtain an available core allocation plan; Executing the target task in the target system according to the available core allocation plan.

2. The method according to claim 1, characterized in that, The selecting resource modules required for running a target task in a target system includes: Determining network card status information, storage end status information, and processor status information in the target system; According to the network card status information, the storage end status information, the processor status information, as well as the task type, task data volume, and processing duration of the target task, selecting network card resources, storage end resources, and processor resources required for running the target task in the target system; Taking the network card resources, the storage end resources, and the processor resources as the resource module.

3. The method according to claim 2, wherein According to the network card status information, the storage end status information, the processor status information, as well as the task type, task data volume, and processing duration of the target task, selecting network card resources, storage end resources, and processor resources required for running the target task in the target system includes: Sorting each network card in the target system according to the network card status information to obtain a network card sequence; Sorting each storage end in the target system according to the storage end status information to obtain a storage end sequence; Sorting each processor in the target system according to the processor status information to obtain a processor sequence; According to the task type, task data volume, and processing duration of the target task, respectively selecting the network card resources, the storage end resources, and the processor resources in the network card sequence, the storage end sequence, and the processor sequence.

4. The method according to claim 1, wherein Determining an initial core allocation plan for the resource module based on historical core allocation information includes: Allocating processor cores to the resource module from multiple dimensions according to the historical core allocation information to obtain multiple optional allocation plans; the multiple dimensions include at least two of instruction set simplification, queue optimization, interrupt balance, and kernel parameter tuning; Selecting the one with the highest processing efficiency among the multiple optional allocation plans as the initial core allocation plan.

5. The method according to claim 4, wherein According to the historical core allocation information, allocating processor cores to the resource module from the dimension of instruction set simplification includes: Optimizing the number and content of instructions executed by a single core from the dimension of instruction set simplification according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

6. The method according to claim 4, characterized in that According to the historical core allocation information, allocating processor cores to the resource module from the dimension of queue optimization includes: Optimizing the queue depth executed by a single core from the dimension of queue optimization according to the historical core allocation information, and determining the number of processor cores that can be allocated to the resource module according to the corresponding optimization results.

7. The method according to claim 4, characterized in that, According to the historical core allocation information, allocating processor cores to the resource module from the dimension of interrupt balance includes: Optimize the interruption time of a single core from the dimension of interruption balance according to the historical core allocation information, and determine the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

8. The method according to claim 4, characterized in that, Allocate the number of processor cores to the resource module from the dimension of kernel parameter tuning according to the historical core allocation information, including: Optimize the frequency and / or energy consumption of a single core from the dimension of kernel parameter tuning according to the historical core allocation information, and determine the number of processor cores that can be allocated to the resource module according to the corresponding optimization result.

9. The method according to claim 1, characterized in that, Optimize the initial core allocation scheme according to the simulated operation data, including: Simulate the operation of the initial core allocation scheme in the target system to obtain the simulated operation data; According to the simulated operation data, adjust the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the initial core allocation scheme to optimize the initial core allocation scheme.

10. The method according to claim 9, characterized in that, According to the simulated operation data, adjust the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the initial core allocation scheme, including: According to the simulated operation data, determine the first data volume processed by the network card resources and the second data volume processed by the storage end resources in the initial core allocation scheme; If the first data volume is greater than the second data volume and the difference between the first data volume and the second data volume is greater than a preset difference threshold, make the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage end resources.

11. The method according to claim 10, wherein After making the number of processor cores corresponding to the network card resources in the initial core allocation scheme less than the number of processor cores corresponding to the storage end resources, it further includes: Replace the initial core allocation scheme with the adjusted initial core allocation scheme, and execute the step of simulating the operation of the initial core allocation scheme in the target system to obtain the simulated operation data, so as to optimize the core allocation scheme again based on the new simulated operation data.

12. The method according to claim 11, wherein It further includes: In the case where the difference between the first data volume and the second data volume is not greater than the preset difference threshold, increase the number of processor cores corresponding to the network card resources, storage end resources, and processor resources in the current core allocation scheme respectively, and calculate the task processing duration of the target task after each increase; Take the core allocation scheme corresponding to the shortest task processing duration as the available core allocation scheme.

13. The method according to any one of claims 1 to 12, characterized in that, It further includes: Judge whether the available processor cores in the target system meet the available core allocation scheme; If the available processor cores meet the available core allocation scheme, execute the step of executing the target task in the target system according to the available core allocation scheme until the target task is completed.

14. The method according to claim 13, wherein It further includes: If the available processor cores do not meet the available core allocation scheme, execute the target task in the target system, dynamically adjust the available core allocation scheme according to the execution situation, and apply the adjusted core allocation scheme.

15. The method according to claim 14, wherein Dynamically adjust the available core allocation scheme according to the execution situation, including: Determine the utilization rate of the processor cores occupied by the IO stack of the target task; If the utilization rate is higher than the preset upper limit value, adjust the part of the processor cores corresponding to the storage - side resources in the current core allocation scheme to the IO stack.

16. The method according to claim 15, wherein If the utilization rate is lower than the preset lower limit value, adjust the processor cores occupied by the IO stack to the storage - side resources in the current core allocation scheme.

17. A data processing device, characterized in that, Comprising: A selection module, configured to select a resource module required for running a target task in a target system; A determination module, configured to determine an initial core allocation scheme for the resource module based on historical core allocation information; An optimization module, configured to obtain simulation operation data of the initial core allocation scheme, and optimize the initial core allocation scheme according to the simulation operation data to obtain an available core allocation scheme; An execution module, configured to execute the target task in the target system according to the available core allocation scheme.

18. An electronic device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 16.

19. A non-volatile storage medium, characterized in that, For saving a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the method according to any one of claims 1 to 16.

Images