A task processing optimization system based on cloud computing and big data

Abstract

The invention provides a task processing optimization system based on cloud computing and big data, which belongs to the field of task optimization and solves the problem of unreasonable task scheduling. Specifically, the system comprises the following steps: a data acquisition module acquires and processes total process parameters and candidate information to obtain adjustable parameters, adjustable processes and target processors; a performance matching module calculates the comprehensive performance index of the target processor and constructs a function matrix TSC of the target processor; a target processor is allocated to threads of the adjustable process according to the comprehensive performance index and the matrix TSC; a task matching module summarizes the allocated threads of the target processor and uses a roulette scheduling algorithm to allocate an execution number to threads of each adjustable process; a continuous monitoring module is used to monitor and process newly added task processes of users; the invention acquires and analyzes user-related task scheduling information, allocates a processor to each task, and improves the execution efficiency of tasks.

Description

Technical Field

[0001] The present invention discloses a task processing optimization system based on cloud computing and big data, and relates to the field of task optimization. Background Art

[0002] The existing systems for task processing optimization have the following deficiencies:

[0003] Network latency: Task processing optimization systems based on cloud computing usually need to access cloud resources through the network; if the network latency is high, it may affect the real-time and efficiency of task processing. Especially when accessing across regions, the network latency problem may be more prominent.

[0004] Management difficulty: Enterprises need to have a professional technical team to manage and maintain the task processing optimization system; at the same time, enterprises also need to establish a complete operation and maintenance process and specifications to ensure the stable operation and efficient management of the system.

[0005] Costs fluctuate with demand: The cost of cloud computing also fluctuates with the demand for task processing; if the demand for task processing suddenly increases, the enterprise needs to pay higher fees to purchase computing resources. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a task processing optimization system based on cloud computing and big data, aiming to solve the problem of unreasonable task scheduling.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: A task processing optimization system based on cloud computing and big data includes:

[0008] Data acquisition module: used to obtain the process identifier PID, process status, memory parameters (memory size occupied by the process) and parent process ID (PPID) corresponding to all processes of the user (in the computer) to obtain the total process parameters; obtain the relevant information of the candidate processors in the cloud computing platform to obtain the candidate information, and determine the candidate processor sequence according to the candidate information; screen the processes according to the process identifier PID and the parent process ID (PPID) to obtain adjustable parameters and adjustable processes; determine the target processor in the candidate processor sequence according to the adjustable parameters;

[0009] Performance matching module: used to obtain the performance index of the target processor and calculate the comprehensive performance index of each target processor; obtain the target processor node number, (processor) serial number and conversion engine information corresponding to each target processor, construct the function matrix of the target processor, recorded as matrix TSC; according to the comprehensive performance index of each target processor, matrix TSC and the number of nodes of each target processor, allocate the target processor to the thread of each adjustable process;

[0010] Task matching module: used to summarize the assigned threads of each target processor and use the roulette scheduling algorithm to allocate the execution number of threads for each adjustable process;

[0011] Continuous monitoring module: used to monitor the progress of the user's newly added tasks and process them (call the data acquisition module, performance matching module and task matching module).

[0012] Furthermore, the workflow of the data acquisition module is as follows:

[0013] Process A1: Use the tasklist command to obtain the PIDs of all user processes, recorded as the search PID; based on the search PID, use the readlink() function to read the process parameters of all processes in the / proc / [pid] / cmdline file to obtain the total process parameters;

[0014] Process parameters include: process identifier PID, process status, memory parameters and PPID;

[0015] Process states include: running state, waiting state, and terminated state;

[0016] Process A2: Obtain relevant information of the candidate processors in the cloud computing platform to obtain the candidate information; determine the candidate processor sequence according to the candidate information; Process A2 includes:

[0017] Process A21: obtaining a processor in the cloud computing platform whose processor status is idle as a processor to be selected; obtaining information of the processor to be selected as information to be selected;

[0018] The candidate information includes: CPU main frequency, number of CPU cores, CPU cache size and CPU process of the candidate processor;

[0019] Process A22: Count the number of processors to be selected, recorded as n; record the CPU main frequency, number of CPU cores, CPU cache size and CPU process of the processors to be selected as oMF, oMN, oMS and oML respectively;

[0020] The candidate information of the first candidate processor: oMF1, oMN1, oMS1 and oML1;

[0021] The candidate information of the second candidate processor: oMF2, oMN2, oMS2 and oML2;

[0022] Similarly, the candidate information of the nth candidate processor is: oMF n 、oMN n 、oMS n and oML n ;

[0023] Process A23: Define calculation formula 11:

[0024] MP(i)=(oMF i *oMi i *oMS i ) / oML i ; Wherein, i represents 1 to n;

[0025] MP(i), oMF i 、oMi i 、oMS i and oML i , respectively represent the CPU comprehensive performance index, CPU main frequency, CPU core number, CPU cache size and CPU process of the i-th candidate processor;

[0026] Substitute the candidate information into the calculation formula 11 to calculate the CPU comprehensive performance indexes MP(1) to MP(n) corresponding to the 1st to nth candidate processors;

[0027] Process A24: Arrange the 1st to nth processors to be selected in descending order of MP(1) to MP(n) to obtain a candidate processor sequence;

[0028] Process A3: In the total process parameters, the process parameters whose process status is running are summarized as set I, and the total number of elements in set I is recorded as z1;

[0029] Summarize the process parameters whose process status is waiting state as set II, and the total number of elements in set II is recorded as z2;

[0030] The process parameters whose process status is the termination state are summarized as set III, and the total number of elements in set III is denoted as z3.

[0031] Furthermore, the subsequent process of process A3 is as follows:

[0032] Process A4: Define judgment formula 12: PPID (Ⅱ) i ∈Ⅰ[PID(Ⅰ)1~PID(Ⅰ) z1 ]; where i represents 1 to z2, PPID(Ⅱ) i Indicates the parent process ID of the i-th process in set II; I [PID (I) 1 ~ PID (I) z1 ] represents the process ID corresponding to the 1st to z1th process parameters in set I;

[0033] Definition formula 13: PPID(Ⅱ) i ∈Ⅲ[PID(Ⅲ)1~PID(Ⅲ) z3 ]; among them, Ⅲ[PID(Ⅲ)1~PID(Ⅲ) z3] represents the process ID corresponding to the 1st to z3th process parameters in set III;

[0034] Process A5: Substitute the process parameters in set II into judgment formula 12 and judgment formula 13, and collect the process parameters that do not satisfy judgment formula 12 and judgment formula 13 as adjustable parameters; count the number of adjustable parameters, recorded as cn;

[0035] Process A6: Read the memory parameters corresponding to the adjustable parameters in the total process parameters, recorded as CS1~CS cn ; Calculate CS1~CS cn The sum of is denoted as ACS;

[0036] Process A7: Get the memory size of the 1st to nth candidate processors in the candidate processor sequence, denoted as ZS1~ZS n ;

[0037] Define the cumulative value AZS (1~j) of the memory size of the candidate processor;

[0038] Wherein, j is a natural number, j≤n; AZS(1~j) represents: the cumulative value of the memory size of the candidate processors from the 1st to the jth in the candidate processor sequence; ZS i Indicates: the memory size of the i-th candidate processor in the candidate processor sequence;

[0039] Process A8: Set the initial value of j to 2, and calculate (the cumulative value of the memory size of the 1st to jth candidate processors) AZS(1~j);

[0040] If AZS(1~j)≥ACS, then the 1st to jth candidate processors are used as target processors;

[0041] If AZS(1~j)<ACS, then j is increased by 1, and AZS(1~j) (the accumulated value of the memory size of the 1st to the jth candidate processor) is calculated again, and the size of AZS(1~j) is compared with ACS until AZS(1~j)≥ACS.

[0042] Process A9: Among all the processes, the processes corresponding to the adjustable parameters are summarized as adjustable processes and enter the performance matching module.

[0043] Furthermore, the workflow of the performance matching module is as follows:

[0044] The workflow of the performance matching module is as follows:

[0045] Process B1: Let the number of target processors be m; calculate the comprehensive performance indexes P1 to P1 corresponding to the 1st to mth target processors. m ;

[0046] Process B2: Obtain the target processor node number, serial number and conversion engine information corresponding to each target processor, and construct the function matrix of the target processor, which is recorded as matrix TSC; Process B2 includes:

[0047] Process B21: Obtain the target processor node numbers of the 1st to mth target processors, denoted as T1, T2~T m ;

[0048] Obtain the serial numbers and conversion engine information corresponding to the first to mth target processors, and obtain the conversion service types corresponding to the first to mth target processors according to the serial numbers and conversion engine information of the target processors;

[0049] Process B22: Count the number of all conversion service types, denoted as k; k ≥ m;

[0050] Define a set S to represent all the conversion services of the 1st to mth target processors, set S = {S1, S2 ~ S k};

[0051] Process B23: construct a k×m two-dimensional matrix oTSC with all matrix elements being 0;

[0052] According to the correspondence between the conversion service types of the 1st to the mth target processors in the set S, the two-dimensional matrix oTSC is filled to obtain the matrix TSC;

[0053] Process B3: The number of adjustable processes is recorded as Rn; according to the comprehensive performance index of each target processor, the matrix TSC and the number of nodes of each target processor, the target processor is allocated to the 1st to Rnth adjustable processes.

[0054] Furthermore, the specific working process of the process B11 is as follows:

[0055] Process B11: Obtaining the performance indexes of the 1st to mth target processors;

[0056] The performance index includes: the CPU frequency, number of CPU cores, RAM size and RAM frequency of the target processor;

[0057] The CPU main frequency, number of CPU cores, RAM size and RAM main frequency of the target processor are denoted as MF, MN, RS and RF respectively;

[0058] The quadruple {MF1, MN1, RS1, RF1} corresponding to the first target processor;

[0059] The quadruple corresponding to the second target processor is {MF2, MN2, RS2, RF2};

[0060] By analogy, the quadruple {MF m ,MN m ,RS m ,RF m};

[0061] Process B12: MF1~MF m , sorted in descending order, we get index sequence 1;

[0062] MN1~MN m , arranged in descending order, we get index sequence 2;

[0063] Set RS1~RS m , arranged in descending order, we get index sequence 3;

[0064] RF1~RF m , arranged in descending order, we get index sequence 4;

[0065] Process B13: Read the first and last items corresponding to index sequences 1 to 4 respectively, and record them as max(MF), min(MF); max(MN), min(MN); max(RS), min(RS); max(RF), min(RF);

[0066] Define conversion formulas 21 to 24 to map the performance indexes corresponding to the 1st to mth target processors into the interval [0,1];

[0067] Conversion formula 21: Among them, MF i and They represent the CPU main frequency and the processing value of the CPU main frequency corresponding to the i-th target processor respectively;

[0068] Conversion formula 22: Among them, MN i and They represent the number of CPU cores and the processing value of the number of CPU cores corresponding to the i-th target processor respectively;

[0069] Conversion formula 23: Among them, RS i and They represent the RAM size and the processing value of the RAM size corresponding to the i-th target processor respectively;

[0070] Conversion formula 24: Among them, RF i and They represent the RAM main frequency and the processing value of the RAM main frequency corresponding to the i-th target processor respectively;

[0071] Among them, i in conversion formulas 21 to 24 represents 1 to m.

[0072] Furthermore, the subsequent process of process B13 is as follows:

[0073] Process B14: MF1 to MF corresponding to the 1st to mth target processors m MN1~MN m , RS1~RS m and RF1~RF m , respectively, into the conversion formulas 21 to 24, and calculate the CPU main frequency (average) processing values ​​corresponding to the 1st to mth target processors

[0074] CPU core number processing value

[0075] RAM size processing value

[0076] RAM main frequency processing value

[0077] Process B15: Calculate the comprehensive performance index P1 of the first target processor;

[0078] Process B151: Process the CPU main frequency corresponding to the first target processor CPU core number processing value RAM size processing value and RAM frequency processing value Denoted as a1, a2, a3 and a4 respectively;

[0079] Process B152: Use the MaCh.random() function to randomly generate weight parameters μ1, μ2, μ3, and μ4 in the interval (0,1); the constraints satisfied by μ1, μ2, μ3, and μ4 are as follows:

[0080] μ1+μ2+μ3+μ4=1;

[0081] μ1 / a1=μ2 / a2=μ3 / a3=μ4 / a4=1;

[0082] Process B153: Read the performance indexes MF1, MN1, RS1 and RF1 corresponding to the first target processor;

[0083] Calculate the comprehensive performance index P1 of the first target processor. The calculation formula of P1 is as follows:

[0084] P1=MF1*μ1+MN1*μ2+RS1*μ3+RF1*μ4;

[0085] Process B16: Repeat the same process of calculating the comprehensive performance index P1 of the first target processor to calculate the comprehensive performance indexes P2 to P3 corresponding to the second to mth target processors. m .

[0086] Furthermore, the specific process of process B3 is as follows:

[0087] Process B31: define a 1×m one-dimensional matrix P to represent the comprehensive performance of the 1st to mth target processors; the matrix P = [1 / P1 1 / P2~1 / Pm]; where m represents the number of target processors;

[0088] Process B32: Allocate a target processor for the first adjustable process;

[0089] Process B33: Repeat the same process of allocating target processors for the first adjustable process, and allocate target processors for the second to Rnth adjustable processes.

[0090] Furthermore, the specific process of the process B32 is as follows:

[0091] Process B321: in the virtual machine, simulate and execute the first adjustable process;

[0092] In the virtual machine, use Toolhelp32 snapshot to obtain all threads of the first adjustable process, count the total number of threads, and record it as tt;

[0093] Use Thread32First and Thread32Next functions to obtain the operation codes and operation times corresponding to the 1st to ttth threads;

[0094] Process B322: according to the operation codes of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task service matrix of the first adjustable process, which is recorded as matrix F;

[0095] According to the operation time of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task time matrix of the first adjustable process, denoted as matrix V;

[0096] Process B323: Multiply the matrix V by the matrix P to obtain a k×m two-dimensional matrix, denoted as matrix ETC;

[0097] The elements in the matrix ETC are denoted as ETC(j,i); where j represents 1 to k and i represents 1 to m; ETC(j,i) represents: the processing time of the jth thread on the ith processor in the first adjustable process;

[0098] Process B324: Obtain the target processor node numbers T1, T2~T1 of the 1st to mth target processors m;

[0099] Transpose the matrix TSC to get the matrix TSC (T) ;

[0100] Combine the matrix ETC with the matrix TSC (T) Multiply them together to obtain a k×k two-dimensional matrix, denoted as matrix PTC;

[0101] The elements in the matrix ETC are denoted as ETC(y,x), where x and y both represent 1 to k. PTC(y,x) means: assign the yth thread in the first adjustable process to the processor node with the number T. x in the target processor;

[0102] Process B325: traverse the elements in the matrix ETC: ETC(1,1)~ETC(m,m);

[0103] Define judgment formula 25: ETC(m,n)≠null∩ETC(m,n)≠0; where m and m both represent 1 to k, and ETC(m,n) represents the element in the mth row and mth column of the matrix ETC;

[0104] Process B326: Determine the target processor of the first thread; set the initial value of m to 1, and set the representation range of n to 1 to k;

[0105] Substitute ETC(1,1) to ETC(1,m) in the matrix ETC into judgment formula 25; mark the elements that satisfy judgment formula 25 as the elements to be selected;

[0106] Select the candidate element with the smallest element value as the determination element, read the target processor corresponding to the determination element as the allocation processor; allocate the first thread to the allocation processor;

[0107] Process B327: Repeat the same process of determining the target processor for the first thread, and assign target processors to the second to ttth threads in the first adjustable process.

[0108] Compared with the prior art, the present invention has the following beneficial effects:

[0109] Distributed computing and parallel processing: The present invention can utilize the distributed computing capabilities of the cloud computing platform to decompose large-scale tasks (i.e., processes) into multiple subtasks (i.e., threads) and process them in parallel on multiple computing nodes; this computing mode improves the efficiency and speed of task processing, and is particularly suitable for processing massive data and tasks with high computational complexity.

[0110] Elastic expansion: The present invention can automatically adjust computing resources according to the task load to achieve elastic expansion of resources; when the task volume increases, the system can quickly increase computing resources to meet the demand, ensuring the continuity and efficiency of task processing.

[0111] Reduce hardware and maintenance costs: By transferring task processing to the cloud, the present invention eliminates the need for enterprises to purchase and maintain expensive hardware equipment, thereby reducing hardware costs and operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0113] Figure 1 It is a schematic diagram of the system of the present invention;

[0114] Figure 2 Optimize the schematic diagram for the task of the present invention;

[0115] Figure 3 This is a schematic diagram of data processing of the present invention. DETAILED DESCRIPTION

[0116] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0117] See also Figure 1 and Figure 2 A task processing optimization system based on cloud computing and big data includes: a data acquisition module, a performance matching module, a task matching module, a continuous monitoring module, a database and a server; wherein the data acquisition module, the performance matching module, the task matching module and the continuous monitoring module are connected to the database and the server respectively.

[0118] It should be noted that the "task" in the present invention means a "computer process", or "process" for short.

[0119] Data acquisition module: used to obtain the process identifier PID, process status, memory parameters (memory size occupied by the process) and parent process ID (PPID) corresponding to all processes of the user (in the computer) to obtain the total process parameters; obtain the relevant information of the candidate processors in the cloud computing platform to obtain the candidate information, and determine the candidate processor sequence according to the candidate information; screen the processes according to the process identifier PID and the parent process ID (PPID) to obtain adjustable parameters and adjustable processes; determine the target processor in the candidate processor sequence according to the adjustable parameters;

[0120] Process A: The workflow of the data acquisition module is as follows:

[0121] Process A1: Use the tasklist command to obtain the PIDs of all processes of the user (in the computer), which are recorded as the search PIDs; according to the search PIDs, use the readlink() function to read the process parameters of all processes in the / proc / [pid] / cmdline file to obtain the total process parameters;

[0122] The process parameters (used in the present invention) include: process identifier PID, process status, memory parameters (memory size occupied by the process) and parent process ID (PPID);

[0123] The process states (used in the present invention) include: running state, waiting state, and terminated state;

[0124] It should be noted that the "data acquisition module" of the present invention only involves process scheduling, not thread scheduling, in the process of acquiring process parameters, to ensure that the life cycle of each process is continuous and complete; therefore, the "terminated state" in the data acquisition module indicates that a process is completely executed.

[0125] Process A2: Obtain relevant information of the candidate processors in the cloud computing platform to obtain the candidate information; determine the candidate processor sequence according to the candidate information;

[0126] Process A21: obtaining a processor in the cloud computing platform whose processor status is idle as a processor to be selected; obtaining information of the processor to be selected as information to be selected;

[0127] The candidate information includes: CPU main frequency, number of CPU cores, CPU cache size and CPU process of the candidate processor;

[0128] Process A22: Count the number of processors to be selected, recorded as n; record the CPU main frequency, number of CPU cores, CPU cache size and CPU process of the processors to be selected as oMF, oMN, oMS and oML respectively;

[0129] The candidate information of the first candidate processor: oMF1, oMN1, oMS1 and oML1;

[0130] The candidate information of the second candidate processor: oMF2, oMN2, oMS2 and oML2;

[0131] Similarly, the candidate information of the nth candidate processor is: oMF n 、oMN n 、oMS n and oML n ;

[0132] Process A23: Define (CPU comprehensive performance) calculation formula 11:

[0133] MP(i)=(oMFi *oMi i *oMS i ) / oML i ; Wherein, i represents 1 to n;

[0134] MP(i), oMF i 、oMi i 、oMS i and oML i , respectively represent the CPU comprehensive performance index, CPU main frequency, CPU core number, CPU cache size and CPU process of the i-th candidate processor;

[0135] Substitute the candidate information (of the 1st to nth candidate processors) into the calculation formula 11 to calculate the CPU comprehensive performance indexes MP(1) to MP(n) corresponding to the 1st to nth candidate processors;

[0136] Process A24: Arrange the 1st to nth processors to be selected in descending order of MP(1) to MP(n) to obtain a candidate processor sequence;

[0137] Process A3: In the total process parameters, the process parameters whose process status is running are summarized as set I, and the total number of elements in set I is recorded as z1;

[0138] Summarize the process parameters whose process status is waiting state as set II, and the total number of elements in set II is recorded as z2;

[0139] Summarize the process parameters whose process status is the termination state as set III, and the total number of elements in set III is recorded as z3;

[0140] Process A4: Define judgment formula 12: PPID (Ⅱ) i ∈Ⅰ[PID(Ⅰ)1~PID(Ⅰ) z1 ]; where i represents 1 to z2, PPID(Ⅱ) i Indicates the parent process ID of the i-th process in set II; I [PID (I) 1 ~ PID (I) z1 ] represents the process ID corresponding to the 1st to z1th process parameters in set I;

[0141] Definition formula 13: PPID(Ⅱ) i ∈Ⅲ[PID(Ⅲ)1~PID(Ⅲ) z3 ]; among them, Ⅲ[PID(Ⅲ)1~PID(Ⅲ) z3 ] represents the process ID corresponding to the 1st to z3th process parameters in set III;

[0142] Process A5: Substitute the process parameters in set II into judgment formula 12 and judgment formula 13, and collect the process parameters that do not satisfy judgment formula 12 and judgment formula 13 as adjustable parameters; count the number of adjustable parameters, recorded as cn;

[0143] Process A6: Read the memory parameters corresponding to the adjustable parameters in the total process parameters, recorded as CS1~CS cn ; Calculate CS1~CS cn The sum of is denoted as ACS;

[0144] Process A7: Get the memory size (RAM size + remaining ROM size) of the first to nth candidate processors in the candidate processor sequence, recorded as ZS1~ZS n ;

[0145] Define the cumulative value AZS (1~j) of the memory size of the candidate processor;

[0146] Wherein, j is a natural number, j≤n; AZS(1~j) represents: the cumulative value of the memory size of the candidate processors from the 1st to the jth in the candidate processor sequence; ZS i Indicates: the memory size of the i-th candidate processor in the candidate processor sequence;

[0147] Process A8: Set the initial value of j to 2, and calculate (the cumulative value of the memory size of the 1st to jth candidate processors) AZS(1~j);

[0148] If AZS(1~j)≥ACS, then the 1st to jth candidate processors are used as target processors;

[0149] If AZS(1~j)<ACS, then j is increased by 1, and AZS(1~j) (the accumulated value of the memory size of the 1st to the jth candidate processor) is calculated again, and the size of AZS(1~j) is compared with ACS until AZS(1~j)≥ACS.

[0150] Process A9: Among all the processes, the processes corresponding to the adjustable parameters are summarized as adjustable processes and enter the performance matching module.

[0151] Performance matching module: used to obtain the performance index of the target processor and calculate the comprehensive performance index of each target processor; obtain the target processor node number, (processor) serial number and conversion engine information corresponding to each target processor, construct the function matrix of the target processor, recorded as matrix TSC; allocate the target processor to each adjustable process according to the comprehensive performance index of each target processor, matrix TSC and the number of nodes of each target processor;

[0152] Process B: Please refer to Figure 3,The workflow of the performance matching module is as follows:

[0153] Process B1: Let the number of target processors be m; calculate the comprehensive performance indexes P1 to P1 corresponding to the 1st to mth target processors. m ;

[0154] Process B11: Obtaining the performance indexes of the 1st to mth target processors;

[0155] The performance index includes: the CPU frequency, number of CPU cores, RAM size and RAM frequency of the target processor;

[0156] The CPU main frequency, number of CPU cores, RAM size and RAM main frequency of the target processor are denoted as MF, MN, RS and RF respectively;

[0157] The quadruple {MF1, MN1, RS1, RF1} corresponding to the first target processor;

[0158] The quadruple corresponding to the second target processor is {MF2, MN2, RS2, RF2};

[0159] By analogy, the quadruple {MF m ,MN m ,RS m ,RF m};

[0160] Process B12: MF1~MF m , arranged in descending order, we get the (performance) index sequence 1;

[0161] MN1~MN m , arranged in descending order, to obtain (performance) index sequence 2;

[0162] Set RS1~RS m , arranged in descending order, we get the (performance) index sequence 3;

[0163] RF1~RF m , arranged in descending order, we get the (performance) index sequence 4;

[0164] Process B13: Read the first and last items corresponding to index sequences 1 to 4 respectively, and record them as max(MF), min(MF); max(MN), min(MN); max(RS), min(RS); max(RF), min(RF);

[0165] Define conversion formulas 21 to 24 to map the performance indexes corresponding to the 1st to mth target processors into the interval [0,1];

[0166] Conversion formula 21: Among them, MF i and They represent the CPU main frequency and the processing value of the CPU main frequency (averaged) corresponding to the i-th target processor respectively;

[0167] Conversion formula 22: Among them, MN i and They represent the number of CPU cores and the processed value of the number of CPU cores (averaged) corresponding to the i-th target processor respectively;

[0168] Conversion formula 23: Among them, RS i and They represent the RAM size and the processed value of RAM size (averaged) corresponding to the i-th target processor respectively;

[0169] Conversion formula 24: Among them, RF i and They represent the RAM main frequency and the processing value of RAM main frequency (averaged) corresponding to the i-th target processor respectively;

[0170] Wherein, i in conversion formulas 21 to 24 represents 1 to m;

[0171] Process B14: MF1 to MF corresponding to the 1st to mth target processors m MN1~MN m , RS1~RS m and RF1~RF m , respectively, into the conversion formulas 21 to 24, and calculate the CPU main frequency (average) processing values ​​corresponding to the 1st to mth target processors

[0172] Number of CPU cores (averaged)

[0173] RAM size (average) processing value

[0174] RAM main frequency (average) processing value

[0175] Process B15: Calculate the comprehensive performance index P1 of the first target processor;

[0176] Process B151: Process the CPU main frequency (average) corresponding to the first target processor Number of CPU cores (averaged) RAM size (average) processing value and RAM main frequency (average) processing value Denoted as a1, a2, a3 and a4 respectively;

[0177] Process B152: Use the MaCh.random() function to randomly generate weight parameters μ1, μ2, μ3, and μ4 in the interval (0,1); the constraints satisfied by μ1, μ2, μ3, and μ4 are as follows:

[0178] μ1+μ2+μ3+μ4=1;

[0179] μ1 / a1=μ2 / a2=μ3 / a3=μ4 / a4=1;

[0180] Process B153: Read the performance indexes MF1, MN1, RS1 and RF1 corresponding to the first target processor;

[0181] Calculate the comprehensive performance index P1 of the first target processor. The calculation formula of P1 is as follows:

[0182] P1=MF1*μ1+MN1*μ2+RS1*μ3+RF1*μ4;

[0183] Process B16: Repeat the same process of calculating the comprehensive performance index P1 of the first target processor to calculate the comprehensive performance indexes P2 to P3 corresponding to the second to mth target processors. m ;

[0184] Process B2: Obtain the target processor node number, (processor) serial number and conversion engine information corresponding to each target processor, and construct the function matrix of the target processor, which is recorded as matrix TSC;

[0185] Process B21: Obtain the target processor node numbers of the 1st to mth target processors, denoted as T1, T2~T m ;

[0186] Obtain the (processor) serial numbers and conversion engine information corresponding to the first to mth target processors, and obtain the conversion service types corresponding to the first to mth target processors according to the (first to mth) target processor serial numbers and conversion engine information;

[0187] Process B22: Count the number of all (type) conversion service types, denoted as k; k ≥ m; (a target processor is installed with at least one or more conversion engines, so the number of all conversion service types of the 1st to mth target processors ≤ the total number of target processors)

[0188] Define a set S to represent all the conversion services of the 1st to mth target processors, set S = {S1, S2 ~ S k};

[0189] Process B23: Refer to Table 1 (matching table of target processor and conversion service), construct a two-dimensional matrix oTSC of k×m with all matrix elements being 0;

[0190] (Explanation of “√” and “×” in Table 1:

[0191] Take the second column in Table 1 as an example; if the first target processor corresponding to the second column (T1) contains the conversion service type S1, then fill in "√" in the space of (2,2), otherwise fill in "×"; and so on, compare whether the first target processor contains the conversion service types S2~S k , fill in “√” or “×” in the spaces from (2,3) to (2, k+1);

[0192] Repeat the process of filling all the blanks in the second column of Table 1 with "√" or "×", and the second to mth target processors and conversion service types S1 to S k According to the corresponding relationship, fill in the blanks in the 3rd to (m+1)th columns with “√” or “×”;)

[0193] According to the corresponding relationship of the conversion service types of the 1st to the mth target processors in the set S, fill in the two-dimensional matrix oTSC (that is, in the two-dimensional matrix oTSC, the positions marked as "√" in the underlined area (k×m) in Table 1 are filled with "1", and the positions marked as "×" are filled with "0") to obtain the matrix TSC;

[0194] Table 1 corresponds to the matrix TSC, (For example only)

[0195]

[0196]

[0197] Table 1 (matching table of target processor and conversion service) (for example only)

[0198] It should be noted that the "conversion engine" in the present invention refers to a component in the target processor used to perform data or instruction conversion tasks. Common conversion engines include: "virtualization engine" and "data flow accelerator";

[0199] "Conversion service" refers to the functions of performing a series of data processing or instruction conversion inside the target processor. Common types of conversion services include: virtualization services, data encryption and decryption services, data compression and decompression services, instruction conversion services, etc.

[0200] Process B3: The number of adjustable processes is recorded as Rn; according to the comprehensive performance index of each target processor, the matrix TSC and the number of nodes of each target processor, the target processor is allocated to the 1st to Rnth adjustable processes;

[0201] Process B31: define a 1×m one-dimensional matrix P to represent the comprehensive performance of the 1st to mth target processors; the matrix P = [1 / P1 1 / P2~1 / Pm]; where m represents the number of target processors;

[0202] Process B32: The number of adjustable processes is recorded as Rn; a target processor is allocated to the first adjustable process;

[0203] Process B321: in the virtual machine, simulate and execute the first adjustable process;

[0204] In the virtual machine, use Toolhelp32 snapshot to obtain all threads of the first adjustable process, and count the total number of threads (of the first adjustable process), recorded as tt;

[0205] Use Thread32First and Thread32Next functions to obtain the operation codes and operation times corresponding to the 1st to ttth threads;

[0206] Process B322: according to the operation codes of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task service matrix of the first adjustable process, which is recorded as matrix F;

[0207] According to the operation time of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task time matrix of the first adjustable process, denoted as matrix V;

[0208] Process B323: Multiply the matrix V by the matrix P to obtain a k×m two-dimensional matrix, denoted as matrix ETC;

[0209] The elements in the matrix ETC are denoted as ETC(j,i); where j represents 1 to k and i represents 1 to m; ETC(j,i) represents: the processing time of the jth thread on the ith processor in the first adjustable process;

[0210] Process B324: Obtain the target processor node numbers T1, T2~T1 of the 1st to mth target processors m ;

[0211] Transpose the matrix TSC to get the matrix TSC (T) (The matrix TSC is k×m, the matrix TSC (T) is m×k);

[0212] Combine the matrix ETC with the matrix TSC (T)Multiply them together to obtain a k×k two-dimensional matrix, denoted as matrix PTC;

[0213] The elements in the matrix ETC are denoted as ETC(y,x), where x and y both represent 1 to k. PTC(y,x) means: assign the yth thread in the first adjustable process to the processor node with the number T. x in the target processor (processing);

[0214] Process B325: traverse the elements in the matrix ETC: ETC(1,1)~ETC(m,m);

[0215] Define judgment formula 25: ETC(m,n)≠null∩ETC(m,n)≠0; where m and m both represent 1 to k, and ETC(m,n) represents the element in the mth row and mth column of the matrix ETC;

[0216] Process B326: Determine the target processor of the first thread; set the initial value of m to 1, and set the representation range of n to 1 to k;

[0217] Substitute ETC(1,1) to ETC(1,m) in the matrix ETC into judgment formula 25; mark the elements that satisfy judgment formula 25 as the elements to be selected;

[0218] Select the candidate element with the smallest element value as the determination element, read the target processor corresponding to the determination element as the allocation processor; allocate the first thread (of the first adjustable process) to the allocation processor;

[0219] Process B327: Repeat the same process of determining the target processor for the first thread (process B326), and allocate target processors for the second to ttth threads in the first adjustable process;

[0220] Process B33: Repeat the same process of allocating target processors for the first adjustable process, and allocate target processors for the second to Rnth adjustable processes.

[0221] Task matching module: used to summarize the assigned threads of each target processor and use the roulette scheduling algorithm to allocate the execution number for the threads of each adjustable process.

[0222] Continuous monitoring module: used to monitor the progress of the user's newly added tasks and process them (call the data acquisition module, performance matching module and task matching module).

[0223] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technical personnel in this field according to actual conditions. For example, if there are weight coefficients and proportional coefficients, their set sizes are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the size of the weight coefficient and the proportional coefficient, it is sufficient as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0224] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A task processing optimization system based on cloud computing and big data, characterized in that: The optimization system includes: Data acquisition module: used to obtain the process identifier PID, process status, memory parameters and parent process ID corresponding to all user processes to obtain total process parameters; obtain relevant information of the candidate processors in the cloud computing platform to obtain the candidate information, and determine the candidate processor sequence according to the candidate information; screen the processes according to the process identifier PID and the parent process ID to obtain adjustable parameters and adjustable processes; determine the target processor in the candidate processor sequence according to the adjustable parameters; Performance matching module: used to obtain the performance index of the target processor and calculate the comprehensive performance index of each target processor; obtain the target processor node number, serial number and conversion engine information corresponding to each target processor, construct the function matrix of the target processor, recorded as matrix TSC; according to the comprehensive performance index of each target processor, matrix TSC and the number of nodes of each target processor, allocate the target processor to the thread of each adjustable process; Task matching module: used to summarize the assigned threads of each target processor and use the roulette scheduling algorithm to allocate the execution number of threads for each adjustable process; Continuous monitoring module: used to monitor the progress of newly added tasks by users and process them; The workflow of the data acquisition module is as follows: Process A1: Use the tasklist command to obtain the PIDs of all user processes, recorded as the search PID; based on the search PID, use the readlink() function to read the process parameters of all processes in the / proc / [pid] / cmdline file to obtain the total process parameters; Process parameters include: process identifier PID, process status, memory parameters and PPID; Process states include: running state, waiting state, and terminated state; Process A2: Obtain relevant information of the candidate processors in the cloud computing platform to obtain the candidate information; determine the candidate processor sequence according to the candidate information; Process A2 includes: Process A21: obtaining a processor in the cloud computing platform whose processor status is idle as a processor to be selected; obtaining information of the processor to be selected as information to be selected; The candidate information includes: CPU main frequency, number of CPU cores, CPU cache size and CPU process of the candidate processor; Process A22: Count the number of processors to be selected, recorded as n; record the CPU main frequency, number of CPU cores, CPU cache size and CPU process of the processors to be selected as oMF, oMN, oMS and oML respectively; The candidate information of the first candidate processor: oMF1, oMN1, oMS1 and oML1; The candidate information of the second candidate processor: oMF2, oMN2, oMS2 and oML2; Similarly, the candidate information of the nth candidate processor is: oMF n 、oMN n 、oMS n and oML n ; Process A23: Define calculation formula 11: MP(i)=(oMF i *oMi i *oMS i ) / oML i ; Wherein, i represents 1 to n; MP(i), oMF i 、oMi i 、oMS i and oML i , respectively represent the CPU comprehensive performance index, CPU main frequency, CPU core number, CPU cache size and CPU process of the i-th candidate processor; Substitute the candidate information into the calculation formula 11 to calculate the CPU comprehensive performance indexes MP(1) to MP(n) corresponding to the 1st to nth candidate processors; Process A24: Arrange the 1st to nth processors to be selected in descending order of MP(1) to MP(n) to obtain a candidate processor sequence; Process A3: In the total process parameters, the process parameters whose process status is running are summarized as set I, and the total number of elements in set I is recorded as z1; Summarize the process parameters whose process status is waiting state as set II, and the total number of elements in set II is recorded as z2; Summarize the process parameters whose process status is the termination state as set III, and the total number of elements in set III is recorded as z3; The subsequent process of process A3 is as follows: Process A4: Define judgment formula 12: PPID (Ⅱ) i ∈Ⅰ[PID(Ⅰ)1~PID(Ⅰ) z1 ]; where i represents 1 to z2, PPID (Ⅱ) i Indicates the parent process ID of the i-th process in set II; I [PID (I) 1 ~ PID (I) z1 ] represents the process ID corresponding to the 1st to z1th process parameters in set I; Definition formula 13: PPID (II) i ∈Ⅲ[PID(Ⅲ)1~PID(Ⅲ) z3 ]; Among them, Ⅲ[PID (Ⅲ) 1~PID (Ⅲ) z3 ] represents the process ID corresponding to the 1st to z3th process parameters in set III; Process A5: Substitute the process parameters in set II into judgment formula 12 and judgment formula 13, and collect the process parameters that do not satisfy judgment formula 12 and judgment formula 13 as adjustable parameters; count the number of adjustable parameters, recorded as cn; Process A6: Read the memory parameters corresponding to the adjustable parameters in the total process parameters, recorded as CS1~CS cn ; Calculate CS1~CS cn The sum of is denoted as ACS; Process A7: Get the memory size of the 1st to nth candidate processors in the candidate processor sequence, denoted as ZS1~ZS n ; Define the cumulative value AZS (1~j) of the memory size of the candidate processor; ; Where j is a natural number, j≤n; AZS(1~j) represents: the cumulative value of the memory size of the candidate processors from the 1st to the jth in the candidate processor sequence; ZS i Indicates: the memory size of the i-th candidate processor in the candidate processor sequence; Process A8: Set the initial value of j to 2, and calculate the cumulative value AZS (1-j) of the memory size of the 1st to jth candidate processors; If AZS (1~j) ≥ ACS, the 1st to jth candidate processors are used as target processors; If AZS (1-j) < ACS, then j is increased by 1, and the cumulative value AZS (1-j) of the memory size of the 1st to jth candidate processors is calculated again, and the size of AZS (1-j) is compared with ACS until AZS (1-j) ≥ ACS. Process A9: Among all the processes, the processes corresponding to the adjustable parameters are summarized as adjustable processes and enter the performance matching module.

2. The task processing optimization system based on cloud computing and big data according to claim 1 is characterized in that: The workflow of the performance matching module is as follows: The workflow of the performance matching module is as follows: Process B1: Let the number of target processors be m; calculate the comprehensive performance indexes P1 to P1 corresponding to the 1st to mth target processors. m ; Process B2: Obtain the target processor node number, serial number and conversion engine information corresponding to each target processor, and construct the function matrix of the target processor, which is recorded as matrix TSC; Process B2 includes: Process B21: Obtain the target processor node numbers of the 1st to mth target processors, denoted as T1, T2~T m ; Obtain the serial numbers and conversion engine information corresponding to the first to mth target processors, and obtain the conversion service types corresponding to the first to mth target processors according to the serial numbers and conversion engine information of the target processors; Process B22: Count the number of all conversion service types, denoted as k; k ≥ m; Define a set S to represent all the conversion services of the 1st to mth target processors, set S = {S1, S2 ~ S k }; Process B23: construct a k×m two-dimensional matrix oTSC with all matrix elements being 0; According to the correspondence between the conversion service types of the 1st to the mth target processors in the set S, the two-dimensional matrix oTSC is filled to obtain the matrix TSC; Process B3: The number of adjustable processes is recorded as Rn; according to the comprehensive performance index of each target processor, the matrix TSC and the number of nodes of each target processor, the target processor is allocated to the 1st to Rnth adjustable processes.

3. The task processing optimization system based on cloud computing and big data according to claim 2 is characterized in that: The specific working process of process B1 is as follows: Process B11: Obtaining the performance indexes of the 1st to mth target processors; The performance index includes: the CPU frequency, number of CPU cores, RAM size and RAM frequency of the target processor; The CPU main frequency, number of CPU cores, RAM size and RAM main frequency of the target processor are denoted as MF, MN, RS and RF respectively; The quadruple {MF1, MN1, RS1, RF1} corresponding to the first target processor; The quadruple corresponding to the second target processor is {MF2, MN2, RS2, RF2}; By analogy, the quadruple {MF m ,MN m ,RS m ,RF m }; Process B12: MF1~MF m , sorted in descending order, we get index sequence 1; MN1~MN m , arranged in descending order, we get index sequence 2; Set RS1~RS m , sorted in descending order, we get index sequence 3; RF1~RF m , arranged in descending order, we get index sequence 4; Process B13: Read the first and last items corresponding to index sequences 1 to 4 respectively, and record them as max(MF), min(MF); max(MN), min(MN); max(RS), min(RS); max(RF), min(RF); Define conversion formulas 21 to 24 to map the performance indexes corresponding to the 1st to mth target processors into the interval [0,1]; Conversion formula 21: ; Among them, MF i and , respectively represent the CPU main frequency and the processing value of the CPU main frequency corresponding to the i-th target processor; Conversion formula 22: MN i and , respectively represent the number of CPU cores and the processing value of the number of CPU cores corresponding to the i-th target processor; Conversion formula 23: Among them, RS i and , respectively represent the RAM size and the processing value of the RAM size corresponding to the i-th target processor; Conversion formula 24: ; Among them, RF i and , respectively represent the RAM main frequency and the processing value of the RAM main frequency corresponding to the i-th target processor; Among them, i in conversion formulas 21 to 24 represents 1 to m.

4. The task processing optimization system based on cloud computing and big data according to claim 3 is characterized in that: The subsequent process of process B13 is as follows: Process B14: MF1 to MF corresponding to the 1st to mth target processors m MN1~MN m , RS1~RS m and RF1~RF m , respectively, into conversion formulas 21 to 24 to calculate the CPU main frequency processing values ​​corresponding to the 1st to mth target processors ~ ; CPU core number processing value ~ ; RAM size processing value ~ ; RAM main frequency processing value ~ ; Process B15: Calculate the comprehensive performance index P1 of the first target processor; Process B151: Process the CPU main frequency corresponding to the first target processor 、CPU core number processing value , RAM size processing value and RAM frequency processing value , denoted as a1, a2, a3 and a4 respectively; Process B152: Use the MaCh.random() function to randomly generate weight parameters μ1, μ2, μ3, and μ4 in the interval (0,1); the constraints satisfied by μ1, μ2, μ3, and μ4 are as follows: μ1+μ2+μ3+μ4=1; μ1 / a1=μ2 / a2=μ3 / a3=μ4 / a4=1; Process B153: Read the performance indexes MF1, MN1, RS1 and RF1 corresponding to the first target processor; Calculate the comprehensive performance index P1 of the first target processor. The calculation formula of P1 is as follows: P1=MF1*μ1+MN1*μ2+RS1*μ3+RF1*μ4; Process B16: Repeat the same process of calculating the comprehensive performance index P1 of the first target processor to calculate the comprehensive performance indexes P2 to P3 corresponding to the second to mth target processors. m .

5. The task processing optimization system based on cloud computing and big data according to claim 2 is characterized in that: The specific process of process B3 is as follows: Process B31: define a 1×m one-dimensional matrix P to represent the comprehensive performance of the 1st to mth target processors; the matrix P = [1 / P1 1 / P2 ~ 1 / Pm]; where m represents the number of target processors; Process B32: Allocate a target processor for the first adjustable process; Process B33: Repeat the same process of allocating target processors for the first adjustable process, and allocate target processors for the second to Rnth adjustable processes.

6. The task processing optimization system based on cloud computing and big data according to claim 5 is characterized in that: The specific process of process B32 is as follows: Process B321: in the virtual machine, simulate and execute the first adjustable process; In the virtual machine, use Toolhelp32 snapshot to obtain all threads of the first adjustable process, count the total number of threads, and record it as tt; Use Thread32First and Thread32Next functions to obtain the operation codes and operation times corresponding to the 1st to ttth threads; Process B322: according to the operation codes of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task service matrix of the first adjustable process, which is recorded as matrix F; According to the operation time of the 1st to ttth threads, create a k×1 one-dimensional matrix to represent the task time matrix of the first adjustable process, denoted as matrix V; Process B323: Multiply the matrix V by the matrix P to obtain a k×m two-dimensional matrix, denoted as matrix ETC; The elements in the matrix ETC are denoted as ETC(j,i); where j represents 1 to k and i represents 1 to m; ETC(j,i) represents: the processing time of the jth thread on the ith processor in the first adjustable process; Process B324: Obtain the target processor node numbers T1, T2~T1 of the 1st to mth target processors m ; Transpose the matrix TSC to get the matrix TSC （T） ; Combine the matrix ETC with the matrix TSC （T） Multiply them together to obtain a k×k two-dimensional matrix, denoted as matrix PTC; The elements in the matrix ETC are denoted as ETC(y,x), where x and y both represent 1 to k. PTC(y,x) means: assign the yth thread in the first adjustable process to the processor node with the number T. x in the target processor; Process B325: traverse the elements in the matrix ETC: ETC(1,1)~ETC(m,m); Define judgment formula 25: ETC(m,n)≠null∩ETC(m,n)≠0; where m and m both represent 1 to k, and ETC(m,n) represents the element in the mth row and mth column of the matrix ETC; Process B326: Determine the target processor of the first thread; set the initial value of m to 1, and set the representation range of n to 1 to k; Substitute ETC(1,1) to ETC(1,m) in the matrix ETC into judgment formula 25; mark the elements that satisfy judgment formula 25 as the elements to be selected; Select the candidate element with the smallest element value as the determination element, read the target processor corresponding to the determination element as the allocation processor; allocate the first thread to the allocation processor; Process B327: Repeat the same process of determining the target processor for the first thread, and assign target processors to the second to ttth threads in the first adjustable process.

Images