Resource allocation processing method and device

Abstract

The embodiment of the invention provides a resource allocation processing method and device, and the method comprises the steps: obtaining a processing request in a data processing process, screening out at least one target algorithm matched with a request type from an algorithm library, constructing a plurality of algorithm tasks corresponding to the target algorithm based on to-be-processed data, and transmitting the algorithm tasks to a server; calculating resource occupation information of each algorithm task and residual resources of each calculation node, performing matching processing on the resource occupation information and the residual resources, determining the calculation node corresponding to each algorithm task according to a matching result, and allocating the algorithm task to the corresponding calculation node. The availability of data parallel processing is improved, and the data processing efficiency is improved.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a resource allocation processing method and device. Background Art

[0002] With the continuous development of Internet technology and the continuous increase in the total amount of data, more and more data is transmitted and stored through various devices, and more and more data is stored in users' personal terminals. Users can store text, images, audio and video, program data and many other data through personal terminals. However, with the continuous increase of personal data, users can also choose to store data in storage applications or storage services to save terminal storage space. However, with the continuous increase of user needs, storage applications or storage services are also facing many problems; In the prior art, storage applications or storage services generally only provide storage services, which is not convenient for users to query and filter data after uploading data. How to enable storage services to better allocate resources to realize various functions has become the focus of attention of all parties. Summary of the invention

[0003] An embodiment of the present specification aims to provide a resource allocation processing method and device to solve the problem of low computing efficiency caused by poor computing power allocation during resource allocation processing.

[0004] To solve the above technical problems, an embodiment of this specification is implemented as follows: In a first aspect, an embodiment of the present specification provides a resource allocation processing method, including: Get processing request; Screening out at least one target algorithm adapted to the request type of the processing request from an algorithm library, and constructing a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; Calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The computing nodes corresponding to the respective algorithm tasks are determined according to the matching results, and the respective algorithm tasks are allocated to the computing nodes.

[0005] In a second aspect, another embodiment of the present specification provides a resource allocation processing device, including: A processing request acquisition module, used to acquire processing requests; An algorithm task construction module, used to select at least one target algorithm adapted to the request type of the processing request from an algorithm library, and to construct a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed contained in the processing request; A resource matching module is used to calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and to match the computing power usage information with the remaining computing power; The algorithm task allocation module is used to determine the computing nodes corresponding to each of the algorithm tasks according to the matching results, and allocate each of the algorithm tasks to the computing nodes.

[0006] In a third aspect, another embodiment of the present specification provides a resource allocation processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the resource allocation processing method as described in the first aspect above.

[0007] In a fourth aspect, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the resource allocation processing method as described in the first aspect above are implemented.

[0008] In a fifth aspect, another embodiment of the present specification provides a computer program product, wherein the computer program product includes a resource allocation processing program, and when the resource allocation processing program is executed by a processor, the steps of the resource allocation processing method described in the first aspect are implemented.

[0009] The resource allocation processing method provided in this embodiment obtains a processing request and selects at least one target algorithm matching the request type in the algorithm library during data processing, selects an algorithm by the request type, removes the influence of specific business data on computing power, constructs multiple algorithm tasks corresponding to the target algorithm based on the data to be processed, calculates the computing power occupancy information of each algorithm task and the remaining computing power of each computing node, performs matching processing on the computing power occupancy information and the remaining computing power, determines the computing node corresponding to each algorithm task according to the matching result, allocates each algorithm task to its own computing node, quickly determines the target algorithm to be used by the request type of the processing request, improves the algorithm determination efficiency, avoids the influence of the relatively large amount of data to be processed on the algorithm determination efficiency, constructs the algorithm task at the same time, matches the resource occupancy information of the algorithm task with the remaining resources of the computing node, improves the data processing efficiency, and improves the pertinence and availability of computing power allocation, and finally allocates an adapted computing node to each algorithm task according to the matching result for data processing, improves the computing power resource allocation efficiency, and thus improves the request processing efficiency of the storage service and / or storage application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 A processing flow chart of a resource allocation processing method provided by one or more embodiments of this specification; Figure 2 A processing flow chart of a resource allocation processing method applied to a data processing scenario provided in one or more embodiments of this specification; FIG3 (a) is a schematic diagram of a resource allocation processing method provided by one or more embodiments of this specification; FIG3( b ) is a schematic diagram of a task allocation process of a resource allocation processing method provided by one or more embodiments of this specification; Figure 4 A schematic diagram of a resource allocation processing device provided for one or more embodiments of this specification; Figure 5 A schematic diagram of the structure of a resource allocation processing device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0011] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0012] This specification provides a resource allocation processing method embodiment: Reference Figure 1 , which shows a processing flow chart of a resource allocation processing method provided by this embodiment, referring to Figure 2 , which shows a processing flow chart of a resource allocation processing method applied to a data processing scenario provided by this embodiment.

[0013] Reference Figure 1 The resource allocation processing method provided in this embodiment specifically includes the following steps S102 to S108.

[0014] Step S102, obtaining a processing request.

[0015] In this embodiment, the processing request may be a data processing request for the data uploaded by the user in the cloud storage service, storage service and / or storage application, that is: the processing request includes a data processing request. Specifically, the data processing request may be a processing request generated according to preset rules for the data uploaded by the user, such as: after detecting the data uploaded by the user, a data classification request is generated for the uploaded data; the data processing request may also be generated and uploaded by the user through a terminal device, such as: the user may select the target data and the corresponding function in the storage service and / or storage application, and generate a data processing request based on the target data and the corresponding function. For example, if the user selects 10 photos in the storage service and / or storage application and selects the photo repair function, a photo repair request is generated based on the 10 photos and the instructions corresponding to the photo repair function, and the photo repair request is submitted.

[0016] In specific implementation, a processing request is obtained to perform subsequent processing according to the request type and the data to be processed contained in the processing request. Specifically, the processing request includes: a data processing request, an image processing request and / or a file processing request.

[0017] During the specific execution process, in order to improve the data processing efficiency, data processing rules can be preset to quickly respond to data processing after detecting the data uploaded by the user, that is, the data request can be generated by itself, and for this purpose, the data uploaded by the user can be obtained and used as the data to be processed, and the data processing request is generated based on the data to be processed and the preset processing rules; in an optional implementation manner provided by this embodiment, obtaining the data processing request includes: Acquire image data uploaded by a user, and use the image data as the data to be processed; An image processing request is generated based on the data to be processed and a preset data processing type.

[0018] For example, image data, text data, and audio and video data uploaded by a user in a storage service and / or a storage application are obtained, and a data classification request for the user uploaded data is generated according to a preset classification processing type.

[0019] For another example, image data uploaded by a user on a cloud storage service is obtained, and an image sorting request for the image data is generated according to a preset image sorting request.

[0020] In addition, the data processing request can also be generated and uploaded by the requester through the terminal device. Specifically, the terminal device can display the page of the storage service and / or storage application, select the target data and the corresponding processing method according to the trigger instruction of the requester, generate the data processing request according to the target data and the corresponding processing method, and submit the data processing request, thereby improving the perception of the requester and enhancing the pertinence of data processing. In an optional implementation manner provided in this embodiment, the data processing request is obtained in the following manner: Determine the target data and the data processing type according to the trigger instruction submitted by the requester to the page storing the application; A data processing request is generated based on the target data and the data processing type to submit the request.

[0021] For example, a user submits a trigger instruction to a page of a cloud storage application displayed on a terminal device, selects multiple images and selects an image repair function. The terminal device receives the trigger instruction and generates an image repair request based on the selected multiple images and the image repair function, and submits the image repair request.

[0022] Step S104: select at least one target algorithm that is compatible with the request type of the processing request from the algorithm library, and construct multiple algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request.

[0023] In specific implementation, the data to be processed and the request type contained in the data processing request are extracted, and the target algorithm adapted to the request type is screened out in the algorithm library to process the data to be processed through the target algorithm, and the data to be processed contained in the processing request is extracted, and the data to be processed is split into multiple sub-data, and multiple algorithm tasks of at least one target algorithm are constructed based on the sub-data to perform the task. Specifically, an adapted algorithm can be set for each request type in advance, and at least one target algorithm adapted to the request type can be screened out in the algorithm library based on the request type of the request, and the sub-data obtained by the split can be associated with at least one target algorithm to construct multiple algorithm tasks for processing the data to be processed through the target algorithm.

[0024] For example, during the algorithm screening process, a data detection algorithm and a data sorting algorithm are preset for a data classification function, and the data detection algorithm and the data sorting algorithm are screened out from the algorithm library according to the classification request; for another example, during the algorithm screening process, an image denoising algorithm and an image adjustment algorithm are preset for an image restoration function, and the image denoising algorithm and the image adjustment algorithm are screened out from the algorithm library according to the image restoration request.

[0025] Step S106, calculating the computing power occupancy information of each algorithm task and calculating the remaining computing power of each computing node, and performing matching processing on the computing power occupancy information and the remaining computing power.

[0026] This embodiment selects the preferred computing power node corresponding to each algorithm task by calculating the computing power usage information of each algorithm task and matching it with the remaining computing power of each computing node. The data processing efficiency is improved and the computing power utilization rate is increased through the optimization method.

[0027] In specific implementation, the computing power occupancy information of each algorithm task is calculated, and the remaining computing power of each computing power node is calculated, and the computing power occupancy information and the remaining computing power are matched to obtain the matching result of the computing power occupancy information and the remaining computing power. Specifically, the computing power requirement value and performance requirement value of each algorithm task can be calculated, and the remaining computing power and remaining performance of each computing node can also be calculated.

[0028] During the specific execution process, a reference mean of the computing power occupation of the algorithm can be set for each algorithm in the algorithm library. The computing power occupation value of each algorithm task is calculated according to the number of algorithm tasks and the number of target algorithms, and the real-time computing power demand value of each algorithm task per unit time is calculated, that is, the performance occupation value, so as to improve the accuracy of the calculation of the computing power occupation information and improve the calculation accuracy; in an optional implementation manner provided in this embodiment, the computing power occupation information of each algorithm task is calculated, including: Calculate the average computing power usage of the at least one target algorithm, and calculate the total computing power usage of each of the algorithm tasks according to the average computing power usage; Calculate the performance occupancy parameter of the total computing power occupancy value within a preset time, and use the total computing power occupancy value and the performance occupancy parameter as the computing power occupancy information.

[0029] Specifically, a weighted calculation is performed according to the reference mean of computing power occupancy of the target algorithm preset in the algorithm library and the reference mean of computing power occupancy of all algorithms in the algorithm library according to the number of algorithms of the algorithm task to obtain the mean of computing power occupancy of at least one target algorithm, and the total computing power occupancy of each algorithm task is calculated through an accumulation algorithm, and the real-time computing power occupancy value of the total computing power occupancy value relative to the preset unit time is calculated, that is, the performance occupancy value, and the total computing power occupancy value of each algorithm task and the corresponding performance occupancy value are used as computing power occupancy information.

[0030] In the specific total computing power occupancy calculation process, the overall average computing power of each algorithm corresponding to the algorithm task can be calculated first, and then the total computing power occupancy value of a single algorithm task can be obtained by correction calculation based on the average computing power and the mean computing power occupancy value of each algorithm, so as to improve the prediction accuracy and data availability of the computing power occupancy value of the algorithm task; in an optional implementation manner provided in this embodiment, the total computing power occupancy value of each algorithm task is calculated, including: Querying the average computing power usage of each of the at least one target algorithm, and determining the historical average computing power usage of the algorithm according to the historical computing power parameters; The computing power occupancy average is calculated based on the computing power occupancy average and the historical occupancy average, and the computing power occupancy average is calculated according to a preset polynomial to obtain the computing power occupancy total value.

[0031] For example, for a single algorithm task, the average computing power occupied by at least one target algorithm corresponding to the algorithm task can be calculated by the following formula: ; in, is the number of algorithms corresponding to the algorithm task, ; is the reference value of computing power occupied by the jth algorithm corresponding to the algorithm task, The average reference value of all algorithm computing power occupied by this algorithm task; Available Indicates the discrete state of the algorithm computing power occupied by the algorithm task, and the average computing power occupied is calculated After that, the total computing power occupied by the algorithm task can be calculated by the following formula: ; Among them, n(t) is the amount of tasks to be processed at the current time point, i∈[1,n(t)], It is a fixed adjustment coefficient, which is set based on historical data fitting.

[0032] It should be noted that the above calculation of the average computing power occupancy and the total computing power occupancy is performed using a single algorithm task as an example. Each algorithm task can refer to the above calculation method to calculate the average computing power occupancy and the total computing power occupancy, which will not be repeated in this embodiment.

[0033] In the specific performance occupancy parameter calculation process, the computing power change curve of the average computing power occupancy of the algorithm task relative to the processing time can be calculated according to the processing time of the historical algorithm task, and the calculation accuracy and perception degree can be improved by means of a curve chart. After the change curve is transformed according to the preset rules, the extreme value is taken to obtain the computing power usage parameter per unit time, that is, the performance occupancy parameter, which improves the reliability of the data; in an optional implementation manner provided in this embodiment, the performance occupancy parameter is calculated, including: Query the processing time of historical algorithm tasks, and calculate the computing power change curve of the computing power occupancy mean value relative to the processing time; The computing power change curve is transformed according to a preset rule and then an extreme value is taken to obtain a computing power usage parameter within the processing time, and the computing power usage parameter is used as the performance occupancy parameter.

[0034] Using the above example, the following formula can be used to calculate the computing power change curve of the algorithm task: ; in, For fixed intervals, It is determined based on the average value of the task processing interval time points in the historical data, and the following formula is used to change the computing power change curve and take extreme value processing: ; Among them, t1 is a preset fixed time length, which is selected and set according to historical data, max(p(t)) is the maximum value of p(t) in the period t~t+t1, x1 and x2 are the first preset weight coefficients, x1+x2=1, which is selected and set after fitting according to historical data, It reflects the predicted change state of the algorithm task after deployment. On the basis of obtaining the computing power change curve, a comprehensive judgment is made through the comprehensive P(t) average state and extreme value state to obtain the performance occupancy parameters.

[0035] It should be noted that the above calculation of the performance occupancy parameter is performed by taking a single algorithm task as an example. The calculation method of the performance occupancy parameter of each algorithm task can refer to the above example, and this embodiment will not be repeated here.

[0036] During the specific execution process, the remaining computing power of each computing node needs to be calculated to improve the pertinence of computing power allocation. Specifically, the current first computing power occupancy value of each computing node can be queried, and the second computing power occupancy value of the data to be processed can be calculated. The remaining computing power and remaining performance of each computing node are calculated according to the first computing power occupancy value and the second computing power occupancy value to improve the reliability of the data. In an optional implementation manner provided in this embodiment, the remaining computing power of each computing node is calculated, including: querying a first occupancy parameter of each computing node, and calculating a second occupancy parameter of a task to be processed of each computing node; The remaining computing power parameter and the remaining performance parameter of each computing node are calculated according to the first occupancy parameter and the second occupancy parameter.

[0037] Specifically, the status value of the computing node can be queried to extract the real-time computing power occupancy of each computing node, and the pending computing power occupancy of each computing node can be calculated based on the amount of data to be processed of each computing node. The remaining computing power value and remaining performance value of each computing node can be calculated based on the real-time computing power occupancy and the pending computing power occupancy.

[0038] In the specific calculation process of the remaining computing power value, a historical computing power occupancy curve can be constructed in advance according to the computing power occupancy of the historical computing power tasks, and a predicted computing power occupancy curve can be constructed according to the real-time computing power occupancy. The predicted computing power occupancy curve is corrected based on the historical computing power occupancy curve to obtain the corrected occupancy curve of each computing node. The remaining computing power is accurately calculated by means of a curve graph to eliminate errors. In an optional implementation manner provided in this embodiment, the remaining computing power parameter is calculated in the following manner: Extracting a historical occupancy curve in the first occupancy parameter and a predicted occupancy curve in the second occupancy parameter, and correcting the predicted occupancy curve based on the historical occupancy curve to obtain a corrected occupancy curve for each computing node; The predicted computing power within a preset time is calculated according to the modified occupancy curve, and the difference between the total computing power of each computing node within the preset time and the predicted computing power is calculated and used as the remaining computing power parameter.

[0039] For example, for a single computing node, the modified occupancy curve can be calculated by using the historical occupancy curve and the predicted occupancy curve according to the following formula: ; in, is the predicted computing power usage curve of the computing node in the period t-t1~t+t1, The actual computing power occupancy curve of the computing node in the period t-t1~t, that is, the historical computing power occupancy curve. After obtaining the modified occupancy curve, the remaining computing power parameters of the computing node can be calculated according to the following formula: ; in, It is the operating performance threshold of the computing node, that is, the designed computing power threshold.

[0040] It should be noted that the calculation method of the remaining computing power parameters of a single computing node provided above can also be applied to each computing node, that is: the calculation method of the remaining computing power parameters of each computing node can be performed according to the above example, and this embodiment will not be repeated here.

[0041] In the specific residual performance parameter calculation process, the preset total performance value of the computing node and the extreme value of the modified occupancy curve within the preset time can be obtained, and the difference between the total performance value and the extreme value is calculated as the residual performance parameter to improve the reliability of the data and the accuracy of the calculation; in an optional implementation manner provided in this embodiment, the residual performance parameter is calculated in the following manner: querying the respective total performance values ​​of the computing nodes within the preset time, and calculating the extreme value of the modified occupancy curve within the preset time; The difference between the total performance value and the extreme value is calculated and used as the remaining performance parameter.

[0042] Using the above example, the remaining performance parameters of a single computing node can be calculated using the following formula: ; in, is the kth computing node in the period t-t1~t The maximum value of y1 and y2 are the second preset weight coefficients, which are selected and determined after fitting according to historical data, y1+y2=1.

[0043] It should be noted that the calculation method of the remaining performance parameters of a single computing node provided above can also be applied to each computing node, that is, the calculation method of the remaining performance parameters of each computing node can be performed with reference to the calculation method of the remaining performance parameters of the single computing node mentioned above, and this embodiment will not be repeated here.

[0044] In the specific matching process, matching can be performed by an optimization method, that is, the algorithm tasks that occupy more computing power are assigned to computing nodes with more remaining computing power for task processing, so as to improve the processing efficiency of the algorithm tasks; in an optional implementation manner provided in this embodiment, matching processing of computing power occupancy information and remaining computing power includes: Sorting the computing nodes according to the remaining computing power, and sorting at least one algorithm task according to the computing power occupancy information; Detect whether the remaining computing power of each computing node reaches the threshold of the computing power occupancy information of the target algorithm task, and use the detection result as the matching result between the target algorithm task and the computing nodes.

[0045] Continuing with the above example, for a single algorithm task, follow Sort the computing nodes from large to small, and judge them in order Is it greater than , and the judgment result is used as the matching result between the algorithm task and each computing node.

[0046] It should be noted that the matching method of a single algorithm task and each computing node provided above can also be applied to each algorithm task, and each algorithm task can also use the above matching method when matching with each computing node, which will not be repeated in this embodiment.

[0047] Step S108, determining the computing nodes corresponding to the respective algorithm tasks according to the matching results, and assigning the respective algorithm tasks to the computing nodes.

[0048] In the specific implementation, according to the matching results of the computing power usage information of each algorithm task and the remaining computing power of each algorithm node, the computing node suitable for each algorithm task is determined and each algorithm task is assigned to the corresponding computing node to execute the algorithm task, that is, to complete the processing request.

[0049] During the specific execution process, the algorithm tasks can be assigned to the computing nodes in order, that is: for algorithm tasks that occupy a large amount of computing power, if there are multiple algorithm nodes that meet the algorithm tasks, the computing node with a larger remaining computing power will be selected for allocation, thereby improving data processing efficiency and improving the computing power utilization of the computing nodes; if a single computing node still has a large amount of remaining computing power after allocating algorithm tasks, algorithm tasks can continue to be allocated to it until the remaining computing power of the remaining computing nodes is greater than the remaining computing power of the computing node. The rationality of computing power allocation is improved by unified data processing, and the stability of the system is improved. That is: each algorithm task can be assigned to one computing node or to multiple technical nodes.

[0050] Using the above example, after sorting the algorithm tasks according to the computing power usage information, for the one with the largest computing power usage, if Greater than , then it is determined that the computing node matches the algorithm task, and further it can be determined that 0.8* Is it greater than , that is: determine whether the remaining computing power of the computing node is greater than the total computing power occupied by each algorithm task. If so, assign each algorithm task to If not, the algorithm tasks ranked lower will be assigned to the remaining computing nodes according to the ranking results of the remaining computing power, and the remaining tasks will be assigned to The compute node corresponding to the first ranking is recalculated And repeat the judgment 0.8* Is it greater than , until the judgment result is yes. For the process of allocating the remaining algorithm tasks to the remaining computing nodes according to the ranking results of the remaining computing power, the ranking results of each computing node according to the remaining computing power can be used to judge each computing node in turn. Is it greater than , if so, assign the algorithm task to the computing node.

[0051] In addition, there is a situation where each computing node does not meet the computing power required by the algorithm task. In this regard, there may be a large number of tasks to be processed on each computing node. The processing time can be preset, and the computing power judgment can be made after the processing time expires. That is: wait for the computing node to process some of the tasks to be processed before making a computing power judgment on the computing node, so as to improve data processing efficiency.

[0052] In summary, the one or more resource allocation processing methods provided in this embodiment obtain a processing request and screen out at least one target algorithm matching the request type in the algorithm library during data processing. The algorithm is selected by the request type, the influence of specific business data on computing power is stripped off, multiple algorithm tasks corresponding to the target algorithm are constructed based on the data to be processed, the computing power occupancy information of each algorithm task is calculated and the remaining computing power of each computing node is calculated, and matching processing of the computing power occupancy information and the remaining computing power is performed. The computing nodes corresponding to each algorithm task are determined according to the matching results, and each algorithm task is assigned to its own computing node. The target algorithm to be used is quickly determined by the request type of the processing request, the algorithm determination efficiency is improved, and the influence of the relatively large amount of data to be processed on the algorithm determination efficiency is avoided. At the same time, the algorithm task is constructed, and the resource occupancy information of the algorithm task is matched with the remaining resources of the computing node, so as to improve the data processing efficiency, improve the pertinence and availability of computing power allocation, and finally, an adapted computing node is assigned to each algorithm task according to the matching results for data processing, so as to improve the computing power resource allocation efficiency, thereby improving the request processing efficiency of the storage service and / or storage application.

[0053] The following combination Figure 2 , taking the application of the resource allocation processing method provided in this embodiment in a data processing scenario as an example, the resource allocation processing method provided in this embodiment is further described. Figure 2 The resource allocation processing method applied to the data processing scenario specifically includes steps S202 to S216.

[0054] Step S202: Obtain a data processing request, and select at least one target algorithm that matches the request type of the data processing request from an algorithm library.

[0055] Step S204: construct multiple algorithm tasks of at least one target algorithm based on the data to be processed included in the data processing request.

[0056] Step S206, calculate the average computing power usage of at least one target algorithm, and calculate the computing power usage value of each algorithm task according to the average computing power usage.

[0057] Step S208, calculate the performance occupancy value of each computing power occupancy value per unit time, and use the computing power occupancy value and performance occupancy value of each algorithm task as the computing power occupancy information of each algorithm task.

[0058] Step S210, query the real-time computing power occupancy information of each computing node, and calculate the predicted computing power occupancy information of the tasks to be processed of each computing node.

[0059] Step S212, calculate the remaining computing power parameters and remaining performance parameters of each computing node according to the real-time computing power occupancy information and the predicted computing power occupancy information, and use the remaining computing power parameters and remaining performance parameters as the remaining computing power.

[0060] Step S214, matching processing is performed on the computing power usage information of each algorithm task and the remaining computing power of each computing node.

[0061] Step S216, determining the computing nodes corresponding to the respective algorithm tasks according to the matching results, and assigning the respective algorithm tasks to the respective corresponding computing nodes.

[0062] It should be noted that any one of steps S202 to S216 or any combination of multiple steps can be selected from steps S102 to S108 to form a new implementation method according to the needs of implementation deployment; and any one or multiple technical features in the technical scheme composed of steps S202 to S216 can be selected from steps S102 to S108 to form a new implementation method according to the needs of actual deployment, or the technical features in one or more optional implementation methods provided by steps S102 to S108 can be selected to combine into a new implementation method, which will not be elaborated here one by one.

[0063] It should also be noted that the one or more resource allocation processing methods provided in this embodiment can also be executed with reference to the flow chart provided in Figure 3 (a). The allocation process of the algorithm tasks in the one or more resource allocation processing methods provided in this embodiment can be executed with reference to the flow chart provided in Figure 3 (b). The specific execution process can be carried out with reference to one or more embodiments provided above, and this embodiment will not be repeated here.

[0064] Figure 4 A schematic diagram of a resource allocation processing device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device comprises: The processing request acquisition module 402 is used to acquire the processing request; An algorithm task construction module 404 is used to select at least one target algorithm adapted to the request type of the processing request from the algorithm library, and to construct multiple algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; The resource matching module 406 is used to calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The algorithm task allocation module 408 is used to determine the computing nodes corresponding to the respective algorithm tasks according to the matching results, and allocate the respective algorithm tasks to the computing nodes.

[0065] The resource allocation processing device provided in this embodiment first runs the processing request acquisition module 402 to obtain the processing request and screens out at least one target algorithm matching the request type in the algorithm library during the data processing process, selects the algorithm by the request type, and removes the influence of specific business data on the computing power, runs the algorithm task construction module 404 to construct multiple algorithm tasks corresponding to the target algorithm based on the data to be processed, and then runs the resource matching module 406 to calculate the computing power occupancy information of each algorithm task and the remaining computing power of each computing node, and performs matching processing on the computing power occupancy information and the remaining computing power, and finally runs the algorithm task allocation module 408 to determine the tasks of each algorithm task according to the matching results. The corresponding computing nodes assign each algorithm task to their own computing nodes, and quickly determine the target algorithm to be used by processing the request type, thereby improving the algorithm determination efficiency and avoiding the impact of relatively large amounts of data to be processed on the algorithm determination efficiency. At the same time, algorithm tasks are constructed, and the resource occupancy information of the algorithm tasks is matched with the remaining resources of the computing nodes to improve data processing efficiency, improve the pertinence and availability of computing power allocation, and finally, according to the matching results, allocate an adapted computing node to each algorithm task for data processing, thereby improving the computing power resource allocation efficiency, and thereby improving the request processing efficiency of storage services and / or storage applications.

[0066] The resource allocation processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0067] Furthermore, an embodiment of the present specification also provides a resource allocation processing device, Figure 5 A schematic diagram of a resource allocation processing device provided in an embodiment of this specification is shown in FIG. Figure 5 As shown, the device includes: a memory 501, a processor 502, a bus 503 and a communication interface 504. The memory 501, the processor 502 and the communication interface 504 communicate through the bus 503, and the communication interface 504 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0068] Figure 5 In the embodiment, the memory 501 stores computer executable instructions that can be run on the processor 502. When the computer executable instructions are executed by the processor 502, the following process is implemented: Get processing request; Screening out at least one target algorithm adapted to the request type of the processing request from an algorithm library, and constructing a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; Calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The computing nodes corresponding to the respective algorithm tasks are determined according to the matching results, and the respective algorithm tasks are allocated to the computing nodes.

[0069] The resource allocation processing device provided in this embodiment obtains a processing request and selects at least one target algorithm matching the request type in the algorithm library through the cooperation of the memory 501, the processor 502, the bus 503 and the communication interface 504 during the data processing process, selects the algorithm by the request type, removes the influence of specific business data on the computing power, constructs multiple algorithm tasks corresponding to the target algorithm based on the data to be processed, calculates the computing power occupancy information of each algorithm task and the remaining computing power of each computing node, performs matching processing on the computing power occupancy information and the remaining computing power, determines the computing nodes corresponding to each algorithm task according to the matching results, and allocates each Algorithm tasks are assigned to their respective computing nodes, and the target algorithm to be used is quickly determined by processing the request type, thereby improving the algorithm determination efficiency and avoiding the impact of relatively large amounts of data to be processed on the algorithm determination efficiency. At the same time, algorithm tasks are constructed, and the resource occupancy information of the algorithm tasks is matched with the remaining resources of the computing nodes, thereby improving data processing efficiency and improving the pertinence and availability of computing power allocation. Finally, according to the matching results, each algorithm task is assigned an adapted computing node for data processing, thereby improving the computing power resource allocation efficiency and thereby improving the request processing efficiency of storage services and / or storage applications.

[0070] The resource allocation processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0071] Furthermore, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Get processing request; Screening out at least one target algorithm adapted to the request type of the processing request from an algorithm library, and constructing a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; Calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The computing nodes corresponding to the respective algorithm tasks are determined according to the matching results, and the respective algorithm tasks are allocated to the computing nodes.

[0072] The computer-readable storage medium provided in this embodiment obtains a processing request and selects at least one target algorithm matching the request type in an algorithm library during data processing, selects an algorithm by request type, removes the influence of specific business data on computing power, constructs multiple algorithm tasks corresponding to the target algorithm based on the data to be processed, calculates computing power occupancy information of each algorithm task and the remaining computing power of each computing node, performs matching processing of computing power occupancy information and remaining computing power, determines the computing nodes corresponding to each algorithm task according to the matching results, allocates each algorithm task to its own computing node, quickly determines the target algorithm to be used by the request type of the processing request, improves the algorithm determination efficiency, avoids the influence of the relatively large amount of data to be processed on the algorithm determination efficiency, improves the algorithm determination efficiency, constructs algorithm tasks at the same time, matches the resource occupancy information of the algorithm tasks with the remaining resources of the computing nodes, improves the data processing efficiency, improves the pertinence and availability of computing power allocation, and finally allocates an adapted computing node to each algorithm task according to the matching results for data processing, improves the computing power resource allocation efficiency, and thus improves the request processing efficiency of the storage service and / or storage application.

[0073] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0074] The computer-readable storage medium provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0075] Furthermore, another embodiment of the present specification also provides a computer program product, which implements the following process when executed by a processor: Get processing request; Screening out at least one target algorithm adapted to the request type of the processing request from an algorithm library, and constructing a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; Calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The computing nodes corresponding to the respective algorithm tasks are determined according to the matching results, and the respective algorithm tasks are allocated to the computing nodes.

[0076] The computer product program provided in this embodiment obtains a processing request and selects at least one target algorithm matching the request type in the algorithm library during data processing, selects an algorithm by the request type, removes the influence of specific business data on computing power, constructs multiple algorithm tasks corresponding to the target algorithm based on the data to be processed, calculates the computing power occupancy information of each algorithm task and the remaining computing power of each computing node, performs matching processing of the computing power occupancy information and the remaining computing power, determines the computing node corresponding to each algorithm task according to the matching result, allocates each algorithm task to its own computing node, quickly determines the target algorithm to be used by the request type of the processing request, improves the algorithm determination efficiency, avoids the influence of the relatively large amount of data to be processed on the algorithm determination efficiency, improves the algorithm determination efficiency, constructs the algorithm task at the same time, matches the resource occupancy information of the algorithm task with the remaining resources of the computing node, improves the data processing efficiency, improves the pertinence and availability of computing power allocation, and finally allocates an adapted computing node to each algorithm task according to the matching result for data processing, improves the computing power resource allocation efficiency, and thus improves the request processing efficiency of the storage service and / or storage application.

[0077] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0078] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-permanent storage in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable storage medium.

[0084] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0086] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0087] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A resource allocation processing method, characterized in that: The method comprises: Get processing request; Screening out at least one target algorithm adapted to the request type of the processing request from an algorithm library, and constructing a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed included in the processing request; Calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and perform matching processing between the computing power usage information and the remaining computing power; The computing nodes corresponding to the respective algorithm tasks are determined according to the matching results, and the respective algorithm tasks are allocated to the computing nodes.

2. The resource allocation processing method according to claim 1, characterized in that: The computing power usage information of each algorithm task is calculated, including: Calculate the average computing power usage of the at least one target algorithm, and calculate the total computing power usage of each of the algorithm tasks according to the average computing power usage; Calculate the performance occupancy parameter of the total computing power occupancy value within a preset time, and use the total computing power occupancy value and the performance occupancy parameter as the computing power occupancy information.

3. The resource allocation processing method according to claim 2, characterized in that: The calculating the average value of computing power occupied by the at least one target algorithm, and calculating the total value of computing power occupied by each of the algorithm tasks according to the average value of computing power occupied, includes: Querying the average computing power usage of each of the at least one target algorithm, and determining the historical average computing power usage of the algorithm according to the historical computing power parameters; The computing power occupancy average is calculated based on the computing power occupancy average and the historical occupancy average, and the computing power occupancy average is calculated according to a preset polynomial to obtain the computing power occupancy total value.

4. The resource allocation processing method according to claim 2, characterized in that: The calculating of the performance occupancy parameter of the total computing power occupancy value within a preset time includes: Query the processing time of historical algorithm tasks, and calculate the computing power change curve of the computing power occupancy mean value relative to the processing time; The computing power change curve is transformed according to a preset rule and then an extreme value is taken to obtain a computing power usage parameter within the processing time, and the computing power usage parameter is used as the performance occupancy parameter.

5. The resource allocation processing method according to claim 1, characterized in that: The calculating the remaining computing power of each computing node includes: querying a first occupancy parameter of each computing node, and calculating a second occupancy parameter of a task to be processed of each computing node; The remaining computing power parameter and the remaining performance parameter of each computing node are calculated according to the first occupancy parameter and the second occupancy parameter.

6. The resource allocation processing method according to claim 5, characterized in that: The remaining computing power parameter is calculated as follows: Extracting a historical occupancy curve in the first occupancy parameter and a predicted occupancy curve in the second occupancy parameter, and correcting the predicted occupancy curve based on the historical occupancy curve to obtain a corrected occupancy curve for each computing node; The predicted computing power within a preset time is calculated according to the modified occupancy curve, and the difference between the total computing power of each computing node within the preset time and the predicted computing power is calculated and used as the remaining computing power parameter.

7. The resource allocation processing method according to claim 6, characterized in that: The remaining performance parameters are calculated as follows: querying the respective total performance values ​​of the computing nodes within the preset time, and calculating the extreme value of the modified occupancy curve within the preset time; The difference between the total performance value and the extreme value is calculated and used as the remaining performance parameter.

8. The resource allocation processing method according to claim 1, characterized in that: The matching process of the computing power usage information and the remaining computing power includes: Sorting the computing nodes according to the remaining computing power, and sorting at least one algorithm task according to the computing power occupancy information; Detect whether the remaining computing power of each computing node reaches the threshold of the computing power occupancy information of the target algorithm task, and use the detection result as the matching result between the target algorithm task and the computing nodes.

9. The resource allocation processing method according to claim 1, characterized in that: The obtaining and processing request comprises: Acquire image data uploaded by a user, and use the image data as the data to be processed; An image processing request is generated based on the data to be processed and a preset data processing type.

10. The resource allocation processing method according to claim 1, characterized in that: The processing request is generated in the following manner: Determine the target data and the data processing type according to the trigger instruction submitted by the requester to the page storing the application; A data processing request is generated based on the target data and the data processing type to submit the request.

11. A resource allocation processing device, characterized in that: The device comprises: A processing request acquisition module is used to acquire processing requests; An algorithm task construction module, used to select at least one target algorithm adapted to the request type of the processing request from an algorithm library, and to construct a plurality of algorithm tasks of the at least one target algorithm based on the data to be processed contained in the processing request; A resource matching module is used to calculate the computing power usage information of each algorithm task and the remaining computing power of each computing node, and to match the computing power usage information with the remaining computing power; The algorithm task allocation module is used to determine the computing nodes corresponding to each of the algorithm tasks according to the matching results, and allocate each of the algorithm tasks to the computing nodes.

12. A resource allocation processing device, characterized in that: The device includes a memory and a processor, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed on the processor, the steps of the resource allocation processing method according to any one of claims 1 to 10 can be implemented.

13. A computer-readable storage medium having computer-executable instructions stored therein, characterized in that: When the computer executable instructions are executed by a processor, the steps of the resource allocation processing method described in any one of claims 1 to 10 can be implemented.

14. A computer program product, characterized in that The computer program product includes a resource allocation processing program, and when the resource allocation processing program is executed by a processor, the steps of the resource allocation processing method according to any one of claims 1 to 10 can be implemented.

Images