Distributed task resource time consumption prediction method based on general scaling law and medium

Abstract

The invention discloses a distributed task resource time consumption prediction method based on a general scaling law and a medium. The method comprises the following steps: acquiring a distributed task to be subjected to time consumption prediction at present, and distributed task description information corresponding to the distributed task to be subjected to time consumption prediction at present; wherein the distributed task description information comprises a distributed task type and distributed task difficulty; obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feeding back a distributed task time consumption prediction sequence result to a user. The problems that an existing pure data driving method often cannot solve the problem outside training data distribution and needs a large amount of data are solved. The accuracy of time consumption prediction of the distributed task resources is improved, and the distribution method of the distributed resources is optimized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and medium for predicting the time consumption of distributed task resources based on a universal scaling law. Background Art

[0002] In the current distributed computing environment, effective task resource allocation is crucial to improving system performance. These strategies include the stability of static resource allocation and the flexibility of dynamic resource allocation, resource allocation based on task requirements and load balancing, adaptability of elastic resource adjustment, intelligence of algorithm optimization, learning ability of adaptive scheduling, and comprehensive consideration of cost-benefit analysis. By comprehensively applying these strategies, the system can use resources more effectively, improve performance, and achieve the best resource allocation solution under different loads and environmental changes.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following defects: Currently, the existing pure data-driven approach can use historical data to train regression models to predict the relationship between the number of resources and task duration under given task conditions. However, the pure data-driven approach often cannot solve problems outside the distribution of training data, and requires a relatively large amount of data. Summary of the invention

[0004] The present invention provides a method and medium for predicting the time consumption of distributed task resources based on a universal scaling law, so as to improve the accuracy of predicting the time consumption of distributed task resources.

[0005] According to one aspect of the present invention, a method for predicting the time consumption of distributed task resources based on a universal scaling law is provided, which includes:

[0006] Obtaining a distributed task that is currently to be predicted for time consumption, and distributed task description information corresponding to the distributed task that is currently to be predicted for time consumption;

[0007] The distributed task description information includes the distributed task type and the distributed task difficulty;

[0008] Obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, distributed task type, and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time-consuming prediction sequence result;

[0009] Feedback of the distributed task time-consuming prediction sequence results to the user is performed.

[0010] According to another aspect of the present invention, a distributed task resource time consumption prediction device based on a general scaling law is provided, which includes:

[0011] A distributed task description information acquisition module, used to acquire the distributed task currently to be predicted for time consumption, and the distributed task description information corresponding to the distributed task currently to be predicted for time consumption;

[0012] The distributed task description information includes the distributed task type and the distributed task difficulty;

[0013] A distributed task time consumption prediction sequence result generation module is used to obtain the current distributed task resource sequence, and input the current distributed task resource sequence, distributed task type and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time consumption prediction sequence result;

[0014] The distributed task time-consuming prediction sequence result feedback module is used to feed back the distributed task time-consuming prediction sequence result to the user.

[0015] According to another aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for predicting the time consumption of distributed task resources based on the general scaling law as described in any embodiment of the present invention is implemented.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed task resource time consumption prediction method based on the general scaling law described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention is to obtain the current distributed task to be predicted for time consumption, and the distributed task description information corresponding to the current distributed task to be predicted for time consumption; wherein the distributed task description information includes the distributed task type and the distributed task difficulty; obtain the current distributed task resource sequence, and input the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feed back the distributed task time consumption prediction sequence result to the user. It solves the problem that the existing pure data-driven method often cannot solve the problem outside the distribution of training data and the problem that a large amount of data is required. It improves the accuracy of the distributed task resource time consumption prediction and optimizes the distributed resource allocation method.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a distributed task resource time consumption prediction method based on a general scaling law provided according to Embodiment 1 of the present invention;

[0021] Figure 2 2 is a schematic diagram of the structure of a distributed task resource time consumption prediction device based on a general scaling law provided according to Embodiment 2 of the present invention;

[0022] Figure 3 It is a structural schematic diagram of an electronic device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1A flowchart of a distributed task resource time consumption prediction method based on a universal scaling law is provided in Example 1 of the present invention. This embodiment is applicable to situations where time consumption prediction is performed on distributed task resources. The method can be executed by a distributed task resource time consumption prediction device based on a universal scaling law. The distributed task resource time consumption prediction device based on a universal scaling law can be implemented in the form of hardware and / or software.

[0027] Correspondingly, such as Figure 1 As shown, the method includes:

[0028] S110, obtaining a distributed task that is currently to be predicted for time consumption, and distributed task description information corresponding to the distributed task that is currently to be predicted for time consumption.

[0029] The distributed task description information includes the distributed task type and the distributed task difficulty.

[0030] The distributed task currently to be time-consuming predicted may be a current distributed task that needs to be processed for time-consuming prediction.

[0031] The distributed task description information may be information describing the distributed task. Specifically, the distributed task description information may include a distributed task type and a distributed task difficulty. The distributed task type may be information describing the distributed task type. The distributed task difficulty may be information describing the difficulty of different distributed tasks.

[0032] In this embodiment, different distributed tasks correspond to different distributed task types and distributed task difficulties. For the distributed task type, which is discrete data, it is necessary to process it through one-hot encoding to obtain continuous data. For the distributed task difficulty, which is continuous data, it is not necessary to process it through one-hot encoding.

[0033] S120, obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, distributed task type, and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time-consuming prediction sequence result.

[0034] Among them, the current distributed task resource sequence can be a sequence of multiple pre-set distributed task resource values. Among them, the general scaling law model can be a model constructed based on the general scaling law. The distributed task time consumption prediction sequence result can be a sequence result composed of multiple distributed task time consumption prediction values, which can be matched with each current distributed task resource value.

[0035] Exemplarily, the current distributed task resource sequence may be set to Z={1, 10, 20, ..., 100}, where Z=100 is the maximum distributed task resource value.

[0036] Furthermore, the current distributed task resource sequence, distributed task type and distributed task difficulty can be input into the general scaling law model to generate a distributed task time consumption prediction sequence result, that is, when Z is equal to 1, 10, 20 or 100, the corresponding distributed task time consumption prediction value is used to generate a distributed task time consumption prediction sequence result from multiple distributed task time consumption prediction values.

[0037] S130: Feedback the distributed task time-consuming prediction sequence results to the user.

[0038] In this embodiment, after the distributed task time consumption prediction sequence results are obtained, they need to be fed back to the user, and the user can perform re-analysis and processing operations based on the distributed task time consumption prediction sequence results.

[0039] Optionally, after inputting the current distributed task resource sequence, distributed task type and distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result, it also includes: generating a current distributed task resource time consumption prediction curve based on the current distributed task resource sequence and the distributed task time consumption prediction sequence result through a pre-set curve drawing tool.

[0040] The current distributed task resource time consumption prediction curve may be a curve describing the relationship between the current distributed task resource sequence and the distributed task time consumption prediction sequence results.

[0041] In this embodiment, when Z is equal to 1, 10, 20 or 100, the corresponding distributed task time consumption prediction values ​​can be respectively used. Assume that Z is equal to 1, D is equal to t1; Z is equal to 10, D is equal to t 10 ; Z is equal to 20, D is equal to t 20 ; Z equals 100, D equals t 100 .

[0042] Correspondingly, a curve drawing tool can be used to generate a current distributed task resource time consumption prediction curve according to the specific correspondence between the current distributed task resource sequence and the distributed task time consumption prediction sequence results.

[0043] The advantage of this setting is that it can more intuitively reflect the corresponding relationship between the current distributed task resource sequence and the distributed task time consumption prediction sequence results, and it is also convenient to determine the optimal distributed task resource value.

[0044] Optionally, after generating the current distributed task resource time consumption prediction curve, it also includes: obtaining the maximum allowable time consumption threshold corresponding to the current distributed task to be predicted; obtaining the minimum curve time consumption and the curve time consumption value with the task resource value equal to 1 according to the current distributed task resource time consumption prediction curve; if the maximum allowable time consumption threshold is less than the minimum curve time consumption, obtaining the target distributed task resource value corresponding to the minimum curve time consumption according to the current distributed task resource time consumption prediction curve; if the maximum allowable time consumption threshold is greater than the curve time consumption value with the task resource value equal to 1, determining the target distributed task resource value to be 1; if the maximum allowable time consumption threshold satisfies the conditions of being not less than the minimum curve time consumption and not greater than the curve time consumption value with the task resource value equal to 1, determining the target distributed task resource value corresponding to the maximum allowable time consumption threshold according to a pre-set binary search method; and feeding back the target distributed task resource value to the user.

[0045] Among them, the maximum allowed time threshold can be the size of the maximum allowed time threshold required for the current distributed task to be predicted. The minimum curve time can be the minimum time value in the current distributed task resource time prediction curve. The curve time value when the task resource value is equal to 1 can be the size of the corresponding time value on the current distributed task resource time prediction curve when the task resource value is equal to 1.

[0046] In this embodiment, it is necessary to further determine the optimal target distributed task resource value.

[0047] Specifically, if the maximum allowed time consumption threshold is less than the minimum time consumption of the curve, the target distributed task resource value corresponding to the minimum time consumption of the curve is obtained according to the current distributed task resource time consumption prediction curve.

[0048] For example, assume that the maximum allowed time threshold is D max =5min, the minimum time consumption of the curve is 10min, and in the current distributed task resource time consumption prediction curve, the target distributed task resource value corresponding to 10min is 33. Since 5min<10min, it can be determined that the target distributed task resource value is 33.

[0049] Further, if the maximum allowed time consumption threshold is greater than the curve time consumption value where the task resource value is equal to 1, the target distributed task resource value is determined to be 1.

[0050] Continuing with the previous example, when the task resource value is equal to 1, the corresponding curve duration value can be obtained as 2 minutes. Since 5 minutes> 2 minutes, it can be determined that the target distributed task resource value is 1.

[0051] Correspondingly, if the maximum allowable time threshold satisfies the requirement of not being less than the minimum curve time and not being greater than the curve time value when the task resource value is equal to 1, the target distributed task resource value corresponding to the maximum allowable time threshold is determined according to the pre-set binary search method.

[0052] Continuing with the previous example, assuming that the minimum curve time is 3 minutes, and the curve time when the task resource value is equal to 1 is 8 minutes, it can be determined that 3min<5min<8min. The target distributed task resource value corresponding to the maximum allowable time threshold can be determined based on the binary search method.

[0053] In this embodiment, before obtaining the distributed task currently to be predicted for time consumption and the distributed task description information corresponding to the distributed task currently to be predicted for time consumption, it is also necessary to train the general scaling law model.

[0054] Specifically, before obtaining the current distributed task to be predicted for time consumption and the distributed task description information corresponding to the current distributed task to be predicted for time consumption, it also includes: obtaining multiple groups of historical distributed tasks, and the historical distributed task types and historical distributed task difficulties corresponding to each of the historical distributed tasks; obtaining the historical distributed task resource sequence and the historical distributed task time sequence; inputting each of the historical distributed task types and historical distributed task difficulties, as well as the historical distributed task resource sequence and the historical distributed task time sequence into a preset initial universal scaling law model for model training, and obtaining the trained universal scaling law model based on a preset mean square error loss function; wherein the initial universal scaling law model is composed of a parallel computing acceleration part, a parallel computing unaccelerable part, and a parallel communication computing time increase part.

[0055] Among them, the historical distributed tasks can be multiple groups of distributed tasks obtained at historical moments. The historical distributed task types can be that each group of historical distributed tasks corresponds to a different task type. The historical distributed task difficulty can be that each group of historical distributed tasks corresponds to a different task difficulty. The historical distributed task resource sequence can be a sequence of multiple distributed task resource values ​​corresponding to a preset historical moment. The historical distributed task time sequence can be a sequence result composed of multiple distributed task time values ​​obtained at historical moments.

[0056] Specifically, the formula corresponding to the initial universal scaling law model is: Among them, C represents the historical distributed task type; S represents the historical distributed task difficulty; Z represents the historical distributed task resource sequence; D represents the historical distributed task time sequence; f represents the neural network model with three different parameters to be trained, and the threshold of f is a positive number; θ1, θ2 and θ3 represent different parameters to be trained; e ε represents the time-consuming error term; where Represents the parallel computing acceleration part; Indicates the non-accelerable part of parallel computing; Indicates the increased time consumption due to parallel communication calculation.

[0057] Specifically, C in the historical distributed task type can be {C1, C2, …, C k}, k means that there are k types of tasks. The S in the historical distributed task difficulty can be {S1,S2,…,S g}, g means that there are g types of task difficulty.

[0058] In detail, each of the historical distributed task types and historical distributed task difficulties, as well as the historical distributed task resource sequence and the historical distributed task time-consuming sequence, are input into a preset initial universal scaling law model for model training, and a trained universal scaling law model is obtained according to a preset mean square error loss function, including: normalizing each of the historical distributed task types by a preset one-hot encoding method to obtain each historical distributed normalized task type; converting the historical distributed task time-consuming sequence by a preset hour unit conversion method to obtain a historical distributed task time-consuming unit conversion sequence; inputting each of the historical distributed normalized task types and historical distributed task difficulties, as well as the historical distributed task resource sequence and the historical distributed task time-consuming unit conversion sequence into a preset initial universal scaling law model for model training, and a trained universal scaling law model is obtained according to a preset mean square error loss function.

[0059] The historical distributed normalized processing task type may be a continuous task type obtained by encoding the historical distributed task type. The historical distributed task time unit conversion sequence may be a sequence obtained by converting the time in the historical distributed task time sequence into hours.

[0060] In this embodiment, since the historical distributed task type is a discrete data type, it is necessary to perform discrete conversion to continuous processing through one-hot encoding to obtain the historical distributed normalized processing task type.

[0061] Furthermore, it is necessary to input the historical distributed normalized processing task type and historical distributed task difficulty, as well as the historical distributed task resource sequence and the historical distributed task time unit conversion sequence, into the initial universal scaling law model for model training. A mean square error loss function can be obtained. If the trained model meets the requirements of the mean square error loss function, a trained universal scaling law model can be obtained.

[0062] In addition, during the model training process, the samples can be divided into a training set and a validation set. After the universal scaling law model is trained, the model performance needs to be verified through the validation set so that the universal scaling law model can be better obtained.

[0063] Optionally, after obtaining the trained universal scaling law model according to the preset mean square error loss function, it also includes: generating a model universal unique identification code according to the trained universal scaling law model; and jointly storing the universal scaling law model and the model universal unique identification code in a model database.

[0064] The model universal unique identification code may be a unique identification code set for the universal scaling law model.

[0065] In this embodiment, after the universal scaling law model is trained, a model universal unique identifier can be matched to the universal scaling law model, and the model universal unique identifier and the universal scaling law model are jointly stored. In this way, the universal scaling law model can be better searched in the model database by the model universal unique identifier.

[0066] The technical solution of the embodiment of the present invention is to obtain the current distributed task to be predicted for time consumption, and the distributed task description information corresponding to the current distributed task to be predicted for time consumption; wherein the distributed task description information includes the distributed task type and the distributed task difficulty; obtain the current distributed task resource sequence, and input the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feed back the distributed task time consumption prediction sequence result to the user. It solves the problem that the existing pure data-driven method often cannot solve the problem outside the distribution of training data and the problem that a large amount of data is required. It improves the accuracy of the distributed task resource time consumption prediction and optimizes the distributed resource allocation method.

[0067] Embodiment 2

[0068] Figure 2A schematic diagram of the structure of a distributed task resource time consumption prediction device based on a universal scaling law provided in the second embodiment of the present invention. The distributed task resource time consumption prediction device based on a universal scaling law provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a distributed task resource time consumption prediction method based on a universal scaling law in the embodiment of the present invention. Figure 2 As shown, the device includes: a distributed task description information acquisition module 210, a distributed task time-consuming prediction sequence result generation module 220 and a distributed task time-consuming prediction sequence result feedback module 230.

[0069] The distributed task description information acquisition module 210 is used to acquire the distributed task currently to be predicted for time consumption, and the distributed task description information corresponding to the distributed task currently to be predicted for time consumption;

[0070] The distributed task description information includes the distributed task type and the distributed task difficulty;

[0071] The distributed task time-consuming prediction sequence result generation module 220 is used to obtain the current distributed task resource sequence, and input the current distributed task resource sequence, the distributed task type and the distributed task difficulty into the pre-trained universal scaling law model to generate the distributed task time-consuming prediction sequence result;

[0072] The distributed task time consumption prediction sequence result feedback module 230 is used to feed back the distributed task time consumption prediction sequence result to the user.

[0073] The technical solution of the embodiment of the present invention is to obtain the current distributed task to be predicted for time consumption, and the distributed task description information corresponding to the current distributed task to be predicted for time consumption; wherein the distributed task description information includes the distributed task type and the distributed task difficulty; obtain the current distributed task resource sequence, and input the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feed back the distributed task time consumption prediction sequence result to the user. It solves the problem that the existing pure data-driven method often cannot solve the problem outside the distribution of training data and the problem that a large amount of data is required. It improves the accuracy of the distributed task resource time consumption prediction and optimizes the distributed resource allocation method.

[0074] On the basis of the above embodiments, it also includes a current distributed task resource time consumption prediction curve generation module, which can be specifically used for: after the current distributed task resource sequence, distributed task type and distributed task difficulty are input into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result, the current distributed task resource time consumption prediction curve is generated according to the current distributed task resource sequence and the distributed task time consumption prediction sequence result through a pre-set curve drawing tool.

[0075] On the basis of the above embodiments, it also includes a target distributed task resource value determination and feedback module, which can be specifically used for: after generating the current distributed task resource time consumption prediction curve, obtaining the maximum allowable time consumption threshold corresponding to the current distributed task to be predicted; according to the current distributed task resource time consumption prediction curve, obtaining the minimum curve time consumption and the curve time consumption value with the task resource value equal to 1; if the maximum allowable time consumption threshold is less than the minimum curve time consumption, then according to the current distributed task resource time consumption prediction curve, obtaining the target distributed task resource value corresponding to the minimum curve time consumption; if the maximum allowable time consumption threshold is greater than the curve time consumption value with the task resource value equal to 1, determining the target distributed task resource value to be 1; if the maximum allowable time consumption threshold satisfies the requirement of not less than the minimum curve time consumption and not greater than the curve time consumption value with the task resource value equal to 1, then determining the target distributed task resource value corresponding to the maximum allowable time consumption threshold according to a pre-set binary search method; and feeding back the target distributed task resource value to the user.

[0076] On the basis of the above embodiments, it also includes a universal scaling law model determination module, which can be specifically used for: obtaining multiple groups of historical distributed tasks, and historical distributed task types and historical distributed task difficulties corresponding to each of the historical distributed tasks, before obtaining the current distributed task to be predicted for time consumption, and the distributed task description information corresponding to the current distributed task to be predicted for time consumption; obtaining historical distributed task resource sequences and historical distributed task time consumption sequences; inputting each of the historical distributed task types and historical distributed task difficulties, as well as the historical distributed task resource sequences and historical distributed task time consumption sequences, into a preset initial universal scaling law model for model training, and obtaining a trained universal scaling law model based on a preset mean square error loss function; wherein the initial universal scaling law model is composed of a parallel computing acceleration part, a parallel computing unaccelerable part, and a parallel communication computing time consumption increase part.

[0077] Based on the above embodiments, the formula corresponding to the initial universal scaling law model is: Among them, C represents the historical distributed task type; S represents the historical distributed task difficulty; Z represents the historical distributed task resource sequence; D represents the historical distributed task time sequence; f represents the neural network model with three different parameters to be trained, and the threshold of f is a positive number; θ1, θ2 and θ3 represent different parameters to be trained; e ε represents the time-consuming error term; where Represents the parallel computing acceleration part; Indicates the non-accelerable part of parallel computing; Indicates the increased time consumption due to parallel communication calculation.

[0078] On the basis of the above embodiments, the distributed task time prediction sequence result generation module 220 can be specifically used for: respectively normalizing each of the historical distributed task types by a preset one-hot encoding method to obtain each historical distributed normalized processing task type; converting the historical distributed task time sequence by a preset hour unit conversion method to obtain a historical distributed task time unit conversion sequence; inputting each of the historical distributed normalized processing task types and the historical distributed task difficulty, as well as the historical distributed task resource sequence and the historical distributed task time unit conversion sequence into a preset initial universal scaling law model for model training, and obtaining a trained universal scaling law model based on a preset mean square error loss function.

[0079] On the basis of the above embodiments, it also includes a model universal unique identification code generation module, which can be specifically used for: after obtaining the trained universal scaling law model according to the preset mean square error loss function, generating a model universal unique identification code according to the trained universal scaling law model; and jointly storing the universal scaling law model and the model universal unique identification code in a model database.

[0080] The distributed task resource time consumption prediction device based on the universal scaling law provided in the embodiment of the present invention can execute the distributed task resource time consumption prediction method based on the universal scaling law provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0081] Embodiment 3

[0082] Figure 3A schematic diagram of the structure of an electronic device 10 that can be used to implement the third embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0083] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0084] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0085] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a distributed task resource time consumption prediction method based on a general scaling law.

[0086] In some embodiments, the distributed task resource time consumption prediction method based on the general scaling law can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the distributed task resource time consumption prediction method based on the general scaling law described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the distributed task resource time consumption prediction method based on the general scaling law by any other appropriate means (for example, by means of firmware).

[0087] The method includes: obtaining a distributed task that is currently to be predicted for time consumption, and distributed task description information corresponding to the distributed task that is currently to be predicted for time consumption; wherein the distributed task description information includes a distributed task type and a distributed task difficulty; obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feeding back the distributed task time consumption prediction sequence result to a user.

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0090] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0093] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0096] Embodiment 4

[0097] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a distributed task resource time consumption prediction method based on a general scaling law when executed by a computer processor, the method comprising: obtaining a distributed task to be predicted for time consumption, and distributed task description information corresponding to the distributed task to be predicted for time consumption; wherein the distributed task description information comprises a distributed task type and a distributed task difficulty; obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, the distributed task type and the distributed task difficulty into a pre-trained general scaling law model to generate a distributed task time consumption prediction sequence result; and feeding back the distributed task time consumption prediction sequence result to a user.

[0098] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the distributed task resource time consumption prediction method based on the general scaling law provided in any embodiment of the present invention.

[0099] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0100] It is worth noting that in the above-mentioned embodiment of the distributed task resource duration prediction device based on the universal scaling law, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0101] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed task resource time consumption prediction method based on the general scaling law, characterized in that: include: Obtaining a distributed task that is currently to be predicted for time consumption, and distributed task description information corresponding to the distributed task that is currently to be predicted for time consumption; The distributed task description information includes the distributed task type and the distributed task difficulty; Obtaining a current distributed task resource sequence, and inputting the current distributed task resource sequence, distributed task type, and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time-consuming prediction sequence result; Feedback of the distributed task time-consuming prediction sequence results to the user is performed.

2. The method according to claim 1, characterized in that After inputting the current distributed task resource sequence, distributed task type and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time-consuming prediction sequence result, the method further includes: According to the current distributed task resource sequence and the distributed task time consumption prediction sequence results, a current distributed task resource time consumption prediction curve is generated by a preset curve drawing tool.

3. The method according to claim 2, characterized in that After generating the current distributed task resource time consumption prediction curve, the method further includes: Obtaining a maximum allowable time consumption threshold corresponding to the current distributed task to be predicted for time consumption; According to the current distributed task resource time consumption prediction curve, obtain the minimum curve time consumption value and the curve time consumption value when the task resource value is equal to 1; If the maximum allowed time consumption threshold is less than the minimum time consumption of the curve, then according to the current distributed task resource time consumption prediction curve, a target distributed task resource value corresponding to the minimum time consumption of the curve is obtained; If the maximum allowed time-consuming threshold is greater than the curve time-consuming value at which the task resource value is equal to 1, then determining the target distributed task resource value to be 1; If the maximum allowed time threshold satisfies the requirement that it is not less than the minimum time value of the curve and not greater than the curve time value when the task resource value is equal to 1, the target distributed task resource value corresponding to the maximum allowed time threshold is determined according to a preset binary search method; The target distributed task resource value is fed back to the user.

4. The method according to claim 1, characterized in that: Before obtaining the distributed task currently to be predicted for time consumption and the distributed task description information corresponding to the distributed task currently to be predicted for time consumption, the method further includes: Acquire multiple groups of historical distributed tasks, and historical distributed task types and historical distributed task difficulties corresponding to each of the historical distributed tasks; Obtain the historical distributed task resource sequence and the historical distributed task time sequence; Input each of the historical distributed task types and historical distributed task difficulties, as well as the historical distributed task resource sequence and the historical distributed task time sequence, into a preset initial universal scaling law model for model training, and obtain a trained universal scaling law model according to a preset mean square error loss function; The initial universal scaling law model is composed of a parallel computing acceleration part, a parallel computing non-accelerable part and a parallel communication computing time-increasing part.

5. The method according to claim 4, characterized in that Also includes: The formula corresponding to the initial universal scaling law model is: Among them, C represents the historical distributed task type; S represents the historical distributed task difficulty; Z represents the historical distributed task resource sequence; D represents the historical distributed task time sequence; f represents the neural network model with three different parameters to be trained, and the threshold of f is a positive number; θ1, θ2 and θ3 represent different parameters to be trained; e ε represents the time-consuming error term; in, Represents the parallel computing acceleration part; Indicates the non-accelerable part of parallel computing; Indicates the increased time consumption due to parallel communication calculation.

6. The method according to claim 5, characterized in that The historical distributed task types and the historical distributed task difficulties, as well as the historical distributed task resource sequences and the historical distributed task time sequences, are input into a preset initial universal scaling law model for model training, and a trained universal scaling law model is obtained according to a preset mean square error loss function, including: Each of the historical distributed task types is normalized by a preset one-hot encoding method to obtain each historical distributed normalized task type; The historical distributed task time-consuming sequence is converted into units by a preset hour unit conversion method to obtain a historical distributed task time-consuming unit conversion sequence; The historical distributed normalized processing task type and the historical distributed task difficulty, as well as the historical distributed task resource sequence and the historical distributed task time-consuming unit conversion sequence, are input into a preset initial universal scaling law model for model training, and a trained universal scaling law model is obtained based on a preset mean square error loss function.

7. The method according to claim 6, characterized in that After obtaining the trained universal scaling law model according to the preset mean square error loss function, the method further includes: Generate a universal unique identification code for the model based on the trained universal scaling law model; The universal scaling law model and the model universal unique identifier are jointly stored in a model database.

8. A distributed task resource time consumption prediction device based on the general scaling law, characterized in that: include: A distributed task description information acquisition module, used to acquire the distributed task currently to be predicted for time consumption, and the distributed task description information corresponding to the distributed task currently to be predicted for time consumption; The distributed task description information includes the distributed task type and the distributed task difficulty; A distributed task time consumption prediction sequence result generation module is used to obtain the current distributed task resource sequence, and input the current distributed task resource sequence, distributed task type and distributed task difficulty into a pre-trained universal scaling law model to generate a distributed task time consumption prediction sequence result; The distributed task time-consuming prediction sequence result feedback module is used to feed back the distributed task time-consuming prediction sequence result to the user.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for predicting the resource time consumption of distributed tasks based on the universal scaling law as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed task resource time consumption prediction method based on the general scaling law as described in any one of claims 1 to 7 when executed.

Images