Resource management method, system and device and storage medium

Abstract

The embodiment of the invention discloses a resource management method, system and device and a storage medium, and relates to the technical field of data processing.The method comprises the steps that at the first moment, multiple pieces of initial operation data of a target task are obtained; determining the influence degree of each piece of initial operation data in the multiple pieces of initial operation data on the actual operation resources; determining at least one piece of target operation data from the multiple pieces of initial operation data according to the influence degree of each piece of initial operation data on the actual operation resources; inputting the at least one piece of target operation data into a pre-trained resource prediction model, and predicting the at least one piece of target operation data by using the pre-trained resource prediction model to obtain a task prediction operation resource required by the target task; and determining a target operation resource corresponding to the target task according to the task prediction operation resource. The target operation resource corresponding to the target task can be quickly determined, so that normal operation of various businesses in financial businesses and medical businesses is maintained.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a resource management method, system, device and storage medium. Background Art

[0002] In the field of big data processing, Apache Spark, with its distributed computing capabilities, is widely used in financial services such as financial risk control modeling and real-time transaction analysis, as well as in many medical services such as medical image processing and genome sequencing. However, traditional Spark resource management mechanisms, when dealing with tasks such as high-frequency financial trading and high-concurrency medical diagnosis, often rely on manual configuration and adjustment of task execution resources. This leads to inefficient task execution resource allocation, which in turn adversely impacts the normal operation of many financial and medical services.

[0003] Application Contents

[0004] In view of this, one of the purposes of this application is to provide a resource management method, system, device and storage medium that can improve the efficiency of determining task operation resources, thereby maintaining the normal operation of various businesses in financial and medical businesses.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a resource management method, the method comprising:

[0007] At a first moment, a plurality of initial running data of the target task is obtained, where the first moment is any moment during the running process of the target task;

[0008] determining the degree of influence of each of the plurality of initial operating data on the actual operating resources;

[0009] determining at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources;

[0010] Inputting at least one target operation data into a pre-trained resource prediction model, and using the pre-trained resource prediction model to predict the at least one target operation data to obtain task predicted operation resources required for the target task, wherein the pre-trained resource prediction model is trained by historical operation data of a plurality of historical tasks;

[0011] Determine the target operating resources corresponding to the target task based on the task's predicted operating resources.

[0012] In one possible implementation, determining the degree of impact of each of the multiple initial operating data on the actual operating resources includes:

[0013] Determining a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy;

[0014] Determining a second evaluation score for each of the plurality of initial operating data according to a second preset evaluation strategy;

[0015] Determining a target evaluation score for each initial operation data according to the first evaluation score and the second evaluation score for each initial operation data;

[0016] Based on the target evaluation score of each initial operation data, the impact degree of each initial operation data on the actual operation resources is determined.

[0017] In one possible implementation, determining a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy includes:

[0018] Discretize multiple initial running data into M intervals;

[0019] Discretize the actual operating resources into N intervals;

[0020] Determine M first distribution frequencies of each initial operating data in the M intervals, and N second distribution frequencies of the actual operating resources in the N intervals;

[0021] Determine the joint distribution frequency of each initial operation data and actual operation resources;

[0022] A first evaluation score for each initial operating data is determined according to the M first distribution frequencies, the N second distribution frequencies, and the joint distribution frequency.

[0023] In one possible implementation, determining a second evaluation score for each of the plurality of initial operating data according to the second preset evaluation strategy includes:

[0024] Determine a first change in each initial operating data between a first moment and a second moment, and a corresponding second change in the actual operating resource between the first moment and the second moment;

[0025] A second evaluation score of each initial operating data is determined based on the first change amount corresponding to each initial operating data and the second change amount corresponding to the actual operating resource.

[0026] In one possible implementation, determining a target evaluation score for each initial operating data according to the first evaluation score and the second evaluation score for each initial operating data includes:

[0027] Normalizing the first evaluation score and the second evaluation score respectively to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score;

[0028] Determine the task type of the target task;

[0029] According to the task type, the target weight value corresponding to the target task is determined from the pre-configured task weight relationship table;

[0030] A target evaluation score is determined based on the first normalized score, the second normalized score and the target weight value.

[0031] In one possible implementation, after normalizing the first evaluation score and the second evaluation score to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score, the method further includes:

[0032] Comparing the score difference between the first normalized score and the second normalized score with a preset threshold to obtain a comparison result;

[0033] When the comparison result indicates that the score difference is greater than a preset threshold, the normalized score with the largest value between the first normalized score and the second normalized score is determined as the target evaluation score.

[0034] In one possible implementation, determining target operating resources corresponding to a target task based on task predicted operating resources includes:

[0035] At a first moment, obtaining resource monitoring data of a processing system processing a target task;

[0036] Determine resource constraints based on resource monitoring data;

[0037] Determine the target operating resources corresponding to the target task based on the task's predicted operating resources and resource constraints.

[0038] In a second aspect, an embodiment of the present application provides a resource management system, the system comprising:

[0039] An acquisition module, configured to acquire a plurality of initial operation data of a target task at a first moment, where the first moment is any moment during the operation of the target task;

[0040] A first determining module is used to determine the impact degree of each initial operating data among the multiple initial operating data on the actual operating resources;

[0041] A second determining module is configured to determine at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources;

[0042] A prediction module is configured to input at least one target operation data into a pre-trained resource prediction model, and use the pre-trained resource prediction model to predict the at least one target operation data to obtain the task predicted operation resources required for the target task, wherein the pre-trained resource prediction model is trained by historical operation data of multiple historical tasks;

[0043] The third determination module is used to determine the target operating resources corresponding to the target task based on the task predicted operating resources.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, it implements the resource management method provided in the first aspect.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by one or more processors, it implements the resource management method provided in the first aspect.

[0046] The resource management method provided in the embodiment of the present application can obtain multiple initial operating data of the target task at the first moment, and determine the degree of influence of each of the multiple initial operating data on the actual operating resources. Then, based on the degree of influence of each initial operating data on the actual operating resources, at least one target operating data can be determined from the multiple initial operating data, and the at least one target operating data can be input into a pre-trained resource prediction model. The pre-trained resource prediction model is used to predict the at least one target operating data to obtain the task prediction operating resources required for the target task. Finally, based on the task prediction operating resources, the target operating resources corresponding to the target task can be quickly determined, thereby maintaining the normal operation of various businesses in financial and medical businesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. It should be understood that the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flowchart of a resource management method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart for determining the degree of influence involved in a resource management method provided in an embodiment of the present application;

[0050] Figure 3 A functional module schematic diagram of a resource management system provided for an embodiment of the present application is shown in FIG. 1.

[0051] Figure 4 A hardware structure schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 2.

[0052] Legend of reference signs:

[0053] 300, a resource management system;

[0054] 310, an acquisition module;

[0055] 320, a first determination module;

[0056] 330, a second determination module;

[0057] 340, a prediction module;

[0058] 350, a third determination module;

[0059] 401, a processor;

[0060] 402, a memory;

[0061] 403, a communication interface;

[0062] 410, a bus. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0065] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0066] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0067] In the description of this application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the invented product is usually placed when in use. It is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on this application.

[0068] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0069] It should be noted that, in the absence of conflict, the features in the embodiments of this application can be combined with each other.

[0070] Furthermore, in the embodiments of the present application, the term "connection" may refer to "electrical connection" or "direct connection." "Electrical connection" may refer to a direct electrical connection between two components or an electrical connection between two components via one or more normally open tubes or other components.

[0071] To facilitate a better understanding of the solutions of the embodiments of the present application, the relevant technologies are first introduced below.

[0072] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0073] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0074] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0075] In order to solve the technical problems in the background technology, the embodiments of the present application provide a resource management method, a resource management system, an electronic device, and a computer-readable storage medium.

[0076] See Figure 1 , Figure 1 A flowchart of an asset management method provided in an embodiment of the present application. The resource management method can be applied to the resource management systems or electronic devices in the following embodiments, where the electronic devices include personal computers, servers, mobile devices, cloud computing platforms, and supercomputers.

[0077] The resource management method will be described below from the perspective of application to electronic devices. The resource management method specifically includes the following steps 110 to 150:

[0078] Step 110 , obtaining a plurality of initial operation data of the target task at a first moment, where the first moment is any moment during the operation of the target task.

[0079] Step 120: Determine the impact of each of the multiple initial operating data on the actual operating resources.

[0080] Step 130 : determining at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources.

[0081] Step 140: Input at least one target operation data into a pre-trained resource prediction model, and use the pre-trained resource prediction model to predict at least one target operation data to obtain the task prediction operation resources required for the target task, wherein the pre-trained resource prediction model is trained by historical operation data of several historical tasks.

[0082] Step 150: Determine target operating resources corresponding to the target task based on the predicted operating resources of the task.

[0083] The resource management method provided in the embodiment of the present application can obtain multiple initial operating data of the target task at the first moment, and determine the degree of influence of each of the multiple initial operating data on the actual operating resources. Then, based on the degree of influence of each initial operating data on the actual operating resources, at least one target operating data can be determined from the multiple initial operating data, and the at least one target operating data can be input into a pre-trained resource prediction model. The pre-trained resource prediction model is used to predict the at least one target operating data to obtain the task prediction operating resources required for the target task. Finally, based on the task prediction operating resources, the target operating resources corresponding to the target task can be quickly determined, thereby maintaining the normal operation of various businesses in financial and medical businesses.

[0084] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. AI refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0085] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0086] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0087] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0088] The following will be Figure 1 Each step of the method is described in detail.

[0089] In step 110, the electronic device can obtain multiple initial operation data corresponding to the target task at any time during the operation of the target task, realize real-time capture of the initial operation data of the target task, and provide a real-time data basis for resource demand analysis of the target task.

[0090] The first moment may be any sampling time point in the life cycle of the target task, such as the third minute or the fifth minute after the target task is started.

[0091] The target task may be a task that currently requires resource management.

[0092] For example, in a financial business scenario, target tasks may include, but are not limited to, real-time transaction monitoring and early warning tasks, financial data cleaning tasks, asset risk prediction tasks, portfolio optimization tasks, credit approval tasks, etc.

[0093] In medical business scenarios, target tasks may include but are not limited to medical data monitoring and analysis tasks, medical image analysis tasks, medical knowledge graph construction tasks, medical resource management and optimization tasks, etc.

[0094] In some embodiments, the target task may be a distributed computing task, such as a Spark task or a Flink task.

[0095] The aforementioned initial operation data may refer to data of different dimensions related to the operation status of the target task.

[0096] In some embodiments, the initial operation data may include task type, task computational complexity, task progress, task failure information, and processing time of the task at different processing stages.

[0097] It should be noted that the initial operating data of the target task obtained by the electronic device at different times may be different. In this way, the electronic device can determine the corresponding target operating resources based on the initial operating data of the target task at different times, which can improve the accuracy and reliability of the determined target operating resources.

[0098] Exemplarily, the processing stages of the target task include three stages, such as the first processing stage, the second processing stage, and the third processing stage. If the target task is in the first stage at the first moment and does not include the end moment of the first processing stage, then the initial running data of the target task does not include the processing time corresponding to the first processing stage.

[0099] If the target task is in the third processing stage at the first moment and does not include the end moment of the third processing stage, then the initial running data of the target task may include processing times corresponding to the first processing stage and the second processing stage.

[0100] In some embodiments, the electronic device may determine a plurality of initial running data of the target task by parsing a log file corresponding to the target task.

[0101] In some embodiments, the first moment can also be a moment corresponding to a preset time interval, for example, a moment corresponding to three minutes after the start of the target task. In this case, the first moment can be regarded as a trigger moment, that is, if the first moment is reached, the electronic device can perform an operation to obtain multiple initial running data of the target task.

[0102] In some embodiments, the first moment may also be the start moment of a processing phase, such as the start moment of the first processing phase, the start moment of the second processing phase, and the start moment of the third processing phase described above. At the start moment of the first processing phase, the electronic device may obtain initial operating data such as the task type and task computational complexity of the target task. At the start moment of the second processing phase, the electronic device may obtain initial operating data such as the task type and task computational complexity of the target task, the processing time consumed in the first processing phase, and the number of failures in the first processing phase.

[0103] In steps 120 and 130, the electronic device can screen the multiple initial operating data by determining the degree of impact of each initial operating data on the actual operating resources, thereby determining at least one target operating data having a significant impact on the actual operating resources. In other words, the electronic device can dynamically determine key operating data, i.e., at least one target operating data, from the multiple initial operating data based on the degree of impact of each initial operating data on the actual operating resources.

[0104] Electronic devices can use numerical indicators to represent the degree of impact.

[0105] For example, for the initial operating data data1, if the value corresponding to the numerical indicator determined by the electronic device is 0, the electronic device may determine that the initial operating data data1 has no impact on the actual operating resources.

[0106] For example, for the initial operating data data1, if the value corresponding to the numerical indicator determined by the electronic device is 0.3, the electronic device may determine that the initial operating data data1 has a weak impact on the actual operating resources.

[0107] For example, for the initial operating data data1, if the value corresponding to the numerical indicator determined by the electronic device is 0.7, the electronic device may determine that the initial operating data data1 has a strong impact on the actual operating resources.

[0108] For example, for the initial operating data data2, if the value corresponding to the numerical indicator determined by the electronic device is 1, the electronic device may determine that the initial operating data data1 has a decisive influence on the actual operating resources.

[0109] The actual running resources mentioned above refer to the system resources actually occupied and used by the target task during its running process.

[0110] In some embodiments, actual operating resources include the number of CPU cores, memory size, disk space, and network bandwidth.

[0111] In some embodiments, the electronic device may use a dual-modal evaluation mechanism to determine the degree of impact of each of the plurality of initial operating data on the actual operating resources, thereby further improving the accuracy and reliability of the impact degree determined by the electronic device.

[0112] In step 140, the electronic device may input at least one target operating data determined in the aforementioned embodiment into a pre-trained resource prediction model to use the pre-trained resource prediction model to determine the task prediction operating resources required for the target task. This eliminates the need to manually set and adjust the operating resources required for the target task, greatly improving the efficiency of determining the operating resources and thereby maintaining the normal operation of various businesses in financial and medical services.

[0113] The pre-trained resource prediction model can predict the operating resources required for any task. The electronic device can train the machine learning model based on the historical operating data of several historical tasks until the training end conditions are met, ending the training and obtaining the pre-trained predicted operating resources.

[0114] In some embodiments, the training end condition includes at least one of the following:

[0115] The training duration is greater than or equal to the preset duration threshold;

[0116] The number of training times is greater than or equal to the preset number threshold;

[0117] The response time is less than or equal to the preset time.

[0118] The above training end conditions can also be modified according to actual needs, such as addition, deletion, etc., and the embodiments of this application will not be introduced one by one here.

[0119] It should be noted that, in the process of training a machine learning model based on the historical operation data of several historical tasks, the electronic device can train the historical operation data of any historical figure according to different combinations of quantity, data type, etc., to support the input of multi-dimensional operation data and realize the prediction of multi-dimensional operation data.

[0120] For example, at a first moment, the electronic device may input the determined task type and task computational complexity into a pre-trained resource prediction model. At a first moment, the electronic device may also input the determined task type, task computational complexity, and processing time corresponding to the previous task processing stage into the pre-trained resource prediction model. At a first moment, the electronic device may also input information on multiple dimensions, such as the determined task type, task computational complexity, processing time corresponding to the previous task processing stage, and number of failures, into the pre-trained resource prediction model to obtain the corresponding predicted task operating resources.

[0121] In step 150 , the electronic device may convert the predicted value, ie, the predicted task operating resource, into an executable physical resource configuration based on the task predicted operating resource to determine the target operating resource.

[0122] Exemplarily, the electronic device converts the predicted running resources of the task into an executable physical resource configuration, and the corresponding physical resource configuration can be expressed as CPU: 8 cores; memory: 64GB; GPU: 2; network bandwidth: 10Gbps.

[0123] See Figure 2 , Figure 2 A flowchart for determining the degree of influence involved in a resource management method provided in an embodiment of the present application.

[0124] In one possible implementation, step 120 of determining the degree of influence of each of the multiple initial operating data on the actual operating resources includes but is not limited to steps 210 to 240:

[0125] Step 210 : Determine a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy.

[0126] Step 220 : Determine a second evaluation score for each of the plurality of initial operating data according to a second preset evaluation strategy.

[0127] Step 230 : determining a target evaluation score for each initial operation data according to the first evaluation score and the second evaluation score for each initial operation data.

[0128] Step 240 : Determine the degree of influence of each initial operating data on the actual operating resources based on the target evaluation score of each initial operating data.

[0129] The embodiment of the present application determines the first evaluation score and the second evaluation score through a dual-modal evaluation mechanism, determines the target evaluation score through the first evaluation score and the second evaluation score, and then determines the degree of influence of each initial operating data on the actual operating resources based on the target evaluation score of each initial operating data, which can improve the accuracy and reliability of the determined degree of influence.

[0130] In some embodiments, the first preset evaluation strategy includes determining a first evaluation score for each of the plurality of initial operating data according to the first preset evaluation strategy based on a Pearson correlation coefficient analysis, including:

[0131] Based on the Pearson correlation coefficient formula, determine the Pearson correlation coefficient between each initial operation data and the actual operation resources;

[0132] The absolute value of the Pearson correlation coefficient corresponding to each initial operating data is determined as the first evaluation score of each initial operating data.

[0133] The first evaluation score determined based on the Pearson correlation coefficient is between 0 and 1.

[0134] The above-mentioned Pearson correlation coefficient formula can be referred to the introduction in the relevant technology, and will not be repeated here in the embodiment of the present application.

[0135] In some embodiments, determining the degree of impact of each initial operating data on actual operating resources based on the target evaluation score of each initial operating data includes:

[0136] The target evaluation score of each initial operation data is determined as the degree of impact of each initial operation data on the actual operation resources;

[0137] or,

[0138] According to the target evaluation score of each initial operation data, the impact degree of each initial operation data on the actual operation resources is determined from the preset mapping relationship table, wherein the preset mapping relationship table includes a plurality of mapping relationships between the impact degree and the evaluation score.

[0139] In one possible implementation, determining a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy includes:

[0140] Discretize multiple initial running data into M intervals;

[0141] Discretize the actual operating resources into N intervals;

[0142] Determine M first distribution frequencies of each initial operating data in the M intervals, and N second distribution frequencies of the actual operating resources in the N intervals;

[0143] Determine the joint distribution frequency of each initial operation data and actual operation resources;

[0144] A first evaluation score for each initial operating data is determined according to the M first distribution frequencies, the N second distribution frequencies, and the joint distribution frequency.

[0145] In some embodiments, the electronic device may discretize the multiple initial operating data and actual operating resources using an equal frequency binning method. For example, the actual operating resources, such as CPU utilization, may be divided into 10 intervals from 0 to 100%.

[0146] In some embodiments, the electronic device may use a probability mass function to calculate the distribution frequencies, such as the first distribution frequency, the second distribution frequency, and the joint distribution frequency.

[0147] In some embodiments, the first evaluation score of the initial operation data may be determined by the following formula:

[0148]

[0149] Wherein, G1 represents the first evaluation score of an initial running data data1;

[0150] P c represents the joint distribution frequency;

[0151] P i represents the i-th first distribution frequency;

[0152] P j represents the jth second distribution frequency.

[0153] For each of the multiple initial operating data, a corresponding first evaluation score can be determined using the above formula (1).

[0154] In one possible implementation, determining a second evaluation score for each of the plurality of initial operating data according to the second preset evaluation strategy includes:

[0155] Determine a first change in each initial operating data between a first moment and a second moment, and a corresponding second change in the actual operating resource between the first moment and the second moment;

[0156] A second evaluation score of each initial operating data is determined based on the first change amount corresponding to each initial operating data and the second change amount corresponding to the actual operating resource.

[0157] It should be noted that the second moment is the forward adjacent moment of the first moment. If the first moment is the initial moment of the target task execution, then the second change in the actual operating resources corresponding to the first moment and the second moment is the specific value of the actual operating resources corresponding to the first moment.

[0158] In some embodiments, the electronic device may determine the second evaluation score based on the following formula:

[0159]

[0160] Wherein, G2 represents the second evaluation score of an initial running data data1;

[0161] △X1 represents the first change of data1;

[0162] △Y1 represents the second change of data1;

[0163] M represents the total number of initial run data.

[0164] In one possible implementation, determining a target evaluation score for each initial operating data according to the first evaluation score and the second evaluation score for each initial operating data includes:

[0165] Normalizing the first evaluation score and the second evaluation score respectively to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score;

[0166] Determine the task type of the target task;

[0167] According to the task type, the target weight value corresponding to the target task is determined from the pre-configured task weight relationship table;

[0168] A target evaluation score is determined based on the first normalized score, the second normalized score and the target weight value.

[0169] This application implements the dynamic determination of the target weight value corresponding to the target task based on the task type of the target task, which can improve the accuracy and reliability of determining the target evaluation score.

[0170] The above-mentioned normalization processing can eliminate the difference between the first evaluation score and the second evaluation score determined based on different preset evaluation strategies, thereby improving the accuracy of determining the target evaluation score.

[0171] Taking the financial business scenario as an example, the above task types may include high-frequency trading, risk control, and end-of-day settlement, with corresponding weight values ​​of 0.8, 0.4, and 0.2, respectively.

[0172] Taking the medical business scenario as an example, the above task types may include emergency impact analysis, gene sequence analysis, and health testing, with corresponding weight values ​​of 0.7, 0.25, and 0.5, respectively.

[0173] If the target task type is high-frequency trading, the target evaluation score G g It can be calculated by the following formula:

[0174] G g =β×NOR1+(1-β)×NOR2 (3);

[0175] Where β represents the target weight value, and β = 0.8;

[0176] NOR1 represents the first normalized score of an initial running data data1;

[0177] NOR2 represents a second normalized score of the initial running data data1.

[0178] In one possible implementation, after normalizing the first evaluation score and the second evaluation score to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score, the method further includes:

[0179] Comparing the score difference between the first normalized score and the second normalized score with a preset threshold to obtain a comparison result;

[0180] When the comparison result indicates that the score difference is greater than a preset threshold, the normalized score with the largest value between the first normalized score and the second normalized score is determined as the target evaluation score.

[0181] The embodiment of the present application takes into account that the evaluation scores determined based on different preset evaluation strategies may differ significantly, and the gap may still be obvious even after normalization (that is, the comparison result in this embodiment indicates that the score difference is greater than the preset threshold). The normalized score with the largest value can be determined as the target evaluation score.

[0182] In some embodiments, if the comparison result determined by the electronic device indicates that the score difference is greater than a preset threshold, the electronic device may generate a warning message, which may be used to remind relevant staff to conduct verification.

[0183] In one possible implementation, determining target operating resources corresponding to a target task based on task predicted operating resources includes:

[0184] At a first moment, obtaining resource monitoring data of a processing system processing a target task;

[0185] Determine resource constraints based on resource monitoring data;

[0186] Determine the target operating resources corresponding to the target task based on the task's predicted operating resources and resource constraints.

[0187] The embodiment of the present application dynamically determines resource constraints based on resource monitoring data of the processing system at different times, which can avoid the phenomenon that pure demand forecasts are inconsistent with actual system capabilities and further improve the accuracy and reliability of the determined target operating resources.

[0188] In some embodiments, the electronic device may determine resource constraints based on physical resource limits, energy consumption limits, data privacy, and the like.

[0189] Taking the physical resource upper limit of the number of available CPU cores as an example, if the number of CPU cores required in the task's predicted running resources is greater than the number of available CPU cores, the number of available CPU cores is limited to the upper limit of the number of CPU cores, forming a resource constraint condition.

[0190] In some embodiments, the target operation data of the target task in the aforementioned embodiment can be used as historical operation data, the electronic device can update the historical training data, and iteratively train and tune the pre-trained resource prediction model in the aforementioned embodiment based on the updated historical training data.

[0191] In some embodiments, after the electronic device obtains a plurality of initial running data of the target task at the first moment, the electronic device further includes:

[0192] Preprocessing is performed on the plurality of initial running data, where the preprocessing includes at least one of data cleaning, data denoising, and data standardization.

[0193] Corresponding to the above method embodiment, the present application embodiment also provides a resource management system, see Figure 3 , Figure 3 This is a functional module diagram of a resource management system provided in an embodiment of the present application, wherein the resource management system 300 includes:

[0194] The acquisition module 310 is configured to acquire a plurality of initial operation data of the target task at a first moment, where the first moment is any moment during the operation of the target task.

[0195] The first determining module 320 is configured to determine the degree of influence of each of the plurality of initial operating data on the actual operating resources.

[0196] The second determining module 330 is configured to determine at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources.

[0197] The prediction module 340 is used to input at least one target operation data into a pre-trained resource prediction model, use the pre-trained resource prediction model to predict the at least one target operation data, and obtain the task prediction operation resources required for the target task, wherein the pre-trained resource prediction model is trained by the historical operation data of several historical tasks.

[0198] The third determining module 350 is configured to determine target operating resources corresponding to the target task according to the task predicted operating resources.

[0199] The resource management system provided by the embodiment of the present application can obtain multiple initial operating data of the target task at the first moment through the acquisition module, and determine the degree of influence of each initial operating data in the multiple initial operating data on the actual operating resources through the first determination module. Then, according to the degree of influence of each initial operating data on the actual operating resources, at least one target operating data is determined from the multiple initial operating data through the second determination module, and the at least one target operating data is input into a pre-trained resource prediction model through the prediction module, and the at least one target operating data is predicted using the pre-trained resource prediction model to obtain the task prediction operating resources required for the target task. Finally, according to the task prediction operating resources, the third determination module can be used to quickly determine the target operating resources corresponding to the target task, thereby maintaining the normal operation of various businesses in financial and medical businesses.

[0200] In a possible implementation, the first determining module 320 is further configured to:

[0201] Determining a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy;

[0202] Determining a second evaluation score for each of the plurality of initial operating data according to a second preset evaluation strategy;

[0203] Determining a target evaluation score for each initial operation data according to the first evaluation score and the second evaluation score for each initial operation data;

[0204] Based on the target evaluation score of each initial operation data, the impact degree of each initial operation data on the actual operation resources is determined.

[0205] In a possible implementation, the first determining module 320 includes a first determining submodule, which is configured to:

[0206] Discretize multiple initial running data into M intervals;

[0207] Discretize the actual operating resources into N intervals;

[0208] Determine M first distribution frequencies of each initial operating data in the M intervals, and N second distribution frequencies of the actual operating resources in the N intervals;

[0209] Determine the joint distribution frequency of each initial operation data and actual operation resources;

[0210] A first evaluation score for each initial operating data is determined according to the M first distribution frequencies, the N second distribution frequencies, and the joint distribution frequency.

[0211] In a possible implementation, the first determining module 320 includes a second determining submodule, where the second determining submodule is configured to:

[0212] Determine a first change in each initial operating data between a first moment and a second moment, and a corresponding second change in the actual operating resource between the first moment and the second moment;

[0213] A second evaluation score of each initial operating data is determined based on the first change amount corresponding to each initial operating data and the second change amount corresponding to the actual operating resource.

[0214] In a possible implementation, the first determining module 320 includes a third determining submodule, where the third determining submodule is configured to:

[0215] Normalizing the first evaluation score and the second evaluation score respectively to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score;

[0216] Determine the task type of the target task;

[0217] According to the task type, the target weight value corresponding to the target task is determined from the pre-configured task weight relationship table;

[0218] A target evaluation score is determined based on the first normalized score, the second normalized score and the target weight value.

[0219] In a possible implementation, the resource management system 300 further includes a comparison module, which is configured to:

[0220] Comparing the score difference between the first normalized score and the second normalized score with a preset threshold to obtain a comparison result;

[0221] When the comparison result indicates that the score difference is greater than a preset threshold, the normalized score with the largest value between the first normalized score and the second normalized score is determined as the target evaluation score.

[0222] In a possible implementation, the third determining module 350 is further configured to:

[0223] At a first moment, obtaining resource monitoring data of a processing system processing a target task;

[0224] Determine resource constraints based on resource monitoring data;

[0225] Determine the target operating resources corresponding to the target task based on the task's predicted operating resources and resource constraints.

[0226] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0227] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0228] Specifically, the processor 401 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0229] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0230] In some embodiments, the memory 402 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods provided according to the embodiments of the present application.

[0231] The processor 401 implements the method provided in the above embodiment by reading and executing the computer program instructions stored in the memory 402 .

[0232] In one example, the electronic device may further include a communication interface 403 and a bus 410. The processor 401, the memory 402, and the communication interface 403 are connected via the bus 410 and communicate with each other.

[0233] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0234] Bus 410 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 410 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0235] In addition, in combination with the methods provided in the above embodiments, embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the methods in the above embodiments is implemented.

[0236] In addition, in conjunction with the methods provided in the above embodiments, embodiments of the present application may be implemented by providing a computer program product. This program product is stored in a storage medium and executed by at least one processor to implement the various processes of the method embodiments provided in the above embodiments, and can achieve similar or identical technical effects. To avoid repetition, these are not described here.

[0237] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0238] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0239] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0240] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0241] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A resource management method, characterized in that: The method comprises: At a first moment, obtaining a plurality of initial running data of a target task, where the first moment is any moment during the running of the target task; determining an impact degree of each of the plurality of initial operating data on actual operating resources; determining at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources; Inputting the at least one target operation data into a pre-trained resource prediction model, and using the pre-trained resource prediction model to predict the at least one target operation data to obtain task predicted operation resources required for the target task, wherein the pre-trained resource prediction model is trained by historical operation data of a plurality of historical tasks; The target operating resources corresponding to the target task are determined according to the predicted operating resources of the task.

2. The method according to claim 1, characterized in that Determining the degree of influence of each of the plurality of initial operating data on the actual operating resources includes: Determining a first evaluation score for each of the plurality of initial operating data according to a first preset evaluation strategy; Determining a second evaluation score for each of the plurality of initial operating data according to a second preset evaluation strategy; determining a target evaluation score for each of the initial operating data according to the first evaluation score and the second evaluation score for each of the initial operating data; Based on the target evaluation score of each of the initial operating data, the degree of influence of each of the initial operating data on the actual operating resources is determined.

3. The method according to claim 2, characterized in that Determining a first evaluation score for each of the plurality of initial operating data according to the first preset evaluation strategy includes: Discretizing the plurality of initial operating data into M intervals; Discretize the actual operating resources into N intervals; Determining M first distribution frequencies of each of the initial operating data in the M intervals, and N second distribution frequencies of the actual operating resources in the N intervals; determining a joint distribution frequency of each of the initial operating data and the actual operating resources; A first evaluation score of each of the initial operating data is determined according to the M first distribution frequencies, the N second distribution frequencies and the joint distribution frequency.

4. The method according to claim 2, characterized in that Determining a second evaluation score of each of the plurality of initial operating data according to the second preset evaluation strategy includes: Determining a first change in each of the initial operating data between the first moment and the second moment, and a corresponding second change in the actual operating resource between the first moment and the second moment; Based on the first change amount corresponding to each of the initial operating data and the second change amount corresponding to the actual operating resources, a second evaluation score of each of the initial operating data is determined.

5. The method according to claim 2, characterized in that Determining a target evaluation score for each of the initial operating data based on the first evaluation score and the second evaluation score for each of the initial operating data includes: Normalizing the first evaluation score and the second evaluation score to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score; Determining the task type of the target task; Determining the target weight value corresponding to the target task from a pre-configured task weight relationship table according to the task type; The target evaluation score is determined according to the first normalized score, the second normalized score and the target weight value.

6. The method according to claim 5, characterized in that After normalizing the first evaluation score and the second evaluation score to obtain a first normalized score corresponding to the first evaluation score and a second normalized score corresponding to the second evaluation score, the method further includes: Comparing a score difference between the first normalized score and the second normalized score with a preset threshold to obtain a comparison result; In a case where the comparison result indicates that the score difference is greater than the preset threshold, the normalized score with the largest value between the first normalized score and the second normalized score is determined as the target evaluation score.

7. The method according to claim 1, characterized in that The step of determining target operating resources corresponding to the target task based on the predicted operating resources of the task includes: At the first moment, obtaining resource monitoring data of a processing system processing the target task; determining resource constraints based on the resource monitoring data; The target operating resources corresponding to the target task are determined according to the task predicted operating resources and the resource constraint conditions.

8. A resource management system, characterized in that: The system comprises: an acquisition module, configured to acquire a plurality of initial operation data of a target task at a first moment, where the first moment is any moment during the operation of the target task; A first determining module is configured to determine the degree of influence of each of the plurality of initial operating data on the actual operating resources; a second determining module, configured to determine at least one target operating data from the plurality of initial operating data according to the degree of influence of each initial operating data on the actual operating resources; a prediction module, configured to input the at least one target operation data into a pre-trained resource prediction model, and use the pre-trained resource prediction model to predict the at least one target operation data to obtain the task predicted operation resources required for the target task, wherein the pre-trained resource prediction model is trained by historical operation data of a plurality of historical tasks; The third determining module is configured to determine target operating resources corresponding to the target task based on the predicted operating resources of the task.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by one or more processors, implements the method according to any one of claims 1 to 7.

Images