Resource scheduling method, device and equipment and readable storage medium

Abstract

The invention discloses a resource scheduling method, device and equipment and a readable storage medium, and the method comprises the steps: determining the weighted popularity of resources in a resource pool based on a consumption record, and determining the popularity of the resources through employing the weighted popularity; selecting hot resources from the resource pool by using the popularity and synchronously backing up the hot resources to machine rooms in different regions; receiving a request of a client for accessing a target resource, and determining a candidate machine room in which the target resource is stored; if the plurality of candidate machine rooms exist, determining a target machine room from the plurality of candidate machine rooms by utilizing the network resource information of the candidate machine rooms; and generating a resource consumption task matched with the target machine room, and feeding back the resource consumption task to the client, so that the client accesses a target resource in the target machine room according to the resource consumption task. The method and the device have the technical effect that the occurrence rate of bandwidth extrusion, delay and the like can be reduced when the request is processed.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a resource scheduling method, apparatus, device, and readable storage medium. Background Art

[0002] Currently, most online cloud storage systems use Object Storage Service (OSS) as their file storage solution. Metadata and file content are stored and managed independently. When a user consumes data, the scheduling service queries the metadata and dispatches the user's request to the appropriate data center. The data center's access gateway reads the file content stored in OSS and returns it to the user. Users can only access file resources from the corresponding data center, which can lead to bandwidth constraints and latency issues when consuming data.

[0003] In summary, how to effectively solve problems such as resource scheduling is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0004] The purpose of this application is to provide a resource scheduling method, apparatus, device and readable storage medium that can reduce the occurrence rate of bandwidth squeeze, delay and the like when processing requests.

[0005] To solve the above technical problems, this application provides the following technical solutions:

[0006] A resource scheduling method, comprising:

[0007] Determining weighted popularity of resources in the resource pool based on consumption records, and determining popularity of the resources using the weighted popularity;

[0008] Selecting hot resources from the resource pool using the popularity and synchronously backing up the hot resources to computer rooms located in different regions;

[0009] Receive a request from a client to access a target resource, and determine a candidate computer room storing the target resource;

[0010] If there are multiple candidate computer rooms, determine the target computer room from the multiple candidate computer rooms using the network resource information of the candidate computer rooms;

[0011] A resource consumption task matching the target computer room is generated, and the resource consumption task is fed back to the client, so that the client accesses the target resource located in the target computer room according to the resource consumption task.

[0012] Preferably, determining a target computer room from a plurality of candidate computer rooms by using the network resource information of the candidate computer rooms comprises:

[0013] Obtaining network resource information of the candidate computer room;

[0014] Eliminating abnormal computer rooms from the candidate computer rooms by using the network resource information;

[0015] After eliminating abnormal computer rooms, using the network resource information, determining a sub-weight of the candidate computer room; the sub-weight is at least one of a bandwidth cost weight, an operation weight, a regional weight, a machine load weight, and an available bandwidth weight;

[0016] Summing the weights of the candidate computer rooms to obtain the scheduling weight of the candidate computer room;

[0017] The candidate computer room with the largest scheduling weight is determined as the target computer room.

[0018] Preferably, determining the weighted heat of resources in the resource pool based on consumption records includes:

[0019] Obtaining consumption records of resources in the resource pool;

[0020] Count the consumption records of each resource by day and get the consumption count for each day within the most recent specified number of days;

[0021] According to the real-time heat calculation strategy, the consumption times of the day are converted into heat to obtain the real-time weighted heat;

[0022] According to the historical heat calculation strategy, the heat conversion process is performed on the consumption times of the previous day to obtain the weighted heat of the previous day.

[0023] Preferably, the number of consumption times on the day is converted into heat according to the real-time heat calculation strategy to obtain the real-time weighted heat, including:

[0024] Normalizing the number of consumption times on the day to obtain an approximate number of consumption times on the day;

[0025] The real-time weighted heat is determined by taking the approximate number of consumption times on the day as the base number and the weighting coefficient corresponding to the resource type as the index.

[0026] Preferably, the heat conversion process is performed on the consumption times of the previous day according to the historical heat calculation strategy to obtain the weighted heat of the previous day, including:

[0027] Using the decay factor corresponding to the resource type as the base and the number of days from the current day as the exponent, calculate the decay weight of the previous day;

[0028] The consumption weight is calculated using the consumption times as the base and the weight coefficient corresponding to the resource type as the exponent;

[0029] Multiply the current day's attenuation weight by the consumption weight to obtain the weighted popularity of the previous day.

[0030] Preferably, determining a candidate computer room storing the target resource includes:

[0031] Obtain metadata of the target resource;

[0032] The metadata is queried from a metadata service to determine the candidate computer room.

[0033] Preferably, it also includes:

[0034] Determining a cold resource from the hot resources using the consumption record;

[0035] Generate a deletion task for deleting the cold resource;

[0036] Sending a deletion task to a computer room storing the cold resource, so as to delete the backup of the cold resource.

[0037] A resource scheduling device, comprising:

[0038] A heat calculation module, configured to determine the weighted heat of resources in the resource pool based on consumption records, and determine the popularity of the resources using the weighted heat;

[0039] A resource backup module, configured to select hot resources from the resource pool using the popularity and synchronously back up the hot resources to computer rooms located in different regions;

[0040] A request receiving module, configured to receive a request from a client to access a target resource and determine a candidate computer room storing the target resource;

[0041] a computer room determination module, configured to determine a target computer room from among the candidate computer rooms by using the network resource information of the candidate computer rooms if there are multiple candidate computer rooms;

[0042] The task generation module is used to generate a resource consumption task that matches the target computer room and feed the resource consumption task back to the client so that the client can access the target resource located in the target computer room according to the resource consumption task.

[0043] An electronic device, comprising:

[0044] memory for storing computer programs;

[0045] A processor is used to implement the steps of the above-mentioned resource scheduling method when executing the computer program.

[0046] A readable storage medium stores a computer program, which implements the steps of the resource scheduling method when executed by a processor.

[0047] By applying the method provided in the embodiment of the present application, the weighted heat of resources in the resource pool is determined based on consumption records, and the popularity of the resources is determined using the weighted heat; the popularity is used to select hot resources from the resource pool and the hot resources are synchronously backed up to computer rooms located in different regions; a request from a client to access a target resource is received, and a candidate computer room storing the target resource is determined; if there are multiple candidate computer rooms, the network resource information of the candidate computer rooms is used to determine the target computer room from the multiple candidate computer rooms; a resource consumption task matching the target computer room is generated, and the resource consumption task is fed back to the client so that the client can access the target resource located in the target computer room according to the resource consumption task.

[0048] In this application, based on consumption records, the weighted popularity of resources in a resource pool is determined. The weighted popularity can be used to determine the popularity of resources. Based on popularity, hot resources can be selected from the resource pool and then synchronized and backed up to data centers in different regions. That is, hot resources are stored in different data centers. When a client requests access to a target resource, candidate data centers that store the target resource are first determined. If multiple candidate data centers exist, this indicates that the target resource is a hot resource. Network resource information corresponding to the multiple candidate data centers can then be obtained. Based on this network resource information, a target data center is selected from the multiple candidate data centers, and a resource consumption task matching the target data center is generated. Ultimately, the resource consumption task is fed back to the client, allowing the client to access the target resource located in the target data center. In other words, this application processes consumption records to select hot resources and synchronizes and backs up the hot resources to different data centers. When a user accesses a hot resource, a target data center can be selected from the different candidate data centers based on the network resources corresponding to the candidate data centers, and the request is dispatched to the target data center. That is, this application can reduce the occurrence rate of bandwidth squeeze, delay, etc. when processing requests, and can effectively improve the user experience.

[0049] Correspondingly, the embodiments of the present application also provide a resource scheduling device, equipment and readable storage medium corresponding to the above-mentioned resource scheduling method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a flowchart of an implementation method of a resource scheduling method in an embodiment of the present application;

[0052] Figure 2 This is a schematic diagram of resource scheduling consumption;

[0053] Figure 3 This is a schematic diagram of resource scheduling consumption in an embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of a resource distribution process in an embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of a resource scheduling process in an embodiment of the present application;

[0056] Figure 6 This is a schematic diagram of a resource cleanup process in an embodiment of the present application;

[0057] Figure 7 This is a schematic diagram of the structure of a resource scheduling device in an embodiment of the present application;

[0058] Figure 8 This is a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0059] Figure 9 This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present application.

[0061] Please refer to Figure 1 , Figure 1 This is a flow chart of a resource scheduling method according to an embodiment of the present application, which includes the following steps:

[0062] S101. Determine the weighted heat of resources in a resource pool based on consumption records, and use the weighted heat to determine the popularity of the resources.

[0063] In this embodiment, popularity measures a resource's popularity, frequency of use, and level of attention in user consumption scenarios. This is determined based on resource consumption. Considering that older consumption patterns have a lower value for assessing a resource's current popularity, this embodiment first determines a resource's weighted popularity based on consumption records, and then uses this weighted popularity to determine the resource's popularity.

[0064] Consumption records can record the time of each consumption of each resource. In this embodiment, the weighted popularity of each resource in the resource pool can be determined directly based on the consumption records. Then, the popularity of the resource can be determined based on the weighted popularity.

[0065] Specifically, for the convenience of statistics, in this embodiment, only the weighted popularity of different time periods between the current time and the specified time period can be counted. Then, the popularity of the resource is finally comprehensively evaluated based on the weighted popularity corresponding to these different time periods.

[0066] Among them, the time period can be statistical units such as year, quarter, month, week, day, hour, etc.

[0067] For example, when the unit is day, the weighted popularity of each day from the current date to N days ago can be counted, and then the popularity of the resource can be determined by combining the weighted popularity.

[0068] In a specific implementation of the present application, determining the weighted heat of resources in a resource pool based on consumption records includes:

[0069] Get consumption records of resources in the resource pool;

[0070] Count the consumption records of each resource by day and get the consumption count for each day within the most recent specified number of days;

[0071] According to the real-time heat calculation strategy, the consumption times of the day are converted into heat to obtain the real-time weighted heat;

[0072] According to the historical heat calculation strategy, the heat conversion process is performed on the consumption times of the previous day to obtain the weighted heat of the previous day.

[0073] For ease of description, the above steps are combined and explained below.

[0074] First, obtain the consumption records of resources in the resource pool. Then, count the consumption of each resource on a daily basis within a specified range of days (e.g., the last N days, where N is an integer greater than 1). In other words, count how many times each resource was consumed on each day. For example, if the last specified range is the last 7 days, the number of times each resource was consumed on each day within the last 7 days will be obtained.

[0075] Consumption times at different times relative to the current date contribute differently to popularity. Therefore, in this embodiment, different popularity calculation strategies are designed for different dates. Specifically, for the current day's consumption times, the real-time popularity calculation strategy can be used to directly convert the current day's consumption times to obtain weighted popularity. For the previous day's consumption times within the statistical range, the historical popularity calculation strategy can be used to convert the previous day's consumption times to obtain weighted popularity within the statistical range.

[0076] In a specific embodiment of the present application, the number of consumption times on the day is converted into heat according to the real-time heat calculation strategy to obtain the real-time weighted heat, including:

[0077] Normalize the number of consumption times on the day to get the approximate number of consumption times on the day;

[0078] The real-time weighted heat is determined by taking the approximate number of consumption times on the day as the base number and the weighting coefficient corresponding to the resource type as the index.

[0079] For ease of description, the above steps are combined and explained below.

[0080] Considering the time of statistics collection, the statistical results for the number of consumption times per day can vary significantly. For example, if the statistics cutoff time is in the early morning, and if the statistics cutoff time is in the afternoon, the statistical results for the number of consumption times per day will differ significantly (i.e., one counts the corresponding number of consumption times from early morning to afternoon, while the other lacks statistics for this time period). Therefore, in this embodiment, the currently counted number of consumption times per day is normalized based on the current statistical situation to obtain an approximate number of consumption times per day. In other words, based on the current statistical situation, an estimate of the total number of consumption times per day is made to obtain the approximate number of consumption times per day, thereby preventing the accuracy of the popularity index from being affected by differences in statistical time.

[0081] In the actual normalization process, it can be assumed that the consumption situation and the distribution of consumption throughout the day are approximate. Therefore, the current number of consumptions can be counted and the approximate number of consumptions for the day can be estimated.

[0082] For example, the approximate number of daily consumption times = daily consumption times / daily statistical interval * 24 * 3600, where the daily statistical interval is in seconds. In other words, based on the daily consumption times and the daily statistical interval, first calculate the number of consumption times per unit time, then multiply by the number of seconds in a day to obtain an estimate of the total daily consumption times (i.e., the approximate number of daily consumption times). Of course, in actual applications, different gain coefficients can be set for different time periods of the day based on business circumstances. For example, different gain coefficients can be set for work hours and leisure hours to best estimate the actual number of daily consumption times.

[0083] Then, to determine the real-time weighted heat, the approximate number of consumptions on that day can be used as the base number, and the weighted coefficient corresponding to the resource type can be used as the index for exponential calculation, and the result can be used as the real-time weighted heat.

[0084] Different resource types can have different weighting coefficients. For example, for video resources, the weighting coefficient for original image resources is 0.35, and the weighting coefficient for transcoded resources is 0.45 (the parameters are configurable). The weighting system can be set based on experience or actual resource consumption statistics (for example, if statistics show that transcoded resources contribute more to popularity than original image resources, a higher weighting coefficient can be set for transcoded resources).

[0085] For example, to obtain a resource's daily consumption records and calculate its real-time weighted popularity, the formula is: Real-time weighted popularity = (Number of consumptions per day / Statistics interval for that day * 24 * 3600). Because the calculation time varies from day to day, the number of consumptions is first converted to the total number of consumptions per day. For example, if a concept art resource was consumed 1000 times by 2:00 PM (the interval from midnight = 14 * 3600 seconds), the real-time weighted popularity of the resource would be (1000 / (14 * 3600) * 24 * 3600) = 0.35.

[0086] In a specific embodiment of the present application, the heat conversion process is performed on the consumption times of the previous day according to the historical heat calculation strategy to obtain the weighted heat of the previous day, including:

[0087] Using the decay factor corresponding to the resource type as the base and the number of days from the current day as the exponent, calculate the decay weight of the previous day;

[0088] The consumption weight is calculated using the number of consumption times as the base and the weighted coefficient corresponding to the resource type as the exponent;

[0089] Multiply the attenuation weight of the day by the consumption weight to get the weighted heat of the previous day.

[0090] For ease of description, the above steps are combined and explained below.

[0091] The impact of past consumption on the current popularity of a resource will gradually decay. Therefore, in this embodiment, a decay factor (used to describe the degree to which the number of consumption of a resource gradually weakens or decreases over time) and a weighting system can be combined to determine the weighted popularity of past days.

[0092] The following uses the weighted popularity statistics of the last 8 days as an example to illustrate the determination of the weighted popularity of the previous 7 days.

[0093] You can obtain the historical consumption records of resources from T-7 to T-1 (where T is the current date, T-1 is the day before the current date, and T-7 is the day before the current date), totaling 7 days, and calculate the weighted popularity of each day. The calculation formula is as follows:

[0094] Historical weighted heat on day N = decay factor N *Number of consumption 加权系数 ;

[0095] The attenuation factors and weighting coefficients corresponding to different resource types can be set to different values. For example, the attenuation factor of the original painting resource = 0.5, the weighting coefficient = 0.35; the attenuation factor of the transcoding resource = 0.5, the weighting coefficient = 0.45 (parameters are configurable). With the help of this formula, the weighted heat of the resources for each of the past 7 days (that is, the weighted heat of the previous days) can be calculated, and exponential smoothing is performed based on different indexes. The farther away from the present time and the fewer the consumption times, the smaller the calculated weighted heat; the closer to the present time and the more the consumption times, the greater the calculated weighted heat. For example, the weighted heat of the original painting resource on the T-2 day (that is, the day before yesterday) = 0.5 2 *Number of consumption 0.35 .

[0096] In this embodiment, after the weighted heat is obtained, the popularity of the resource can be determined based on the weighted heat.

[0097] In actual applications, considering the large number of resources and the long time it takes to deploy resources, it's important to filter resources with high popularity today and historically. This way, resources can be more effectively utilized after deployment, as their popularity has decreased after deployment. To understand resource popularity decay, you can first perform a rough screening based on real-time weighted popularity. For example, you can first remove resources in the resource pool whose real-time weighted popularity is less than the minimum weighted popularity. For example, the minimum weighted popularity for original image resources is 200, and the minimum weighted popularity for transcoding resources is 300 (the parameters here are configurable and adjustable).

[0098] When determining popularity, the calculated weighted heats may be accumulated, and the accumulated result is used as the popularity.

[0099] Taking the weighted popularity statistics of the last 8 days as an example, when determining popularity, for the same resource, the accumulation of the weighted popularity of the past 7 days and the real-time weighted popularity of the day is used as the current resource popularity.

[0100] S102: Select hot resources from the resource pool using popularity and synchronize the hot resources to computer rooms in different regions.

[0101] After obtaining the popularity, hot resources can be selected from the resource pool based on the popularity, and then the hot resources can be synchronously backed up to computer rooms in different regions. In this way, the same hot resource can be used in computer rooms in different regions to balance the access services corresponding to the hot resources, increase the bandwidth of the hot resources, and accelerate the response speed to user needs.

[0102] Corresponding access gateways, object storage and other equipment are installed in the computer room.

[0103] Specifically, you can use the popularity-based adaptive time window algorithm (ATW) to filter hot resources to be deployed. The algorithm is as follows:

[0104] Input: resource pool Q to be screened, including file GCID and file size FileSize;

[0105] CS: total size of resource pool resources;

[0106] L: upper limit of the cache capacity of the computer room;

[0107] W: the window length at the current moment;

[0108] T: The popularity of the resource calculated in the previous step (depending on the load of the computer room, only resources with a popularity exceeding a certain threshold can be included);

[0109] QT: Sort the resources in T by popularity from high to low;

[0110] Then, execute the loop code to iterate over each resource:

[0111] CS = 0;

[0112] for GCID, FileSize in QT;

[0113] If CS + FileSize < L, GCID is put into the list to be deployed.

[0114] Among them, the resources in the list to be deployed are hot resources

[0115] After selecting the hot resources, you can deploy the same hot resources to computer rooms in different regions.

[0116] Of course, when backing up and deploying hot resources, you can also use a load balancing strategy to evenly distribute the selected hot resources by region and computer room.

[0117] S103: Receive a request from a client to access a target resource, and determine a candidate computer room storing the target resource.

[0118] When a request for accessing a target resource is received from a client, the candidate computer room currently storing the target resource is first identified.

[0119] In a specific implementation of the present application, determining a candidate computer room storing target resources includes:

[0120] Get metadata of target resource;

[0121] Query metadata from the metadata service to determine candidate computer rooms.

[0122] The metadata service stores metadata corresponding to the target resource stored in which computer room. That is, in this embodiment, the metadata of the target resource can be obtained first, and then the metadata can be queried from the metadata service to determine the candidate computer room.

[0123] If there is only one candidate data center, meaning the target resource exists in only one data center, a resource consumption task matching that data center can be directly generated and fed back to the client. Based on the resource consumption task, the client can then access the target resource from the data center that stores the target resource.

[0124] When there are two or more candidate computer rooms, step S102 may be executed.

[0125] S104: If there are multiple candidate computer rooms, determine a target computer room from the multiple candidate computer rooms using the network resource information of the candidate computer rooms.

[0126] The term "multiple candidate computer rooms" refers to the target resource being stored in two or more candidate computer rooms. The target resource is a hot resource. To provide a better user experience, in this embodiment, the target computer room that is currently most suitable for serving the client can be selected from these candidate computer rooms.

[0127] Specifically, the target computer room that is currently most suitable for providing consumption services of the target resources can be selected based on the network resource information of the candidate computer rooms. For example, based on the network resource information, a target computer room with the best current performance and the least tasks can be selected.

[0128] In a specific embodiment of the present application, using the network resource information of the candidate computer rooms to determine the target computer room from multiple candidate computer rooms includes:

[0129] Obtain network resource information of candidate computer rooms;

[0130] Use network resource information to eliminate abnormal computer rooms from candidate computer rooms;

[0131] After eliminating abnormal computer rooms, using network resource information, determine the sub-weights of candidate computer rooms; the sub-weights are at least one of bandwidth cost weight, operation weight, regional weight, machine load weight, and available bandwidth weight;

[0132] Sum the weights of the candidate computer rooms to obtain the scheduling weight of the candidate computer room;

[0133] The candidate computer room with the largest scheduling weight is determined as the target computer room.

[0134] For ease of description, the above steps are combined and explained below.

[0135] The network resource information includes but is not limited to machine load, used bandwidth, total bandwidth, bandwidth cost and other information.

[0136] Based on this network resource information, abnormal computer rooms among the candidate computer rooms can be eliminated first. The abnormal computer rooms can be computer rooms that cannot provide external services, such as computer rooms whose bandwidth has reached the total bandwidth or computer rooms with failures.

[0137] After eliminating abnormal computer rooms, corresponding sub-weights can be set for each resource in the network resource information. These sub-weights are then summed to obtain the scheduling weights of the candidate computer rooms. The candidate computer room with the largest scheduling weight is selected as the target computer room.

[0138] For example, you can query network resource information such as machine load, used bandwidth, total bandwidth, and bandwidth cost for candidate data centers. Then, eliminate data centers with machine load exceeding 60% and bandwidth usage exceeding 80% on-demand and 90% on-retrieval. Then, calculate the scheduling weights for the remaining data centers.

[0139] Bandwidth cost weight = 50 * (1 – bandwidth cost of the data center / maximum bandwidth cost of the data center);

[0140] Operator weight: Same operator = 15, Different operator = 0;

[0141] Regional weight: same province = 10, same region = 5, other = 0;

[0142] Machine load weight = 15 * (1 – machine load);

[0143] Available bandwidth weight = 10 * (1 – used bandwidth ratio);

[0144] Final scheduling weight = the sum of the above sub-weights.

[0145] Sort the remaining candidate computer rooms by scheduling weight, and select the candidate computer room with the highest scheduling weight as the target computer room.

[0146] S105: Generate a resource consumption task that matches the target computer room, and feed back the resource consumption task to the client, so that the client can access the target resource located in the target computer room according to the resource consumption task.

[0147] After the candidate computer rooms are identified, a resource consumption task can be generated and returned to the client. In this way, the client can consume the target resources by accessing the target computer room based on the resource consumption task.

[0148] As for how the resource consumption task is specifically generated and how the client accesses the target resource from the target computer room, you can refer to the relevant resource access solution, which will not be described in detail here.

[0149] In a specific embodiment of the present application, it also includes:

[0150] Using consumption records, identify cold resources from hot resources;

[0151] Generate a deletion task to delete cold resources;

[0152] Send the deletion task to the data center where the cold resources are stored to delete the backup of the cold resources.

[0153] For ease of description, the above steps are combined and explained below.

[0154] In this embodiment, consumption records can be used to identify not only hot resources but also cold resources. For example, cold resource thresholds can be set based on consumption count, popularity, or heat weight, thereby clearly identifying cold resources. Deletion tasks for the cold resources are then generated and sent to the data centers storing the cold resources. The data centers that receive the deletion tasks can then delete the corresponding cold resources. This enables resource heat management, allowing for continuous adjustment of resource backup status to meet current needs.

[0155] By applying the method provided in the embodiment of the present application, the weighted heat of resources in the resource pool is determined based on consumption records, and the popularity of the resources is determined using the weighted heat; the popularity is used to select hot resources from the resource pool and the hot resources are synchronously backed up to computer rooms located in different regions; a request from a client to access a target resource is received, and a candidate computer room storing the target resource is determined; if there are multiple candidate computer rooms, the network resource information of the candidate computer rooms is used to determine the target computer room from the multiple candidate computer rooms; a resource consumption task matching the target computer room is generated, and the resource consumption task is fed back to the client so that the client can access the target resource located in the target computer room according to the resource consumption task.

[0156] In this application, based on consumption records, the weighted popularity of resources in a resource pool is determined. The weighted popularity can be used to determine the popularity of resources. Based on popularity, hot resources can be selected from the resource pool and then synchronized and backed up to data centers in different regions. That is, hot resources are stored in different data centers. When a client requests access to a target resource, candidate data centers that store the target resource are first determined. If multiple candidate data centers exist, this indicates that the target resource is a hot resource. Network resource information corresponding to the multiple candidate data centers can then be obtained. Based on this network resource information, a target data center is selected from the multiple candidate data centers, and a resource consumption task matching the target data center is generated. Ultimately, the resource consumption task is fed back to the client, allowing the client to access the target resource located in the target data center. In other words, this application processes consumption records to select hot resources and synchronizes and backs up the hot resources to different data centers. When a user accesses a hot resource, a target data center can be selected from the different candidate data centers based on the network resources corresponding to the candidate data centers, and the request is dispatched to the target data center. That is, this application can reduce the occurrence rate of bandwidth squeeze, delay, etc. when processing requests, and can effectively improve the user experience.

[0157] To facilitate those skilled in the art to better understand and implement the resource scheduling method provided in the embodiment of the present application, the resource scheduling method is described in detail below with reference to a specific scenario as an example.

[0158] Please refer to Figure 2 ,exist Figure 2 In the data center, the same file resource is only stored in one copy, that is, it only exists in one computer room. When users consume, they can only go to the corresponding computer room to obtain file resources. It is impossible to implement more refined and richer scheduling strategies based on computer room load, bandwidth cost, user region, etc.

[0159] Please refer to Figure 3 ,exist Figure 3 In this scenario, hot resources are deployed across multiple data centers. Resource scheduling is responsible for dispatching user requests to the most appropriate one among these data centers, while resource cleanup is responsible for removing redundant copies of resources with low usage and low contribution from data centers. In other words, the resource selection problem is essentially a CDN PUSH (a content push mechanism in a content delivery network (CDN)) deployment problem. Resource distribution proactively deploys hot resources to redundant data centers based on user access statistics.

[0160] Please refer to Figure 4 ,Resource deployment, including the determination of thermal resources and computer rooms, and the implementation of resource deployment based on tasks.

[0161] Please refer to Figure 5Regarding resource scheduling, we must first identify the computer rooms where the resources are located. Then, based on the storage, bandwidth and other information of these computer rooms, after eliminating abnormal computer rooms, we calculate the corresponding scheduling weights, select the computer room with the largest scheduling weight as the target computer room, and generate corresponding consumption tasks to achieve resource scheduling.

[0162] Please refer to Figure 6 ,Regarding resource cleaning and resource distribution types, the popularity of resources can be ,calculated, and the least popular and least contributing batch of resources ,can be deleted from the corresponding computer room.

[0163] It can be seen that the resource scheduling method provided in the embodiment of the present application mainly realizes redundant storage, resource distribution and resource cleaning.

[0164] Among them, redundant storage: popular resources are distributed to multiple computer rooms for storage in multiple copies. When consuming, users can choose the most reasonable computer room to use based on factors such as the machine load, bandwidth cost, region, operator, etc., while ensuring service quality, reducing bandwidth costs and improving user experience.

[0165] Resource distribution: Historical consumption records from the previous N days are combined with real-time consumption records for the current day to calculate weighted popularity. Popularity is then calculated based on the weighted popularity. An adaptive time window algorithm based on popularity is then used to screen resources for deployment. This solves the problem of delayed resource distribution and can effectively handle sudden traffic increases caused by rapid consumption growth in a short period of time.

[0166] Resource cleanup: For multi-copy resources, after deployment, their usage efficiency and popularity are calculated based on consumption records, and low-efficiency resources are cleaned up in a timely manner to save storage costs.

[0167] That is to say, in actual applications, executing the resource scheduling method provided in the embodiments of the present application can improve user experience: the multi-cloud scheduling system can choose to use a computer room with low load, sufficient bandwidth, and closer to the user, which can effectively reduce bandwidth squeezing, delays, and other problems during user requests, and significantly improve the user's consumption experience, especially in online video playback scenarios, which can significantly reduce playback freezes and other problems; reduce costs: due to the same-province and same-region scheduling strategies, the traffic for cross-provincial and cross-regional settlements can be significantly reduced, reducing costs; making resource deployment more timely: the historical consumption records of the previous N days and the real-time consumption records of the day are combined to calculate the weighted heat, and the popularity is calculated based on the weighted heat. Then, an adaptive time window algorithm based on popularity is used to screen the resources to be deployed, solving the problem of delayed resource distribution.

[0168] Corresponding to the above method embodiment, the embodiment of the present application further provides a resource scheduling device. The resource scheduling device described below and the resource scheduling method described above can refer to each other.

[0169] See also Figure 7 As shown, the device includes the following modules:

[0170] The heat calculation module 101 is used to determine the weighted heat of resources in the resource pool based on consumption records, and use the weighted heat to determine the popularity of the resources;

[0171] The resource backup module 102 is used to select hot resources from the resource pool based on popularity and synchronize the hot resources to computer rooms in different regions;

[0172] The request receiving module 103 is used to receive a request from a client to access a target resource and determine a candidate computer room storing the target resource;

[0173] The computer room determination module 104 is configured to determine a target computer room from among multiple candidate computer rooms by using the network resource information of the candidate computer rooms if there are multiple candidate computer rooms;

[0174] The task generation module 105 is used to generate a resource consumption task that matches the target computer room and feed the resource consumption task back to the client so that the client can access the target resource in the target computer room according to the resource consumption task.

[0175] The device provided in the embodiment of the present application is used to determine the weighted heat of resources in the resource pool based on consumption records, and use the weighted heat to determine the popularity of the resources; use the popularity to select hot resources from the resource pool and synchronize the hot resources to computer rooms located in different regions; receive a request from a client to access a target resource, and determine a candidate computer room that stores the target resource; if there are multiple candidate computer rooms, use the network resource information of the candidate computer rooms to determine the target computer room from the multiple candidate computer rooms; generate a resource consumption task that matches the target computer room, and feed the resource consumption task back to the client so that the client can access the target resource located in the target computer room according to the resource consumption task.

[0176] In this application, based on consumption records, the weighted popularity of resources in a resource pool is determined. The weighted popularity can be used to determine the popularity of resources. Based on popularity, hot resources can be selected from the resource pool and then synchronized and backed up to data centers in different regions. That is, hot resources are stored in different data centers. When a client requests access to a target resource, candidate data centers that store the target resource are first determined. If multiple candidate data centers exist, this indicates that the target resource is a hot resource. Network resource information corresponding to the multiple candidate data centers can then be obtained. Based on this network resource information, a target data center is selected from the multiple candidate data centers, and a resource consumption task matching the target data center is generated. Ultimately, the resource consumption task is fed back to the client, allowing the client to access the target resource located in the target data center. In other words, this application processes consumption records to select hot resources and synchronizes and backs up the hot resources to different data centers. When a user accesses a hot resource, a target data center can be selected from the different candidate data centers based on the network resources corresponding to the candidate data centers, and the request is dispatched to the target data center. That is, this application can reduce the occurrence rate of bandwidth squeeze, delay, etc. when processing requests, and can effectively improve the user experience.

[0177] In a specific embodiment of the present application, the computer room determination module is specifically used to obtain network resource information of the candidate computer rooms;

[0178] Use network resource information to eliminate abnormal computer rooms from candidate computer rooms;

[0179] After eliminating abnormal computer rooms, using network resource information, determine the sub-weights of candidate computer rooms; the sub-weights are at least one of bandwidth cost weight, operation weight, regional weight, machine load weight, and available bandwidth weight;

[0180] Sum the weights of the candidate computer rooms to obtain the scheduling weight of the candidate computer room;

[0181] The candidate computer room with the largest scheduling weight is determined as the target computer room.

[0182] In a specific implementation of the present application, the heat calculation module is specifically used to obtain consumption records of resources in the resource pool;

[0183] Count the consumption records of each resource by day and get the consumption count for each day within the most recent specified number of days;

[0184] According to the real-time heat calculation strategy, the consumption times of the day are converted into heat to obtain the real-time weighted heat;

[0185] According to the historical heat calculation strategy, the heat conversion process is performed on the consumption times of the previous day to obtain the weighted heat of the previous day.

[0186] In a specific embodiment of the present application, the heat calculation module is specifically used to normalize the number of consumption times on the day to obtain the approximate number of consumption times on the day;

[0187] The real-time weighted heat is determined by taking the approximate number of consumption times on the day as the base number and the weighting coefficient corresponding to the resource type as the index.

[0188] In a specific embodiment of the present application, the heat calculation module is specifically used to calculate the attenuation weight of the previous day using the attenuation factor corresponding to the resource type as the base and the number of days from the current day as the index;

[0189] The consumption weight is calculated using the number of consumption times as the base and the weighted coefficient corresponding to the resource type as the exponent;

[0190] Multiply the attenuation weight of the day by the consumption weight to get the weighted heat of the previous day.

[0191] In a specific implementation of the present application, the request receiving module is specifically used to obtain metadata of the target resource;

[0192] Query metadata from the metadata service to determine candidate computer rooms.

[0193] In a specific embodiment of the present application, it also includes:

[0194] The cold resource removal module is used to identify cold resources from hot resources using consumption records;

[0195] A deletion task for deleting the cold resource is generated; the deletion task is sent to the computer room storing the cold resource so as to delete the backup of the cold resource.

[0196] Corresponding to the above method embodiment, an embodiment of the present application further provides an electronic device. The electronic device described below and the resource scheduling method described above can refer to each other.

[0197] See also Figure 8 As shown, the electronic device includes:

[0198] Memory 332, for storing computer programs;

[0199] The processor 322 is configured to implement the steps of the resource scheduling method of the above method embodiment when executing a computer program.

[0200] For details, please refer to Figure 9 , Figure 9This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. This electronic device may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) (for example, one or more processors) and memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 may be temporary storage or permanent storage. The program stored in the memory 332 may include one or more modules (not shown), each of which may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 to execute the series of instruction operations in the memory 332 on the electronic device 301.

[0201] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input / output interfaces 358 , and / or one or more operating systems 341 .

[0202] The steps in the resource scheduling method described above can be implemented by the structure of an electronic device.

[0203] Corresponding to the above method embodiment, an embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the resource scheduling method described above can refer to each other.

[0204] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the resource scheduling method of the above method embodiment.

[0205] The readable storage medium may specifically be any readable storage medium capable of storing program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0206] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0207] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0208] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0209] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0210] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A resource scheduling method, characterized in that: include: Determining weighted popularity of resources in the resource pool based on consumption records, and determining popularity of the resources using the weighted popularity; Selecting hot resources from the resource pool using the popularity and synchronously backing up the hot resources to computer rooms located in different regions; Receive a request from a client to access a target resource, and determine a candidate computer room storing the target resource; If there are multiple candidate computer rooms, determine the target computer room from the multiple candidate computer rooms using the network resource information of the candidate computer rooms; A resource consumption task matching the target computer room is generated, and the resource consumption task is fed back to the client, so that the client accesses the target resource located in the target computer room according to the resource consumption task.

2. The method according to claim 1, characterized in that Determining a target computer room from a plurality of candidate computer rooms by using the network resource information of the candidate computer rooms includes: Obtaining network resource information of the candidate computer room; Eliminating abnormal computer rooms from the candidate computer rooms by using the network resource information; After eliminating abnormal computer rooms, using the network resource information, determining a sub-weight of the candidate computer room; the sub-weight is at least one of a bandwidth cost weight, an operation weight, a regional weight, a machine load weight, and an available bandwidth weight; Summing the weights of the candidate computer rooms to obtain the scheduling weight of the candidate computer room; The candidate computer room with the largest scheduling weight is determined as the target computer room.

3. The method according to claim 1, characterized in that Determine the weighted popularity of resources in the resource pool based on consumption records, including: Obtaining consumption records of resources in the resource pool; Count the consumption records of each resource by day and get the consumption count for each day within the most recent specified number of days; According to the real-time heat calculation strategy, the consumption times of the day are converted into heat to obtain the real-time weighted heat; According to the historical heat calculation strategy, the heat conversion process is performed on the consumption times of the previous day to obtain the weighted heat of the previous day.

4. The method according to claim 3, characterized in that According to the real-time heat calculation strategy, the number of consumption times on the day is converted into heat to obtain real-time weighted heat, including: Normalizing the number of consumption times on the day to obtain an approximate number of consumption times on the day; The real-time weighted heat is determined by taking the approximate number of consumption times on the day as the base number and the weighting coefficient corresponding to the resource type as the index.

5. The method according to claim 3, characterized in that According to the historical heat calculation strategy, the heat conversion process is performed on the consumption times of the previous day to obtain the weighted heat of the previous day, including: Using the decay factor corresponding to the resource type as the base and the number of days from the current day as the exponent, calculate the decay weight of the previous day; The consumption weight is calculated using the consumption times as the base and the weight coefficient corresponding to the resource type as the exponent; Multiply the current day's attenuation weight by the consumption weight to obtain the weighted popularity of the previous day.

6. The method according to claim 1, characterized in that Determining a candidate computer room storing the target resource includes: Obtain metadata of the target resource; The metadata is queried from a metadata service to determine the candidate computer room.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Determining a cold resource from the hot resources using the consumption record; Generate a deletion task for deleting the cold resource; Sending a deletion task to a computer room storing the cold resource, so as to delete the backup of the cold resource.

8. A resource scheduling device, characterized in that: include: A heat calculation module, configured to determine the weighted heat of resources in the resource pool based on consumption records, and determine the popularity of the resources using the weighted heat; A resource backup module, configured to select hot resources from the resource pool using the popularity and synchronously back up the hot resources to computer rooms located in different regions; A request receiving module, configured to receive a request from a client to access a target resource and determine a candidate computer room storing the target resource; a computer room determination module, configured to determine a target computer room from among the candidate computer rooms by using the network resource information of the candidate computer rooms if there are multiple candidate computer rooms; The task generation module is used to generate a resource consumption task that matches the target computer room and feed the resource consumption task back to the client so that the client can access the target resource located in the target computer room according to the resource consumption task.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the resource scheduling method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the resource scheduling method according to any one of claims 1 to 7.

Images