Load balancing control method and device

Abstract

The present invention provides a load balancing control method and device. The method comprises: receiving a user's query request, wherein the query request comprises a target file list and time information for starting to read, and the target file list comprises file identifiers of files requested to be read by the user; determining the total information volume of files requested to be read by the user according to the target file list; determining the information volume limit value according to the file reading information and resource occupancy information of the user within a set historical date; when the total information volume is greater than the information volume limit value, filtering some file identifiers from the target file list to obtain a permitted file list, wherein the total information volume of files in the permitted file list is less than or equal to the information volume limit value. The present invention can realize load balancing control for a big data system and improve the utilization efficiency of resources in the big data system.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a load balancing control method and device. Background Art

[0002] With the further development of communication network technology, big data technology has been further applied. Big data systems can provide users with services such as the collection, transmission, storage, calculation and analysis of massive data. For example, the Hadoop architecture is one of the architectures of the big data system, and the Hadoop distributed file system (HDFS) is implemented based on the Hadoop architecture. HDFS has the advantages of high fault tolerance, high throughput, and low cost, and is widely used in big data computing, storage and other service scenarios.

[0003] In the process of providing computing services to users in the big data system, the current management strategy is only to divide permissions in the big data system. For example, for HDFS, only user A is allowed to access and read directory A in HDFS to complete the computing service. However, the current management strategy does not take into account the resource usage and access time of users during the access and reading process, resulting in load imbalance in the current big data system. For example, in the same period, such as the busy period, multiple users access or read files from the big data system at the same time, resulting in a high load on the big data system during this period, and the resources of the big data system may be exhausted, which reduces the data transmission speed when the big data system provides computing services to multiple users, affecting the user experience. Summary of the invention

[0004] The present invention provides a load balancing control method and device, which can realize load balancing control of a big data system and improve the utilization efficiency of resources in the big data system.

[0005] In a first aspect, the present invention provides a load balancing control method, comprising: receiving a user's query request, wherein the query request includes a target file list and time information for starting to read, and the target file list includes file identifiers of files requested by the user to read; determining the total amount of information of the files requested by the user to read based on the target file list; determining an information amount limit based on the user's file reading information and resource occupancy information within a set historical date; when the total amount of information is greater than the information amount limit, filtering some file identifiers from the target file list to obtain a permitted file list, wherein the total amount of information of the files in the permitted file list is less than or equal to the information amount limit.

[0006] The present invention provides a load balancing control method. After receiving a user's query request, the information volume limit is determined according to the user's file reading information and resource occupancy information within a set historical date; the target file list requested by the user is identified according to the information volume limit, and when the total information volume requested by the user is greater than the information volume limit, the files in the target file list requested by the user are restricted. In this way, the load balancing control method provided by the present invention can restrict the information volume requested by the user according to the file reading information and resource occupancy information within the historical date, thereby realizing load balancing control of the big data system and improving the utilization efficiency of resources in the big data system.

[0007] Exemplarily, in the process of Hadoop cluster providing services to users, users need to read data files in HDFS to complete computing services. Assuming that the user requests to read file list A during the busy period, the control method of the present invention can determine the information volume limit based on the file reading information and resource occupancy information within the historical date, identify the file list A, and obtain a list of licensed files whose total information volume is less than or equal to the information volume limit, thereby limiting the file reading situation of users during the busy period, extending or shifting the time for providing computing services to users, so that the resources in the Hadoop cluster can be balanced and controlled, and the utilization efficiency of the resources in the Hadoop cluster can be improved.

[0008] In a possible implementation, the time information for starting reading includes the date and time point for starting reading; the information amount limit is determined based on the file reading information and resource occupancy information of the user within the set historical date, including: determining the average reading speed and resource occupancy deviation coefficient of the user in a set time period within the set historical date based on the file reading information and resource occupancy information of the user within the set historical date, the set historical date includes the previous N days before the date, N is a positive integer, the starting point of the set time period is the time point, and the resource occupancy deviation coefficient is used to represent the deviation between the resource occupancy at the time point in the set time period within the set historical date and the average resource occupancy in the set time period; based on the average reading speed and the resource occupancy deviation coefficient, determining the information amount limit allowed to be read by the user within a certain period of time, the certain period being one or more periods of the same length as the set period starting from the time point of the date.

[0009] In a possible implementation, determining the information amount limit that a user is allowed to read within a certain period of time according to the average reading speed and the resource occupancy deviation coefficient includes: determining the information amount limit according to the following formula.

[0010] L M =(1+r)×v0×t0;

[0011] Among them, L Mis the maximum limit of file information volume, r is the resource usage deviation coefficient, v0 is the average reading speed, and t0 is the duration of a certain period.

[0012] In a possible implementation, based on the file reading information and resource occupancy information of the user within a set historical date, the average reading speed and resource occupancy deviation coefficient of the user in a set time period within the set historical date are determined, including: determining the instantaneous reading speed of the user's computing model at multiple moments in the set time period within the set historical date based on the file reading information of the user's computing model within the set historical date; determining the average reading speed based on the instantaneous reading speeds at multiple moments; determining the instantaneous resource occupancy of the user's computing model at multiple moments in the set time period within the set historical date and the resource occupancy at a time point in each day based on the resource occupancy information of the user's computing model within the set historical date; determining the average resource occupancy of the set time period within the set historical date based on the instantaneous resource occupancy at multiple moments; and determining the resource occupancy deviation coefficient based on the average resource occupancy and the resource occupancy.

[0013] In one possible implementation, a resource occupancy deviation coefficient is determined based on the average resource occupancy and the resource occupancy, including: subtracting the resource occupancy at a time point in each day within a set historical date from the average resource occupancy to obtain multiple difference values; and determining the ratio of the average value of the multiple difference values ​​to the average resource occupancy as the resource occupancy deviation coefficient.

[0014] In a possible implementation, the query request also includes a user identifier, and the method also includes: querying the user's tariff information based on the user identifier to determine the user's remaining resource amount, the remaining resource amount is used to represent the remaining readable file information amount in the user's tariff information; determining the resource occupancy of the set time period based on the average resource occupancy and the length of the set time period, the resource occupancy of the set time period is used to characterize the amount of file information read by the user within the set time period; determining the length of a certain period based on the remaining resource amount, the resource occupancy of the set time period, and the length of the set time period.

[0015] In one possible implementation, the total amount of information of the files requested by the user to be read is determined based on the target file list, including: querying the storage information of the data files in the distributed file system based on the target file list to determine the amount of information of each file in the target file list; and determining the total amount of information of the files requested by the user to be read based on the amount of information of each file.

[0016] In a possible implementation, the resource occupancy information includes any one of the following: processor resource occupancy information, memory resource occupancy information, and disk resource occupancy information. The method further includes: sending a query response to the user, the query response including a license file list.

[0017] In a second aspect, an embodiment of the present invention provides a load balancing control device, comprising: a communication module and a processing module, the communication module being used to receive a user's query request, wherein the query request includes a target file list and time information for starting to read, and the target file list includes file identifiers of files requested to be read by the user; the processing module being used to determine the total amount of information of files requested to be read by the user based on the target file list; determining an information amount limit based on the file reading information and resource occupancy information of the user within a set historical date; and when the total amount of information is greater than the information amount limit, filtering some file identifiers from the target file list to obtain a permitted file list, wherein the total amount of information of files in the permitted file list is less than or equal to the information amount limit.

[0018] In a possible implementation, the time information for starting reading includes the date and time point for starting reading; the processing module is specifically used to determine the average reading speed and resource occupancy deviation coefficient of the user in a set time period within the set historical date based on the file reading information and resource occupancy information of the user within the set historical date, the set historical date includes the previous N days before the date, N is a positive integer, the starting point of the set time period is the time point, and the resource occupancy deviation coefficient is used to represent the deviation between the resource occupancy at the time point in the set time period within the set historical date and the average resource occupancy in the set time period; based on the average reading speed and the resource occupancy deviation coefficient, determine the limit on the amount of information allowed to be read by the user within a certain period of time, the certain period being one or more periods of the same length as the set period starting from the time point of the date.

[0019] In a possible implementation, the processing module is specifically configured to determine the information amount limit according to the following formula.

[0020] L M =(1+r)×v0×t0;

[0021] Among them, L M is the maximum limit of file information volume, r is the resource usage deviation coefficient, v0 is the average reading speed, and t0 is the duration of a certain period.

[0022] In one possible implementation, the processing module is specifically used to determine the instantaneous reading speed of the user's computing model at multiple moments in a set time period within the set historical date based on the file reading information of the user's computing model within the set historical date; determine the average reading speed based on the instantaneous reading speeds at multiple moments; determine the instantaneous resource occupancy of the user's computing model at multiple moments in the set time period within the set historical date and the resource occupancy at a time point in each day based on the resource occupancy information of the user's computing model within the set historical date; determine the average resource occupancy of the set time period within the set historical date based on the instantaneous resource occupancy at multiple moments; and determine the resource occupancy deviation coefficient based on the average resource occupancy and the resource occupancy.

[0023] In one possible implementation, the processing module is specifically used to subtract the resource occupancy at a time point on each day within a set historical date from the average resource occupancy to obtain multiple difference values; and the ratio of the average value of the multiple difference values ​​to the average resource occupancy is determined as the resource occupancy deviation coefficient.

[0024] In one possible implementation, the query request also includes a user identifier, and the processing module is also used to query the user's tariff information based on the user identifier, determine the user's remaining resource amount, and the remaining resource amount is used to represent the remaining readable file information amount in the user's tariff information; determine the resource occupancy of the set time period based on the average resource occupancy and the length of the set time period, and the resource occupancy of the set time period is used to characterize the amount of file information read by the user within the set time period; determine the length of a certain period based on the remaining resource amount, the resource occupancy of the set time period, and the length of the set time period.

[0025] In a possible implementation, the processing module is specifically used to query the storage information of the data files in the distributed file system according to the target file list, determine the amount of information of each file in the target file list; and determine the total amount of information of the files requested by the user to read according to the amount of information of each file.

[0026] In a possible implementation, the resource occupancy information includes any one of the following: processor resource occupancy information, memory resource occupancy information, and disk resource occupancy information. The communication module is further configured to send a query response to the user, the query response including a license file list.

[0027] In a third aspect, an embodiment of the present invention provides a load balancing control device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented.

[0028] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 This is a schematic diagram of an application scenario of a load balancing control method provided by an embodiment of the present invention;

[0031] Figure 2 It is a flow chart of a load balancing control method provided by an embodiment of the present invention;

[0032] Figure 3 It is a flow chart of another load balancing control method provided by an embodiment of the present invention;

[0033] Figure 4 is a structural schematic diagram of a control device provided by an embodiment of the present invention;

[0034] Figure 5 It is a structural schematic diagram of another control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0036] In the description of the present invention, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" and "plurality" refer to two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0037] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0038] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include other steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following will be described through specific embodiments in conjunction with the accompanying drawings of the present invention.

[0040] Figure 1 FIG. 1 is a schematic diagram of an application scenario of the load balancing control method according to an embodiment of the present invention. Figure 1 As shown in the figure, the big data system takes the hadoop cluster as an example. The hadoop cluster includes HDFS and MapReduce. MapReduce provides parallel application development and implements distributed computing and parallel task processing on the cluster. HDFS provides support for file operations and storage during MapReduce task processing. HDFS and MapReduce in the hadoop cluster interact with each other to provide computing services to users.

[0041] In the embodiment of the present application, the user includes a user device and a computing model of the user. The user device is used to control the startup, operation, and fault repair of the computing service required by the user. The computing model of the user is used to execute the computing service required by the user. In some embodiments, Figure 1 As shown, the user device and the user's computing model can be set separately. The user's computing model is set in the MapReduce of the hadoop cluster. The user sends the computing task to the hadoop cluster. After the hadoop cluster receives the computing task, the user's computing model completes the calculation by reading the data files stored in HDFS, obtains the calculation result, and sends the calculation result to the user device for the user to view.

[0042] In other embodiments, the user device and the user's computing model may be arranged together (not shown in the figure). The user's computing model interacts with the hadoop cluster, reads the data files stored in the HDFS to complete the calculation, and obtains the calculation result.

[0043] The following are three scenarios in which the Hadoop cluster provides computing services to users.

[0044] Scenario 1: Physical examination center A sends the first computing task to the hadoop cluster at 8 pm on Monday. The first computing task is to analyze and process the chest X-ray images taken during the physical examination of 100 medical examinees. After receiving the first computing task, the hadoop cluster needs to analyze and process the chest X-ray images of the 100 medical examinees to determine the first analysis result, where the first analysis result includes information such as the health status of the 100 medical examinees and the causes of possible lesions. Afterwards, the hadoop cluster sends the first analysis result to the physical examination center A to complete the processing of the first computing task.

[0045] Scenario 2: Traffic department B sends the second computing task to the hadoop cluster at 8 pm on Monday. The second computing task is to analyze and process the videos and images taken in the traffic monitoring system. After receiving the second computing task, the hadoop cluster analyzes and processes the videos and images to determine the second analysis result, where the second analysis result includes possible violation information in the videos and images, such as violation type, violation location, license plate information, etc. Afterwards, the hadoop cluster sends the second analysis result to traffic department B to complete the processing of the second computing task.

[0046] Scenario 3: Network company C sends a third computing task to the hadoop cluster at 8 p.m. on Monday. The third computing task is to analyze and process the network log information provided by network company C. After receiving the third computing task, the hadoop cluster analyzes and processes the network log information to determine the third analysis result, where the third analysis result includes the type of possible network security events, the time of occurrence, the corresponding IP address and other information. Afterwards, the hadoop cluster sends the third analysis result to network company C to complete the processing of the third computing task.

[0047] In the above three scenarios, the physical examination center A, the transportation department B, and the network company C all send computing tasks to the hadoop cluster at 8 pm on Monday. When the hadoop cluster executes the above computing tasks, the user's computing program unit completes the analysis and processing of the computing tasks by accessing the data files stored in HDFS.

[0048] However, the control strategy in the Hadoop cluster only processes permissions for users. For example, user A can only read data files in directory A in HDFS, and user B can only read data files in directory B in HDFS. As a result, multiple users may read data files in the same directory at the same time, resulting in excessive resource usage of the Hadoop cluster during that time, or even resource exhaustion, affecting the reading speed of users when accessing data files in HDFS.

[0049] For example, both the physical examination center A and the transportation department B are image processing tasks, and need to access the data files in the same directory in HDFS. In addition, at this moment, there may be other users accessing the data files in the directory. At this time, the data transmission speed of HDFS when serving multiple users decreases, affecting the user experience.

[0050] In order to solve the above technical problems, an embodiment of the present invention provides a load balancing control method, by setting a control device in the hadoop cluster, before the user needs to access the data files in the HDFS, the control device reviews the file list that the user needs to access. During the review process, the control device calculates the information volume limit in the time period that needs to be accessed, reduces the information volume limit in the busy time period of the hadoop cluster, and increases the information volume limit in the idle time period of the hadoop cluster. In this way, the control device can adjust and limit the file list that the user is allowed to access in different time periods, respectively, to reduce the amount of file information that the user is allowed to access during the busy period, and increase the amount of file information that the user is allowed to access during the idle period. The control device can transfer the user's computing tasks in the busy period to the idle period, realize the load balancing of the busy period and the idle period in the hadoop cluster, and improve the utilization efficiency of resources in the HDFS.

[0051] In some embodiments, the control device in the present invention may be a server. The server is respectively connected to HDFS and MapReduce. Exemplarily, a Hadoop cluster may be provided with a separate server, which is connected to a local area network in the Hadoop cluster to achieve communication with HDFS and MapReduce. In order to improve the security of data in the Hadoop cluster, the server may set a unique port for the user or the user's computing program unit to communicate, so as to implement the load balancing control method provided by the present invention. For example, the port may be port 20771.

[0052] The load balancing control method provided in the embodiment of the present application is described in detail below in conjunction with other drawings in the present application.

[0053] like Figure 2As shown, the embodiment of the present invention provides a load balancing control method, the execution subject is Figure 1 According to the control device shown in FIG. , the method includes steps S201 - S205 .

[0054] S201: The control device receives a query request from a user.

[0055] The query request includes a target file list and time information for starting to read, and the target file list includes file identifiers of files requested to be read by the user.

[0056] In the embodiment of the present application, the query request includes a list of target files requested to be read. The list of target files includes the file identifiers of the files requested to be read by the user. For example, assuming that the user's computing task requires reading 100 files in directory A, the list of target files in the query request is a list of files consisting of the file identifiers of the 100 files.

[0057] In the embodiment of the present application, the query request also includes the time information of starting to read. The time information includes the date and the time point. For example, assuming that the user's computing task needs to read the files in the file list at 20:00 on the 21st, the date in the time information of starting to read is the 21st, and the time point is 20:00.

[0058] In the embodiment of the present application, the query request is used to request a query for a list of permitted files in the target file list requested to be read that the user is allowed to start reading at that point in time.

[0059] Exemplarily, the permitted file list may be some of the files in the file list, that is, the resource usage of the Hadoop cluster is too high, and the file list accessed by the user needs to be restricted. For example, the permitted file list may include file identifiers of 80 files out of 100 files.

[0060] In another exemplary embodiment, the permitted file list may be all files in the file list, that is, the current resource occupancy rate of the Hadoop cluster can meet the user's need to access the files in the file list, and there is no need to limit the file list accessed by the user. For example, the permitted file list may include the file identifiers of all files in 100 files.

[0061] In some embodiments, the query request also includes a user identifier. Exemplarily, the user identifier may be a registered account number stored in the user's tariff information, or the user identifier may also be a user name set by the user. This application does not limit this.

[0062] In some embodiments, the query request also includes the user's authority information. Exemplarily, the query request may include the user's account number and password. The control device verifies the user's authority based on the account number and password.

[0063] As a possible implementation manner, the user equipment directly sends a query request to the control device, and correspondingly, the control device receives the query request sent by the user equipment.

[0064] As another possible implementation, the user equipment sends the computing task to the computing model, and the computing model sends a query request to the control device. Correspondingly, the control device receives the user's query request sent by the computing model.

[0065] In some embodiments, the user may send the query request to the control device through a web service application programming interface (webservice API).

[0066] S202: The control device determines the total amount of information of the files requested to be read by the user according to the target file list.

[0067] As a possible implementation method, the control device can query the storage information of the data files in the distributed file system based on the file list, determine the information volume of each file in the file list; and determine the total information volume of the files requested by the user to read based on the information volume of each file.

[0068] As another possible implementation, the control device may periodically scan the storage information of the data files in the distributed file system and save the scanning results. Thereafter, the control device determines the total amount of information of the file requested to be read by the user according to the pre-stored scanning results.

[0069] S203: The control device determines the information volume limit value according to the file reading information and resource occupancy information of the user within a set historical date.

[0070] In the embodiment of the present application, the historical date is set to include the N days before the date in the query request, where N is a positive integer. The end date of the historical date is set to be the day before the date in the query request. Exemplarily, the specified historical date may include 7 days. Assuming that the date in the query request is the 21st, the specified historical date is from the 14th to the 20th.

[0071] In some embodiments, the file reading information includes the reading speed and time of the file. Exemplarily, the file reading information may include the instantaneous reading speed at multiple moments.

[0072] In some embodiments, the resource occupancy information includes any one of the following: processor resource occupancy information, memory resource occupancy information, and disk resource occupancy information.

[0073] S204: When the total amount of information is greater than the information amount limit, the control device selects some file identifiers from the target file list to obtain a permitted file list.

[0074] The total amount of information of the files in the license file list is less than or equal to the information limit.

[0075] It can be understood that if the total amount of information is greater than the information limit, it means that the Hadoop cluster is in a busy period, the resource usage in the current period is high, and it is necessary to limit the amount of file information that users are allowed to access during this period.

[0076] As a possible implementation method, the control device can obtain the file information amount of each file in the file list, and determine the first M files in the P files in the file list as the permitted file list according to the order of the files in the file list, where the file information amount of the M files is less than the information amount limit, P and M are both positive integers, and P>M.

[0077] As another possible implementation method, the control device can obtain the file information volume of each file in the file list, randomly select M files from the P files in the file list to determine as the permitted file list, wherein the file information volume of the M files is less than the information volume limit, P and M are both positive integers, and P>M.

[0078] The present invention provides a load balancing control method. After receiving a user's query request, the information volume limit is determined according to the user's file reading information and resource occupancy information within a set historical date; the target file list requested by the user is identified according to the information volume limit, and when the total information volume requested by the user is greater than the information volume limit, the files in the target file list requested by the user are restricted. In this way, the load balancing control method provided by the present invention can restrict the information volume requested by the user according to the file reading information and resource occupancy information within the historical date, thereby realizing load balancing control of the big data system and improving the utilization efficiency of resources in the big data system.

[0079] Exemplarily, in the process of Hadoop cluster providing services to users, users need to read data files in HDFS to complete computing services. Assuming that the user requests to read file list A during the busy period, the control method of the present invention can determine the information volume limit based on the file reading information and resource occupancy information within the historical date, identify the file list A, and obtain a list of licensed files whose total information volume is less than or equal to the information volume limit, thereby limiting the file reading situation of users during the busy period, extending or shifting the time for providing computing services to users, so that the resources in the Hadoop cluster can be balanced and controlled, and the utilization efficiency of the resources in the Hadoop cluster can be improved.

[0080] Optionally, step S203 may be specifically implemented through steps A1-A2.

[0081] A1. The control device determines the average reading speed and resource occupancy deviation coefficient of the user in a set period within the set historical date according to the file reading information and resource occupancy information of the user within the set historical date.

[0082] In the embodiment of the present application, the historical date is set to include the N days before the date, N is a positive integer, and the starting point of the set time period is the time point.

[0083] In some embodiments, the starting point of the set time period is the time point in the query request. For example, the length of the set time period can be 1 hour or half an hour, which is not limited. Assuming that the time point in the query request is 20 o'clock, the set time period can be the period from 20 o'clock to 21 o'clock on each day from the 14th to the 20th.

[0084] In some embodiments, the average reading speed may be a preset value. For example, the control device may preset the average reading speed of the user to be 200 G / h.

[0085] In some embodiments, the average reading speed may be determined according to the user's tariff information. Exemplarily, the control device may query the user's tariff information for the upper limit of resource occupation per unit time, and determine the average reading speed according to the upper limit of resource occupation per unit time.

[0086] In other embodiments, the average speed may also be determined based on the file reading information of the user's calculation model within a set historical date to determine the average reading speed.

[0087] Exemplarily, the average reading speed may be determined according to steps A11 - A12 .

[0088] A11. The control device determines the instantaneous reading speed of the user's computing model at multiple moments in a set period within the set historical date based on the file reading information of the user's computing model within the set historical date.

[0089] A12. The control device determines the average reading speed according to the instantaneous reading speeds at multiple moments.

[0090] For example, the control device may determine the average value of instantaneous reading speeds at multiple moments as the average reading speed.

[0091] For another example, the control device determines a fitting function according to instantaneous reading speeds at multiple moments, calculates an integral value of the fitting function over time, and determines a ratio of the integral value to time as an average reading speed for a specified period of time.

[0092] In the embodiment of the present application, the resource occupancy deviation coefficient is used to indicate the deviation between the resource occupancy at a time point in a set period within a set historical date and the average resource occupancy in the set period.

[0093] It can be understood that if the resource occupancy deviation coefficient is a negative value, the resource occupancy at the time point is less than the average resource occupancy of the set period, indicating that the set period is a leisure period. If the resource occupancy deviation coefficient is a positive value, the resource occupancy at the time point is greater than the average resource occupancy of the set period, indicating that the set period is a busy period.

[0094] In some embodiments, the resource occupancy deviation coefficient may be determined according to steps A13-A15.

[0095] A13. The control device determines the user's calculation model according to the resource occupancy information of the user's calculation model within the set historical date, the instantaneous resource occupancy at multiple moments in the set time period within the set historical date, and the resource occupancy at a time point every day.

[0096] In some embodiments, the average resource occupancy is used to represent the average occupancy at each moment in a set time period. For example, the average resource occupancy may be the average resource occupancy at each moment in the time period from 20:00 to 21:00.

[0097] In some embodiments, the resource occupancy at the time point is used to represent the instantaneous resource occupancy at the time point, for example, the instantaneous resource occupancy at 20:00 on the 14th.

[0098] A14. The control device determines the average resource occupancy during a set period of time within a set historical date based on the instantaneous resource occupancy at multiple times.

[0099] Exemplarily, the control device may calculate an average value of instantaneous resource occupancy at multiple moments, and determine the average value as the average resource occupancy for a set period of time.

[0100] As another example, the control device can set the instantaneous occupancy of resources at multiple moments in a specified time period within a historical date, determine a fitting function, calculate the integral value of the fitting function over time, and determine the ratio of the integral value to time as the average resource occupancy for the set time period.

[0101] A15. The control device determines the resource occupancy deviation coefficient based on the average resource occupancy and the resource occupancy.

[0102] Exemplarily, the control device may subtract the resource occupancy at a time point on each day within a set historical date from the average resource occupancy to obtain multiple difference values; and determine the ratio of the average value of the multiple difference values ​​to the average resource occupancy as the resource occupancy deviation coefficient.

[0103] As another example, the control device may determine the resource occupancy deviation coefficient according to the following formula.

[0104]

[0105] Among them, r is the resource occupancy deviation coefficient, is the average resource usage in the set time period, pi is the resource usage at the i-th day in the set historical date, and n is the number of days in the set historical date.

[0106] A2. The control device determines the limit of the amount of information that the user is allowed to read within a certain period of time based on the average reading speed and the resource occupancy deviation coefficient.

[0107] In the embodiment of the present application, a certain period is one or more periods of the same length as the set period starting from a date time point. For example, assuming that the time information in the query request is 20:00 on the 21st, the certain period can be from 20:00 on the 21st to 23:00 on the 21st.

[0108] In some embodiments, the duration of a certain period of time may be a preset duration. For example, the control device may preset the user's single query duration to be 3 hours.

[0109] In other embodiments, the duration of a certain period of time may be the remaining access time in the user's tariff information. For example, the user's tariff information contains a total of 20 hours of duration, 17 hours have been used, and the remaining unused access time is 3 hours.

[0110] In other embodiments, the duration of a certain period of time may also be determined comprehensively based on resource usage within a set historical date and user fee information.

[0111] Exemplarily, the control device may query the user's tariff information according to the user identifier in the query request, determine the user's remaining resource amount, determine the resource occupancy of the set period according to the average resource occupancy and the duration of the set period, and determine the duration of a certain period according to the remaining resource amount, the resource occupancy of the set period, and the duration of the set period. The remaining resource amount is used to indicate the amount of file information that can be read remaining in the user's tariff information; the resource occupancy of the set period is used to indicate the amount of file information read by the user within the set period.

[0112] For example, the control device may determine the duration of a certain period of time according to the following formula.

[0113]

[0114] Among them, t0 is the duration of a certain period, Q is the remaining resources of the user, and p a is the resource usage in the set time period, and t1 is the duration of the set time period.

[0115] As a possible implementation manner, the information amount limit can be determined according to the following formula.

[0116] L M =(1+r)×v0×t0;

[0117] Among them, L M is the information limit, r is the resource usage deviation coefficient, v0 is the average reading speed, and t0 is the duration of a certain period.

[0118] In this way, the present invention adjusts the limit on the amount of information that users are allowed to read within a certain period of time through the resource occupancy deviation coefficient, thereby reducing the amount of file information that users are allowed to access during busy periods and increasing the amount of file information that users are allowed to access during idle periods. This makes it possible to balance the load during busy periods in the big data system and transfer it to idle periods for processing, thereby achieving load balancing control of the big data system and improving the utilization efficiency of resources in the big data system.

[0119] Optionally, the load balancing control method provided by the embodiment of the present invention further includes step S205.

[0120] S205: When the total amount of information is less than or equal to the information amount limit, the control device determines all file identifiers in the target file list as a permitted file list.

[0121] It can be understood that the total amount of information is less than or equal to the information limit, which means that the Hadoop cluster is in an idle period and the resource usage of the current period is low, allowing the user to read the files in the target file list during this period to complete the computing task.

[0122] Optional, such as Figure 3As shown, the load balancing control method provided by the embodiment of the present invention further includes step S206.

[0123] S206: The control device sends a query response to the user. Correspondingly, the user receives the query response sent by the control device.

[0124] In the embodiment of the present application, the query response includes a license file list.

[0125] In some embodiments, the query response may also include a file address of a file in the permitted file list, so that the user can access the file in the permitted file list.

[0126] It should be noted that after receiving the license file list in the query response, the user can use the license file list as the input of the calculation model, and the calculation model completes the calculation task to obtain the calculation result.

[0127] Exemplarily, when the Hadoop cluster is in an idle period, the Hadoop cluster does not restrict the target file list, allowing the user to access all files in the file list, so that the user can complete the computing task and obtain the computing result during the idle period.

[0128] In another exemplary embodiment, when the Hadoop cluster is in a busy period, the Hadoop cluster restricts the target file list and allows the user to access the files in the license file list, so that during the idle period, the user completes a part of the computing task according to the files in the license file list. When the Hadoop cluster allows the user to access other files in the target file list except the license file list, the remaining part of the computing task is completed.

[0129] In this way, the load balancing method provided in the embodiment of the present application can limit the amount of file information accessed by users during busy hours, thereby transferring the load during busy hours to off-peak hours, thereby achieving load balancing control in HDFS and improving the utilization efficiency of resources in HDFS.

[0130] Optionally, the control device may also periodically detect data files in HDFS, user resource occupancy information, user historical access information, and user fee information, and save the detection results. Thus, after receiving a user's query request, the control device directly determines the license file list based on the detection results.

[0131] Optionally, after receiving the user's query request, the control device can also query the database of the Hadoop cluster to obtain the user's resource occupancy information, the user's historical access information, the user's fee information, etc., so as to complete the load balancing control method of the present invention.

[0132] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0133] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0134] Figure 4 The schematic diagram of the structure of a control device provided by an embodiment of the present invention is shown. The control device 400 is used to implement the load balancing control method described in the above embodiment. The control device 400 includes: a communication module 401 and a processing module 402.

[0135] The communication module 401 is used to receive a query request from a user, wherein the query request includes a target file list and time information for starting to read, and the target file list includes file identifiers of files requested to be read by the user.

[0136] Processing module 402 is used to determine the total amount of information of the files requested to be read by the user based on the target file list; determine the information amount limit based on the file reading information and resource usage information of the user within a set historical date; when the total amount of information is greater than the information amount limit, filter some file identifiers from the target file list to obtain a licensed file list, wherein the total amount of information of the files in the licensed file list is less than or equal to the information amount limit.

[0137] In some embodiments, the time information for starting reading includes the date and time point for starting reading; the processing module 402 is specifically used to determine the average reading speed and resource occupancy deviation coefficient of the user in a set time period within the set historical date based on the file reading information and resource occupancy information of the user within the set historical date, the set historical date includes the previous N days before the date, N is a positive integer, the starting point of the set time period is the time point, and the resource occupancy deviation coefficient is used to represent the deviation between the resource occupancy at the time point in the set time period within the set historical date and the average resource occupancy in the set time period; based on the average reading speed and the resource occupancy deviation coefficient, determine the limit on the amount of information allowed to be read by the user within a certain period of time, the certain period being one or more periods of the same length as the set period starting from the time point of the date.

[0138] In some embodiments, the processing module 402 is specifically configured to determine the information volume limit according to the following formula.

[0139] L M =(1+r)×v0×t0;

[0140] Among them, L Mis the maximum limit of file information volume, r is the resource usage deviation coefficient, v0 is the average reading speed, and t0 is the duration of a certain period.

[0141] In some embodiments, processing module 402 is specifically used to determine the instantaneous reading speed of the user's computing model at multiple moments in a set time period within the set historical date based on the file reading information of the user's computing model within the set historical date; determine the average reading speed based on the instantaneous reading speeds at multiple moments; determine the instantaneous resource occupancy of the user's computing model at multiple moments in the set time period within the set historical date and the resource occupancy at a time point in each day based on the resource occupancy information of the user's computing model within the set historical date; determine the average resource occupancy of the set time period within the set historical date based on the instantaneous resource occupancy at multiple moments; determine the resource occupancy deviation coefficient based on the average resource occupancy and the resource occupancy.

[0142] In some embodiments, the processing module 402 is specifically used to subtract the resource occupancy at a time point in each day within a set historical date from the average resource occupancy to obtain multiple difference values; and the ratio of the average value of the multiple difference values ​​to the average resource occupancy is determined as the resource occupancy deviation coefficient.

[0143] In some embodiments, the query request also includes a user identifier, and the processing module 402 is also used to query the user's tariff information based on the user identifier, and determine the user's remaining resource amount, where the remaining resource amount is used to represent the remaining readable file information amount in the user's tariff information; determine the resource occupancy of the set time period based on the average resource occupancy and the length of the set time period, where the resource occupancy of the set time period is used to characterize the amount of file information read by the user within the set time period; determine the length of a certain period based on the remaining resource amount, the resource occupancy of the set time period, and the length of the set time period.

[0144] In some embodiments, the processing module 402 is specifically used to query the storage information of the data files in the distributed file system according to the target file list, determine the amount of information of each file in the target file list; and determine the total amount of information of the files requested by the user to read according to the amount of information of each file.

[0145] In some embodiments, the communication module 401 is further configured to send a query response to the user, where the query response includes a license file list.

[0146] Figure 5 FIG. 1 is a schematic diagram of the structure of another control device provided by an embodiment of the present invention. Figure 5As shown, the control device 400 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above method embodiment are implemented, for example Figure 2 Alternatively, when the processor 501 executes the computer program 503, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 4 The functions of the communication module 401 and the processing module 402 are shown.

[0147] Exemplarily, the computer program 503 may be divided into one or more modules / units, which are stored in the memory 502 and executed by the processor 501 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 503 in the control device 400. For example, the computer program 503 may be divided into Figure 4 The communication module 401 and the processing module 402 are shown.

[0148] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0149] The memory 502 may be an internal storage unit of the control device 400, such as a hard disk or memory of the control device 400. The memory 502 may also be an external storage device of the control device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the control device 400. Further, the memory 502 may also include both an internal storage unit of the control device 400 and an external storage device. The memory 502 is used to store the computer program and other programs and data required by the control device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0150] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the method embodiments of the above-mentioned power control system can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0157] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A load balancing control method, characterized in that: include: Receive a query request from a user, wherein the query request includes a target file list and a time information for starting to read, the target file list includes file identifiers of files requested to be read by the user; the time information for starting to read includes a date and time point for starting to read; Determining the total amount of information of the files requested to be read by the user according to the target file list; Determine the information volume limit value according to the file reading information and resource occupation information of the user within a set historical date; When the total amount of information is greater than the information amount limit, filtering some file identifiers from the target file list to obtain a permitted file list, wherein the total amount of information of the files in the permitted file list is less than or equal to the information amount limit; The determining of the information volume limit value according to the file reading information and resource occupancy information of the user within the set historical date comprises: determining the average reading speed and resource occupancy deviation coefficient of the user in a set period within the set historical date according to the file reading information and resource occupancy information of the user within the set historical date, the set historical date includes the first N days before the date, N is a positive integer, the starting point of the set period is the time point, and the resource occupancy deviation coefficient is used to represent the deviation between the resource occupancy at the time point and the average resource occupancy of the set period in the set period within the set historical date; determining the information volume limit value allowed to be read by the user within a certain period according to the average reading speed and the resource occupancy deviation coefficient, the certain period being one or more periods of the same length as the set period starting from the time point of the date; Determining the information amount limit value allowed to be read by the user within a certain period of time according to the average reading speed and the resource occupation deviation coefficient includes: determining the information amount limit value according to the following formula; L M =(1+r)×v0×t0; where L M is the maximum limit of the file information volume, r is the resource occupation deviation coefficient, v0 is the average reading speed, and t0 is the duration of the certain period; The method of determining the average reading speed and resource occupancy deviation coefficient of the user in a set time period within a set historical date based on the file reading information and resource occupancy information of the user within a set historical date includes: determining the instantaneous reading speed of the user's computing model at multiple moments in a set time period within a set historical date based on the file reading information of the user's computing model within the set historical date; determining the average reading speed based on the instantaneous reading speeds at the multiple moments; determining the instantaneous resource occupancy of the user's computing model at multiple moments in a set time period within the set historical date and the resource occupancy at the time points of each day based on the resource occupancy information of the user's computing model within the set historical date; determining the average resource occupancy of the set time period within the set historical date based on the instantaneous resource occupancy at multiple moments; subtracting the resource occupancy at the time points of each day within the set historical date from the average resource occupancy to obtain multiple difference values; and determining the ratio of the average value of the multiple difference values ​​to the average resource occupancy as the resource occupancy deviation coefficient.

2. The method according to claim 1, characterized in that The query request also includes a user identifier, and the method further includes: According to the user identifier, query the user's fee information to determine the remaining resource amount of the user, where the remaining resource amount is used to indicate the remaining readable file information amount in the user's fee information; Determine the resource occupancy of the set period according to the average resource occupancy and the duration of the set period, wherein the resource occupancy of the set period is used to characterize the amount of file information read by the user within the set period; The duration of the certain period is determined according to the remaining resource amount, the resource occupancy amount of the set period, and the duration of the set period.

3. The method according to claim 1, characterized in that The step of determining the total amount of information of the files requested to be read by the user according to the target file list includes: According to the target file list, query the storage information of the data files in the distributed file system to determine the information volume of each file in the target file list; The total information volume of the files requested to be read by the user is determined according to the information volume of each file.

4. The method according to any one of claims 1 to 3, characterized in that The resource occupancy information includes any one of the following: processor resource occupancy information, memory resource occupancy information and disk resource occupancy information; The method further includes: sending a query response to the user, wherein the query response includes the license file list.

5. A load balancing control device, characterized in that: The control device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images