Bandwidth adjustment method, system, device and storage medium

Abstract

Embodiments of the present application provide a bandwidth adjustment method, system, device and storage medium. The attribute description information of the current computing nodes accessing the data lake can be counted, and the expected bandwidth of each computing node can be predicted. Under each storage unit in the data lake, the total expected bandwidth corresponding to the currently carried computing nodes can be calculated respectively, and the bandwidth quota of the storage unit can be adjusted according to the total expected bandwidth and the actual bandwidth of the storage unit. In this way, the bandwidth adjustment system can serve as an intermediate medium between the computing nodes and the data lake, and can perceive the dynamic bandwidth demand of the computing nodes accessing the data lake in real time, and can adjust the bandwidth quota of the storage unit in the data lake based on the perceived dynamic bandwidth demand. Accordingly, the bandwidth quota of each storage unit in the data lake can be dynamically adjusted, so that the bandwidth provided by the storage unit in the data lake can be adapted to the bandwidth demand of the computing nodes, thereby more reasonably providing bandwidth for the computing nodes accessing the data lake.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and in particular to a bandwidth adjustment method, system, device and storage medium. BACKGROUND

[0002] A data lake can be interpreted as a highly scalable data storage area that stores a large amount of raw data in a raw format until it needs to be used. The data lake can store all types of data without fixed limitations on account size or files, and without defining specific purposes. Data comes from different sources and can be structured, semi-structured, or even unstructured, and data can be queried on demand. The data lake can seamlessly interface with multiple computing and analysis platforms, and according to different use scenarios, a corresponding computing engine can be selected to process and analyze the data stored in the data lake, thereby breaking down the silos and mining business value.

[0003] Bandwidth is an important resource of the data lake. At present, the data lake usually provides bandwidth according to the storage amount, for example, if the storage amount of a tenant in the data lake is 16TB, the tenant is provided with a bandwidth capacity of 100MBps by default. However, this is too rigid for the tenants of the data lake and cannot meet the flexible bandwidth requirements of the tenants, and problems such as redundancy and waste of the bandwidth provided to the tenants or insufficient bandwidth affecting the data analysis efficiency of the tenants often occur. SUMMARY

[0004] Aspects of the present application provide a bandwidth adjustment method, system, device and storage medium to more reasonably provide bandwidth for computing nodes accessing a data lake.

[0005] The bandwidth adjustment method provided by the present application embodiment comprises:

[0006] From the access request received from the data lake, the attribute description information of the computing node currently accessing the data lake is counted;

[0007] Based on the attribute description information, the expected bandwidth of the computing node is predicted respectively;

[0008] The expected bandwidth is taken as a basis to determine the expected total bandwidth corresponding to the computing node currently carried in the storage unit in the data lake and to adjust the bandwidth quota according to the expected total bandwidth and the actual bandwidth of the storage unit.

[0009] The bandwidth adjustment method provided by the present application embodiment comprises:

[0010] The expected bandwidth of the computing node currently accessing the data lake is predicted, and the expected bandwidth is predicted based on the attribute description information of the computing node;

[0011] Under the storage unit in the data lake, the expected bandwidth total amount corresponding to the currently bearing computing nodes is respectively calculated;

[0012] The actual bandwidth amount of the storage unit is acquired;

[0013] Under the storage unit, the bandwidth quota is respectively adjusted according to the actual bandwidth amount and the expected bandwidth total amount.

[0014] Embodiments of the present application also provide an electronic device, comprising a memory and a processor;

[0015] The memory is used for storing one or more computer instructions;

[0016] The processor is coupled with the memory and is used for executing the one or more computer instructions, so as to:

[0017] From the access request received from the data lake, the attribute description information of the computing nodes currently accessing the data lake is counted;

[0018] Based on the attribute description information, the expected bandwidth amount of the computing nodes is respectively predicted;

[0019] The expected bandwidth amount is taken as a basis to determine the expected bandwidth total amount corresponding to the currently bearing computing nodes under the storage unit in the data lake and to adjust the bandwidth quota according to the expected bandwidth total amount and the actual bandwidth amount of the storage unit.

[0020] Embodiments of the present application also provide an electronic device, comprising a memory and a processor;

[0021] The memory is used for storing one or more computer instructions;

[0022] The processor is coupled with the memory and is used for executing the one or more computer instructions, so as to:

[0023] The expected bandwidth amount predicted for the computing nodes currently accessing the data lake is acquired, the expected bandwidth amount being predicted based on the attribute description information of the computing nodes;

[0024] Under the storage unit in the data lake, the expected bandwidth total amount corresponding to the currently bearing computing nodes is respectively calculated;

[0025] The actual bandwidth amount of the storage unit is acquired;

[0026] Under the storage unit, the bandwidth quota is respectively adjusted according to the actual bandwidth amount and the expected bandwidth total amount.

[0027] Embodiments of the present application also provide a bandwidth adjusting system, comprising:

[0028] a statistics node applying the bandwidth adjustment method, and a dynamic adjustment node applying the bandwidth adjustment method; the statistics node provides an expected bandwidth amount predicted for a computing node accessing a data lake to the dynamic adjustment node, so that the dynamic adjustment node adjusts a bandwidth quota for a storage unit in the data lake.

[0029] The embodiments of the present application further provide a computer readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the bandwidth adjustment method.

[0030] In the embodiments of the present application, a bandwidth adjustment system applied to a data lake scenario is provided, which can statistically acquire attribute description information of each computing node currently accessing the data lake, and predict an expected bandwidth amount of each computing node; under each storage unit in the data lake, an expected bandwidth total amount corresponding to the computing nodes currently carried by the storage unit can be calculated, and a bandwidth quota of the storage unit can be adjusted according to the expected bandwidth total amount and an actual bandwidth amount of the storage unit. In this way, the bandwidth adjustment system can serve as an intermediate medium between the computing nodes and the data lake, and can perceive dynamic bandwidth demands of the computing nodes accessing the data lake in real time, and can adjust the bandwidth quota of the storage unit in the data lake based on the perceived dynamic bandwidth demands. Accordingly, in the embodiments of the present application, the bandwidth adjustment system can dynamically adjust the bandwidth quota of each storage unit in the data lake, so that the bandwidth amount provided by the storage unit in the data lake can be adapted to the bandwidth demands of the computing nodes, thereby more reasonably providing bandwidth for the computing nodes accessing the data lake. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0032] Figure 1 a structure schematic diagram of a bandwidth adjustment system according to an example embodiment of the present application;

[0033] Figure 2 a flowchart of a bandwidth adjustment method according to another example embodiment of the present application;

[0034] Figure 3 a flowchart of a statistics stage according to yet another example embodiment of the present application;

[0035] Figure 4 a flowchart of a dynamic adjustment stage according to yet another example embodiment of the present application;

[0036] Figure 5A flowchart of an execution stage according to another example embodiment of the present application;

[0037] Figure 6 A flowchart of a bandwidth adjustment method according to another example embodiment of the present application;

[0038] Figure 7 A structural diagram of an electronic device according to another example embodiment of the present application;

[0039] Figure 8 A flowchart of another bandwidth adjustment method according to another example embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0041] At present, data lakes usually provide bandwidth according to storage capacity, which is too rigid and cannot meet the flexible bandwidth requirements of tenants. Therefore, in some embodiments of the present application: a bandwidth adjustment system applied to a data lake scenario is provided, which can count the attribute description information of each computing node currently accessing the data lake and predict the expected bandwidth of each computing node; under each storage unit in the data lake, the expected total bandwidth corresponding to the computing nodes currently carried by the storage unit can be calculated, and the bandwidth quota of the storage unit can be adjusted according to the expected total bandwidth and the actual bandwidth of the storage unit. In this way, the bandwidth adjustment system can serve as an intermediate medium between the computing nodes and the data lake, and can perceive the dynamic bandwidth requirements of the computing nodes accessing the data lake in real time, and can adjust the bandwidth quota of the storage unit in the data lake based on the perceived dynamic bandwidth requirements. Accordingly, in the embodiments of the present application, the bandwidth adjustment system can dynamically adjust the bandwidth quota of each storage unit in the data lake, so that the bandwidth provided by the storage unit in the data lake can be adapted to the bandwidth requirements of the computing nodes, thereby more reasonably providing bandwidth for the computing nodes accessing the data lake.

[0042] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0043] Figure 1 A structural diagram of a bandwidth adjustment system according to an example embodiment of the present application. As shown in the figure, the system includes a counting node and a dynamic adjustment node. Figure 1

[0044] ​The statistical node is configured to: count attribute description information of each computing node currently accessing the data lake from an access request received by the data lake; and predict an expected bandwidth of each computing node based on the attribute description information.

[0045] The dynamic adjustment node is configured to: calculate an expected total bandwidth of computing nodes currently carried under each storage unit in the data lake; obtain an actual bandwidth of each storage unit; and adjust a bandwidth quota under each storage unit according to the actual bandwidth and the expected total bandwidth.

[0046] The bandwidth adjustment system provided by the embodiment can be applied to a data lake scenario to provide a bandwidth adjustment solution for the data lake. The data lake can be interpreted as a highly scalable data storage area, and the data lake can include a large number of storage units. The storage unit can be understood as the smallest storage unit in the data lake. In different data lake products, the definition name of the storage unit can be different. The storage unit can be a bucket, and the embodiment does not limit the name, specification, and storage form of the storage unit. In addition, the original flow control system (a system for managing bandwidth) of the data lake usually controls bandwidth in units of storage units. For example, tenant A rents a plurality of buckets in the data lake, and the flow control system can configure a fixed bandwidth for tenant A on each bucket according to the storage amount of tenant A on each bucket.

[0047] The data lake can seamlessly connect to various computing and analysis platforms, and according to different analysis requirements, a corresponding computing engine can be selected to process and analyze the data stored in the data lake. At present, the data lake and the computing and analysis platform usually belong to different operators, which makes it difficult to connect the computing and storage links. Therefore, the data lake can only configure bandwidth for tenants according to the storage amount used by the tenants, which leads to the problem of unreasonable bandwidth configuration. It should be understood that the data lake usually cannot accurately perceive the bandwidth requirements of the tenants, and it is even more impossible to configure bandwidth according to the bandwidth requirements of the tenants.

[0048] To solve this dilemma, the embodiment provides a bandwidth adjustment system to make the data lake provide bandwidth to the computing nodes more reasonably. In the embodiment, due to the flexible and changeable computing requirements of the tenants, the computing nodes accessing the data lake are also dynamically changed, which includes but is not limited to the change of the number of computing nodes and the change of the storage units accessed by the computing nodes. Therefore, the bandwidth adjustment system provided by the embodiment can dynamically adjust the bandwidth provided by the data lake to the computing nodes as the computing nodes dynamically change. Considering that the computing nodes are essentially accessing the storage units in the data lake, and the computing nodes carried on the same storage unit share the bandwidth provided by the storage unit, therefore, in the embodiment, the bandwidth quota of each storage unit in the data lake can be dynamically adjusted, so that the bandwidth obtained by the computing nodes carried on the storage unit can be indirectly adjusted.

[0049] In the embodiment, in order to avoid introducing impact on the tenant side, all nodes included in the bandwidth adjustment system can be deployed in the data lake, of course, the embodiment is not limited thereto, and the nodes in the bandwidth adjustment system can also be deployed outside the data lake. In addition, considering the huge number of access requests occurring in the data lake, in order to ensure the efficiency of bandwidth adjustment, each node in the bandwidth adjustment system can be implemented as a node in the cloud, of course, the embodiment is not limited thereto, and each node in the bandwidth adjustment system can also be implemented as a processing module in a computing device, and the embodiment does not limit the physical implementation form of each node in the bandwidth adjustment system.

[0050] Reference Figure 1 In the embodiment, the statistics node can obtain the attribute description information of each computing node currently accessing the data lake from the access requests received by the data lake. The statistics node can be deployed in the original access access layer of the data lake to more conveniently analyze the access requests received by the data lake. In addition, the statistics node can be implemented by using a distributed cluster, and the distributed cluster can include at least one control node and a plurality of worker nodes controlled by each control node. In the case where the number of control nodes is one, the control node can take over all the computing nodes accessing the data lake; and in the case where the number of control nodes is more than one, the plurality of control nodes can take charge of the computing nodes under different tenants. For example, the control nodes are three, which are a, b and c, and if the tenants of the data lake are 3 million, the 3 million tenants can be divided into three groups, and the three control nodes can take charge of one group respectively. Each control node can call the plurality of worker nodes controlled thereby to complete the statistical work.

[0051] In this embodiment, the statistical attribute description information can include but is not limited to the identifier of the tenant to which the computing node belongs, the identifier of the accessed storage unit, the node type or the node address, etc. Of course, the attribute description information can also include other information that can represent the identity of the computing node or is helpful for predicting the expected bandwidth of the computing node, and is not limited to this. In addition, in this embodiment, a communication protocol can be predetermined between the statistical node and the computing node, and the attribute description information required by the computing node to carry in the access request can be specified in the communication protocol. In this way, the computing node can carry the attribute description information in the access request when initiating the access request to the data lake, so that the statistical node can parse the attribute description information from the access request.

[0052] On this basis, the statistical node can predict the expected bandwidth of each computing node based on the attribute description information. In this embodiment, the statistical node and the computing node are connected, and various parameters for predicting bandwidth can be obtained, so that the statistical node can conveniently and quickly predict the expected bandwidth of each computing node currently accessing the data lake based on the obtained parameters. The prediction scheme adopted by the statistical node in this embodiment is not limited, and the prediction scheme can be flexibly set as needed.

[0053] An exemplary prediction scheme can be: obtaining a hardware parameter of a target node in any of the computing nodes based on the node type of the target node; calculating a first predicted bandwidth of the target node according to the hardware parameter; perceiving the type of an application program initiating the current access request on the target node; searching for a second predicted bandwidth corresponding to the combination of the type of the application program and the node type of the target node from an experience library; and determining the expected bandwidth corresponding to the target node according to the first predicted bandwidth and the second predicted bandwidth. The hardware parameter can include but is not limited to the number of CPU cores, the number of memories, the network bandwidth capability, etc. In addition, the experience library can include the bandwidth capability values that various application programs can reach when running on different types of computing nodes, and the bandwidth capability values in the experience library can be experience values, from which appropriate experience values can be selected as the second predicted bandwidth.

[0054] In this exemplary prediction scheme, the expected bandwidth of the target node is calculated from the hardware and software angles. It should be understood that this is only an exemplary prediction scheme, and the scheme for predicting the expected bandwidth of the computing node in this embodiment is not limited to this.

[0055] It should be noted that in this embodiment, the access request received by the data lake can come from multiple computing nodes, involve multiple tenants, and the computing nodes or tenants accessing the data lake can be dynamically changed. For this purpose, in this embodiment, the statistical node can periodically perform statistical work to timely perceive these changes. In order to more conveniently perceive these changes, in an exemplary implementation scheme: the statistical node can also maintain an access request statistics table, the access request statistics table including a field corresponding to the attribute description and a field corresponding to the expected bandwidth amount; and the attribute description information and the expected bandwidth amount of each computing node accessing the data lake are dynamically updated in the access request statistics table. An exemplary access request statistics table is as follows, Table-1:

[0056] Tenant Bucket Compute Node IP Node Type Extension Field Expected Bandwidth Amount Tenant A A-Bucket 1 192.168.0.2 I2 Normal 1 Gbps Tenant A A-Bucket 1 192.168.0.3 I2 Normal 1 Gbps Tenant A A-Bucket 1 192.168.1.2 G2 Normal 2 Gbps Tenant A A-Bucket 2 192.168.1.3 G2 Normal 2 Gbps Tenant A A-Bucket 2 192.168.100.2 D1 Normal 0.5 Gbps … … … … … … Tenant B B-Bucket 3 192.168.200.2 G2 Normal 2 Gbps … … … … … …

[0057] Based on this, the statistical node can trigger the dynamic adjustment node in Figure 1 to start working when it is monitored that the access request statistics table changes. Wherein, the change of the access request statistics table refers to the increase or decrease of records in the table or the change of the value under the field, the increase or decrease of records representing that the computing nodes accessing the data lake have undergone elastic scaling, and the change of the value under the field representing that the access state of the existing computing nodes has changed. Of course, this is a preferred scheme, and the embodiment is not limited thereto, and the dynamic adjustment node can also not be triggered by the statistical node, but autonomously arrange its working time, for example, the dynamic adjustment node can also periodically work, etc.

[0058] Referring to Figure 1For the dynamic adjustment node, the expected total bandwidth of each storage unit in the data lake can be calculated respectively. That is, the dynamic adjustment node can convert the bandwidth demand (expected bandwidth of the computing node, etc.) perceived from the computing node side under the node dimension or the tenant dimension into bandwidth requirements under the storage unit in the data lake, thereby realizing the connection between computing and storage. Alternatively, the dynamic adjustment node can calculate the sum of the expected bandwidth of all computing nodes currently carried by the target unit in each storage unit as the expected total bandwidth of the target unit. A single storage unit usually carries computing nodes of a tenant, in which case the dynamic adjustment node can sum the expected bandwidth of each computing node carried by the storage unit to obtain the expected total bandwidth of the storage unit. For example, if bucket B in the data lake currently carries 5 computing nodes of tenant A, the expected bandwidth of each computing node can be summed to obtain the expected total bandwidth of bucket B. A single storage unit can carry computing nodes of multiple tenants, and there can be bandwidth priority between tenants. In the bandwidth resource allocation process, the tenants will also be preempted for bandwidth according to the bandwidth priority, and have obtained bandwidth from the storage unit in proportion. In this case, the dynamic adjustment node can temporarily weaken the concept of tenant in the dynamic adjustment stage, and also sum the expected bandwidth of each computing node carried by the storage unit to obtain the expected total bandwidth of the storage unit. Of course, the dynamic adjustment node can also calculate the expected total bandwidth of each tenant corresponding to the storage unit respectively to prepare for the allocation of the adjusted bandwidth resource of the storage unit to multiple tenants in the subsequent bandwidth resource allocation process.

[0059] Reference Figure 1 The dynamic adjustment node can also obtain the actual bandwidth of each storage unit. As mentioned earlier, the data lake originally has a flow control system deployed, so the flow control system can be reused to measure the actual bandwidth of each storage unit, and the dynamic adjustment node can obtain the actual bandwidth of each storage unit from the flow control system. Of course, the present embodiment is not limited thereto, and the dynamic adjustment node can also independently measure the actual bandwidth of each storage unit.

[0060] In addition, in the present embodiment, the dynamic adjustment node can also obtain the original bandwidth quota of each storage unit, where the original bandwidth quota refers to the bandwidth quota obtained by each storage unit after the last dynamic adjustment operation. In this dynamic adjustment operation, the bandwidth quota of each storage unit will be adjusted again.

[0061] In an alternative implementation, the dynamic adjustment node can maintain a dynamic adjustment record table, which can include, but is not limited to, a tenant identification field, a storage unit identification field, an actual bandwidth amount field, an original bandwidth quota field, and an expected total bandwidth field. An exemplary dynamic adjustment record table is as follows, Table-2:

[0062]

[0063]

[0064] On this basis, referring to Figure 1 , the dynamic adjustment node can also adjust the bandwidth quota under each storage unit according to the actual bandwidth amount and the expected total bandwidth, respectively. An exemplary adjustment scheme can be:

[0065] Under each first-type storage unit, the required bandwidth increment is calculated, the expected total bandwidth of the first-type storage unit is greater than the original bandwidth quota and the actual bandwidth amount reaches the original bandwidth quota;

[0066] Under each second-type storage unit, the supported bandwidth decrement is calculated, the actual bandwidth amount of the second-type storage unit is less than the original bandwidth quota;

[0067] In the case where the total value of the bandwidth increment corresponding to the first-type storage units is less than the total value of the bandwidth decrement corresponding to the second-type storage units, the bandwidth quota of the first-type storage units is adjusted to the expected total bandwidth and the bandwidth quota of the second-type storage units is adjusted to the actual bandwidth amount.

[0068] In this exemplary adjustment scheme, the bandwidth increment of each first-type storage unit can be calculated, bandwidth increment = expected total bandwidth - original bandwidth quota. The bandwidth increment corresponding to each first-type storage unit is summed to obtain the total value of the bandwidth increment corresponding to the first-type storage units. Similarly, the bandwidth decrement corresponding to each second-type storage unit can also be calculated, bandwidth decrement = original bandwidth quota - actual bandwidth amount. The bandwidth decrement corresponding to each second-type storage unit is summed to obtain the total value of the bandwidth decrement corresponding to the second-type storage units.

[0069] In the example scheme, the first type of storage unit and the second type of storage unit can be found from each storage unit of the data lake respectively. Specifically, the storage units whose expected total bandwidth is greater than the original bandwidth quota and whose actual bandwidth amount reaches the original bandwidth quota are found as the first type of storage unit. Taking the above dynamic adjustment record table as an example, A-barrel 1 and A-barrel 2 therein are the first type of storage unit. The storage units whose actual bandwidth amount is less than the original bandwidth quota are found as the second type of storage unit. Taking the above dynamic adjustment record table as an example, B-barrel 3, B-barrel 4, C-barrel 5 and C-barrel 6 therein are the second type of storage unit.

[0070] The dynamic adjustment node increases the bandwidth quota of the first type of storage unit and decreases the bandwidth quota of the second type of storage unit. In this process, the relationship between the total bandwidth increment value corresponding to the first type of storage unit and the total bandwidth decrement value corresponding to the second type of storage unit needs to be considered. In the case where the total bandwidth increment value corresponding to the first type of storage unit is less than the total bandwidth decrement value corresponding to the second type of storage unit, it is represented that the bandwidth resource that can be released by the second type of storage unit can meet the bandwidth increment demand of the first type of storage unit, and therefore, the bandwidth quota of the first type of storage unit can be adjusted to the expected total bandwidth and the bandwidth quota of the second type of storage unit can be adjusted to the actual bandwidth amount.

[0071] In the case where the total bandwidth increment value corresponding to the first type of storage unit is greater than the total bandwidth decrement value corresponding to the second type of storage unit, an increment coefficient can be calculated according to the current remaining bandwidth amount of the data lake and the total bandwidth increment value. The bandwidth quota of each first type of storage unit is adjusted according to the increment coefficient and the expected total bandwidth. Exemplarily, the increment coefficient can be = (the current remaining bandwidth amount of the data lake + the total bandwidth decrement value corresponding to the second type of storage unit) / the total bandwidth increment value. It should be understood that, in the case where the total bandwidth increment value corresponding to the first type of storage unit is greater than the total bandwidth decrement value corresponding to the second type of storage unit, it is represented that the bandwidth resource that can be released by the second type of storage unit cannot meet the bandwidth increment demand of the first type of storage unit, and new bandwidth pressure appears in the data lake. The inventive concept here is that the dynamic adjustment node adjusts the bandwidth quota of each first type of storage unit according to the bandwidth service upper limit of the data lake, so as to ensure that the adjusted bandwidth quota will not cause the total bandwidth configured out of the data lake to exceed the bandwidth service upper limit of the data lake. The implementation of the increment coefficient is not limited to the above. In addition, the scheme for calculating the increment coefficient is not limited to the above specific logic, and the present embodiment does not limit this.

[0072] At this point, the bandwidth quota corresponding to each storage unit in the data lake can be dynamically adjusted.

[0073] In this embodiment, the bandwidth adjustment system can further include an execution node. The execution node can obtain the adjusted bandwidth quota of each storage unit from the dynamic adjustment node, and send the bandwidth quota to a flow control system in the data lake, so that the flow control system adjusts the bandwidth of the computing nodes it carries according to the bandwidth quota of each storage unit.

[0074] In addition, the execution node can also determine the tenants to which each storage unit belongs, and calculate the sum of the bandwidth quotas obtained by the used storage units under each tenant as the tenant-level bandwidth quota. Here, there is a special case mentioned earlier, that is, the case where the same storage unit carries multiple tenants. In this case, the execution node can determine the bandwidth quota that can be allocated to each of the multiple tenants on the storage unit according to the bandwidth preemption logic mentioned above, and the allocated bandwidth quota can be incorporated into the tenant-level bandwidth quota.

[0075] Optionally, the execution node can maintain a bandwidth quota record table, which can include but is not limited to a configuration level field, a tenant identifier field, a storage unit identifier field, and a bandwidth quota field. After the bandwidth quota after this adjustment takes effect, the execution node can update the bandwidth quota in the bandwidth quota record table to the original bandwidth quota field in the dynamic adjustment record table for use by the dynamic adjustment node in the next adjustment phase. An exemplary bandwidth quota record table is as follows, Table-3:

[0076] Configuration Level Tenant Bucket Bandwidth Quota Tenant Level Tenant A None 50 Bucket Level Tenant A A-Bucket 1 20 Bucket Level Tenant A A-Bucket 2 30 … Tenant Level Tenant B None 40 Bucket Level Tenant B B-Bucket 3 20 Bucket Level Tenant B B-Bucket 4 20

[0077] Based on this, the bandwidth quota of each storage unit in the data lake is dynamically adjusted, and the adjusted bandwidth quota is adapted to the bandwidth demand of the computing nodes carried thereon. As for how to allocate the bandwidth quota on the storage unit among the computing nodes carried thereon, the bandwidth preemption mechanism, the average allocation mechanism, the bandwidth balancing mechanism, etc. can be set to ensure that each computing node obtains the required bandwidth amount, and this embodiment does not limit this.

[0078] To sum up, in the embodiment, a bandwidth adjustment system applied to a data lake scenario is provided, which can count attribute description information of each computing node currently accessing the data lake and predict expected bandwidth amounts of the computing nodes; under each storage unit in the data lake, a total expected bandwidth amount corresponding to the currently carried computing nodes can be calculated respectively, and the bandwidth quota of the storage unit can be adjusted according to the total expected bandwidth amount and an actual bandwidth amount of the storage unit currently. In this way, the bandwidth adjustment system can serve as an intermediate medium between the computing nodes and the data lake, perceive dynamic bandwidth demands of the computing nodes accessing the data lake in real time, and adjust the bandwidth quota of the storage unit in the data lake based on the perceived dynamic bandwidth demands. Accordingly, in the embodiment, the bandwidth adjustment system can dynamically adjust the bandwidth quota of each storage unit in the data lake, so that the bandwidth amount provided by the storage unit in the data lake is adapted to the bandwidth demands of the computing nodes, thereby more reasonably providing bandwidth for the computing nodes accessing the data lake.

[0079] Figure 2 A flowchart of a bandwidth adjustment method provided for another exemplary embodiment of the application is shown, which can be implemented by the aforementioned bandwidth adjustment system. Referring to Figure 2 , the method can include:

[0080] Step 200, counting attribute description information of each computing node currently accessing the data lake from access requests received by the data lake;

[0081] Step 201, respectively predicting expected bandwidth amounts of the computing nodes based on the attribute description information;

[0082] Step 202, respectively calculating total expected bandwidth amounts corresponding to the currently carried computing nodes under each storage unit in the data lake;

[0083] Step 203, obtaining actual bandwidth amounts of the storage units currently;

[0084] Step 204, respectively adjusting the bandwidth quota according to the actual bandwidth amount and the total expected bandwidth amount under each storage unit.

[0085] In an optional embodiment, the attribute description information includes one or more of an identifier of a tenant to which the computing node belongs, an identifier of the accessed storage unit, a node type, or a node address.

[0086] In an optional embodiment, the step of respectively calculating the expected bandwidth amounts of the computing nodes based on the attribute description information includes:

[0087] obtaining a hardware parameter of a target node in any of the computing nodes based on the node type of the target node;

[0088] calculating a first predicted bandwidth amount of the target node according to the hardware parameter;

[0089] perceive a type of an application that initiates the current access request on the target node;

[0090] from the experience library, find a second predicted bandwidth amount corresponding to a combination of the type of the application and the node type of the target node;

[0091] determine an expected bandwidth amount corresponding to the target node according to the first predicted bandwidth amount and the second predicted bandwidth amount.

[0092] In an optional embodiment, the method can further include:

[0093] maintain an access request statistics table, the access request statistics table including a field describing a property and a field corresponding to an expected bandwidth amount;

[0094] dynamically update the property description information and the expected bandwidth amount of each computing node accessing the data lake in the access request statistics table.

[0095] In an optional embodiment, the method can further include:

[0096] in a case where a change in the access request statistics table is monitored, trigger an operation of calculating an expected bandwidth total amount corresponding to currently bearing computing nodes under each storage unit in the data lake and a subsequent operation.

[0097] In an optional embodiment, the step of calculating the expected bandwidth total amount corresponding to the currently bearing computing nodes under each storage unit in the data lake includes:

[0098] calculating a sum of the expected bandwidth amounts of all computing nodes currently bearing by a target unit as an expected bandwidth total amount corresponding to the target unit under the target unit.

[0099] In an optional embodiment, the step of adjusting the bandwidth quota under each storage unit according to the actual bandwidth amount and the expected bandwidth total amount includes:

[0100] calculating a required bandwidth increment under each first-type storage unit, the expected bandwidth total amount of the first-type storage unit being greater than an original bandwidth quota and the actual bandwidth amount reaching the original bandwidth quota;

[0101] calculating a supported bandwidth decrement under each second-type storage unit, the actual bandwidth amount of the second-type storage unit being less than the original bandwidth quota;

[0102] in a case where a total value of the bandwidth increments corresponding to the first-type storage units is less than a total value of the bandwidth decrements corresponding to the second-type storage units, adjusting the bandwidth quota of the first-type storage units to the expected bandwidth total amount and adjusting the bandwidth quota of the second-type storage units to the actual bandwidth amount.

[0103] In an optional embodiment, the method can further include:

[0104] In the case that the total bandwidth increment value corresponding to the first type of storage unit is greater than the total bandwidth decrement value corresponding to the second type of storage unit, calculating an increment coefficient according to the current residual bandwidth amount of the data lake and the total bandwidth increment value, the increment coefficient being less than 1;

[0105] Adjusting the bandwidth quota of each first type of storage unit according to the increment coefficient and the total expected bandwidth amount.

[0106] In an optional embodiment, the method can further include:

[0107] Determining the tenant to which each storage unit belongs;

[0108] Calculating the sum value of the bandwidth quota obtained by the used storage units under each tenant as the tenant-level bandwidth quota.

[0109] In an optional embodiment, the method can further include:

[0110] Sending the latest bandwidth quota obtained by each storage unit to the flow control system in the data lake, so that the flow control system adjusts the bandwidth amount of the computing nodes carried according to the bandwidth quota of each storage unit.

[0111] Accordingly, the embodiment provides a bandwidth adjustment method applied to a data lake scenario, which can count the attribute description information of each computing node currently accessing the data lake and predict the expected bandwidth amount of each computing node; under each storage unit in the data lake, the total expected bandwidth amount corresponding to the computing nodes currently carried by the storage unit can be calculated, and the bandwidth quota of the storage unit can be adjusted according to the total expected bandwidth amount and the actual bandwidth amount of the storage unit. In this way, the bandwidth adjustment system can serve as an intermediate medium between the computing nodes and the data lake, and can perceive the dynamic bandwidth demand of the computing nodes accessing the data lake in real time, and can adjust the bandwidth quota of the storage unit in the data lake based on the perceived dynamic bandwidth demand. Accordingly, in the embodiment of the present application, the bandwidth adjustment system can dynamically adjust the bandwidth quota of each storage unit in the data lake, so that the bandwidth amount provided by the storage unit in the data lake is adapted to the bandwidth demand of the computing nodes, thereby more reasonably providing the computing nodes accessing the data lake with bandwidth.

[0112] The bandwidth adjustment method will be divided into a statistical stage, a dynamic adjustment stage and an execution stage, and each stage will be exemplarily described.

[0113] Figure 3 A flowchart of the statistical stage is provided for another exemplary embodiment of the present application. Referring to Figure 3 , the exemplary processing logic of the statistical stage can include:

[0114] 1. In the request access layer, count all requests;

[0115] 2. In the single machine, count the requests of each tenant according to the event interval t;

[0116] 3. Each single machine sends the tenant request statistics to the designated node N, and the node N is responsible for updating the information of the corresponding tenant in Table-1;

[0117] 4. If the tenant request statistics have changed, update the corresponding records in Table-2 through dynamic adjustment of the node; if not, the single machine processes the next time interval t and returns to step 2.

[0118] In this phase, the access request statistics are performed at a specified frequency, i.e., at a time interval t. The statistics are independently counted by a large number of single machines in the data lake, storage layer access layer, which improves efficiency through local processing; then, according to the distributed principle, a specified control node is used to process the statistics of a specific tenant, and the control node is responsible for updating the data in Table-1, thereby reducing the performance problems caused by a large number of single machines accessing Table-1 concurrently. Table-1 can be stored locally on the control node.

[0119] In order to reduce the performance impact of updating Table-2 every time each tenant changes, after multiple control nodes have processed all tenant information, it is determined whether there is an overall update change in Table-1. If there is a change, update the corresponding records in Table-2 in batches, thereby improving the efficiency of data exchange between nodes and reducing the performance impact.

[0120] Figure 4 A flowchart of a dynamic adjustment phase provided for another exemplary embodiment of the present application is shown. Referring to Figure 4 , the exemplary processing logic of the dynamic adjustment phase can include:

[0121] 1. Analyze Table-2 according to a time interval t';

[0122] 2. If the "total expected bandwidth of the computing node" is greater than the "bandwidth quota" and the "actual bandwidth" has reached the "bandwidth quota", it means that the bandwidth needs to be increased, and the total bandwidth X that each tenant hopes to increase is calculated;

[0123] 3. If the "actual bandwidth" is less than the "bandwidth quota", it means that the bandwidth can be reduced, and the total bandwidth Y that can be reduced is calculated;

[0124] 4. If X-Y is less than 0, perform step 5; if X-Y is not less than 0, perform step 6;

[0125] 5. The system has no new bandwidth pressure, and Table-3 is adjusted as needed;

[0126] 6. Determine whether the added amount exceeds the system pressure of the data lake. If yes, execute 7; if no, execute 8;

[0127] 7. The system has no added bandwidth pressure. Adjust Table-3 as needed.

[0128] 8. According to the system capacity, proportionally reduce the bandwidth increment, and send the adjusted bandwidth quota to Table-3.

[0129] In this phase, the dynamic adjustment node scans Table-2 to handle scenarios of increasing and decreasing bandwidth:

[0130] • Analyze the records to find records where "total expected bandwidth of computing nodes" is greater than "bandwidth quota" and "actual bandwidth" has reached "bandwidth quota", indicating that bandwidth needs to be increased, and calculate the total bandwidth X that each tenant hopes to increase after aggregation.

[0131] • Analyze the table records to find records where "actual bandwidth" is less than "bandwidth quota", indicating that bandwidth can be reduced, and calculate the total bandwidth Y that can be reduced after aggregation of each tenant.

[0132] • Then analyze the trend of this round of dynamic adjustment. If "X-Y is less than 0", the system has no new bandwidth pressure and Table-3 is adjusted as needed; otherwise, the system has new bandwidth pressure, and it needs to be determined whether it exceeds the system pressure.

[0133] If "the added amount exceeds the system pressure", proportionally reduce the bandwidth increment according to the system capacity, and send the adjusted bandwidth quota to Table-3; otherwise, the system has no new bandwidth pressure, and Table-3 is adjusted as needed.

[0134] If Table-3 has content updates, the bandwidth actual flow control module will complete the reconfiguration of bandwidth in the system.

[0135] Figure 5 A flowchart of the execution phase is provided for another exemplary embodiment of the present application. Referring to Figure 5 , the exemplary processing logic of the execution phase can include:

[0136] 1. The execution node reads Table-3 update records at a time interval t".

[0137] 2. The bandwidth actual flow control module writes the values of Table-3 records into the flow control system.

[0138] 3. Confirm that the flow control takes effect in the system, and the flow control quota value is applied to a single machine or cluster.

[0139] Through the above process, the dynamic adjustment of bandwidth according to the elastic scaling of computing nodes can be realized, fully utilizing the computing capacity and making the computing engine more convenient to use.

[0140] To sum up, the bandwidth adjustment scheme provided by the embodiment can produce beneficial effects in at least the following aspects:

[0141] The field definition of the tenant-level access request statistics table. Through the table, the node and expected bandwidth information can be efficiently mastered, helping subsequent dynamic bandwidth adjustment.

[0142] The tenant-level access request statistics adopts single-machine local statistics, and the tenant records are updated by the proxy node, thereby reducing the concurrent influence and improving the system performance.

[0143] The dynamic adjustment node can dynamically adjust the bandwidth according to the increase or decrease of bandwidth demand, in combination with the system bandwidth capacity, thereby flexibly meeting the business demand.

[0144] The dynamic adjustment node updates the table according to the actual change, thereby realizing the minimum granularity of table update and reducing the performance problem caused by large-scale processing of fields.

[0145] The execution node can guarantee the effectiveness of the configuration through the two stages of writing and confirming.

[0146] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 200 to 204 can be device A; for another example, the execution subject of steps 200 and 201 can be device A, and the execution subject of steps 202 to 204 can be device B; and the like.

[0147] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or performed in parallel without the order in which they appear in this text. The serial numbers of the operations, such as 201, 202, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of “first”, “second” and the like in this text are used to distinguish different storage units, etc., and do not represent the sequence, nor do they limit the types of “first” and “second”.

[0148] It should be noted that the technical details in the above bandwidth adjustment method embodiments can refer to the related descriptions in the system embodiments described above, and will not be repeated here to save space, but this should not cause any loss of the protection scope of the present application.

[0149] In some other possible designs, Figure 6 A flowchart of a bandwidth adjustment method provided by another exemplary embodiment of the present application is shown in FIG. 6. Figure 6In these possible designs, the method can include:

[0150] Step 600, from the access request received from the data lake, statistics the attribute description information of the computing node currently accessing the data lake;

[0151] Step 601, based on the attribute description information, respectively predict the expected bandwidth amount of the computing node;

[0152] Step 602, take the expected bandwidth amount as the basis to determine the expected total bandwidth amount corresponding to the currently bearing computing node under the storage unit in the data lake and adjust the bandwidth quota according to the expected total bandwidth amount and the actual bandwidth amount of the storage unit at present.

[0153] Wherein, the attribute description information can include one or more of the identification of the tenant, the identification of the accessed storage unit, the node type or the node address.

[0154] Wherein, the step of calculating the expected bandwidth amount of the computing node based on the attribute description information can include:

[0155] Based on the node type of any target node in the computing node, obtain the hardware parameter of the target node;

[0156] According to the hardware parameter, calculate the first predicted bandwidth amount of the target node;

[0157] Perceive the type of the application program initiating the current access request on the target node;

[0158] From the experience library, find the second predicted bandwidth amount corresponding to the combination of the type of the application program and the node type of the target node;

[0159] According to the first predicted bandwidth amount and the second predicted bandwidth amount, determine the expected bandwidth amount corresponding to the target node.

[0160] Optionally, the method can further include:

[0161] Maintain an access request statistics table, which includes fields related to attribute description and fields corresponding to expected bandwidth amount;

[0162] Dynamically update the attribute description information and the expected bandwidth amount of the computing node accessing the data lake in the access request statistics table.

[0163] Figure 7 A structural schematic diagram of an electronic device provided by another exemplary embodiment of the present application is shown in FIG. 7. Figure 7 The electronic device can include a memory 70, a processor 71, a communication component 72, a power supply component 73, etc. Figure 6The related method can be implemented by an electronic device, which can be a cloud server, a cloud node, a server cluster, etc., and the physical implementation form of the electronic device is not limited. For different physical implementation forms, the components in the electronic device can also be implemented according to the corresponding deployment requirements, which will not be described here.

[0164] It is worth noting that, Figure 6 The bandwidth adjustment method shown can be implemented independently without the limitation of other system participants, and in addition, Figure 6 The expected bandwidth amount generated in the bandwidth adjustment method shown can also be applied to other application scenarios other than the bandwidth adjustment scenario, and the present embodiment does not limit this.

[0165] In still some possible designs, Figure 8 Another flowchart of a bandwidth adjustment method provided for another exemplary embodiment of the present application is shown in FIG. 8. Figure 8 In these possible designs, the method can include:

[0166] Step 800: obtaining an expected bandwidth amount predicted for a computing node currently accessing a data lake, the expected bandwidth amount being predicted based on attribute description information of the computing node;

[0167] Step 801: calculating, under a storage unit in the data lake, an expected bandwidth total amount corresponding to the currently carried computing node respectively;

[0168] Step 802: obtaining an actual bandwidth amount of the storage unit currently;

[0169] Step 803: adjusting, under the storage unit, a bandwidth quota respectively according to the actual bandwidth amount and the expected bandwidth total amount.

[0170] Among them, the step of calculating, under the storage unit in the data lake, an expected bandwidth total amount corresponding to the currently carried computing node respectively, can include:

[0171] Calculating, under a target unit in the storage unit, a sum of expected bandwidth amounts of all computing nodes currently carried by the target unit as an expected bandwidth total amount corresponding to the target unit.

[0172] Among them, the step of adjusting, under the storage unit, a bandwidth quota respectively according to the actual bandwidth amount and the expected bandwidth total amount, can include:

[0173] Under a first type of storage unit, the required bandwidth increment is calculated respectively, and the expected bandwidth total amount of the first type of storage unit is greater than the original bandwidth quota and the actual bandwidth amount reaches the original bandwidth quota;

[0174] Under a second type of storage unit, the supported bandwidth decrement is calculated respectively, and the actual bandwidth amount of the second type of storage unit is less than the original bandwidth quota.

[0175] In a case where the total bandwidth increment value corresponding to the first type of storage unit is less than the total bandwidth decrement value corresponding to the second type of storage unit, the bandwidth quota of the first type of storage unit is adjusted to the total expected bandwidth and the bandwidth quota of the second type of storage unit is adjusted to the actual bandwidth.

[0176] Optionally, the method can further include:

[0177] In a case where the total bandwidth increment value corresponding to the first type of storage unit is greater than the total bandwidth decrement value corresponding to the second type of storage unit, an increment coefficient is calculated according to the current remaining bandwidth of the data lake and the total bandwidth increment value, the increment coefficient being less than 1.

[0178] The bandwidth quota of the first type of storage unit is adjusted according to the increment coefficient and the total expected bandwidth.

[0179] Optionally, the method can further include:

[0180] A tenant to which the storage unit belongs is determined.

[0181] The sum of the bandwidth quotas obtained by the used storage units is calculated as a tenant-level bandwidth quota under the tenant.

[0182] Optionally, the method can further include:

[0183] The latest bandwidth quota obtained by the storage unit is sent to a flow control system in the data lake, so that the flow control system adjusts the bandwidth of the computing node carried according to the bandwidth quota of the storage unit.

[0184] Based on Figure 7 The structural schematic diagram of the electronic device provided, Figure 8 The related method can be implemented by the electronic device, which can be a cloud server, a cloud node, a server cluster, etc., and the physical implementation form of the electronic device is not limited. It should be noted that Figure 8 The bandwidth adjustment method shown can be independently implemented without the limitation of other system participants, in addition, Figure 8 The expected bandwidth required in the bandwidth adjustment method shown can be obtained through various possible ways, and the embodiment does not limit the way of obtaining.

[0185] Correspondingly, the embodiment of the present application further provides a computer readable storage medium storing a computer program, which can implement each step that can be executed by the electronic device in the method embodiment when the computer program is executed.

[0186] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0187] The present application is described in reference to the flowchart illustrations and / or block diagrams according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams.

[0188] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams. Figure One one or more functions specified in the flowchart illustrations and / or block diagrams.

[0190] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0191] The memory can include non-persistent memory, random access memory (RAM), and / or non-volatile memory, such as read only memory (ROM) or flash memory, among others. The memory is an example of computer-readable media.

[0192] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0193] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0194] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A bandwidth adjustment method, comprising: From the access requests received from the data lake, statistically analyze the attribute description information of each computing node currently connected to the data lake; Based on the attribute description information, the expected bandwidth of each computing node is predicted respectively. The desired bandwidth is provided to the dynamic adjustment node, which is used to determine the total desired bandwidth corresponding to the currently supported computing node under each storage unit in the data lake. Obtain the current actual bandwidth of each storage unit; For each Class I storage unit, the required bandwidth increment is calculated, where the expected total bandwidth of the Class I storage unit is greater than the original bandwidth quota and the actual bandwidth reaches the original bandwidth quota. For each Class II storage unit, the supported bandwidth reduction is calculated, where the actual bandwidth of the Class II storage unit is less than the original bandwidth quota. In each storage unit, if the total bandwidth increment corresponding to the Class I storage unit is less than the total bandwidth reduction corresponding to the Class II storage unit, the bandwidth quota of each Class I storage unit is adjusted to its corresponding expected total bandwidth, and the bandwidth quota of each Class II storage unit is adjusted to its corresponding actual bandwidth.

2. The method according to claim 1, wherein the attribute description information includes one or more of the following: the identifier of the tenant, the identifier of the accessed storage unit, the node type, or the node address.

3. The method according to claim 2, wherein calculating the expected bandwidth of each computing node based on the attribute description information includes: Based on the node type of any target node among the computing nodes, obtain the hardware parameters of the target node; Based on the hardware parameters, calculate the first predicted bandwidth of the target node; Sensing the type of application that initiated the current access request on the target node; From the experience base, find the second predicted bandwidth corresponding to the combination of the application type and the node type of the target node; Based on the first predicted bandwidth and the second predicted bandwidth, the expected bandwidth corresponding to the target node is determined.

4. The method according to claim 1, further comprising: Maintain an access request statistics table, which includes fields related to attribute descriptions and fields corresponding to expected bandwidth. The access request statistics table is dynamically updated with the attribute description information and expected bandwidth of the computing nodes connected to the data lake.

5. A bandwidth adjustment method, comprising: Obtain the expected bandwidth for each computing node currently connected to the data lake, the expected bandwidth being predicted based on the attribute description information of each computing node; Under each storage unit in the data lake, calculate the total expected bandwidth corresponding to the currently supported computing node; Obtain the current actual bandwidth of each storage unit; For each of the first type of storage units, the required bandwidth increment is calculated separately. The expected total bandwidth of the first type of storage unit is greater than the original bandwidth quota and the actual bandwidth reaches the original bandwidth quota. Under each second-class storage unit, the supported bandwidth reduction is calculated separately, and the actual bandwidth of the second-class storage unit is less than the original bandwidth quota. In each storage unit, if the total bandwidth increment corresponding to the first type of storage unit is less than the total bandwidth reduction corresponding to the second type of storage unit, the bandwidth quota of each first type of storage unit will be adjusted to its corresponding expected total bandwidth, and the bandwidth quota of each second type of storage unit will be adjusted to its corresponding actual bandwidth.

6. The method according to claim 5, wherein calculating the total expected bandwidth corresponding to the currently supported computing node under each storage unit in the data lake includes: Under each target unit in the storage unit, the sum of the expected bandwidth of all computing nodes currently supported by the target unit is calculated as the total expected bandwidth corresponding to the target unit.

7. The method according to claim 5, further comprising: If the total bandwidth increment corresponding to the first type of storage unit is greater than the total bandwidth reduction corresponding to the second type of storage unit, an increment coefficient is calculated based on the current remaining bandwidth of the data lake and the total bandwidth increment, and the increment coefficient is less than 1. The bandwidth quota of the first type of storage unit is adjusted according to the incremental coefficient and the expected total bandwidth.

8. The method according to claim 5, further comprising: Determine the tenant to which the storage unit belongs; Under each tenant, the sum of the bandwidth quotas obtained by the storage units used is calculated and used as the tenant-level bandwidth quota.

9. The method according to claim 5, further comprising: The latest bandwidth quota obtained under the storage unit is sent to the flow control system in the data lake, so that the flow control system can adjust the bandwidth of the computing nodes it supports according to the bandwidth quota of the storage unit.

10. An electronic device, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: From the access requests received from the data lake, statistically analyze the attribute description information of each computing node currently connected to the data lake; Based on the attribute description information, the expected bandwidth of each computing node is predicted respectively. The desired bandwidth is provided to the dynamic adjustment node, which is used to determine the total desired bandwidth corresponding to the currently supported computing node under each storage unit in the data lake. Obtain the current actual bandwidth of each storage unit; For each Class I storage unit, the required bandwidth increment is calculated, where the expected total bandwidth of the Class I storage unit is greater than the original bandwidth quota and the actual bandwidth reaches the original bandwidth quota. For each Class II storage unit, the supported bandwidth reduction is calculated, where the actual bandwidth of the Class II storage unit is less than the original bandwidth quota. In each storage unit, if the total bandwidth increment corresponding to the Class I storage unit is less than the total bandwidth reduction corresponding to the Class II storage unit, the bandwidth quota of each Class I storage unit is adjusted to its corresponding expected total bandwidth, and the bandwidth quota of each Class II storage unit is adjusted to its corresponding actual bandwidth.

11. An electronic device, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: Obtain the expected bandwidth for each computing node currently connected to the data lake, the expected bandwidth being predicted based on the attribute description information of each computing node; Under each storage unit in the data lake, calculate the total expected bandwidth corresponding to the currently supported computing node; Obtain the current actual bandwidth of each storage unit; For each of the first type of storage units, the required bandwidth increment is calculated separately. The expected total bandwidth of the first type of storage unit is greater than the original bandwidth quota and the actual bandwidth reaches the original bandwidth quota. Under each second-class storage unit, the supported bandwidth reduction is calculated separately, and the actual bandwidth of the second-class storage unit is less than the original bandwidth quota. In each storage unit, if the total bandwidth increment corresponding to the first type of storage unit is less than the total bandwidth reduction corresponding to the second type of storage unit, the bandwidth quota of each first type of storage unit will be adjusted to its corresponding expected total bandwidth, and the bandwidth quota of each second type of storage unit will be adjusted to its corresponding actual bandwidth.

12. A bandwidth adjustment system, comprising: A statistical node employing the bandwidth adjustment method as described in any one of claims 1-4, and a dynamic adjustment node employing the bandwidth adjustment method as described in any one of claims 5-9; the statistical node provides the expected bandwidth amount predicted for the computing nodes accessing the data lake to the dynamic adjustment node, so that the dynamic adjustment node can adjust the bandwidth quota for the storage units in the data lake.

13. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the bandwidth adjustment method according to any one of claims 1-9.

Images