Load migration method and device and electronic equipment

Abstract

The invention discloses a load migration method and device and electronic equipment, and relates to the technical field of cluster resource management and intelligent operation and maintenance, and the load migration method comprises the steps: obtaining historical time sequence data corresponding to each node in a cluster, the historical time sequence data corresponding to the node comprises running state information corresponding to the node at each time point of the historical time sequence; predicting node load information of each node in a future predetermined time period according to the historical time sequence data corresponding to each node; loads of nodes in the cluster comprise data fragments; determining a load migration decision for the nodes in the cluster according to the node load information of each node in a future predetermined time period; migrating partial data fragments of the first node in the cluster to at least one second node according to the load migration decision; wherein the node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cluster resource management and intelligent operation and maintenance, and particularly relates to a load migration method and device and electronic equipment. BACKGROUND

[0002] When a traditional OpenSearch cluster is expanded, a load such as a shard is allocated based on manual experience or static rules, which can easily lead to uneven load of CPUs or disks between cluster nodes (the actual difference can reach more than 40%), which can cause system performance to decline and affect stability, and at the same time, system hardware resources cannot be effectively utilized, resulting in resource waste. SUMMARY

[0003] To this end, the present application discloses the following technical solutions:

[0004] A load migration method, comprising:

[0005] obtaining historical time series data corresponding to each node in the cluster respectively, the historical time series data corresponding to each node including running state information corresponding to each node at each time point in the historical time series respectively;

[0006] predicting node load information of each node in a future predetermined time period according to the historical time series data corresponding to each node respectively; the load of the nodes in the cluster includes data shards;

[0007] determining a load migration decision for the nodes in the cluster according to the node load information of each node in the future predetermined time period;

[0008] migrating part of data shards of a first node in the cluster to at least one second node according to the load migration decision;

[0009] wherein the node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

[0010] Optionally, the predicting node load information of each node in a future predetermined time period according to the historical time series data corresponding to each node respectively comprises:

[0011] in a load prediction model based on a time window sliding manner, predicting a node load peak of each node in a future predetermined time period after a current time point according to historical time series data corresponding to each node at the current time point.

[0012] Optionally, the determining a load migration decision for the nodes in the cluster according to the node load information of each node in the future predetermined time period comprises:

[0013] determining a migration score corresponding to each node according to node load information of each node in a future predetermined time period; the migration score corresponding to each node is positively correlated with node load of the node load information of the node in the future predetermined time period;

[0014] determining a first node and a second node to be subjected to data shard migration according to the migration score corresponding to each node;

[0015] The first node is a node in the cluster whose corresponding migration score reaches a first threshold value, and the second node is a node in the cluster whose corresponding migration score does not reach a second threshold value; the first threshold value is not lower than the second threshold value.

[0016] Optionally, the data shard on the node includes an index shard stored on the node; the index shard is a data shard obtained by dividing an index.

[0017] Optionally, the load migration method further comprises:

[0018] determining an access mode corresponding to an index stored on each node in the cluster;

[0019] performing hierarchical processing on the index shard on each node according to the access mode corresponding to the index stored on each node.

[0020] Optionally, the determining of the access mode corresponding to the index stored on each node in the cluster comprises:

[0021] determining the access mode corresponding to the index according to a query frequency and / or an update frequency of the index stored on each node in the cluster.

[0022] Optionally, the hierarchical processing of the index shard on each node according to the access mode corresponding to the index stored on each node comprises:

[0023] storing the index shard on each node into storage hardware corresponding to performance matching the access mode of the index to which the index shard belongs, and allocating processing hardware corresponding to performance matching the access mode of the index to which the index shard belongs to the index shard.

[0024] Optionally, the load migration method further comprises:

[0025] determining a health degree of each node in the cluster;

[0026] in response to the health degree of the target node indicating that the target node fails, recovering the target node by synchronizing incremental load generated during the failure to the target node.

[0027] A load migration apparatus comprises:

[0028] An acquisition module is configured to obtain historical time series data corresponding to each node in a cluster respectively, wherein the historical time series data corresponding to each node comprises running state information corresponding to each node at each time point in a historical time series respectively;

[0029] A prediction module is configured to predict node load information of each node in a future predetermined time period according to the historical time series data corresponding to each node respectively; the load of each node in the cluster comprises data shards;

[0030] A determination module is configured to determine a load migration decision for the nodes in the cluster according to the node load information of each node in the future predetermined time period;

[0031] A migration processing module is configured to migrate part of data shards of a first node in the cluster to at least one second node according to the load migration decision;

[0032] wherein the node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

[0033] An electronic device comprises:

[0034] a memory configured to store at least a set of computer instructions;

[0035] a processor configured to implement any of the load migration methods provided above by executing the set of instructions stored in the memory.

[0036] A storage medium carrying one or more sets of computer instructions, which, when executed by an electronic device, can enable the electronic device to implement any of the load migration methods provided above. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0038] Figure 1 is a flowchart of the load migration method provided by the present application;

[0039] Figure 2 is an implementation architecture diagram of the load migration based on the load prediction model provided by the present application;

[0040] Figure 3 is another flowchart of the load migration method provided by the present application;

[0041] Figure 4 is still another flowchart of the load migration method provided by the present application;

[0042] Figure 5 is a component structure diagram of the load migration device provided by the present application;

[0043] Figure 6 is a component structure diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0045] The embodiments of the present application provide a load migration method, device and electronic device, which are used to solve the problem that the load distribution in the traditional OpenSearch cluster is unbalanced, which leads to the decline of system performance and affects stability, and the system hardware resources cannot be effectively utilized, thereby causing resource waste. The load migration method can be applied to, but is not limited to, electronic devices in a plurality of general-purpose or special-purpose computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, and the like.

[0046] Referring to the method flowchart shown in Figure 1 The load migration method provided by the embodiments of the present application can include the following steps 101 to 104, which will be described in detail below.

[0047] Step 101, obtaining historical time series data corresponding to each node in the cluster respectively.

[0048] The cluster can be, but is not limited to, an OpenSearch cluster.

[0049] Optionally, in this step, the historical time series data corresponding to each node in the cluster at the current time point respectively can be obtained, for example, the historical time series data of each node in the cluster within a certain corresponding time period (for example, n hours, n days, n weeks or n months before the current time point) before the current time point is obtained.

[0050] Optionally, the historical time series data corresponding to the nodes in the cluster includes running state information corresponding to each node at each time point in the historical time series.

[0051] The running state information can include, but is not limited to, CPU (Central Processing Unit) usage, memory usage, disk I / O (Input / Output), query QPS (Queries-per-second), and / or index rate, etc.

[0052] In step 102, node load information of each node in a future predetermined time period is predicted according to the historical time series data corresponding to each node.

[0053] The node load information of each node in the future predetermined time period can include, but is not limited to, a node load peak value of each node in the future predetermined time period.

[0054] After obtaining the historical time series data corresponding to each node at the current time point in the cluster based on step 101, optionally, the node load peak value of each node in a future predetermined time period after the current time point can be predicted according to the historical time series data corresponding to each node at the current time point, for example, a future 2-hour load peak value of each node after the current time point.

[0055] In implementation, optionally, a load prediction model can be constructed in advance, and the load prediction model is used to predict the node load peak value of each node in a future predetermined time period after the current time point.

[0056] The load prediction model can be, but is not limited to, a neural network model constructed by model training on an LSTM (Long Short-Term Memory) neural network. The input data of the model includes the historical time series data corresponding to each node at the current time point in the cluster, and the output data includes the node load peak value of each node in a future predetermined time period after the current time point.

[0057] In actual application, the load prediction model can specifically predict the node load peak value of each node in a future predetermined time period after the current time point according to the historical time series data corresponding to each node at the current time point based on a time window sliding manner.

[0058] In the embodiments of the present application, the load of the nodes in the cluster includes data shards.

[0059] Optionally, the data shards on the nodes include, but are not limited to, index shards stored on the nodes, wherein the index shards are data shards obtained by dividing an index.

[0060] Step 103, determining a load migration decision for the nodes in the cluster according to the node load information of each node in a future predetermined time period.

[0061] After predicting the node load information of each node in a future predetermined time period, the embodiment of the present application continues to determine a load migration decision for the nodes in the cluster according to the node load information of each node in a future predetermined time period. This process can but not limited to be further implemented as the following steps 11)-12):

[0062] 11) determining a migration score corresponding to each node respectively according to at least the node load information of each node in a future predetermined time period; the migration score corresponding to the node is positively correlated with the node load of the node represented by the node load information of the node in the future predetermined time period.

[0063] Specifically, but not limited to, the migration score corresponding to each node respectively can be determined according to the node load information of each node in the cluster in a future predetermined time period (for example, the node load peak of each node in a future predetermined time period after the current time point), and in combination with any one or more of the CPU load, disk usage and network delay of the node.

[0064] An exemplary calculation formula of the migration score of the node in the cluster is provided as follows:

[0065] Migration Score = a * CPU load + β * disk usage + γ * network delay + δ * node load.

[0066] Wherein, Migration Score represents the migration score of the node, a, β, γ and δ represent weight coefficients respectively, and in the implementation, the weight coefficients can but not limited to be optimized by gradient descent. The node load in the formula can include but not limited to the node load peak of the node in a future predetermined time period after the current time point.

[0067] 12) determining a first node and a second node to be migrated for data sharding according to the migration score corresponding to each node respectively.

[0068] Wherein, the first node is a node in the cluster corresponding to a migration score reaching a first threshold, and the second node is a node in the cluster corresponding to a migration score not reaching a second threshold; the first threshold is not lower than the second threshold.

[0069] It is easy to understand that the first node to be migrated is the node in the cluster that is high load in the future predetermined period of time and thus needs to migrate part of the data shards; the second node to be migrated is the node in the cluster that is low load in the future predetermined period of time and thus can migrate part of the data shards, so as to realize load balancing among the nodes in the cluster by dynamically migrating / scheduling the data shards among the different nodes in advance based on the different load conditions of the different nodes in the cluster in the future predetermined period of time, and avoid the problem of uneven load among the nodes in the cluster.

[0070] As described above, the migration score of a node is positively correlated with the node load of the node in the future predetermined period of time, which is represented by the node load information of the node. Based on this, after determining the migration scores of the nodes in the cluster, the nodes in the cluster whose corresponding migration scores reach a first threshold value can be determined as the first nodes to be migrated, and the nodes in the cluster whose corresponding migration scores do not reach a second threshold value can be determined as the second nodes that can be migrated, so as to form the load migration decision.

[0071] Step 104, migrating part of the data shards of the first node in the cluster to at least one second node according to the load migration decision.

[0072] In this embodiment, the node load of the first node in the future predetermined period of time is higher than the node load of the second node in the future predetermined period of time. For example, the peak value of the node load of the first node in the future predetermined period of time is higher than the peak value of the node load of the second node in the future predetermined period of time.

[0073] On the basis of step 103, this step further migrates part of the data shards of the first node in the cluster to at least one second node based on the load migration decision, so as to avoid the problem of uneven load among the nodes caused by the fact that the first node is too high in load and the second node is relatively too low in load in the future predetermined period of time (such as the next two hours).

[0074] After completing the processing of step 104, the load migration process in the method provided in this application can be ended, or the method can be waited until the next cycle is reached to trigger the method of this application again to perform dynamic load prediction and load migration processing based on the load prediction.

[0075] Referring to Figure 2, the application provides a load migration implementation architecture based on a load prediction model. In implementation, the load prediction model can be constructed by training a neural network such as LSTM based on a time series database. The time series database can include historical time series data of cluster nodes and load information of corresponding nodes in a future time period, which can be used as model training samples and test samples for model training and performance testing. Before model training based on sample data, the sample data can be preprocessed such as invalid data elimination and normalization. After the load prediction model is obtained through model training and testing, the load prediction model can be applied to new data prediction based on the method of the application to predict node load information of each node in the cluster in a future time period, and based on the prediction result, related intervention processing is performed on the cluster nodes, such as making a load migration decision based on the prediction result, and migrating the load of the nodes in the cluster based on the load migration decision.

[0076] In summary, the load migration method provided by the embodiments of the application realizes dynamic load prediction of nodes in a cluster and dynamic data shard scheduling based on the dynamic load prediction, can perform data shard migration / scheduling between different nodes in advance based on the dynamically predicted load information of different nodes in the cluster in a future predetermined time period, to realize early intervention of the cluster nodes, ensure load balancing between the cluster nodes, and avoid the problem of uneven node load, thereby effectively solving the problem of uneven load between cluster nodes caused by traditional OpenSearch clusters that rely on manual experience or static rule-based shard allocation, avoiding system performance degradation and affecting system stability caused by uneven load, and effectively utilizing system hardware resources to avoid resource waste.

[0077] In an optional embodiment, referring to the method flowchart shown in Figure 3 The load migration method provided by the application can further include the following processing.

[0078] Step 301, determining an access mode corresponding to an index stored on each node in the cluster.

[0079] In implementation, the query frequency (e.g., determined based on the number of _search requests) and / or the update frequency (e.g., determined based on the number of _bulk operations) of the indexes stored in each node in the cluster can be counted, and the access mode of the indexes stored in each node can be determined based on the query frequency and / or the update frequency of the indexes stored in each node.

[0080] In determining the access mode of the indexes based on the query frequency and / or the update frequency of the indexes stored in each node, a hotness index of the indexes stored in each node can be calculated based on the query frequency and / or the update frequency of the indexes stored in each node, and the access mode of the indexes can be determined based on the hotness index of the indexes stored in each node.

[0081] An exemplary calculation formula of the hotness index of the indexes is as follows:

[0082] Hotness index = lg(query_count + 1) x a + lg(update_count + 1) x b.

[0083] wherein query_count represents the query frequency of the indexes in the node; update_count represents the update frequency of the indexes in the node, i.e., the data update frequency of the shards in the indexes; a and b are weight coefficients, and exemplarily, a = 0.7 and b = 0.3.

[0084] The access mode of the indexes can include, but is not limited to, a hot access mode, a warm access mode and a cold access mode, denoted as Hot, Warm and Cold respectively. The hotness index corresponding to Hot, Warm and Cold decreases in turn, and after the hotness index of the indexes in the node is determined, the access mode of the indexes in the node can be determined based on the hotness index of the indexes in the node and the predetermined determination criteria of each access mode.

[0085] An example of the determination criteria of each access mode is provided as follows:

[0086] Hotness index ≥ 80, the access mode of the indexes is Hot;

[0087] 30 ≤ hotness index < 80, the access mode of the indexes is Warm;

[0088] Hotness index < 30, the access mode of the indexes is Cold.

[0089] In step 302, the shards of the indexes in each node are processed in layers based on the access mode of the indexes stored in each node.

[0090] Optionally, the index shards on each node are hierarchically processed according to the access mode corresponding to the index stored on the node, which can be further implemented as: storing the index shards on each node to the storage hardware corresponding to the performance matching the access mode of the index to which the index shard belongs, and allocating the processing hardware corresponding to the performance matching the access mode of the index to which the index shard belongs to the index shard.

[0091] The higher the index hotness index represented by the access mode of the index to which the index shard belongs, the higher the performance of the storage hardware matching the access mode of the index to which the index shard belongs. For example, the Hot access mode corresponds to NVMe SSD (Solid State Disk), the Warm access mode corresponds to SATA (Serial ATA) SSD, and the Cold access mode corresponds to HDD (Hard Disk Drive).

[0092] Similarly, the higher the index hotness index represented by the access mode of the index to which the index shard belongs, the higher the performance of the processing hardware matching the access mode of the index to which the index shard belongs. For example, the Hot access mode corresponds to high-performance CPU, the Warm access mode corresponds to medium-performance CPU, and the Cold access mode corresponds to low-performance CPU.

[0093] Referring to Table 1, an exemplary hierarchical strategy for hierarchical processing of index shards is provided, which can be but is not limited to the hierarchical strategy according to which the index shards on each node are stored to the storage hardware corresponding to the performance matching the access mode of the index to which the index shard belongs, and the processing hardware corresponding to the performance matching the access mode of the index to which the index shard belongs is allocated to the index shard.

[0094] Table 1

[0095] Hotness threshold Storage type / access pattern Hardware configuration ≥80 Hot node NVMe SSD + high CPU 30～80 Warm node SATA SSD + medium CPU <30 Cold node HDD + low CPU

[0096] The execution order of steps 301-302 and steps 101-104 is not limited, for example, steps 301-302 can be executed in sequence after step 104, or steps 301-302 can be executed in sequence before step 101, or steps 301-302 can be executed during the execution of steps 101-104, depending on the actual application.

[0097] In summary, the embodiment realizes automatic identification of index data access mode by performing access hotness analysis on indexes on cluster nodes, and intelligently layers cold and hot data represented by different access modes on the basis, realizes configuration of corresponding performance storage hardware and / or processing hardware for cold and hot data represented by different access modes, thereby further effectively utilizing hardware resources of the cluster nodes, and meanwhile improving data access / processing performance of the cluster system.

[0098] In an optional embodiment, referring to the method flowchart shown in Figure 4 The load migration method provided by the application can further include the following processing:

[0099] Step 401, determine the health degree of each node in the cluster.

[0100] Specifically, but not limited to, the health degree of the node can be determined according to the remaining space ratio of the disk on the node, the heap memory free ratio and / or the node performance.

[0101] The node performance can be determined by comprehensive operation on the CPU, memory and other hardware performance of the node, but is not limited thereto.

[0102] An exemplary calculation formula of the health degree index of the node in the cluster is provided below: HealthScore=

[0103] 1-(0.5*disk_free_rate+0.3*heap_free+0.2*node_responsiveness).

[0104] Wherein, HealthScore represents the health degree of the node, disk_free_rate represents the remaining space ratio of the disk on the node, heap_free represents the heap memory free ratio of the node, and node_responsiveness represents the node performance.

[0105] Step 402, in response to the health degree of the target node representing that the target node fails, restore the target node by synchronizing the incremental load generated during the failure to the target node.

[0106] When the health degree of the node meets the value requirement representing that the node fails, it can be determined that the node fails.

[0107] In response to the health degree of the target node indicating that the target node fails, the embodiment adopts an incremental recovery mechanism to recover the target node. Specifically, when the target node fails, based on the incremental recovery mechanism, only the incremental load generated during the failure is synchronized to the target node, for example, only the incremental data shard generated during the failure is synchronized to the target node, instead of synchronizing the full amount of data to the target node.

[0108] The execution sequence of steps 401-402 and steps 101-104 is not limited, for example, steps 401-402 can be executed in sequence after step 104, or steps 401-402 can be executed in sequence before step 101, or steps 401-402 can be executed during the execution of steps 101-104, depending on the actual application.

[0109] The embodiment determines the health degree of each node in the cluster, and in response to the health degree of the target node indicating that the target node fails, synchronizes the incremental load generated during the failure to the target node, thereby achieving data recovery of the failed node based on the incremental recovery mechanism, thereby effectively reducing the recovery time of the failed node in the cluster and accelerating the recovery rate of the failed node.

[0110] To sum up, the load migration method provided by the embodiment of the application has at least the following technical advantages compared with the known technology:

[0111] 1. Resource utilization rate is improved: through dynamic sharding scheduling, the load difference between cluster nodes can be reduced from 40% in the traditional scheme to within 10%.

[0112] 2. Cost optimization: cold and hot data layering reduces storage costs by 35% (TPC-OpenSearch benchmark test).

[0113] 3. Fault recovery acceleration: the incremental recovery mechanism shortens the 1TB data recovery time from 6 hours to 2.5 hours.

[0114] In addition, it can also reduce the query delay of node data in the cluster, the hardware cost and the frequency of human intervention in operation and maintenance. See Table 2 for related comparative experimental data of the traditional scheme and the scheme of the application (according to the data volume of 1T, three nodes cluster prediction).

[0115] Table 2

[0116] Metrics Traditional solution The present application Query latency (P99) 850 ms 320 ms Hardware cost / TB / month $120 $78 Frequency of operational manual intervention 3 times / day 0.2 times / day

[0117] 4. This application supports DB configuration management based on hierarchical design and page control from data collection to model training to DB (Data Base) management. It can realize on-demand management of Opensearch and other DBs based on this application solution, and supports one-click startup of AI customized management and one-click restoration of traditional management.

[0118] Corresponding to the above-mentioned load migration method, the embodiment of the present application further provides a load migration device, the composition structure of which is as follows: Figure 5 As shown, including:

[0119] An acquisition module 501 is used to obtain historical time series data corresponding to each node in the cluster, where the historical time series data corresponding to the node includes the operation status information corresponding to each time point in the historical time series;

[0120] Prediction module 502, used to predict the node load information of each node in a predetermined time period in the future based on the historical time series data corresponding to each node; the load of the nodes in the cluster includes data sharding;

[0121] A determination module 503 is configured to determine a load migration decision for nodes in the cluster based on node load information of each node in a predetermined future time period;

[0122] A migration processing module 504 is configured to migrate part of the data shards of the first node in the cluster to at least one second node according to the load migration decision;

[0123] The node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

[0124] In an optional embodiment, the prediction module 502 is specifically used to: in the load prediction model based on the time window sliding method, according to the historical time series data corresponding to each node at the current time point, predict the node load peak of each node in the future predetermined time period after the current time point.

[0125] In an optional implementation manner, the determination module 503 is specifically configured to:

[0126] Determine the migration score corresponding to each node at least based on the node load information of each node in a predetermined future time period; the migration score corresponding to the node is positively correlated with the node load represented by the node load information in the predetermined future time period;

[0127] Determine the first and second nodes for data shard migration based on the migration scores of each node.

[0128] The first node is a node in the cluster whose migration score reaches a first threshold, and the second node is a node in the cluster whose migration score does not reach a second threshold.

[0129] In an optional implementation, the data shard on the node includes an index shard stored on the node; the index shard is a data shard obtained by partitioning an index.

[0130] In an optional implementation, the apparatus further includes a hierarchical processing module configured to:

[0131] determine an access pattern corresponding to the index stored on each node in the cluster;

[0132] perform hierarchical processing on the index shard on each node according to the access pattern corresponding to the index stored on each node.

[0133] In an optional implementation, the hierarchical processing module, when determining the access pattern corresponding to the index stored on each node in the cluster, is specifically configured to:

[0134] determine the access pattern corresponding to the index according to a query frequency and / or an update frequency of the index stored on each node in the cluster.

[0135] In an optional implementation, the hierarchical processing module, when performing hierarchical processing on the index shard on each node according to the access pattern corresponding to the index stored on each node, is specifically configured to:

[0136] store the index shard on each node in storage hardware whose performance matches the access pattern of the index to which the index shard belongs, and allocate processing hardware whose performance matches the access pattern of the index to which the index shard belongs to the index shard.

[0137] In an optional implementation, the apparatus further includes a fault processing module configured to:

[0138] determine the health degree of each node in the cluster;

[0139] in response to the health degree of the target node indicating that the target node has failed, recover the target node by synchronizing the incremental load generated during the failure to the target node.

[0140] Embodiments of the present application also disclose an electronic device, and a component structure of the electronic device, as shown in Figure 6 at least includes:

[0141] a memory 10 configured to store a computer instruction set;

[0142] The computer instruction set can be implemented in the form of a computer program.

[0143] The processor 20 is configured to implement the load migration method according to any one of the above method embodiments by executing the computer instruction set in the memory.

[0144] The processor 20 can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices, etc.

[0145] Optionally, the electronic device further includes a memory, a cache, and the like.

[0146] Optionally, the electronic device further includes a camera assembly, and / or is connected with an external camera assembly.

[0147] In addition, the electronic device can further include a communication interface, a communication bus, and the like. The memory, the processor, and the communication interface complete communication with each other through the communication bus.

[0148] The communication interface is configured to enable communication between the electronic device and other devices. The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, and the like.

[0149] The embodiments of the present application also disclose a storage medium, which carries one or more computer instruction sets. When the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the load migration method according to any one of the above method embodiments.

[0150] It should be noted that each of the embodiments in the present specification describes in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0151] For the convenience of description, the above system or device is described in various modules or units in terms of functions. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0152] Those skilled in the art can clearly understand the application can be implemented by means of software plus necessary universal hardware platforms from the description of the above embodiments. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the application.

[0153] Finally, it should be noted that in this document, relational terms such as first and second and top and bottom and and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0154] The above only describes the preferred embodiments of the application, and it should be pointed out that for those skilled in the art, without departing from the principles of the application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A load migration method, comprising: Obtaining historical time series data corresponding to each node in the cluster, wherein the historical time series data corresponding to the node includes the operating status information corresponding to each time point in the historical time series; Predicting node load information of each node in a predetermined future time period based on historical time series data corresponding to each node; the load of the nodes in the cluster includes data sharding; Determining a load migration decision for nodes in the cluster based on node load information of each node in a predetermined future time period; Migrating part of the data shards of the first node in the cluster to at least one second node according to the load migration decision; The node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

2. The load migration method according to claim 1, wherein the step of predicting the node load information of each node in a predetermined future time period based on the historical time series data corresponding to each node comprises: The load prediction model is based on a time window sliding method. According to the historical time series data corresponding to each node at the current time point, the node load peak of each node in the future predetermined time period after the current time point is predicted.

3. The load migration method according to claim 1, wherein determining the load migration decision for the nodes in the cluster based on the node load information of each node in a predetermined future time period comprises: Determining a migration score corresponding to each node at least based on node load information of each node in a predetermined time period in the future; The migration score corresponding to the node is positively correlated with the node load represented by the node load information of the node in the future predetermined time period; Determine the first and second nodes for data shard migration based on the migration scores of each node. The first node is a node in the cluster whose corresponding migration score reaches a first threshold, and the second node is a node in the cluster whose corresponding migration score does not reach a second threshold; The first threshold is not lower than the second threshold.

4. According to the load migration method according to any one of claims 1-3, the data shards on the node include index shards stored on the node; the index shards are data shards obtained by splitting the index.

5. The load migration method according to claim 4, further comprising: Determining an access mode corresponding to an index stored on each node in the cluster; The index shards on each node are layered according to the access mode corresponding to the index stored on each node.

6. The load migration method according to claim 5, wherein determining the access mode corresponding to the index stored on each node in the cluster comprises: An access mode corresponding to an index is determined according to a query frequency and / or an update frequency corresponding to the index stored on each node in the cluster.

7. The load migration method according to claim 5, wherein the layered processing of the index shards on each node according to the access mode corresponding to the index stored on each node comprises: The index shards on each node are stored in storage hardware whose corresponding performance matches the access mode of the index to which the index shards belong, and the index shards are allocated processing hardware whose corresponding performance matches the access mode of the index to which the index shards belong.

8. The load migration method according to claim 1, further comprising: Determining the health of each node in the cluster; In response to a health status indication of the target node indicating that the target node fails, the target node is recovered by synchronizing an incremental load generated during the failure to the target node.

9. A load migration device, comprising: An acquisition module is used to obtain the historical time series data corresponding to each node in the cluster. The historical time series data corresponding to the node includes the operating status information corresponding to each time point in the historical time series; A prediction module, configured to predict the node load information of each node in a predetermined future time period based on the historical time series data corresponding to each node; the load of the nodes in the cluster includes data sharding; A determination module, configured to determine a load migration decision for nodes in the cluster based on node load information of each node in a future predetermined time period; A migration processing module, configured to migrate part of the data shards of the first node in the cluster to at least one second node according to the load migration decision; The node load of the first node in the future predetermined time period is higher than the node load of the second node in the future predetermined time period.

10. An electronic device comprising: a memory for storing at least one set of computer instructions; A processor, configured to implement the load migration method according to any one of claims 1 to 8 by executing the instruction set stored in the memory.

Images