Data migration method and device, computer program product and storage medium

Abstract

The invention discloses a data migration method and device, a computer program product and a storage medium. In the data migration process of migrating first fragment data in a first index to a second index, a first data change request for the fragment data in the first index is received; the fragment number of the first index is smaller than that of the second index; adding the first data change request to a third index; and under the condition that the fragmented data in the first index is migrated to the second index, executing a data change operation corresponding to the first data change request in the third index on the fragmented data of the second index. According to the method and the device, flexible and non-perceptual operation of data migration and index fragment expansion can be realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, device, computer program product and storage medium. Background Art

[0002] In current Elasticsearch cluster applications, when creating an index, you need to specify the number of shards for the index, and once created, the number of shards for the index cannot be changed. To increase the number of shards, the usual approach is to create a new index and then use the Elasticsearch native API (Application Programming Interface) to migrate the data from the original index to the new index.

[0003] However, during the data migration process, applications can generally only perform data query operations, and cannot perform data change operations such as insertion, deletion, and update, because data change operations may cause inconsistency between the old and new indexes after migration. Therefore, during data migration, applications must suspend data change operations. When the amount of data in the original index is large, the migration operation will take a long time, resulting in a corresponding increase in the time it takes for applications to suspend data changes, which in turn makes the system less flexible and less available. Summary of the invention

[0004] In response to the above technical problems, embodiments of the present application provide a data migration method, device, computer program product, and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a data migration method, including:

[0006] In a data migration process of migrating first shard data in a first index to a second index, receiving a first data change request for shard data in the first index; the number of shards in the first index is less than the number of shards in the second index;

[0007] adding the first data change request to a third index;

[0008] In a case where the shard data in the first index has been migrated to the second index, a data change operation corresponding to the first data change request in the third index is performed on the shard data in the second index.

[0009] In a second aspect, an embodiment of the present application provides a data migration device, including:

[0010] A request receiving module, used to receive a first data change request for shard data in a first index during a data migration process of migrating first shard data in a first index to a second index; the number of shards in the first index is less than the number of shards in the second index;

[0011] A request adding module, used for adding the first data change request to a third index;

[0012] A data migration module is used to perform a data change operation corresponding to the first data change request in the third index on the shard data in the second index when the shard data in the first index has been migrated to the second index.

[0013] In a third aspect, an embodiment of the present application also provides a computer program product, including a memory storing multiple instructions; a processor loads instructions from the memory to execute the steps of any data migration method provided in the embodiment of the present application.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps of any data migration method provided in the embodiment of the present application.

[0015] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any data migration method provided in an embodiment of the present application.

[0016] By adopting the scheme of the embodiment of the present application, firstly, in the data migration process of migrating the shard data in the first index to the second index, a first data change request for the shard data in the first index can be received, and the above first data change request can be added to the third index, so that the received data change request can be temporarily stored in the third index without interrupting the reception of the data change request. Then, the shard data of the first index with the first number of shards is migrated to the second index with the second number of shards. Since the number of shards of the first index is less than the number of shards of the second index, the data migration operation of shard expansion can be implemented for the first index. The data change requests for the shard data in the first index will be recorded in the third index process and responded to in time, so that any first data change request for the shard data in the first index will not be suspended or omitted during the data migration process of the shard data. Finally, in the process of transferring the first data change request to the second index, the first data change operation for the shard data in the first index is processed in the second index to complete the data migration of the first index.

[0017] In this way, since the first data change request recorded in the third index is promptly transferred to the second index and quickly responded to and converted into the first data change operation for the shard data in the second index, the actual interruption time caused by business response can be greatly reduced, thereby realizing user-imperceptible data migration and shard expansion operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of an application environment of the data migration method provided in an embodiment of the present application;

[0020] Figure 2 It is one of the schematic diagrams of the existing data migration methods;

[0021] Figure 3 This is the second schematic diagram of the existing data migration method;

[0022] Figure 4 This is the third schematic diagram of the existing data migration method;

[0023] Figure 5 This is a schematic diagram of an embodiment of the data migration method provided in the embodiments of the present application;

[0024] Figure 6 This is a schematic diagram of an implementation process of the data migration method provided in an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of recording a first data change request through a third index provided in an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of determining an unprocessed first data change request provided in an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a process of writing a second data change request into a second index provided in an embodiment of the present application;

[0028] Fig.10 is a structural schematic diagram of a data migration device provided in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the internal structure of the computer program product provided in the embodiments of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0031] The data migration method in one embodiment of the present application can be run on a local terminal device or a server. When the method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0032] In order to better understand a data migration method, device, electronic device and storage medium provided in an embodiment of the present application, an application environment applicable to the embodiment of the present application is first described below.

[0033] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing an application environment of a data migration method provided in an embodiment of the present application. As an implementation mode, the data migration method provided in an embodiment of the present application can be applied to an electronic device. The electronic device can be as follows: Figure 1 The server 110 shown in the figure can communicate with the terminal device 120 through a network. The network is a medium for providing a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0034] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative, and any number of servers, networks and terminal devices may be provided as required. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0035] In some embodiments, the terminal device 120 may obtain a data migration request for the first index and a data change request for the shard data in the first index inputted by the user during the data migration process. Further, the terminal device 120 sends the above data migration request and data change request to the server 110 via the network. After the server 110 receives the above data migration request and data change request, the data migration process may be performed on the shard data of the first index using the data migration method described in the embodiment of the present application.

[0036] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0037] The data migration method of this embodiment can be applied to any scenario that needs to be executed, and the method can be applied to products in these scenarios, for example, the application of Elasticsearch cluster. Before introducing the data migration method of this embodiment, the relevant technical data in the common scenarios of the data migration method of this embodiment are first explained.

[0038] Take the Elasticsearch application as an example. Elasticsearch is an open source distributed search and analysis engine, which is usually used to handle real-time search, log analysis and data analysis tasks of large-scale data. Elasticsearch is built on Apache Lucene, a library for full-text search. Lucene manages indexing and search functions, provides a RESTful API interface and distributed multi-tenant capabilities, enabling it to scale horizontally on multiple servers.

[0039] In existing Elasticsearch applications, when creating an Elasticsearch index, you need to determine the number of shards in the index. A shard is the basic unit used to divide and manage data in an index. An index can include multiple shards, each of which is a part of the index and is responsible for storing data and processing queries. The number of shards refers to the total number of shards in the index. The relationship between shards and data is that each shard contains a part of the index's data, and data can be distributed, stored, and processed on multiple servers. Therefore, the setting of the number of shards can determine the distribution of data and the efficiency of queries.

[0040] In the scenario of this application, each shard is essentially a Lucene index instance, which is not only a storage unit for data, but also has indexing and search capabilities. Each shard can independently execute queries and manage data.

[0041] After creating an index, if you do not want to use the default value of the number of shards, you need to specify the number of shards for the index in advance, for example, the number of shards can be 6, 7, etc. The selection of the number of shards is crucial in the early stages. Once the index is created, the currently specified number of shards cannot be changed. You can only create a new index to specify a different number of shards.

[0042] After understanding the above content, you can understand that in order to increase the number of shards of the Elasticsearch index, the current practice is to create a new index, then set the number of shards of the new index, and then use the native API (Application Programming Interface) of Elasticsearch to migrate the data of the original index to the new index. Specifically, you can first create a new index. The structure of the new index is consistent with the original index, but the number of shards can be adjusted according to new requirements. For example, you can increase the number of shards from 5 to 6, and then use the reindex API of the Elasticsearch application to migrate the data of the original index to the new index. The specific instructions can be expressed as follows:

[0043]

[0044] like Figure 2 As shown in FIG, before data migration, the user can initiate a first data change request for operations such as inserting, updating, and deleting shard data in the original index, and the original index can also respond to the first data change request in a timely manner and perform the corresponding data change operation. Figure 3 As shown in the figure, during the data migration process, the application can only perform query operations, and cannot perform operations involving data changes such as insertion, deletion, and update, and therefore cannot receive the first data change request of the above data change operations. Because during the data migration process, data changes to the original index (such as inserting new data, deleting or updating existing data) will cause inconsistency between the new and old indexes, thus affecting the integrity and consistency of the data. Therefore, Figure 4 As shown, in order to ensure the correctness and consistency of data migration, the application must stop receiving all first data change requests during data migration, interrupt and suspend all data change operations, and then reopen to receive the user's first data change request and perform corresponding data change operations after the data migration is completed.

[0045] In the scenario of massive data migration, when the amount of data in the original index is large, the time of the migration operation will be significantly extended. For example, if the original index contains millions or even hundreds of millions of records, the data migration process may take hours or even longer. During this period, the application must stop all data change operations, resulting in the inability of the business system to operate normally. This situation will seriously affect the availability and user experience of the system, especially for business scenarios that require high availability and real-time data processing. Long business downtime will have a significant negative impact. In addition, some other problems may arise during the data migration process, such as migration failure, data loss and other risks. In order to reduce these risks, additional data backup and recovery operations are usually required, which further increases the complexity of the system and the operation and maintenance costs.

[0046] In summary, the existing data migration operation has significant shortcomings. During the data migration process after creating a new index, all data change operations need to be suspended, resulting in the business system being unavailable for a long time and being unable to achieve imperceptible operations, which seriously affects the system's flexibility, availability and user experience.

[0047] Please refer to Figure 5 The above technical problems can be solved by using the specific process of the data migration method provided in the embodiment of the present application, including the following steps 201 to 204, which are specifically introduced below.

[0048] Step 201, in the process of migrating the first shard data in the first index to the second index, receiving a first data change request for the shard data in the first index. The first index may be an original index in the Elasticsearch application, or simply referred to as the original index. An index is a basic unit for storing and organizing data, and each index may contain multiple shards.

[0049] Among them, sharding is the basic unit for dividing and managing data. Through sharding, data can be distributed on different nodes to improve the efficiency of query and writing. It can be understood that in the data migration scenario of this application, the processing details of the sharded data in the index will directly affect the efficiency and accuracy of the index expansion and data migration process.

[0050] The first index can have a first number of shards, and the second index can have a second number of shards. The number of shards is a key configuration parameter for shards, which directly affects the performance, scalability, and data management efficiency of the application. When creating an index, you need to specify the number of shards for the index. Since the number of shards is fixed after the index is created and cannot be changed directly, you need to reasonably set the number of shards based on the amount of data and expected expansion requirements when creating the index.

[0051] In the scenario of the embodiment of the present application, as the amount of data increases and business needs change, the first index can be sharded and expanded to improve the processing capacity of the system, that is, the number of shards in the shards in the first index is increased. However, since the number of shards of the index is fixed, the first number of shards of the first index cannot be directly modified. Therefore, by creating a new index and migrating the shard data of all shards in the first index to a new second index, by setting the second number of shards of the new second index, the data of the first number of shards in the first index can be migrated to the second index with the second number of shards, so as to achieve sharding expansion of the first index.

[0052] It can be understood that sharding expansion of the first index requires creating a new second index first. The sharding expansion of the first index can be achieved by specifying that the number of second shards of the second index is greater than the number of first shards of the first index, and migrating the shard data of the first index to the second index.

[0053] In some embodiments, the shard data of the first index may be migrated to the second index in the following manner:

[0054] Creating a second index having the second number of shards;

[0055] Transferring the shard data of the first index to the fourth index;

[0056] The shard data transferred to the fourth index is migrated to the second index.

[0057] The fourth index may be a temporary index, which is equivalent to a transfer station in the process of data migration of the shard data of the first index. In some embodiments, a fourth index may be temporarily created through an application program interface (e.g., a reindex API) of an Elasticsearch application or an ES plug-in to store the shard data transferred from the first index.

[0058] Specifically, before migrating data from the first index to the second index, the shard data of the first index can be transferred to the fourth index to ensure that the shard data of the first index will not be lost before the final migration. In this way, the data can be verified and processed during the migration of the shard data of the first index to avoid data inconsistency or loss caused by directly performing data migration operations on the first index and the second index. In addition, the entire process can also perform data migration without interrupting business operations, minimizing the impact on the system and improving application availability.

[0059] In some embodiments, after all the shard data of the first index is transferred to the fourth index, the Elasticsearch application or ES plug-in can use the corresponding application program interface again to migrate the data in the fourth index to the final target index, that is, the second index. In this way, the second index has the new number of shards and all the shard data of the first index, so as to achieve the shard expansion and data migration of the first index.

[0060] It should also be noted that during the data migration process, the fourth index can not only be used to store data, but also to perform some management and maintenance work, such as data verification, data processing, and resource management. For example, after migrating the shard data to the fourth index, the shard data can be verified to ensure that all shard data is completely and accurately transferred to the fourth index. It is also possible to perform necessary data processing and preprocessing in the fourth index so that the shard data can be smoothly migrated to the second index. After the fourth index is used, it can be deleted to release system resources and ensure that system performance is not affected.

[0061] In some embodiments, a data snapshot may be created for the shard data of the first index, and then the data snapshot may be transferred to the fourth index to transfer the shard data of the first index to the fourth index.

[0062] Among them, data snapshot is a mechanism for backing up and restoring index data to ensure the integrity, security and consistency of data during migration and expansion. Figure 6 As shown, a data snapshot can be created for the shard data of the first index first, and then the data snapshot of the shard data can be transferred to the fourth index serving as a temporary index.

[0063] In the specific implementation, creating a data snapshot for the shard data is equivalent to backing up the shard data in the current state. The backed-up shard data can be subsequently loaded into the new index by transferring the snapshot to ensure the integrity and consistency of the shard data of the first index during the data migration process. In addition, during the process of creating and transferring snapshots, the system can continue to process business requests, reduce business suspension time, and improve system availability.

[0064] In the above manner, this application uses the data snapshot function of Elasticsearch to create a data snapshot for the shard data of the first index, and transfer the data snapshot to the fourth index, thereby achieving secure backup and reliable migration of data. The data snapshot not only ensures the consistency and integrity of the data during the entire data migration process, but also reduces business interruption time, improves system reliability and user experience, and improves the efficiency and security of Elasticsearch index shard expansion and data migration.

[0065] Among them, the first data change request refers to a data change request for the first shard data received during the data migration process of migrating the first shard data to the second index, such as various types of data change requests such as inserting, deleting, and updating the first shard data.

[0066] Step 202: Add the first data change request to the third index.

[0067] In some embodiments, the third index is an index for recording and managing first data change requests. The role of the third index is to ensure that during the index sharding expansion and data migration process, all first data change requests generated and received in the above process are accurately captured and synchronized, thereby ensuring the consistency of data migration.

[0068] In some embodiments, before step 201, the present application may receive an operation request for an alias of the first index, and parse out a second index associated with the alias of the first index and the number of shards of the second index from the operation request.

[0069] Among them, alias is the auxiliary name of the index, indicating the reference and association to the index. An alias can point to one or more indices, allowing requests to be dynamically redirected to different indices without changing the request code. Elasticsearch applications can directly access aliases, and the alias backend can correspond to the read indexes of multiple Elasticsearch applications, but an alias is only allowed to correspond to one write index of the Elasticsearch application.

[0070] As an example only, in the data migration scenario of the embodiment of the present application, the data migration method of the present application can be executed through the ES plug-in of the Elasticsearch application, for example, the ES plug-in can be an Elasticsearch 8.11.4 plug-in with a version of "8.11.4". Before use, the plug-in program can be installed in a corresponding directory, for example, the directory is "Elasticsearch-8.11.4 / plugins". After the installation is complete, the Elasticsearch 8.11.4 plug-in can be loaded by starting or restarting the process of the Elasticsearch application.

[0071] Among them, the operation request for the alias can be any operation such as adding, deleting, updating and querying the index alias. Since the alias has an associated index, association means that the alias can point to one or more actual indexes, so that users or applications can access the associated index through the alias without directly referencing the actual name of the index. In this way, without affecting the business logic, the index referenced by the alias can be adjusted, changed or rebuilt through the operation request for the alias, which can improve the flexibility and maintainability of the system.

[0072] As an example only, assuming that the alias of the first index is "index alias" and points to the first index "index old", a specific operation request can be expressed as:

[0073]

[0074] In some embodiments, information can be extracted from the operation information to identify the first index associated with the alias, and the number of shards to be set when creating the first index can be parsed. For example, the "number_of_shards" field in the operation request can be parsed, that is, the number of shards, and the parsing result is 6, indicating that the number of shards set when creating the first index is 6.

[0075] In some embodiments, the first data change request may be added to the third index in the following manner:

[0076] First, enable the interception plug-in to intercept the first data change request through the interception plug-in;

[0077] Then, the first data change requests are sequentially added to the third index according to the interception order of the first data change requests, and a request identifier is sequentially added to each of the first data change requests.

[0078] Among them, the interception plug-in can be the ES plug-in mentioned above, or it can be a sub-plug-in in the ES plug-in specifically used to intercept the first data change request. In some embodiments, the interception operation of the first data change request can be implemented through a hook. Hook is an execution mechanism for software development that allows custom processing logic to be inserted when a specific event or operation occurs. Through hooks, new functions or behaviors can be added without modifying the original code. Among them, the interception order refers to the order in which each first data change request is intercepted and captured. For example, the first data change request A and the first data change request B are intercepted, and the first data change request A is intercepted one second before the first data change request B, then the corresponding interception order is the first data change request A, the first data change request B.

[0079] In some embodiments, the interception plug-in can not only intercept and capture the first data change request for the shard data of the first index, but also cache the intercepted and captured first data change request. Specifically, the interception plug-in can add the first data change request to the third index in sequence according to the interception order for the first data change request, and add a request identifier for each first data change request in sequence.

[0080] The request identifier can be a number value or an ID value. Figure 7 As shown, a corresponding request identifier can be added to each first data change request in the order in which it is intercepted, for example, the ID can be 1, 2, 3, ..., n, etc. It should be noted that the first data change request can be a request for one shard data or a request for multiple shard data, for example Figure 7 The first data change request corresponding to ID1 is a request for multiple shard data, specifically a request to insert multiple shard data "(Zhang San, 800), (Li Si, 440), (Wang Wu, 50) and (Li Er, 200)", while the first data change request corresponding to ID2 is a request for one shard data, specifically a request to update the shard data "(Zhang San, 800)" to "(Zhang San, 900)".

[0081] It can be understood that the third index is equivalent to a transaction record index specifically used to record and cache the first data change request. In this way, in the process of data migration of the first index, there is no need to interrupt the reception of the first data change request for the shard data of the first index. Instead, it is recorded and cached in real time in the third index to facilitate subsequent timely response to the above first data change request. Compared with the existing method of first suspending the reception of the first data change request, and then resuming the reception and delayed response to the first data change request after the first index data migration is completed, the duration of business interruption caused by the first data change request can be significantly reduced. Usually, this duration can be ignored, thereby improving the data migration efficiency of the index and realizing user-imperceptible data migration operations.

[0082] In some embodiments, after the first data change request is added to the third index, in the case where consecutive first data change requests are for inserting data, data change operations corresponding to the first data change requests may be combined and executed.

[0083] It can be understood that in the scenario of the present application, if each data insertion operation is executed independently, it will not only increase the system operation overhead, but may also cause the system to freeze, crash, etc. Therefore, the third index can solve this problem through dynamic transaction merging.

[0084] Specifically, when it is determined that the first consecutive data change request is to insert data, for example, when it is determined based on ID sorting that adjacent consecutive IDs are all data insertion requests, the data insertion operations corresponding to the above multiple data insertion requests can be merged and executed later. Assuming that the time required for a single data insertion operation is T milliseconds, the time required to merge and batch insert N data in this way will be significantly less than (N×T) milliseconds.

[0085] In some embodiments, the data change operation corresponding to the first data change request may be combined and executed in the following manner:

[0086] First, obtain the first traffic data generated by the cluster where the first index is located in the first period;

[0087] Next, based on the first traffic data, predicting second traffic data generated by the cluster in a second period;

[0088] Finally, if it is determined according to the second traffic data that the second time period is in a low traffic period, then in the case where the continuous first data change requests are to insert data, the steps of executing the data change operations corresponding to the first data change requests are combined.

[0089] In a specific implementation, the first traffic data generated by the cluster in the first period can be recorded by a component or device for cluster monitoring. For example, in a high-concurrency scenario, the QPS and disk bandwidth of data change operations in the cluster can be obtained in real time as the first traffic data, and the collected first traffic data can be stored as structured information for subsequent traffic prediction.

[0090] In some embodiments, a suitable prediction model can be selected to predict the second traffic data of the second time period in the future based on the historical characteristics of the first traffic data. For example, if the traffic generated by the cluster has periodic fluctuations, the time series analysis method can be used to predict the second traffic data; if the traffic changes of the cluster are complex, a deep learning model (such as LSTM) can be used to predict the second traffic data. Key indicators such as the QPS and disk load of the cluster in the second time period can be determined based on the predicted second traffic data.

[0091] Specifically, if the predicted QPS and disk load in the second traffic data are both lower than a certain threshold, it can be determined that the second time period is in a low-traffic period. For example, when the QPS of the second time period is lower than 1000 times / second and the disk bandwidth is lower than 50MB / s, the second time period can be regarded as a low-traffic period. If it is detected during the low-traffic period that the continuous first data change requests are all of the type of inserting data, the insert operations corresponding to these consecutive insert requests can be merged and executed as a batch insert task. For example, two insert data operations with ID 1 and ID 2 can be merged into an insert operation that inserts two data at a time.

[0092] Correspondingly, if it is determined according to the second traffic data that the second time period is in the traffic peak period, the step of merging the data change operations corresponding to the first data change requests to execute when the continuous first data change requests are insert data is suspended.

[0093] Specifically, if the QPS and disk load in the predicted second traffic data are both higher than a certain threshold, it can be determined that the second period is in a low traffic period. For example, when the QPS of the second period is higher than 5000 times / second and the disk bandwidth is greater than 90% of the upper limit, the second period can be regarded as a peak traffic period.

[0094] During peak traffic periods, you can immediately pause the merge execution of consecutive insertion operations to avoid the system occupying too many processing resources and causing lags, excessive load, etc. It should be noted that when the traffic falls from the peak period to the low-peak period, you can restart the merge execution of consecutive insertion operations.

[0095] Through the above methods, this application effectively combines traffic prediction and dynamic transaction merging operations, which can not only improve data migration efficiency, but also ensure the stability and performance of the system during peak hours by flexibly switching task execution mechanisms during high and low traffic periods.

[0096] Step 203: When the shard data in the first index has been migrated to the second index, a data change operation corresponding to the first data change request in the third index is performed on the shard data in the second index.

[0097] like Figure 6 As shown, all the shard data of the first index backed up in the fourth index can be migrated to the second index first, and then all the first data change requests in the third index can be transferred to the second index, so as to ensure that the first data change requests for the shard data in the first index during the data migration process (such as requests to insert, delete, and update data) can be correctly recorded and processed, and applied to the newly created second index.

[0098] In some embodiments, according to the request identifier of each first data change request, the corresponding data change operation can be performed on each shard data in the second index in sequence according to the order of the request identifier to complete the data migration of the first index.

[0099] Since the request identifier of each first data change request is added according to the order of interception, for example, the first data change request with ID 1 is intercepted earlier than the first data change request with ID 2, in order to ensure the accuracy and consistency of data change processing, after the first data change request is transferred to the second index, the corresponding data change operation must also be performed in the order of the request identifiers.

[0100] like Figure 7 As shown, taking the shard data "(Zhang San, 800)" as an example, in the first data change request of ID1, a request is made to insert the newly added shard data "(Zhang San, 800)", then the data change operation corresponding to the first data change request of ID1 is executed first, and the shard data "(Zhang San, 800)" is accurately inserted into the second index. Furthermore, the first data change request of ID2 corresponds to a data update request, requesting to update the shard data "(Zhang San, 800)" to the shard data "(Zhang San, 900)", then the data change operation corresponding to ID2 is continued to be executed on the basis of the shard data "(Zhang San, 800)", and the shard data "(Zhang San, 800)" is updated to "(Zhang San, 900)". It can be understood that if the data change operation is not performed in the order of the request identifier, the shard data "(Zhang San, 800)" has not yet been inserted into the second index, and the data change operation of ID2 to update the shard data "(Zhang San, 800)" to "(Zhang San, 900)" cannot be performed.

[0101] Correspondingly, the first data change request of ID3 corresponds to a data deletion request. After the data change operation of ID2 is completed, the data change operation of ID3 is continued to be executed to delete the shard data "(Zhang San, 900)" from the second index. Similarly, if the data change operation is not executed in the order of the request identifiers, the shard data "(Zhang San, 800)" may not have been inserted into the second index, and the operation of deleting the shard data "(Zhang San, 900)" of ID3 cannot be executed. Or the data change operation of ID2 to update the shard data to "(Zhang San, 900)", that is, to delete the shard data "(Zhang San, 800)", will cause confusion and errors in the data change of the second index. The above method of executing the data change operation in the order of the request identifiers of the first data change request can ensure the accuracy and consistency of data migration.

[0102] In some embodiments, after the first shard data in the first index has been migrated to the second index, in response to a data migration request for the second shard data in the first index, the data change operation corresponding to the first data change request in the third index is synchronously performed on the first shard data of the second index, as well as the data migration of the second shard data of the first index.

[0103] It is understandable that after the first shard data of the first index is successfully migrated to the second index, a data migration request for the second shard data newly written to the first index may also be received, and the third index may also accumulate unprocessed data change requests for the first shard data. In this case, the data change operation for the first shard data and the data migration operation for the second shard data can be performed synchronously to reduce the pressure on the system.

[0104] Specifically, when the above situation occurs, the second shard data can be distinguished from the first shard data, and does not participate in operations such as being intercepted, creating snapshots, and migrating to a temporary fourth index, but is directly migrated to the second index and executed synchronously with the data change operation of the first shard data, thereby saving the process and resources for migrating the second shard data and improving data migration efficiency without affecting data consistency.

[0105] However, it should be noted that this situation only applies to scenarios with a small number of unprocessed data change requests. It can be understood that if the data processing pressure in the second index is large, if the second shard data is directly written to the second index, it may cause greater resource pressure and load pressure on the second index, causing system freezes and crashes. Therefore, the above two operations can be performed simultaneously when there are only a small number of unprocessed data change requests.

[0106] It is understandable that the traditional shard expansion process requires downtime or limits on data change operations to ensure data consistency during data migration. However, the method of the present application can simultaneously perform migration of newly written shard data and change operations on already written shard data when capturing newly written shard data, thereby significantly reducing or eliminating downtime caused by data change operations.

[0107] In some embodiments, the application may also include:

[0108] If a second data change request for the shard data of the first index is detected, and the number of unprocessed first data change requests in the second index is greater than or equal to a preset threshold, adding the second data change request to the third index;

[0109] If the number of unprocessed first data change requests in the second index is less than the preset threshold, the second data change request is sent to the second index.

[0110] It can be understood that, normally, in the process of migrating the sharded data of the first index, after all the first data change requests recorded in the third index are transferred to the second index, each first data change request can be immediately responded to in the second index in the order of the request identifier to perform the data change operation on the sharded data.

[0111] However, there is another situation where, after all the first data change requests recorded in the third index are transferred to the second index, the second data change request for the shard data may still be detected, that is, new first data change requests may continue to be detected. For this situation, two processing methods can be used. The first processing method is to add the second data change request to the third index for recording in a conventional manner, and the second processing method is to send the second data change request directly to the second index for execution. Which of the above processing methods to use can be determined in combination with the number of unprocessed data processing requests in the second index.

[0112] In some embodiments, the number of unprocessed first data change requests in the second index may be determined first. Figure 8 As shown, after all the first data change requests in the third index are transferred to the second index, the total number of pending first data change requests can be counted first, for example, there are 1,000 pending first data change requests. Since the first data change requests transferred to the second index will receive a timely response to perform the corresponding data change operation on the data of the second index, when a new second data change request is detected, the system has usually processed a portion of the first data change requests, and the number of processed first data change requests can be obtained in real time, and then the number of unprocessed first data change requests can be determined in combination with the total number of pending first data change requests. For example, for the above 1,000 pending first data change requests, when the second data change request is received, 995 first data change requests have been processed, and then it can be determined that the number of unprocessed first data change requests is 5.

[0113] like Fig. 9 As shown, in some embodiments, if the number of unprocessed first data change requests in the second index is greater than or equal to a preset threshold, the second data change request is added to the third index; if the number of unprocessed first data change requests in the second index is less than the preset threshold, the second data change request may not be sent to the third index for recording, but may be sent directly to the second index to perform a target data change operation on the shard data of the second index in the second index.

[0114] It can be understood that the preset threshold can be an indicator used by the system to measure the data synchronization status between the first index and the second index, representing the gap between the processed first data change request in the second index and all data change requests in the first index.

[0115] In some embodiments, assuming that the preset threshold is 5, when the number of unprocessed first data change requests in the second index is greater than or equal to 5, it means that there are still many unprocessed or incompletely processed data change requests in the second index. In this case, the second data change request can be added to the third index to ensure that the second data change request does not directly affect the current data processing process of the second index, thereby alleviating the load pressure of data processing in the second index and avoiding performance problems caused by excessive data volume during the processing of the second index.

[0116] When the number of unprocessed first data change requests in the second index is less than 5, it means that the first data change requests for the first index have been almost completely processed in the second index. After intercepting the second data change request, the alias of the first index can be immediately pointed to the second index to ensure that the load pressure of the second index to perform the data change operation is not large. The new second data change request is directly added to the second index for processing, thereby improving the efficiency of data migration and avoiding inconsistency in data migration.

[0117] Through the above method, the present application can ensure data consistency during the dynamic expansion of index shards, and dynamically adjust the alias pointing of the old index by judging the number of unprocessed first data change requests in the new index, so that the first data change request can be processed seamlessly, thereby improving the system availability and user experience.

[0118] In some embodiments, the method of the present application may further include:

[0119] First, obtaining a first quantity of inserted data as a first data change request within a third time period;

[0120] When the first preset condition is met, the preset threshold is less than the first threshold, and the first preset condition includes that the first number is greater than the preset number, the third time period is in the peak traffic period, and there is data update in the first data change request;

[0121] When the second preset condition is met, the preset threshold is the number of the first data change requests, and the second preset condition includes that the first number is less than the preset number, the third time period is a low traffic period, and there is no data update in the first data change request;

[0122] When the first preset condition and the second preset condition are not met, the preset threshold is less than the second threshold.

[0123] As an example only, the first threshold can be 10, the second threshold can be 5, and the preset number can be 100. Then the first preset condition can be: when the number of data change requests for inserting data is greater than 100, the third time period is during a peak traffic period, and there is a data update in the first data change request, the preset threshold is set to be less than 10.

[0124] Correspondingly, the second preset condition can be: when the number of data change requests for inserting data is less than 100, the third time period is in a low traffic period, and there is no data update in the first data change request, the preset threshold is set to the number of unprocessed first data change requests, for example, it can be set to 20.

[0125] Correspondingly, when the first preset condition and the second preset condition are not met, the preset threshold can be directly set to be less than the second threshold, for example, the preset threshold is less than 5.

[0126] This application comprehensively considers the type and number of data requests to be processed, the peak and off-peak conditions of traffic, and dynamically adjusts the preset thresholds based on preset conditions. Through this flexible adjustment mechanism, it can not only ensure the performance of the system during peak traffic, but also make full use of resources to improve efficiency during off-peak periods, while ensuring the overall stability of the system under other complex conditions.

[0127] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0128] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more data migration devices provided below can refer to the limitations of the data migration method above, and the specific limitations will not be repeated here.

[0129] The data migration device provided in this embodiment can be integrated into a terminal device. Fig.10 As shown, the data migration device may include:

[0130] The request receiving module 301 is used to receive a first data change request for the shard data in the first index during the data migration process of migrating the first shard data in the first index to the second index; the number of shards in the first index is less than the number of shards in the second index;

[0131] A request adding module 302, configured to add the first data change request to the third index;

[0132] The data migration module 303 is used to perform a data change operation corresponding to the first data change request in the third index on the shard data in the second index when the shard data in the first index has been migrated to the second index.

[0133] In some embodiments, the request adding module 302 is further specifically configured to:

[0134] When the consecutive first data change requests are for inserting data, the data change operations corresponding to the first data change requests are merged and executed.

[0135] In some embodiments, the request adding module 302 is further specifically used for: when the consecutive first data change requests are all data insertion requests, merging and executing the data change operations corresponding to the first data change requests, and further for:

[0136] Obtain first traffic data generated by the cluster where the first index is located in the first period;

[0137] Predicting, based on the first traffic data, second traffic data generated by the cluster in the second time period;

[0138] If it is determined according to the second traffic data that the second time period is in a low traffic peak period, then in the case where the continuous first data change requests are insert data, the steps of executing the data change operations corresponding to the first data change requests are combined.

[0139] In some embodiments, the request adding module 302 is further specifically configured to:

[0140] If it is determined according to the second traffic data that the second time period is in a traffic peak period, the step of merging the data change operations corresponding to the first data change requests when the consecutive first data change requests are insert data is suspended.

[0141] In some embodiments, the data migration module 303 is further specifically used for:

[0142] In response to the data migration request of the second shard data in the first index, the data change operation corresponding to the first data change request in the third index is synchronously performed on the first shard data of the second index, and the data migration of the second shard data of the first index is performed.

[0143] In some embodiments, the data migration module 303 is further specifically used for:

[0144] If a second data change request for the shard data of the first index is detected, and the number of unprocessed first data change requests in the second index is greater than or equal to a preset threshold, adding the second data change request to the third index;

[0145] If the number of unprocessed first data change requests in the second index is less than a preset threshold, the second data change request is sent to the second index.

[0146] In some embodiments, the data migration module 303 is further specifically used for:

[0147] Obtaining a first quantity of inserted data as a first data change request within a third time period;

[0148] When the first preset condition is met, the preset threshold is less than the first threshold, and the first preset condition includes that the first number is greater than the preset number, the third time period is in the peak traffic period, and there is data update in the first data change request;

[0149] When the second preset condition is met, the preset threshold is the number of unprocessed first data change requests, and the second preset condition includes that the first number is less than the preset number, the third time period is a low traffic period, and there is no data update in the first data change request;

[0150] When the first preset condition and the second preset condition are not met, the preset threshold is less than the second threshold.

[0151] By adopting the scheme of the embodiment of the present application, firstly, in the data migration process of migrating the shard data in the first index to the third index, a first data change request for the shard data in the first index having the first number of shards can be received, and the above data change request can be added to the second index, so that the received data change request can be temporarily stored in the second index without interrupting the reception of the data change request. Then, the shard data of the first index having the first number of shards is migrated to the third index having the second number of shards. Since the first number of shards is less than the second number of shards, the data migration operation of shard expansion can be implemented for the first index. The data change requests for the shard data in the first index will be recorded and responded to in the process of the second index, so that any data change requests for the shard data in the first index will not be suspended or omitted during the data migration process of the shard data. Finally, the data change request is transferred to the third index to process the data change operation for the shard data in the first index in the third index, and the data migration of the first index is completed. In this way, the data change requests recorded in the second index are promptly transferred to the third index and quickly converted into data change operations for the sharded data, thereby greatly reducing the actual interruption time caused by business response, thereby achieving user-imperceptible data migration and shard expansion operations.

[0152] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0153] In some embodiments, the computer program product is a terminal device, for example, and its internal structure diagram can be as follows: Fig.11 As shown. The computer program product includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer program product is used to provide computing and control capabilities. The memory of the computer program product includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer program product is used to exchange information between the processor and an external device. The communication interface of the computer program product is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data migration method is implemented. The display unit of the computer program product is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer program product can be a touch layer covered on the display screen, or it can be a button, trackball or touchpad set on the computer program product shell, or it can be an external keyboard, touchpad or mouse, etc.

[0154] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the computer program product to which the present application scheme is applied. The specific computer program product may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0156] Since the computer program stored in the computer-readable storage medium can execute any one of the data migration methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the data migration methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0157] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer program product reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer program product executes the method provided in various optional implementations in the above embodiments.

[0158] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0159] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0160] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0161] In the above-mentioned data migration device, computer-readable storage medium, computer program product, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described data migration device, computer-readable storage medium, computer program product, computer program product and its corresponding units can refer to the description of the data migration method in the above embodiment, and will not be repeated here.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above is a detailed introduction to a data migration method, device, computer program product, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data migration method, characterized in that: The method comprises: In a data migration process of migrating first shard data in a first index to a second index, receiving a first data change request for shard data in the first index; the number of shards in the first index is less than the number of shards in the second index; adding the first data change request to a third index; In a case where the shard data in the first index has been migrated to the second index, a data change operation corresponding to the first data change request in the third index is performed on the shard data in the second index.

2. The method according to claim 1, characterized in that After adding the first data change request to the third index, the method further includes: In the case where the continuous first data change requests are for inserting data, the data change operations corresponding to the first data change requests are executed in a merged manner.

3. The method according to claim 2, characterized in that The method further includes: when the consecutive first data change requests are all data insertion requests, merging and executing data change operations corresponding to the first data change requests, including: Obtain first traffic data generated by the cluster where the first index is located in the first time period; Predicting, based on the first traffic data, second traffic data generated by the cluster in a second time period; If it is determined according to the second traffic data that the second time period is in a low traffic period, then in the case where the consecutive first data change requests are for inserting data, the steps of executing the data change operations corresponding to the first data change requests are combined.

4. The method according to claim 3, characterized in that The method further comprises: If it is determined according to the second traffic data that the second time period is in a traffic peak period, the step of merging the data change operations corresponding to the first data change requests when the consecutive first data change requests are insert data is suspended.

5. The method according to claim 1, characterized in that After the first shard data in the first index has been migrated to the second index, the method further includes: In response to a data migration request for the second shard data in the first index, a data change operation corresponding to the first data change request in the third index is synchronously performed on the first shard data of the second index, as well as data migration of the second shard data of the first index.

6. The method according to claim 1, characterized in that After adding the first data change request to the third index, the method further includes: If a second data change request for the shard data of the first index is detected, and the number of unprocessed first data change requests in the second index is greater than or equal to a preset threshold, adding the second data change request to the third index; If the number of unprocessed first data change requests in the second index is less than the preset threshold, the second data change request is sent to the second index.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining a first quantity of inserted data by the first data change request within a third time period; When a first preset condition is met, the preset threshold is less than the first threshold, and the first preset condition includes that the first number is greater than a preset number, the third time period is a traffic peak period, and there is data update in the first data change request; When a second preset condition is met, the preset threshold is the number of the first data change requests that have not been processed, and the second preset condition includes that the first number is less than the preset number, the third time period is a low traffic period, and there is no data update in the first data change request; When the first preset condition and the second preset condition are not met, the preset threshold is less than the second threshold.

8. A data migration device, characterized in that: The device comprises: A request receiving module, used to receive a first data change request for shard data in a first index during a data migration process of migrating first shard data in a first index to a second index; the number of shards in the first index is less than the number of shards in the second index; A request adding module, used for adding the first data change request to a third index; A data migration module is used to perform a data change operation corresponding to the first data change request in the third index on the shard data in the second index when the shard data in the first index has been migrated to the second index.

9. A computer program product, characterized in that It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the data migration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the data migration method according to any one of claims 1 to 7.

11. A computer software, characterized in that: The computer software includes a computer program, and the computer program is executed by a processor to perform the steps of the data migration method according to any one of claims 1 to 7.

Images